google: Use regional instance templates for workers

* Use regional instance templates for the worker node regional
managed instance groups. Regional instance templates are kept in
the associated region, whereas the older "global" instance templates
were kept in a particular region (regardless of where the MIG region)
so outages in a region X could affect clusters in a region Y which
is undesired

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,10 +0,0 @@
-High level description of the change.
-
-* Specific change
-* Specific change
-
-## Testing
-
-Describe your work to validate the change works.
-
-rel: issue number (if applicable)

.github/dependabot.yaml

@@ -4,6 +4,3 @@ updates:
   directory: "/"
   schedule:
     interval: weekly
-  pull-request-branch-name:
-    separator: "-"
-  open-pull-requests-limit: 3

.github/release.yaml

@@ -0,0 +1,12 @@
+changelog:
+  categories:
+    - title: Contributions
+      labels:
+        - '*'
+      exclude:
+        labels:
+          - dependencies
+          - no-release-note
+    - title: Dependencies
+      labels:
+        - dependencies

.github/workflows/publish.yaml

@@ -0,0 +1,12 @@
+name: publish
+on:
+  push:
+    branches:
+      - release-docs
+jobs:
+  mkdocs:
+    name: mkdocs
+    uses: poseidon/matchbox/.github/workflows/mkdocs-pages.yaml@main
+    # Add content write for GitHub Pages
+    permissions:
+      contents: write

.gitignore

@@ -0,0 +1,2 @@
+site/
+venv/

CHANGES.md

@@ -4,7 +4,600 @@ Notable changes between versions.
 
 ## Latest
 
+## v1.31.0
+
+* Kubernetes [v1.31.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.31.md#v1310)
+* Use Cilium kube-proxy replacement mode when `cilium` networking is chosen ([#1501](https://github.com/poseidon/typhoon/pull/1501))
+* Fix invalid flannel-cni container image for those using `flannel` networking ([#1497](https://github.com/poseidon/typhoon/pull/1497))
+
+### AWS
+
+* Use EC2 resource-based hostnames instead of IP-based hostnames ([#1499](https://github.com/poseidon/typhoon/pull/1499))
+  * The Amazon DNS server can resolve A and AAAA queries to IPv4 and IPv6 node addresses
+* Tag controller node EBS volumes with a name based on the controller node name
+
+### Google
+
+* Use `google_compute_region_instance_template` instead of `google_compute_instance_template`
+  * Google's regional instance template metadata is kept in the associated region for greater resiliency. The "global" instance templates were kept in a single region
+
+## v1.30.4
+
+* Kubernetes [v1.30.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#v1304)
+* Update Cilium from v1.15.7 to [v1.16.1](https://github.com/cilium/cilium/releases/tag/v1.16.1)
+* Update CoreDNS from v1.11.1 to v1.11.3
+* Remove `enable_aggregation` variable for Kubernetes Aggregation Layer, always set to true
+* Remove `cluster_domain_suffix` variable, always use "cluster.local"
+* Remove `enable_reporting` variable for analytics, always set to false
+
+## v1.30.3
+
+* Kubernetes [v1.30.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#v1303)
+* Update Cilium from v1.15.6 to [v1.15.7](https://github.com/cilium/cilium/releases/tag/v1.15.7)
+* Update flannel from v0.25.4 to [v0.25.5](https://github.com/flannel-io/flannel/releases/tag/v0.25.5)
+
+### AWS
+
+* Configure controller and worker disks ([#1482](https://github.com/poseidon/typhoon/pull/1482))
+  * Add `controller_disk_type`, `controller_disk_size`, and `controller_disk_iops` variables
+  * Add `worker_disk_type`, `worker_disk_size`, and `worker_disk_iops` variables
+  * Remove `disk_type`, `disk_size`, and `disk_iops` variables
+  * Fix propagating settings to worker disks, previously ignored
+* Configure CPU pricing model for burstable instance types ([#1482](https://github.com/poseidon/typhoon/pull/1482))
+  * Add `controller_cpu_credits` and `worker_cpu_credits` variables (`standard` or `unlimited`)
+* Configure controller or worker instance architecture ([#1485](https://github.com/poseidon/typhoon/pull/1485))
+  * Add `controller_arch` and `worker_arch` variables (`amd64` or `arm64`)
+  * Remove `arch` variable
+
+```diff
+module "cluster" {
+  ...
+- arch      = "amd64"
+- disk_type = "gp3"
+- disk_size = 30
+- disk_iops = 3000
+
++ controller_arch        = "amd64"
++ controller_disk_size   = 15
++ controller_cpu_credits = "standard"
++ worker_arch            = "amd64"
++ worker_disk_size       = 22
++ worker_cpu_credits     = "unlimited"
+}
+```
+
+### Azure
+
+* Configure the virtual network and subnets with IPv6 private address space
+  * Change `host_cidr` variable (string) to a `network_cidr` object with `ipv4` and `ipv6` fields that list CIDR strings. Leave the variable unset to use the defaults. (**breaking**)
+* Add support for dual-stack Kubernetes Ingress Load Balancing
+  * Add a public IPv6 frontend, 80/443 rules, and a worker-ipv6 backend pool
+  * Change the `controller_address_prefixes` output from a list of strings to an object with `ipv4` and `ipv6` fields. Most Azure resources can't accept a mix, so these are split out (**breaking**)
+  * Change the `worker_address_prefixes` output from a list of strings to an object with `ipv4` and `ipv6` fields. Most Azure resources can't accept a mix, so these are split out (**breaking**)
+  * Change the `backend_address_pool_id` output (and worker module input) from a string to an object with `ipv4` and `ipv6` fields that list ids (**breaking**)
+* Configure nodes to have outbound IPv6 internet connectivity (analogous to IPv4 SNAT)
+  * Configure controller nodes to have a public IPv6 address
+  * Configure worker nodes to use outbound rules and the load balancer for SNAT
+* Extend network security rules to allow IPv6 traffic, analogous to IPv4
+* Rename `region` variable to `location` to align with Azure platform conventions ([#1469](https://github.com/poseidon/typhoon/pull/1469))
+* Change worker pools from uniform to flexible orchestration mode ([#1473](https://github.com/poseidon/typhoon/pull/1473))
+* Add options to allow workers nodes to use ephemeral local disks ([#1473](https://github.com/poseidon/typhoon/pull/1473))
+  * Add `controller_disk_type` and `controller_disk_size` variables
+  * Add `worker_disk_type`, `worker_disk_size`, and `worker_ephemeral_disk` variables
+* Reduce the number of public IPv4 addresses needed for the Azure load balancer ([#1470](https://github.com/poseidon/typhoon/pull/1470))
+* Configure controller or worker instance architecture for Flatcar Linux ([#1485](https://github.com/poseidon/typhoon/pull/1485))
+  * Add `controller_arch` and `worker_arch` variables (`amd64` or `arm64`)
+  * Remove `arch` variable
+
+```diff
+module "cluster" {
+  ...
+- region = "centralus"
++ location = "centralus"
+  # optional
+- host_cidr = "10.0.0.0/16"
++ network_cidr = {
++   ipv4 = ["10.0.0.0/16"]
++ }
+
+  # instances
++ controller_disk_type = "StandardSSD_LRS"
++ worker_ephemeral_disk = true
+}
+```
+
+### Google Cloud
+
+* Allow configuring controller and worker disks ([#1486](https://github.com/poseidon/typhoon/pull/1486))
+  * Add `controller_disk_size` and `worker_disk_size` variables
+  * Remove `disk_size` variable
+
+## v1.30.2
+
+* Kubernetes [v1.30.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#v1302)
+* Update CoreDNS from v1.9.4 to v1.11.1
+* Update Cilium from v1.15.5 to [v1.15.6](https://github.com/cilium/cilium/releases/tag/v1.15.6)
+* Update flannel from v0.25.1 to [v0.25.4](https://github.com/flannel-io/flannel/releases/tag/v0.25.4)
+
+## v1.30.1
+
+* Kubernetes [v1.30.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#v1301)
+* Add firewall rules and security group rules for Cilium and Hubble metrics ([#1449](https://github.com/poseidon/typhoon/pull/1449))
+* Update Cilium from v1.15.3 to [v1.15.5](https://github.com/cilium/cilium/releases/tag/v1.15.5)
+* Update flannel from v0.24.4 to [v0.25.1](https://github.com/flannel-io/flannel/releases/tag/v0.25.1)
+* Introduce `components` variabe to enable/disable/configure pre-installed components ([#1453](https://github.com/poseidon/typhoon/pull/1453))
+* Add Terraform modules for `coredns`, `cilium`, and `flannel` components
+
+### Azure
+
+* Add `controller_security_group_name` output for adding custom security rules ([#1450](https://github.com/poseidon/typhoon/pull/1450))
+* Add `controller_address_prefixes` output for adding custom security rules ([#1450](https://github.com/poseidon/typhoon/pull/1450))
+
+## v1.30.0
+
+* Kubernetes [v1.30.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#v1300)
+* Update etcd from v3.5.12 to [v3.5.13](https://github.com/etcd-io/etcd/releases/tag/v3.5.13)
+* Update Cilium from v1.15.2 to [v1.15.3](https://github.com/cilium/cilium/releases/tag/v1.15.3)
+* Update Calico from v3.27.2 to [v3.27.3](https://github.com/projectcalico/calico/releases/tag/v3.27.3)
+
+## v1.29.3
+
+* Kubernetes [v1.29.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md#v1293)
+* Update Cilium from v1.15.1 to [v1.15.2](https://github.com/cilium/cilium/releases/tag/v1.15.2)
+* Update flannel from v0.24.2 to [v0.24.4](https://github.com/flannel-io/flannel/releases/tag/v0.24.4)
+
+## v1.29.2
+
+* Kubernetes [v1.29.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md#v1292)
+* Update etcd from v3.5.10 to [v3.5.12](https://github.com/etcd-io/etcd/releases/tag/v3.5.12)
+* Update Cilium from v1.14.3 to [v1.15.1](https://github.com/cilium/cilium/releases/tag/v1.15.1)
+* Update Calico from v3.26.3 to [v3.27.2](https://github.com/projectcalico/calico/releases/tag/v3.27.2)
+  * Fix upstream incompatibility with Fedora CoreOS ([calico#8372](https://github.com/projectcalico/calico/issues/8372))
+* Update flannel from v0.22.2 to [v0.24.2](https://github.com/flannel-io/flannel/releases/tag/v0.24.2)
+* Add an `install_container_networking` variable (default `true`) ([#1421](https://github.com/poseidon/typhoon/pull/1421))
+  * When `true`, the chosen container `networking` provider is installed during cluster bootstrap
+  * Set `false` to self-manage the container networking provider. This allows flannel, Calico, or Cilium
+  to be managed via Terraform (like any other Kubernetes resources). Nodes will be NotReady until you
+  apply the self-managed container networking provider. This may become the default in future.
+  * Continue to set `networking` to one of the three supported container networking providers. Most
+  require custom firewall / security policies be present across nodes so they have some infra tie-ins.
+
+## v1.29.1
+
+* Kubernetes [v1.29.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md#v1291)
+
+### AWS
+
+* Continue to support AWS IMDSv1 ([#1412](https://github.com/poseidon/typhoon/pull/1412))
+
+### Known Issues
+
+* Calico and Fedora CoreOS cannot be used together currently ([calico#8372](https://github.com/projectcalico/calico/issues/8372))
+
+## v1.29.0
+
+* Kubernetes [v1.29.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md#v1290)
+
+### Known Issues
+
+* Calico and Fedora CoreOS cannot be used together currently ([calico#8372](https://github.com/projectcalico/calico/issues/8372))
+
+## v1.28.4
+
+* Kubernetes [v1.28.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#v1284)
+
+## v1.28.3
+
+* Kubernetes [v1.28.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#v1283)
+* Update etcd from v3.5.9 to [v3.5.10](https://github.com/etcd-io/etcd/releases/tag/v3.5.10)
+* Update Cilium from v1.14.2 to [v1.14.3](https://github.com/cilium/cilium/releases/tag/v1.14.3)
+* Workaround problems in Cilium v1.14's partial `kube-proxy` implementation ([#365](https://github.com/poseidon/terraform-render-bootstrap/pull/365))
+* Update Calico from v3.26.1 to [v3.26.3](https://github.com/projectcalico/calico/releases/tag/v3.26.3)
+
+### Google Cloud
+
+* Allow upgrading Google Cloud Terraform provider to v5.x
+
+## v1.28.2
+
+* Kubernetes [v1.28.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#v1282)
+* Update Cilium from v1.14.1 to [v1.14.2](https://github.com/cilium/cilium/releases/tag/v1.14.2)
+
+### Azure
+
+* Add optional `azure_authorized_key` variable
+  * Azure obtusely inspects public keys, requires RSA keys, and forbids more secure key formats (e.g. ed25519)
+  * Allow passing a dummy RSA key via `azure_authorized_key` (delete the private key) to satisfy Azure validations, then the usual `ssh_authorized_key` variable can new newer formats (e.g. ed25519)
+
+## v1.28.1
+
+* Kubernetes [v1.28.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#v1281)
+
+## v1.28.0
+
+* Kubernetes [v1.28.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#v1280)
+* Update Cilium from v1.13.4 to [v1.14.1](https://github.com/cilium/cilium/releases/tag/v1.14.1)
+* Update flannel from v0.22.0 to [v0.22.2](https://github.com/flannel-io/flannel/releases/tag/v0.22.2)
+
+## v1.27.4
+
+* Kubernetes [v1.27.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#v1274)
+
+## v1.27.3
+
+* Kubernetes [v1.27.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#v1273)
+* Update etcd from v3.5.7 to [v3.5.9](https://github.com/etcd-io/etcd/releases/tag/v3.5.9)
+* Update Cilium from v1.13.2 to [v1.13.4](https://github.com/cilium/cilium/releases/tag/v1.13.4)
+* Update Calico from v3.25.1 to [v3.26.1](https://github.com/projectcalico/calico/releases/tag/v3.26.1)
+* Update flannel from v0.21.2 to [v0.22.0](https://github.com/flannel-io/flannel/releases/tag/v0.22.0)
+
+### AWS
+
+* Allow upgrading AWS Terraform provider to v5.x ([#1353](https://github.com/poseidon/typhoon/pull/1353))
+
+### Azure
+
+* Enable boot diagnostics for controller and worker VMs ([#1351](https://github.com/poseidon/typhoon/pull/1351))
+
+## v1.27.2
+
+* Kubernetes [v1.27.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#v1272)
+
+### Fedora CoreOS
+
+* Update Butane Config version from v1.4.0 to v1.5.0
+  * Require any custom Butane [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) update to v1.5.0
+* Require Fedora CoreOS `37.20230303.3.0` or newer (with ignition v2.15)
+* Require poseidon/ct v0.13+ (**action required**)
+
+## v1.27.1
+
+* Kubernetes [v1.27.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#v1271)
+* Update etcd from v3.5.7 to [v3.5.8](https://github.com/etcd-io/etcd/releases/tag/v3.5.8)
+* Update Cilium from v1.13.1 to [v1.13.2](https://github.com/cilium/cilium/releases/tag/v1.13.2)
+* Update Calico from v3.25.0 to [v3.25.1](https://github.com/projectcalico/calico/releases/tag/v3.25.1)
+
+## v1.26.3
+
+* Kubernetes [v1.26.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#v1263)
+* Update Cilium from v1.12.6 to [v1.13.1](https://github.com/cilium/cilium/releases/tag/v1.13.1)
+
+### Bare-Metal
+
+* Add `oem_type` variable for Flatcar Linux ([#1302](https://github.com/poseidon/typhoon/pull/1302))
+
+## v1.26.2
+
+* Kubernetes [v1.26.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#v1262)
+* Update Cilium from v1.12.5 to [v1.12.6](https://github.com/cilium/cilium/releases/tag/v1.12.6)
+* Update flannel from v0.20.2 to [v0.21.2](https://github.com/flannel-io/flannel/releases/tag/v0.21.2)
+
+### Bare-Metal
+
+* Add a `worker` module to allow customizing individual worker nodes ([#1295](https://github.com/poseidon/typhoon/pull/1295))
+
+### Known Issues
+
+* Fedora CoreOS [issue](https://github.com/coreos/fedora-coreos-tracker/issues/1423) fix is progressing through channels
+
+## v1.26.1
+
+* Kubernetes [v1.26.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#v1261)
+* Update etcd from v3.5.6 to [v3.5.7](https://github.com/etcd-io/etcd/releases/tag/v3.5.7)
+* Update Cilium from v1.12.4 to [v1.12.5](https://github.com/cilium/cilium/releases/tag/v1.12.5)
+* Update Calico from v3.24.5 to [v3.25.0](https://github.com/projectcalico/calico/releases/tag/v3.25.0)
+* Update CoreDNS from v1.9.3 to [v1.9.4](https://github.com/poseidon/terraform-render-bootstrap/pull/341)
+
+## v1.26.0
+
+* Kubernetes [v1.26.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#v1260)
+* Update etcd from v3.5.5 to [v3.5.6](https://github.com/etcd-io/etcd/releases/tag/v3.5.6)
+* Update Cilium from v1.12.3 to [v1.12.4](https://github.com/cilium/cilium/releases/tag/v1.12.4)
+* Update flannel from v0.15.1 to [v0.20.2](https://github.com/flannel-io/flannel/releases/tag/v0.20.2)
+* Reminder: Modules are no longer published to the [Terraform Module Registry](https://registry.terraform.io/search/modules?q=poseidon) ([#1282](https://github.com/poseidon/typhoon/pull/1282))
+  * See [#1282](https://github.com/poseidon/typhoon/pull/1282) and [v1.25.4](https://github.com/poseidon/typhoon/releases/tag/v1.25.4) for details
+
+### AWS
+
+* Migrate AWS launch configurations to launch templates ([#1275](https://github.com/poseidon/typhoon/pull/1275))
+  * Starting Dec 31, 2022 AWS won't add new instance types/families to launch configurations
+
+### Addons
+
+* Update ingress-nginx from v1.3.1 to [v1.5.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.5.1)
+* Update Prometheus from v2.40.1 to [v2.40.5](https://github.com/prometheus/prometheus/releases/tag/v2.40.5)
+* Update node-exporter from v1.3.1 to [v1.5.0](https://github.com/prometheus/node_exporter/releases/tag/v1.5.0)
+* Update kube-state-metrics from v2.6.0 to [v2.7.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.7.0)
+* Update Grafana from v9.2.4 to [v9.3.1](https://github.com/grafana/grafana/releases/tag/v9.3.1)
+
+## v1.25.4
+
+* Kubernetes [v1.25.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#v1254)
+* Update Calico from v3.24.1 to [v3.24.5](https://github.com/projectcalico/calico/releases/tag/v3.24.5)
+* Allow Kubelet kubeconfig to drain nodes, if desired ([#330](https://github.com/poseidon/terraform-render-bootstrap/pull/330))
+* Re-enable Kubelet Graceful Node Shutdown ([#1261](https://github.com/poseidon/typhoon/pull/1261))
+  * Introduce companion project [poseidon/scuttle](https://github.com/poseidon/scuttle)
+* Link to new Mastodon account for release announcements
+  * [@typhoon@fosstodon.org](https://fosstodon.org/@typhoon)
+  * [@poseidon@fosstodon.org](https://fosstodon.org/@poseidon)
+* Deprecate publishing to the [Terraform Module Registry](https://registry.terraform.io/search/modules?q=poseidon)
+  * Typhoon docs have always shown using Git-based module sources, not the Terraform Module Registry
+  * Module usage should be `source = "git::https://github.com/poseidon/typhoon/...` not `source = poseidon/kubernetes/...`
+  * Terraform's Module Registry requires subtree mirroring typhoon to special terraform-platform-kubernetes repos, only supports release versions (no commit SHAs or forks), only ever contained Flatcar Linux modules (not Fedora CoreOS) for historical reasons
+  * Note, this does not affect Terraform Providers like `poseidon/matchbox` or `poseidon/ct`, the registry works well for providers
+
+### Fedora CoreOS
+
+* Remove unused `Wants=network.target` from `etcd-member.service` ([#1254](https://github.com/poseidon/typhoon/pull/1254))
+
+### Cloud
+
+* Remove defunct `delete-node.service` from worker node configurations ([#1256](https://github.com/poseidon/typhoon/pull/1256))
+
+### Addons
+
+* Update Prometheus from v2.39.1 to [v2.40.1](https://github.com/prometheus/prometheus/releases/tag/v2.40.1)
+* Update Grafana from v9.1.7 to [v9.2.4](https://github.com/grafana/grafana/releases/tag/v9.2.4)
+
+## v1.25.3
+
+* Kubernetes [v1.25.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#v1253)
+* Switch Kubernetes registry from `k8s.gcr.io` to `registry.k8s.io` for addons ([#1246](https://github.com/poseidon/typhoon/pull/1246))
+* Update Cilium from v1.12.2 to [v1.12.3](https://github.com/cilium/cilium/releases/tag/v1.12.3) ([#1253](https://github.com/poseidon/typhoon/pull/1253))
+
+### Azure
+
+* Change default Azure `worker_type` from [`Standard_DS1_v2`](https://learn.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series#dsv2-series) to [`Standard_D2as_v5`](https://learn.microsoft.com/en-us/azure/virtual-machines/dasv5-dadsv5-series#dasv5-series) ([#1248](https://github.com/poseidon/typhoon/pull/1248))
+  * Get 2 VCPU, 7 GiB, 12500Mbps (vs 1 VCPU, 3.5GiB, 750 Mbps)
+  * Small increase in pay-as-you-go price ($53.29 -> $62.78)
+  * Small increase in spot price ($5.64/mo -> $7.37/mo)
+  * Change from Intel to AMD EPYC (`D2as_v5` cheaper than `D2s_v5`)
+
+### Flatcar Linux
+
+* Add Flatcar Linux ARM64 support on Azure ([docs](https://typhoon.psdn.io/advanced/arm64/), [#1251](https://github.com/poseidon/typhoon/pull/1251))
+* Switch from Azure Hypervisor gen1 to gen2 (**action required**) ([#1248](https://github.com/poseidon/typhoon/pull/1248))
+  * Run `az vm image terms accept --publish kinvolk --offer flatcar-container-linux-free --plan stable-gen2`
+
+### Docs
+
+* Remove old docs note about not supporting ARM64 with Calico
+  * Typhoon supports ARM64 with `cilium`, `calico`, and `flannel`
+
+### Addons
+
+* Update Prometheus from v2.38.0 to [v2.39.1](https://github.com/prometheus/prometheus/releases/tag/v2.39.1)
+* Update Grafana from v9.1.6 to [v9.1.7](https://github.com/grafana/grafana/releases/tag/v9.1.7)
+
+## v1.25.2
+
+Kubernetes v1.25.2 was skipped since there were minimal changes upstream.
+
+## v1.25.1
+
+* Kubernetes [v1.25.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#v1251)
+* Update etcd from v3.5.4 to [v3.5.5](https://github.com/etcd-io/etcd/releases/tag/v3.5.5)
+* Update Cilium from v1.12.1 to [v1.12.2](https://github.com/cilium/cilium/releases/tag/v1.12.2)
+* Update Calico from v3.23.3 to [v3.24.1](https://github.com/projectcalico/calico/releases/tag/v3.24.1)
+* Revert Kubelet Graceful Node Shutdown on worker nodes ([#1227](https://github.com/poseidon/typhoon/pull/1227))
+  * Fix issue where non-critical pods are left in Error/Completed state on node shutdown
+* Remove feature flag disable workaround for [kubernetes/kubernetes#112081](https://github.com/kubernetes/kubernetes/issues/112081)
+  * Kubernetes [reverted](https://github.com/kubernetes/kubernetes/pull/112078) `LocalStorageCapacityIsolationFSQuotaMonitoring` back to alpha
+* Remove workaround for preventing `search .` propagation in [kubernetes/kubernetes#112135](https://github.com/kubernetes/kubernetes/issues/112135)
+  * Upstream Kubernetes [fix](https://github.com/kubernetes/kubernetes/pull/112157)
+
+### Addons
+
+* Update kube-state-metrics from v2.5.0 to [v2.6.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.6.0)
+* Update ingress-nginx from v1.3.0 to [v1.3.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.3.1)
+* Update Grafana from v9.1.0 to [v9.1.6](https://github.com/grafana/grafana/releases/tag/v9.1.6)
+
+## v1.25.0
+
+* Kubernetes [v1.25.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#v1250)
+  * Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature gate ([#1220](https://github.com/poseidon/typhoon/pull/1220), fixes [kubernetes#112081](https://github.com/kubernetes/kubernetes/issues/112081))
+  * Add workaround to revert adding "search ." to containers' `/etc/resolv.conf` ([#1224](https://github.com/poseidon/typhoon/pull/1224), fixes [kubernetes#112135](https://github.com/kubernetes/kubernetes/issues/112135))
+* Migrate most Kubelet flags to KubeletConfiguration file ([#1219](https://github.com/poseidon/typhoon/pull/1219))
+* Configure Kubelet Graceful Node Shutdown ([#1222](https://github.com/poseidon/typhoon/pull/1222))
+  * Allow up to 30s for critical pods to gracefully shutdown on node shutdown
+  * Allow up to 15s for regular pods to gracefully shutdown on node shutdown
+  * Mark node NotReady promptly on node shutdown
+  * Lengthen systemd inhibitor lock max delay from 5s to 45s
+
+### Fedora CoreOS
+
+* Change Podman `log-driver` from `journald` to `k8s-file` ([#1221](https://github.com/poseidon/typhoon/pull/1221))
+  * Fix `etcd-member` and Kubelet systemd service log lines appearing twice in journal logs
+
+## v1.24.4
+
+* Kubernetes [v1.24.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1244)
+* Update CoreDNS from v1.8.6 to [v1.9.3](https://github.com/poseidon/terraform-render-bootstrap/pull/318)
+* Update Cilium from v1.11.7 to [v1.12.1](https://github.com/cilium/cilium/releases/tag/v1.12.1)
+* Update Calico from v3.23.1 to [v3.23.3](https://github.com/projectcalico/calico/releases/tag/v3.23.3)
+* Switch Kubernetes registry from `k8s.gcr.io` to `registry.k8s.io` ([#1206](https://github.com/poseidon/typhoon/pull/1206))
+* Remove use of deprecated Terraform [template](https://registry.terraform.io/providers/hashicorp/template) provider ([#1194](https://github.com/poseidon/typhoon/pull/1194))
+
+### Fedora CoreOS
+
+* Remove ineffective `/etc/fedora-coreos/iptables-legacy.stamp` ([#1201](https://github.com/poseidon/typhoon/pull/1201))
+  * Typhoon already uses iptables v1.8.7 (nf_tables) since FCOS 36
+  * Staying on legacy iptables required a file in `/etc/coreos` instead
+
+### Flatcar Linux
+
+* Migrate Flatcar Linux from Ignition spec v2.3.0 to v3.3.0 ([#1196](https://github.com/poseidon/typhoon/pull/1196)) (**action required**)
+  * Flatcar Linux 3185.0.0+ [supports](https://flatcar-linux.org/docs/latest/provisioning/ignition/specification/#ignition-v3) Ignition v3.x specs (which are rendered from Butane Configs, like Fedora CoreOS)
+  * `poseidon/ct` v0.11.0 [supports](https://github.com/poseidon/terraform-provider-ct/pull/131) the `flatcar` Butane Config variant
+  * Require poseidon/ct v0.11+ and Flatcar Linux 3185.0.0+
+* Please modify any Flatcar Linux snippets to use the [Butane Config](https://coreos.github.io/butane/config-flatcar-v1_0/) format (**action required**)
+
+```tf
+variant: flatcar
+version: 1.0.0
+...
+```
+
+### AWS
+
+* [Refresh](https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html) instances in autoscaling group when launch configuration changes ([#1208](https://github.com/poseidon/typhoon/pull/1208)) ([docs](https://typhoon.psdn.io/topics/maintenance/#node-configuration-updates), **important**)
+  * Worker launch configuration changes start an autoscaling group instance refresh to replace instances
+  * Instance refresh creates surge instances, waits for a warm-up period, then deletes old instances
+  * Changing `worker_type`, `disk_*`, `worker_price`, `worker_target_groups`, or Butane `worker_snippets` on existing worker nodes will replace instances
+  * New AMIs or changing `os_stream` will be ignored, to allow Fedora CoreOS or Flatcar Linux to keep themselves updated
+  * Previously, new launch configurations were made in the same way, but not applied to instances unless manually replaced
+* Rename worker autoscaling group `${cluster_name}-worker` ([#1202](https://github.com/poseidon/typhoon/pull/1202))
+  * Rename launch configuration `${cluster_name}-worker` instead of a random id
+
+### Google
+
+* [Roll](https://cloud.google.com/compute/docs/instance-groups/rolling-out-updates-to-managed-instance-groups) instance template changes to worker managed instance groups ([#1207](https://github.com/poseidon/typhoon/pull/1207)) ([docs](https://typhoon.psdn.io/topics/maintenance/#node-configuration-updates), **important**)
+  * Worker instance template changes roll out by gradually replacing instances
+  * Automatic rollouts create surge instances, wait for health checks, then delete old instances (0 unavailable instances)
+  * Changing `worker_type`, `disk_size`, `worker_preemptible`, or Butane `worker_snippets` on existing worker nodes will replace instances
+  * New compute images or changing `os_stream` will be ignored, to allow Fedora CoreOS or Flatcar Linux to keep themselves updated
+  * Previously, new instance templates were made in the same way, but not applied to instances unless manually replaced
+* Add health checks to worker managed instance groups (i.e. "autohealing") ([#1207](https://github.com/poseidon/typhoon/pull/1207))
+  * Use health checks to probe kube-proxy every 30s
+  * Replace worker nodes that fail the health check 6 times (3min)
+* Name `kube-apiserver` and `worker` health checks consistently ([#1207](https://github.com/poseidon/typhoon/pull/1207))
+  * Use name `${cluster_name}-apiserver-health` and `${cluster_name}-worker-health`
+* Rename managed instance group from `${cluster_name}-worker-group` to `${cluster_name}-worker` ([#1207](https://github.com/poseidon/typhoon/pull/1207))
+* Fix bug provisioning clusters with multiple controller nodes ([#1195](https://github.com/poseidon/typhoon/pull/1195))
+
+### Addons
+
+* Update Prometheus from v2.37.0 to [v2.38.0](https://github.com/prometheus/prometheus/releases/tag/v2.38.0)
+* Update Grafana from v9.0.3 to [v9.1.0](https://github.com/grafana/grafana/releases/tag/v9.1.0)
+
+## v1.24.3
+
+* Kubernetes [v1.24.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1243)
+* Update Cilium from v1.11.6 to [v1.11.7](https://github.com/cilium/cilium/releases/tag/v1.11.7)
+
+### Addons
+
+* Update ingress-nginx from v1.2.1 to [v1.3.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.3.0)
+* Update Prometheus from v2.36.1 to [v2.37.0](https://github.com/prometheus/prometheus/releases/tag/v2.37.0)
+* Update Grafana from v8.5.6 to [v9.0.3](https://github.com/grafana/grafana/releases/tag/v9.0.3)
+
+### Notes
+
+* Poseidon repos will soon change their default branch from `master` to `main`
+
+## v1.24.2
+
+* Kubernetes [v1.24.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1242)
+* Update Cilium from v1.11.5 to [v1.11.6](https://github.com/cilium/cilium/releases/tag/v1.11.6)
+* Update Calico from v3.22.2 to [v3.23.1](https://github.com/projectcalico/calico/releases/tag/v3.23.1)
+
+### Addons
+
+* Update Prometheus from v2.36.0 to [v2.36.1](https://github.com/prometheus/prometheus/releases/tag/v2.36.1)
+* Update Grafana from v8.5.3 to [v8.5.6](https://github.com/grafana/grafana/releases/tag/v8.5.6)
+* Update kube-state-metrics from v2.4.2 to [v2.5.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.5.0)
+
+## Known Issues
+
+* Skip AWS Terraform provider v4.17.0 to v4.19.0, which had a regression affecting workers joining ([#1173](https://github.com/poseidon/typhoon/issues/1173))
+
+## v1.24.1
+
+* Kubernetes [v1.24.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1241)
+* Update Cilium from v1.11.4 to [v1.11.5](https://github.com/cilium/cilium/releases/tag/v1.11.5)
+
+### Addons
+
+* Update Prometheus from v2.35.0 to [v2.36.0](https://github.com/prometheus/prometheus/releases/tag/v2.36.0)
+* Update Grafana from v8.5.1 to [v8.5.3](https://github.com/grafana/grafana/releases/tag/v8.5.3)
+* Update nginx-ingress from v1.2.0 to [v1.2.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.1)
+
+## v1.24.0
+
+* Kubernetes [v1.24.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1240)
+* Update etcd from v3.5.2 to [v3.5.4](https://github.com/etcd-io/etcd/releases/tag/v3.5.4)
+* Add Kubelet mounts to enable relabeling workload volumes ([#1152](https://github.com/poseidon/typhoon/pull/1152))
+  * StorageClass no longer require explicit SELinux mount contexts
+
+### Addons
+
+* Update nginx-ingress from v1.1.3 to [v1.2.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0)
+* Update Prometheus from v2.34.0 to [v2.35.0](https://github.com/prometheus/prometheus/releases/tag/v2.35.0)
+* Update Grafana from v8.4.5 to [v8.5.1](https://github.com/grafana/grafana/releases/tag/v8.5.1)
+
+## v1.23.6
+
+* Kubernetes [v1.23.6](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1236)
+* Update Cilium from v1.11.2 to [v1.11.4](https://github.com/cilium/cilium/releases/tag/v1.11.4)
+* Rename Cilium DaemonSet from `cilium-agent` to `cilium` to match Cilium CLI tools ([#303](https://github.com/poseidon/terraform-render-bootstrap/pull/303))
+* Update Calico from v3.22.1 to [v3.22.2](https://github.com/projectcalico/calico/releases/tag/v3.22.2)
+* Mount /etc/machine-id from host into Kubelet ([#1143](https://github.com/poseidon/typhoon/pull/1143))
+* Remove deprecated use of `key_algorithm` in `hashicorp/tls` resources
+
+### Azure
+
+* Allow upgrading Azure Terraform provider to v3.x ([#1144](https://github.com/poseidon/typhoon/pull/1144))
+* Rename `worker_address_prefix` output to `worker_address_prefixes`
+
+### Google Cloud
+
+* Fix issue on Flatcar Linux with controller nodes not ignoring os image changes ([#1149](https://github.com/poseidon/typhoon/pull/1149))
+  * Nodes will auto-update, Terraform should not attempt to delete/recreate them
+
+### Addons
+
+* Update nginx-ingress from v1.1.2 to [v1.1.3](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.3)
+* Update Prometheus from v2.33.5 to [v2.34.0](https://github.com/prometheus/prometheus/releases/tag/v2.34.0)
+* Update Grafana from v8.4.4 to [v8.4.5](https://github.com/grafana/grafana/releases/tag/v8.4.5)
+
+## v1.23.5
+
+* Kubernetes [v1.23.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1235)
+* Update Cilium from v1.11.1 to [v1.11.2](https://github.com/cilium/cilium/releases/tag/v1.11.2)
+* Update Calico from v3.21.2 to [v3.22.1](https://github.com/projectcalico/calico/releases/tag/v3.22.1)
+  * Fix [calico#5011](https://github.com/projectcalico/calico/issues/5011), broken since v1.23.0
+
+### Addons
+
+* Refresh Prometheus rules and Grafana dashboards ([#1136](https://github.com/poseidon/typhoon/pull/1136))
+* Update nginx-ingress from v1.1.1 to [v1.1.2](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.2)
+* Update Prometheus from v2.33.3 to [v2.33.5](https://github.com/prometheus/prometheus/releases/tag/v2.33.5)
+* Update Grafana from v8.4.1 to [v8.4.3](https://github.com/grafana/grafana/releases/tag/v8.4.3)
+* Update kube-state-metrics from v2.3.0 to [v2.4.2](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.4.2)
+
+## v1.23.4
+
+* Kubernetes [v1.23.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1234)
+* Update etcd from v3.5.1 to [v3.5.2](https://github.com/etcd-io/etcd/releases/tag/v3.5.2)
+* Change default CNI `networking` provider from `calico` to `cilium` ([#1114](https://github.com/poseidon/typhoon/pull/1114))
+
+### AWS
+
+* Allow upgrading AWS Terraform Provider to v4.x
+
+### Addons
+
+* Align nginx-ingress `--controller-class` with `IngressClass`
+  * Watch only `public` IngressClass objects, better [example](https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/)
+* Update Prometheus from v2.32.1 to [v2.33.3](https://github.com/prometheus/prometheus/releases/tag/v2.33.3)
+* Update Grafana from v8.3.6 to [v8.4.1](https://github.com/grafana/grafana/releases/tag/v8.4.1)
+
+## V1.23.3
+
+* Kubernetes [v1.23.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1233)
+
+### Flatcar Linux
+
+#### Google Cloud
+
+* Switch to using official Kinvolk Flatcar Linux images
+* Promote Typhoon on Flatcar Linux / Google Cloud to stable
+* Change `os_image` to `flatcar-stable`, `flatcar-beta`, or `flatcar-alpha` (**action required**)
+
+## v1.23.2
+
 * Kubernetes [v1.23.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1232)
+* Update Cilium from v1.11.0 to [v1.11.1](https://github.com/cilium/cilium/releases/tag/v1.11.1)
 * Remove Kubelet flag `--network-plugin`. Unused since `docker-shim` isn't used ([#1106](https://github.com/poseidon/typhoon/pull/1106))
 
 ### Fedora CoreOS

README.md

@@ -1,4 +1,11 @@
-# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
+# Typhoon
+
+[![Release](https://img.shields.io/github/v/release/poseidon/typhoon?style=flat-square)](https://github.com/poseidon/typhoon/releases)
+[![Stars](https://img.shields.io/github/stars/poseidon/typhoon?style=flat-square)](https://github.com/poseidon/typhoon/stargazers)
+[![Sponsors](https://img.shields.io/github/sponsors/poseidon?logo=github&style=flat-square)](https://github.com/sponsors/poseidon)
+[![Mastodon](https://img.shields.io/badge/follow-news-6364ff?logo=mastodon&style=flat-square)](https://fosstodon.org/@typhoon)
+
+<img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
 
 Typhoon is a minimal and free Kubernetes distribution.
 
@@ -11,7 +18,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.23.2 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@@ -19,7 +26,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Modules
 
-Typhoon provides a Terraform Module for each supported operating system and platform.
+Typhoon provides a Terraform Module for defining a Kubernetes cluster on each supported operating system and platform.
 
 Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 
@@ -43,11 +50,20 @@ Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/
 | Azure         | Flatcar Linux    | [azure/flatcar-linux/kubernetes](azure/flatcar-linux/kubernetes) | alpha |
 | Bare-Metal    | Flatcar Linux    | [bare-metal/flatcar-linux/kubernetes](bare-metal/flatcar-linux/kubernetes) | stable |
 | DigitalOcean | Flatcar Linux  | [digital-ocean/flatcar-linux/kubernetes](digital-ocean/flatcar-linux/kubernetes) | beta |
-| Google Cloud  | Flatcar Linux  | [google-cloud/flatcar-linux/kubernetes](google-cloud/flatcar-linux/kubernetes) | beta |
+| Google Cloud  | Flatcar Linux  | [google-cloud/flatcar-linux/kubernetes](google-cloud/flatcar-linux/kubernetes) | stable |
 
 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Flatcar Linux (ARM64) | [aws/flatcar-linux/kubernetes](aws/flatcar-linux/kubernetes) | alpha |
+| Azure         | Flatcar Linux (ARM64) | [azure/flatcar-linux/kubernetes](azure/flatcar-linux/kubernetes) | alpha |
+
+Typhoon also provides Terraform Modules for optionally managing individual components applied onto clusters.
+
+| Name    | Terraform Module | Status |
+|---------|------------------|--------|
+| CoreDNS | [addons/coredns](addons/coredns) | beta |
+| Cilium  | [addons/cilium](addons/cilium) | beta |
+| flannel | [addons/flannel](addons/flannel) | beta |
 
 ## Documentation
 
@@ -62,7 +78,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.31.0"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -101,9 +117,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.2
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.31.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.31.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.31.0
 ```
 
 List the pods.
@@ -111,9 +127,10 @@ List the pods.
 ```
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                      READY  STATUS    RESTARTS  AGE
-kube-system   calico-node-1cs8z                         2/2    Running   0         6m
+kube-system   cilium-1cs8z                              1/1    Running   0         6m
-kube-system   calico-node-d1l5b                         2/2    Running   0         6m
+kube-system   cilium-d1l5b                              1/1    Running   0         6m
-kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   cilium-sp9ps                              1/1    Running   0         6m
+kube-system   cilium-operator-68d778b448-g744f          1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   kube-apiserver-controller-0               1/1    Running   0         6m
@@ -156,5 +173,12 @@ Poseidon's Github [Sponsors](https://github.com/sponsors/poseidon) support the i
     <img src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/SVG/DO_Logo_horizontal_blue.svg" width="201px">
 </a>
 <br>
+<br>
+
+<a href="https://deploy.equinix.com/">
+  <img src="https://storage.googleapis.com/poseidon/equinix.png" width="201px">
+</a>
+<br>
+<br>
 
 If you'd like your company here, please contact dghubble at psdn.io.

addons/cilium/cluster-role-binding.tf

@@ -0,0 +1,36 @@
+resource "kubernetes_cluster_role_binding" "operator" {
+  metadata {
+    name = "cilium-operator"
+  }
+
+  role_ref {
+    api_group = "rbac.authorization.k8s.io"
+    kind      = "ClusterRole"
+    name      = "cilium-operator"
+  }
+
+  subject {
+    kind      = "ServiceAccount"
+    name      = "cilium-operator"
+    namespace = "kube-system"
+  }
+}
+
+resource "kubernetes_cluster_role_binding" "agent" {
+  metadata {
+    name = "cilium-agent"
+  }
+
+  role_ref {
+    api_group = "rbac.authorization.k8s.io"
+    kind      = "ClusterRole"
+    name      = "cilium-agent"
+  }
+
+  subject {
+    kind      = "ServiceAccount"
+    name      = "cilium-agent"
+    namespace = "kube-system"
+  }
+}
+

addons/cilium/cluster-role.tf

@@ -0,0 +1,112 @@
+resource "kubernetes_cluster_role" "operator" {
+  metadata {
+    name = "cilium-operator"
+  }
+
+  # detect and restart [core|kube]dns pods on startup
+  rule {
+    verbs      = ["get", "list", "watch", "delete"]
+    api_groups = [""]
+    resources  = ["pods"]
+  }
+
+  rule {
+    verbs      = ["list", "watch"]
+    api_groups = [""]
+    resources  = ["nodes"]
+  }
+
+  rule {
+    verbs      = ["patch"]
+    api_groups = [""]
+    resources  = ["nodes", "nodes/status"]
+  }
+
+  rule {
+    verbs      = ["get", "list", "watch"]
+    api_groups = ["discovery.k8s.io"]
+    resources  = ["endpointslices"]
+  }
+
+  rule {
+    verbs      = ["get", "list", "watch"]
+    api_groups = [""]
+    resources  = ["services"]
+  }
+
+  # Perform LB IP allocation for BGP
+  rule {
+    verbs      = ["update"]
+    api_groups = [""]
+    resources  = ["services/status"]
+  }
+
+  # Perform the translation of a CNP that contains `ToGroup` to its endpoints
+  rule {
+    verbs      = ["get", "list", "watch"]
+    api_groups = [""]
+    resources  = ["services", "endpoints", "namespaces"]
+  }
+
+  rule {
+    verbs      = ["*"]
+    api_groups = ["cilium.io"]
+    resources  = ["ciliumnetworkpolicies", "ciliumnetworkpolicies/status", "ciliumnetworkpolicies/finalizers", "ciliumclusterwidenetworkpolicies", "ciliumclusterwidenetworkpolicies/status", "ciliumclusterwidenetworkpolicies/finalizers", "ciliumendpoints", "ciliumendpoints/status", "ciliumendpoints/finalizers", "ciliumnodes", "ciliumnodes/status", "ciliumnodes/finalizers", "ciliumidentities", "ciliumidentities/status", "ciliumidentities/finalizers", "ciliumlocalredirectpolicies", "ciliumlocalredirectpolicies/status", "ciliumlocalredirectpolicies/finalizers", "ciliumendpointslices", "ciliumloadbalancerippools", "ciliumloadbalancerippools/status", "ciliumcidrgroups", "ciliuml2announcementpolicies", "ciliuml2announcementpolicies/status", "ciliumpodippools"]
+  }
+
+  rule {
+    verbs      = ["create", "get", "list", "update", "watch"]
+    api_groups = ["apiextensions.k8s.io"]
+    resources  = ["customresourcedefinitions"]
+  }
+
+  # Cilium leader elects if among multiple operator replicas
+  rule {
+    verbs      = ["create", "get", "update"]
+    api_groups = ["coordination.k8s.io"]
+    resources  = ["leases"]
+  }
+}
+
+resource "kubernetes_cluster_role" "agent" {
+  metadata {
+    name = "cilium-agent"
+  }
+
+  rule {
+    verbs      = ["get", "list", "watch"]
+    api_groups = ["networking.k8s.io"]
+    resources  = ["networkpolicies"]
+  }
+
+  rule {
+    verbs      = ["get", "list", "watch"]
+    api_groups = ["discovery.k8s.io"]
+    resources  = ["endpointslices"]
+  }
+
+  rule {
+    verbs      = ["get", "list", "watch"]
+    api_groups = [""]
+    resources  = ["namespaces", "services", "pods", "endpoints", "nodes"]
+  }
+
+  rule {
+    verbs      = ["patch"]
+    api_groups = [""]
+    resources  = ["nodes/status"]
+  }
+
+  rule {
+    verbs      = ["create", "get", "list", "watch", "update"]
+    api_groups = ["apiextensions.k8s.io"]
+    resources  = ["customresourcedefinitions"]
+  }
+
+  rule {
+    verbs      = ["*"]
+    api_groups = ["cilium.io"]
+    resources  = ["ciliumnetworkpolicies", "ciliumnetworkpolicies/status", "ciliumclusterwidenetworkpolicies", "ciliumclusterwidenetworkpolicies/status", "ciliumendpoints", "ciliumendpoints/status", "ciliumnodes", "ciliumnodes/status", "ciliumidentities", "ciliumidentities/status", "ciliumlocalredirectpolicies", "ciliumlocalredirectpolicies/status", "ciliumegressnatpolicies", "ciliumendpointslices", "ciliumcidrgroups", "ciliuml2announcementpolicies", "ciliuml2announcementpolicies/status", "ciliumpodippools"]
+  }
+}
+

addons/cilium/config.tf

@@ -0,0 +1,196 @@
+resource "kubernetes_config_map" "cilium" {
+  metadata {
+    name      = "cilium"
+    namespace = "kube-system"
+  }
+  data = {
+    # Identity allocation mode selects how identities are shared between cilium
+    # nodes by setting how they are stored. The options are "crd" or "kvstore".
+    # - "crd" stores identities in kubernetes as CRDs (custom resource definition).
+    #   These can be queried with:
+    #     kubectl get ciliumid
+    # - "kvstore" stores identities in a kvstore, etcd or consul, that is
+    #   configured below. Cilium versions before 1.6 supported only the kvstore
+    #   backend. Upgrades from these older cilium versions should continue using
+    #   the kvstore by commenting out the identity-allocation-mode below, or
+    #   setting it to "kvstore".
+    identity-allocation-mode    = "crd"
+    cilium-endpoint-gc-interval = "5m0s"
+    nodes-gc-interval           = "5m0s"
+
+    # If you want to run cilium in debug mode change this value to true
+    debug = "false"
+    # The agent can be put into the following three policy enforcement modes
+    # default, always and never.
+    # https://docs.cilium.io/en/latest/policy/intro/#policy-enforcement-modes
+    enable-policy = "default"
+
+    # Prometheus
+    enable-metrics                 = "true"
+    prometheus-serve-addr          = ":9962"
+    operator-prometheus-serve-addr = ":9963"
+    proxy-prometheus-port          = "9964" # envoy
+
+    # Enable IPv4 addressing. If enabled, all endpoints are allocated an IPv4
+    # address.
+    enable-ipv4 = "true"
+
+    # Enable IPv6 addressing. If enabled, all endpoints are allocated an IPv6
+    # address.
+    enable-ipv6 = "false"
+
+    # Enable probing for a more efficient clock source for the BPF datapath
+    enable-bpf-clock-probe = "true"
+
+    # Enable use of transparent proxying mechanisms (Linux 5.7+)
+    enable-bpf-tproxy = "false"
+
+    # If you want cilium monitor to aggregate tracing for packets, set this level
+    # to "low", "medium", or "maximum". The higher the level, the less packets
+    # that will be seen in monitor output.
+    monitor-aggregation = "medium"
+
+    # The monitor aggregation interval governs the typical time between monitor
+    # notification events for each allowed connection.
+    #
+    # Only effective when monitor aggregation is set to "medium" or higher.
+    monitor-aggregation-interval = "5s"
+
+    # The monitor aggregation flags determine which TCP flags which, upon the
+    # first observation, cause monitor notifications to be generated.
+    #
+    # Only effective when monitor aggregation is set to "medium" or higher.
+    monitor-aggregation-flags = "all"
+
+    # Specifies the ratio (0.0-1.0) of total system memory to use for dynamic
+    # sizing of the TCP CT, non-TCP CT, NAT and policy BPF maps.
+    bpf-map-dynamic-size-ratio = "0.0025"
+    # bpf-policy-map-max specified the maximum number of entries in endpoint
+    # policy map (per endpoint)
+    bpf-policy-map-max = "16384"
+    # bpf-lb-map-max specifies the maximum number of entries in bpf lb service,
+    # backend and affinity maps.
+    bpf-lb-map-max = "65536"
+
+    # Pre-allocation of map entries allows per-packet latency to be reduced, at
+    # the expense of up-front memory allocation for the entries in the maps. The
+    # default value below will minimize memory usage in the default installation;
+    # users who are sensitive to latency may consider setting this to "true".
+    #
+    # This option was introduced in Cilium 1.4. Cilium 1.3 and earlier ignore
+    # this option and behave as though it is set to "true".
+    #
+    # If this value is modified, then during the next Cilium startup the restore
+    # of existing endpoints and tracking of ongoing connections may be disrupted.
+    # As a result, reply packets may be dropped and the load-balancing decisions
+    # for established connections may change.
+    #
+    # If this option is set to "false" during an upgrade from 1.3 or earlier to
+    # 1.4 or later, then it may cause one-time disruptions during the upgrade.
+    preallocate-bpf-maps = "false"
+
+    # Name of the cluster. Only relevant when building a mesh of clusters.
+    cluster-name = "default"
+    # Unique ID of the cluster. Must be unique across all conneted clusters and
+    # in the range of 1 and 255. Only relevant when building a mesh of clusters.
+    cluster-id = "0"
+
+    # Encapsulation mode for communication between nodes
+    # Possible values:
+    #   - disabled
+    #   - vxlan (default)
+    #   - geneve
+    routing-mode = "tunnel"
+    tunnel       = "vxlan"
+    # Enables L7 proxy for L7 policy enforcement and visibility
+    enable-l7-proxy = "true"
+
+    auto-direct-node-routes = "false"
+
+    # enableXTSocketFallback enables the fallback compatibility solution
+    # when the xt_socket kernel module is missing and it is needed for
+    # the datapath L7 redirection to work properly.  See documentation
+    # for details on when this can be disabled:
+    # http://docs.cilium.io/en/latest/install/system_requirements/#admin-kernel-version.
+    enable-xt-socket-fallback = "true"
+
+    # installIptablesRules enables installation of iptables rules to allow for
+    # TPROXY (L7 proxy injection), itpables based masquerading and compatibility
+    # with kube-proxy. See documentation for details on when this can be
+    # disabled.
+    install-iptables-rules = "true"
+
+    # masquerade traffic leaving the node destined for outside
+    enable-ipv4-masquerade = "true"
+    enable-ipv6-masquerade = "false"
+
+    # bpfMasquerade enables masquerading with BPF instead of iptables
+    enable-bpf-masquerade = "true"
+
+    # kube-proxy
+    kube-proxy-replacement                      = "true"
+    kube-proxy-replacement-healthz-bind-address = ":10256"
+    enable-session-affinity                     = "true"
+
+    # ClusterIPs from host namespace
+    bpf-lb-sock = "true"
+    # ClusterIPs from external nodes
+    bpf-lb-external-clusterip = "true"
+
+    # NodePort
+    enable-node-port             = "true"
+    enable-health-check-nodeport = "false"
+
+    # ExternalIPs
+    enable-external-ips = "true"
+
+    # HostPort
+    enable-host-port = "true"
+
+    # IPAM
+    ipam                        = "cluster-pool"
+    disable-cnp-status-updates  = "true"
+    cluster-pool-ipv4-cidr      = "${var.pod_cidr}"
+    cluster-pool-ipv4-mask-size = "24"
+
+    # Health
+    agent-health-port               = "9876"
+    enable-health-checking          = "true"
+    enable-endpoint-health-checking = "true"
+
+    # Identity
+    enable-well-known-identities = "false"
+    enable-remote-node-identity  = "true"
+
+    # Hubble server
+    enable-hubble                  = var.enable_hubble
+    hubble-disable-tls             = "false"
+    hubble-listen-address          = ":4244"
+    hubble-socket-path             = "/var/run/cilium/hubble.sock"
+    hubble-tls-client-ca-files     = "/var/lib/cilium/tls/hubble/client-ca.crt"
+    hubble-tls-cert-file           = "/var/lib/cilium/tls/hubble/server.crt"
+    hubble-tls-key-file            = "/var/lib/cilium/tls/hubble/server.key"
+    hubble-export-file-max-backups = "5"
+    hubble-export-file-max-size-mb = "10"
+
+    # Hubble metrics
+    hubble-metrics-server      = ":9965"
+    hubble-metrics             = "dns drop tcp flow port-distribution icmp httpV2"
+    enable-hubble-open-metrics = "false"
+
+
+    # Misc
+    enable-bandwidth-manager        = "false"
+    enable-local-redirect-policy    = "false"
+    policy-audit-mode               = "false"
+    operator-api-serve-addr         = "127.0.0.1:9234"
+    enable-l2-neigh-discovery       = "true"
+    enable-k8s-terminating-endpoint = "true"
+    enable-k8s-networkpolicy        = "true"
+    external-envoy-proxy            = "false"
+    write-cni-conf-when-ready       = "/host/etc/cni/net.d/05-cilium.conflist"
+    cni-exclusive                   = "true"
+    cni-log-file                    = "/var/run/cilium/cilium-cni.log"
+  }
+}
+

addons/cilium/daemonset.tf

@@ -0,0 +1,379 @@
+resource "kubernetes_daemonset" "cilium" {
+  wait_for_rollout = false
+
+  metadata {
+    name      = "cilium"
+    namespace = "kube-system"
+    labels = {
+      k8s-app = "cilium"
+    }
+  }
+  spec {
+    strategy {
+      type = "RollingUpdate"
+      rolling_update {
+        max_unavailable = "1"
+      }
+    }
+    selector {
+      match_labels = {
+        k8s-app = "cilium-agent"
+      }
+    }
+    template {
+      metadata {
+        labels = {
+          k8s-app = "cilium-agent"
+        }
+        annotations = {
+          "prometheus.io/port"   = "9962"
+          "prometheus.io/scrape" = "true"
+        }
+      }
+      spec {
+        host_network         = true
+        priority_class_name  = "system-node-critical"
+        service_account_name = "cilium-agent"
+        security_context {
+          seccomp_profile {
+            type = "RuntimeDefault"
+          }
+        }
+        toleration {
+          key      = "node-role.kubernetes.io/controller"
+          operator = "Exists"
+        }
+        toleration {
+          key      = "node.kubernetes.io/not-ready"
+          operator = "Exists"
+        }
+        dynamic "toleration" {
+          for_each = var.daemonset_tolerations
+          content {
+            key      = toleration.value
+            operator = "Exists"
+          }
+        }
+        automount_service_account_token = true
+        enable_service_links            = false
+
+        # Cilium v1.13.1 starts installing CNI plugins in yet another init container
+        # https://github.com/cilium/cilium/pull/24075
+        init_container {
+          name    = "install-cni"
+          image   = "quay.io/cilium/cilium:v1.16.1"
+          command = ["/install-plugin.sh"]
+          security_context {
+            allow_privilege_escalation = true
+            privileged                 = true
+            capabilities {
+              drop = ["ALL"]
+            }
+          }
+          volume_mount {
+            name       = "cni-bin-dir"
+            mount_path = "/host/opt/cni/bin"
+          }
+        }
+
+        # Required to mount cgroup2 filesystem on the underlying Kubernetes node.
+        # We use nsenter command with host's cgroup and mount namespaces enabled.
+        init_container {
+          name  = "mount-cgroup"
+          image = "quay.io/cilium/cilium:v1.16.1"
+          command = [
+            "sh",
+            "-ec",
+            # The statically linked Go program binary is invoked to avoid any
+            # dependency on utilities like sh and mount that can be missing on certain
+            # distros installed on the underlying host. Copy the binary to the
+            # same directory where we install cilium cni plugin so that exec permissions
+            # are available.
+            "cp /usr/bin/cilium-mount /hostbin/cilium-mount && nsenter --cgroup=/hostproc/1/ns/cgroup --mount=/hostproc/1/ns/mnt \"$${BIN_PATH}/cilium-mount\" $CGROUP_ROOT; rm /hostbin/cilium-mount"
+          ]
+          env {
+            name  = "CGROUP_ROOT"
+            value = "/run/cilium/cgroupv2"
+          }
+          env {
+            name  = "BIN_PATH"
+            value = "/opt/cni/bin"
+          }
+          security_context {
+            allow_privilege_escalation = true
+            privileged                 = true
+          }
+          volume_mount {
+            name       = "hostproc"
+            mount_path = "/hostproc"
+          }
+          volume_mount {
+            name       = "cni-bin-dir"
+            mount_path = "/hostbin"
+          }
+        }
+
+        init_container {
+          name    = "clean-cilium-state"
+          image   = "quay.io/cilium/cilium:v1.16.1"
+          command = ["/init-container.sh"]
+          security_context {
+            allow_privilege_escalation = true
+            privileged                 = true
+          }
+          volume_mount {
+            name       = "sys-fs-bpf"
+            mount_path = "/sys/fs/bpf"
+          }
+          volume_mount {
+            name       = "var-run-cilium"
+            mount_path = "/var/run/cilium"
+          }
+          # Required to mount cgroup filesystem from the host to cilium agent pod
+          volume_mount {
+            name              = "cilium-cgroup"
+            mount_path        = "/run/cilium/cgroupv2"
+            mount_propagation = "HostToContainer"
+          }
+        }
+
+        container {
+          name    = "cilium-agent"
+          image   = "quay.io/cilium/cilium:v1.16.1"
+          command = ["cilium-agent"]
+          args = [
+            "--config-dir=/tmp/cilium/config-map"
+          ]
+          env {
+            name = "K8S_NODE_NAME"
+            value_from {
+              field_ref {
+                api_version = "v1"
+                field_path  = "spec.nodeName"
+              }
+            }
+          }
+          env {
+            name = "CILIUM_K8S_NAMESPACE"
+            value_from {
+              field_ref {
+                api_version = "v1"
+                field_path  = "metadata.namespace"
+              }
+            }
+          }
+          env {
+            name = "KUBERNETES_SERVICE_HOST"
+            value_from {
+              config_map_key_ref {
+                name = "in-cluster"
+                key  = "apiserver-host"
+              }
+            }
+          }
+          env {
+            name = "KUBERNETES_SERVICE_PORT"
+            value_from {
+              config_map_key_ref {
+                name = "in-cluster"
+                key  = "apiserver-port"
+              }
+            }
+          }
+          port {
+            name           = "peer-service"
+            protocol       = "TCP"
+            container_port = 4244
+          }
+          # Metrics
+          port {
+            name           = "metrics"
+            protocol       = "TCP"
+            container_port = 9962
+          }
+          port {
+            name           = "envoy-metrics"
+            protocol       = "TCP"
+            container_port = 9964
+          }
+          port {
+            name           = "hubble-metrics"
+            protocol       = "TCP"
+            container_port = 9965
+          }
+          # Not yet used, prefer exec's
+          port {
+            name           = "health"
+            protocol       = "TCP"
+            container_port = 9876
+          }
+          lifecycle {
+            pre_stop {
+              exec {
+                command = ["/cni-uninstall.sh"]
+              }
+            }
+          }
+          security_context {
+            allow_privilege_escalation = true
+            privileged                 = true
+          }
+          liveness_probe {
+            exec {
+              command = ["cilium", "status", "--brief"]
+            }
+            initial_delay_seconds = 120
+            timeout_seconds       = 5
+            period_seconds        = 30
+            success_threshold     = 1
+            failure_threshold     = 10
+          }
+          readiness_probe {
+            exec {
+              command = ["cilium", "status", "--brief"]
+            }
+            initial_delay_seconds = 5
+            timeout_seconds       = 5
+            period_seconds        = 20
+            success_threshold     = 1
+            failure_threshold     = 3
+          }
+          # Load kernel modules
+          volume_mount {
+            name       = "lib-modules"
+            read_only  = true
+            mount_path = "/lib/modules"
+          }
+          # Access iptables concurrently
+          volume_mount {
+            name       = "xtables-lock"
+            mount_path = "/run/xtables.lock"
+          }
+          # Keep state between restarts
+          volume_mount {
+            name       = "var-run-cilium"
+            mount_path = "/var/run/cilium"
+          }
+          volume_mount {
+            name              = "sys-fs-bpf"
+            mount_path        = "/sys/fs/bpf"
+            mount_propagation = "Bidirectional"
+          }
+          # Configuration
+          volume_mount {
+            name       = "config"
+            read_only  = true
+            mount_path = "/tmp/cilium/config-map"
+          }
+          # Install config on host
+          volume_mount {
+            name       = "cni-conf-dir"
+            mount_path = "/host/etc/cni/net.d"
+          }
+          # Hubble
+          volume_mount {
+            name       = "hubble-tls"
+            mount_path = "/var/lib/cilium/tls/hubble"
+            read_only  = true
+          }
+        }
+        termination_grace_period_seconds = 1
+
+        # Load kernel modules
+        volume {
+          name = "lib-modules"
+          host_path {
+            path = "/lib/modules"
+          }
+        }
+        # Access iptables concurrently with other processes (e.g. kube-proxy)
+        volume {
+          name = "xtables-lock"
+          host_path {
+            path = "/run/xtables.lock"
+            type = "FileOrCreate"
+          }
+        }
+        # Keep state between restarts
+        volume {
+          name = "var-run-cilium"
+          host_path {
+            path = "/var/run/cilium"
+            type = "DirectoryOrCreate"
+          }
+        }
+        # Keep state for bpf maps between restarts
+        volume {
+          name = "sys-fs-bpf"
+          host_path {
+            path = "/sys/fs/bpf"
+            type = "DirectoryOrCreate"
+          }
+        }
+        # Mount host cgroup2 filesystem
+        volume {
+          name = "hostproc"
+          host_path {
+            path = "/proc"
+            type = "Directory"
+          }
+        }
+        volume {
+          name = "cilium-cgroup"
+          host_path {
+            path = "/run/cilium/cgroupv2"
+            type = "DirectoryOrCreate"
+          }
+        }
+        # Read configuration
+        volume {
+          name = "config"
+          config_map {
+            name = "cilium"
+          }
+        }
+        # Install CNI plugin and config on host
+        volume {
+          name = "cni-bin-dir"
+          host_path {
+            path = "/opt/cni/bin"
+            type = "DirectoryOrCreate"
+          }
+        }
+        volume {
+          name = "cni-conf-dir"
+          host_path {
+            path = "/etc/cni/net.d"
+            type = "DirectoryOrCreate"
+          }
+        }
+        # Hubble TLS (optional)
+        volume {
+          name = "hubble-tls"
+          projected {
+            default_mode = "0400"
+            sources {
+              secret {
+                name     = "hubble-server-certs"
+                optional = true
+                items {
+                  key  = "ca.crt"
+                  path = "client-ca.crt"
+                }
+                items {
+                  key  = "tls.crt"
+                  path = "server.crt"
+                }
+                items {
+                  key  = "tls.key"
+                  path = "server.key"
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+

addons/cilium/deployment.tf

@@ -0,0 +1,163 @@
+resource "kubernetes_deployment" "operator" {
+  wait_for_rollout = false
+  metadata {
+    name      = "cilium-operator"
+    namespace = "kube-system"
+  }
+  spec {
+    replicas = 1
+    strategy {
+      type = "RollingUpdate"
+      rolling_update {
+        max_unavailable = "1"
+      }
+    }
+    selector {
+      match_labels = {
+        name = "cilium-operator"
+      }
+    }
+    template {
+      metadata {
+        labels = {
+          name = "cilium-operator"
+        }
+        annotations = {
+          "prometheus.io/scrape" = "true"
+          "prometheus.io/port"   = "9963"
+        }
+      }
+      spec {
+        host_network         = true
+        priority_class_name  = "system-cluster-critical"
+        service_account_name = "cilium-operator"
+        security_context {
+          seccomp_profile {
+            type = "RuntimeDefault"
+          }
+        }
+        toleration {
+          key      = "node-role.kubernetes.io/controller"
+          operator = "Exists"
+        }
+        toleration {
+          key      = "node.kubernetes.io/not-ready"
+          operator = "Exists"
+        }
+        topology_spread_constraint {
+          max_skew           = 1
+          topology_key       = "kubernetes.io/hostname"
+          when_unsatisfiable = "DoNotSchedule"
+          label_selector {
+            match_labels = {
+              name = "cilium-operator"
+            }
+          }
+        }
+        automount_service_account_token = true
+        enable_service_links            = false
+        container {
+          name    = "cilium-operator"
+          image   = "quay.io/cilium/operator-generic:v1.16.1"
+          command = ["cilium-operator-generic"]
+          args = [
+            "--config-dir=/tmp/cilium/config-map",
+            "--debug=$(CILIUM_DEBUG)"
+          ]
+          env {
+            name = "K8S_NODE_NAME"
+            value_from {
+              field_ref {
+                api_version = "v1"
+                field_path  = "spec.nodeName"
+              }
+            }
+          }
+          env {
+            name = "CILIUM_K8S_NAMESPACE"
+            value_from {
+              field_ref {
+                api_version = "v1"
+                field_path  = "metadata.namespace"
+              }
+            }
+          }
+          env {
+            name = "KUBERNETES_SERVICE_HOST"
+            value_from {
+              config_map_key_ref {
+                name = "in-cluster"
+                key  = "apiserver-host"
+              }
+            }
+          }
+          env {
+            name = "KUBERNETES_SERVICE_PORT"
+            value_from {
+              config_map_key_ref {
+                name = "in-cluster"
+                key  = "apiserver-port"
+              }
+            }
+          }
+          env {
+            name = "CILIUM_DEBUG"
+            value_from {
+              config_map_key_ref {
+                name     = "cilium"
+                key      = "debug"
+                optional = true
+              }
+            }
+          }
+          port {
+            name           = "metrics"
+            protocol       = "TCP"
+            host_port      = 9963
+            container_port = 9963
+          }
+          port {
+            name           = "health"
+            container_port = 9234
+            protocol       = "TCP"
+          }
+          liveness_probe {
+            http_get {
+              scheme = "HTTP"
+              host   = "127.0.0.1"
+              port   = "9234"
+              path   = "/healthz"
+            }
+            initial_delay_seconds = 60
+            timeout_seconds       = 3
+            period_seconds        = 10
+          }
+          readiness_probe {
+            http_get {
+              scheme = "HTTP"
+              host   = "127.0.0.1"
+              port   = "9234"
+              path   = "/healthz"
+            }
+            timeout_seconds   = 3
+            period_seconds    = 15
+            failure_threshold = 5
+          }
+          volume_mount {
+            name       = "config"
+            read_only  = true
+            mount_path = "/tmp/cilium/config-map"
+          }
+        }
+
+        volume {
+          name = "config"
+          config_map {
+            name = "cilium"
+          }
+        }
+      }
+    }
+  }
+}
+

addons/cilium/service-account.tf

@@ -0,0 +1,15 @@
+resource "kubernetes_service_account" "operator" {
+  metadata {
+    name      = "cilium-operator"
+    namespace = "kube-system"
+  }
+  automount_service_account_token = false
+}
+
+resource "kubernetes_service_account" "agent" {
+  metadata {
+    name      = "cilium-agent"
+    namespace = "kube-system"
+  }
+  automount_service_account_token = false
+}

addons/cilium/variables.tf

@@ -0,0 +1,17 @@
+variable "pod_cidr" {
+  type        = string
+  description = "CIDR IP range to assign Kubernetes pods"
+  default     = "10.2.0.0/16"
+}
+
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}
+
+variable "enable_hubble" {
+  type        = bool
+  description = "Run the embedded Hubble Server and mount hubble-server-certs Secret"
+  default     = true
+}

addons/cilium/versions.tf

@@ -0,0 +1,8 @@
+terraform {
+  required_providers {
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = "~> 2.8"
+    }
+  }
+}

addons/coredns/cluster-role.tf

@@ -0,0 +1,37 @@
+resource "kubernetes_cluster_role" "coredns" {
+  metadata {
+    name = "system:coredns"
+  }
+  rule {
+    api_groups = [""]
+    resources = [
+      "endpoints",
+      "services",
+      "pods",
+      "namespaces",
+    ]
+    verbs = [
+      "list",
+      "watch",
+    ]
+  }
+  rule {
+    api_groups = [""]
+    resources = [
+      "nodes",
+    ]
+    verbs = [
+      "get",
+    ]
+  }
+  rule {
+    api_groups = ["discovery.k8s.io"]
+    resources = [
+      "endpointslices",
+    ]
+    verbs = [
+      "list",
+      "watch",
+    ]
+  }
+}

addons/coredns/config.tf

@@ -0,0 +1,30 @@
+resource "kubernetes_config_map" "coredns" {
+  metadata {
+    name      = "coredns"
+    namespace = "kube-system"
+  }
+  data = {
+    "Corefile" = <<-EOF
+      .:53 {
+          errors
+          health {
+            lameduck 5s
+          }
+          ready
+          log . {
+              class error
+          }
+          kubernetes ${var.cluster_domain_suffix} in-addr.arpa ip6.arpa {
+              pods insecure
+              fallthrough in-addr.arpa ip6.arpa
+          }
+          prometheus :9153
+          forward . /etc/resolv.conf
+          cache 30
+          loop
+          reload
+          loadbalance
+      }
+  EOF
+  }
+}

addons/coredns/deployment.tf

@@ -0,0 +1,151 @@
+resource "kubernetes_deployment" "coredns" {
+  wait_for_rollout = false
+  metadata {
+    name      = "coredns"
+    namespace = "kube-system"
+    labels = {
+      k8s-app              = "coredns"
+      "kubernetes.io/name" = "CoreDNS"
+    }
+  }
+  spec {
+    replicas = var.replicas
+    strategy {
+      type = "RollingUpdate"
+      rolling_update {
+        max_unavailable = "1"
+      }
+    }
+    selector {
+      match_labels = {
+        k8s-app = "coredns"
+        tier    = "control-plane"
+      }
+    }
+    template {
+      metadata {
+        labels = {
+          k8s-app = "coredns"
+          tier    = "control-plane"
+        }
+      }
+      spec {
+        affinity {
+          node_affinity {
+            preferred_during_scheduling_ignored_during_execution {
+              weight = 100
+              preference {
+                match_expressions {
+                  key      = "node.kubernetes.io/controller"
+                  operator = "Exists"
+                }
+              }
+            }
+          }
+          pod_anti_affinity {
+            preferred_during_scheduling_ignored_during_execution {
+              weight = 100
+              pod_affinity_term {
+                label_selector {
+                  match_expressions {
+                    key      = "tier"
+                    operator = "In"
+                    values   = ["control-plane"]
+                  }
+                  match_expressions {
+                    key      = "k8s-app"
+                    operator = "In"
+                    values   = ["coredns"]
+                  }
+                }
+                topology_key = "kubernetes.io/hostname"
+              }
+            }
+          }
+        }
+        dns_policy          = "Default"
+        priority_class_name = "system-cluster-critical"
+        security_context {
+          seccomp_profile {
+            type = "RuntimeDefault"
+          }
+        }
+        service_account_name = "coredns"
+        toleration {
+          key    = "node-role.kubernetes.io/controller"
+          effect = "NoSchedule"
+        }
+        container {
+          name  = "coredns"
+          image = "registry.k8s.io/coredns/coredns:v1.11.3"
+          args  = ["-conf", "/etc/coredns/Corefile"]
+          port {
+            name           = "dns"
+            container_port = 53
+            protocol       = "UDP"
+          }
+          port {
+            name           = "dns-tcp"
+            container_port = 53
+            protocol       = "TCP"
+          }
+          port {
+            name           = "metrics"
+            container_port = 9153
+            protocol       = "TCP"
+          }
+          resources {
+            requests = {
+              cpu    = "100m"
+              memory = "70Mi"
+            }
+            limits = {
+              memory = "170Mi"
+            }
+          }
+          security_context {
+            capabilities {
+              add  = ["NET_BIND_SERVICE"]
+              drop = ["all"]
+            }
+            read_only_root_filesystem = true
+          }
+          liveness_probe {
+            http_get {
+              path   = "/health"
+              port   = "8080"
+              scheme = "HTTP"
+            }
+            initial_delay_seconds = 60
+            timeout_seconds       = 5
+            success_threshold     = 1
+            failure_threshold     = 5
+          }
+          readiness_probe {
+            http_get {
+              path   = "/ready"
+              port   = "8181"
+              scheme = "HTTP"
+            }
+          }
+          volume_mount {
+            name       = "config"
+            mount_path = "/etc/coredns"
+            read_only  = true
+          }
+        }
+        volume {
+          name = "config"
+          config_map {
+            name = "coredns"
+            items {
+              key  = "Corefile"
+              path = "Corefile"
+            }
+          }
+        }
+      }
+    }
+  }
+}
+

addons/coredns/service-account.tf

@@ -0,0 +1,24 @@
+resource "kubernetes_service_account" "coredns" {
+  metadata {
+    name      = "coredns"
+    namespace = "kube-system"
+  }
+  automount_service_account_token = false
+}
+
+
+resource "kubernetes_cluster_role_binding" "coredns" {
+  metadata {
+    name = "system:coredns"
+  }
+  role_ref {
+    api_group = "rbac.authorization.k8s.io"
+    kind      = "ClusterRole"
+    name      = "system:coredns"
+  }
+  subject {
+    kind      = "ServiceAccount"
+    name      = "coredns"
+    namespace = "kube-system"
+  }
+}

addons/coredns/service.tf

@@ -0,0 +1,31 @@
+resource "kubernetes_service" "coredns" {
+  metadata {
+    name      = "coredns"
+    namespace = "kube-system"
+    labels = {
+      "k8s-app"            = "coredns"
+      "kubernetes.io/name" = "CoreDNS"
+    }
+    annotations = {
+      "prometheus.io/scrape" = "true"
+      "prometheus.io/port"   = "9153"
+    }
+  }
+  spec {
+    type       = "ClusterIP"
+    cluster_ip = var.cluster_dns_service_ip
+    selector = {
+      k8s-app = "coredns"
+    }
+    port {
+      name     = "dns"
+      protocol = "UDP"
+      port     = 53
+    }
+    port {
+      name     = "dns-tcp"
+      protocol = "TCP"
+      port     = 53
+    }
+  }
+}

addons/coredns/variables.tf

@@ -0,0 +1,15 @@
+variable "replicas" {
+  type        = number
+  description = "CoreDNS replica count"
+  default     = 2
+}
+
+variable "cluster_dns_service_ip" {
+  description = "Must be set to `cluster_dns_service_ip` output by cluster"
+  default     = "10.3.0.10"
+}
+
+variable "cluster_domain_suffix" {
+  description = "Must be set to `cluster_domain_suffix` output by cluster"
+  default     = "cluster.local"
+}

addons/coredns/versions.tf

@@ -0,0 +1,9 @@
+terraform {
+  required_providers {
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = "~> 2.8"
+    }
+  }
+}
+

addons/flannel/cluster-role-binding.tf

@@ -0,0 +1,18 @@
+resource "kubernetes_cluster_role_binding" "flannel" {
+  metadata {
+    name = "flannel"
+  }
+
+  role_ref {
+    api_group = "rbac.authorization.k8s.io"
+    kind      = "ClusterRole"
+    name      = "flannel"
+  }
+
+  subject {
+    kind      = "ServiceAccount"
+    name      = "flannel"
+    namespace = "kube-system"
+  }
+}
+

addons/flannel/cluster-role.tf

@@ -0,0 +1,24 @@
+resource "kubernetes_cluster_role" "flannel" {
+  metadata {
+    name = "flannel"
+  }
+
+  rule {
+    api_groups = [""]
+    resources  = ["pods"]
+    verbs      = ["get"]
+  }
+
+  rule {
+    api_groups = [""]
+    resources  = ["nodes"]
+    verbs      = ["list", "watch"]
+  }
+
+  rule {
+    api_groups = [""]
+    resources  = ["nodes/status"]
+    verbs      = ["patch"]
+  }
+}
+

addons/flannel/config.tf

@@ -0,0 +1,44 @@
+resource "kubernetes_config_map" "config" {
+  metadata {
+    name      = "flannel-config"
+    namespace = "kube-system"
+    labels = {
+      k8s-app = "flannel"
+      tier    = "node"
+    }
+  }
+
+  data = {
+    "cni-conf.json" = <<-EOF
+      {
+        "name": "cbr0",
+        "cniVersion": "0.3.1",
+        "plugins": [
+          {
+            "type": "flannel",
+            "delegate": {
+              "hairpinMode": true,
+              "isDefaultGateway": true
+            }
+          },
+          {
+            "type": "portmap",
+            "capabilities": {
+              "portMappings": true
+            }
+          }
+        ]
+      }
+    EOF
+    "net-conf.json" = <<-EOF
+      {
+        "Network": "${var.pod_cidr}",
+        "Backend": {
+          "Type": "vxlan",
+          "Port": 4789
+        }
+      }
+    EOF
+  }
+}
+

addons/flannel/daemonset.tf

@@ -0,0 +1,167 @@
+resource "kubernetes_daemonset" "flannel" {
+  metadata {
+    name      = "flannel"
+    namespace = "kube-system"
+    labels = {
+      k8s-app = "flannel"
+    }
+  }
+  spec {
+    strategy {
+      type = "RollingUpdate"
+      rolling_update {
+        max_unavailable = "1"
+      }
+    }
+    selector {
+      match_labels = {
+        k8s-app = "flannel"
+      }
+    }
+    template {
+      metadata {
+        labels = {
+          k8s-app = "flannel"
+        }
+      }
+      spec {
+        host_network         = true
+        priority_class_name  = "system-node-critical"
+        service_account_name = "flannel"
+        security_context {
+          seccomp_profile {
+            type = "RuntimeDefault"
+          }
+        }
+        toleration {
+          key      = "node-role.kubernetes.io/controller"
+          operator = "Exists"
+        }
+        toleration {
+          key      = "node.kubernetes.io/not-ready"
+          operator = "Exists"
+        }
+        dynamic "toleration" {
+          for_each = var.daemonset_tolerations
+          content {
+            key      = toleration.value
+            operator = "Exists"
+          }
+        }
+        init_container {
+          name    = "install-cni"
+          image   = "quay.io/poseidon/flannel-cni:v0.4.2"
+          command = ["/install-cni.sh"]
+          env {
+            name = "CNI_NETWORK_CONFIG"
+            value_from {
+              config_map_key_ref {
+                name = "flannel-config"
+                key  = "cni-conf.json"
+              }
+            }
+          }
+          volume_mount {
+            name       = "cni-bin-dir"
+            mount_path = "/host/opt/cni/bin/"
+          }
+          volume_mount {
+            name       = "cni-conf-dir"
+            mount_path = "/host/etc/cni/net.d"
+          }
+        }
+
+        container {
+          name  = "flannel"
+          image = "docker.io/flannel/flannel:v0.25.5"
+          command = [
+            "/opt/bin/flanneld",
+            "--ip-masq",
+            "--kube-subnet-mgr",
+            "--iface=$(POD_IP)"
+          ]
+          env {
+            name = "POD_NAME"
+            value_from {
+              field_ref {
+                field_path = "metadata.name"
+              }
+            }
+          }
+          env {
+            name = "POD_NAMESPACE"
+            value_from {
+              field_ref {
+                field_path = "metadata.namespace"
+              }
+            }
+          }
+          env {
+            name = "POD_IP"
+            value_from {
+              field_ref {
+                field_path = "status.podIP"
+              }
+            }
+          }
+          security_context {
+            privileged = true
+          }
+          resources {
+            requests = {
+              cpu = "100m"
+            }
+          }
+          volume_mount {
+            name       = "flannel-config"
+            mount_path = "/etc/kube-flannel/"
+          }
+          volume_mount {
+            name       = "run-flannel"
+            mount_path = "/run/flannel"
+          }
+          volume_mount {
+            name       = "xtables-lock"
+            mount_path = "/run/xtables.lock"
+          }
+        }
+
+        volume {
+          name = "flannel-config"
+          config_map {
+            name = "flannel-config"
+          }
+        }
+        volume {
+          name = "run-flannel"
+          host_path {
+            path = "/run/flannel"
+          }
+        }
+        # Used by install-cni
+        volume {
+          name = "cni-bin-dir"
+          host_path {
+            path = "/opt/cni/bin"
+          }
+        }
+        volume {
+          name = "cni-conf-dir"
+          host_path {
+            path = "/etc/cni/net.d"
+            type = "DirectoryOrCreate"
+          }
+        }
+        # Acces iptables concurrently
+        volume {
+          name = "xtables-lock"
+          host_path {
+            path = "/run/xtables.lock"
+            type = "FileOrCreate"
+          }
+        }
+      }
+    }
+  }
+}
+

addons/flannel/service-account.tf

@@ -0,0 +1,7 @@
+resource "kubernetes_service_account" "flannel" {
+  metadata {
+    name      = "flannel"
+    namespace = "kube-system"
+  }
+}
+

addons/flannel/variables.tf

@@ -0,0 +1,11 @@
+variable "pod_cidr" {
+  type        = string
+  description = "CIDR IP range to assign Kubernetes pods"
+  default     = "10.2.0.0/16"
+}
+
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}

addons/flannel/versions.tf

@@ -0,0 +1,8 @@
+terraform {
+  required_providers {
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = "~> 2.8"
+    }
+  }
+}

addons/grafana/dashboards-etcd.yaml

@@ -140,7 +140,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(grpc_server_started_total{job=\"$cluster\",grpc_type=\"unary\"}[5m]))",
+                  "expr": "sum(rate(grpc_server_started_total{job=\"$cluster\",grpc_type=\"unary\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "RPC Rate",
@@ -149,7 +149,7 @@ data:
                   "step": 2
                 },
                 {
-                  "expr": "sum(rate(grpc_server_handled_total{job=\"$cluster\",grpc_type=\"unary\",grpc_code!=\"OK\"}[5m]))",
+                  "expr": "sum(rate(grpc_server_handled_total{job=\"$cluster\",grpc_type=\"unary\",grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "RPC Failed Rate",
@@ -430,7 +430,7 @@ data:
               "steppedLine": true,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\"$cluster\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\"$cluster\"}[$__rate_interval])) by (instance, le))",
                   "hide": false,
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}} WAL fsync",
@@ -439,7 +439,7 @@ data:
                   "step": 4
                 },
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\"$cluster\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\"$cluster\"}[$__rate_interval])) by (instance, le))",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}} DB fsync",
                   "metric": "etcd_disk_backend_commit_duration_seconds_bucket",
@@ -617,7 +617,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(etcd_network_client_grpc_received_bytes_total{job=\"$cluster\"}[5m])",
+                  "expr": "rate(etcd_network_client_grpc_received_bytes_total{job=\"$cluster\"}[$__rate_interval])",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}} Client Traffic In",
                   "metric": "etcd_network_client_grpc_received_bytes_total",
@@ -703,7 +703,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(etcd_network_client_grpc_sent_bytes_total{job=\"$cluster\"}[5m])",
+                  "expr": "rate(etcd_network_client_grpc_sent_bytes_total{job=\"$cluster\"}[$__rate_interval])",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}} Client Traffic Out",
                   "metric": "etcd_network_client_grpc_sent_bytes_total",
@@ -789,7 +789,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(etcd_network_peer_received_bytes_total{job=\"$cluster\"}[5m])) by (instance)",
+                  "expr": "sum(rate(etcd_network_peer_received_bytes_total{job=\"$cluster\"}[$__rate_interval])) by (instance)",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}} Peer Traffic In",
                   "metric": "etcd_network_peer_received_bytes_total",
@@ -878,7 +878,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(etcd_network_peer_sent_bytes_total{job=\"$cluster\"}[5m])) by (instance)",
+                  "expr": "sum(rate(etcd_network_peer_sent_bytes_total{job=\"$cluster\"}[$__rate_interval])) by (instance)",
                   "hide": false,
                   "interval": "",
                   "intervalFactor": 2,
@@ -972,7 +972,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(etcd_server_proposals_failed_total{job=\"$cluster\"}[5m]))",
+                  "expr": "sum(rate(etcd_server_proposals_failed_total{job=\"$cluster\"}[$__rate_interval]))",
                   "intervalFactor": 2,
                   "legendFormat": "Proposal Failure Rate",
                   "metric": "etcd_server_proposals_failed_total",
@@ -988,7 +988,7 @@ data:
                   "step": 2
                 },
                 {
-                  "expr": "sum(rate(etcd_server_proposals_committed_total{job=\"$cluster\"}[5m]))",
+                  "expr": "sum(rate(etcd_server_proposals_committed_total{job=\"$cluster\"}[$__rate_interval]))",
                   "intervalFactor": 2,
                   "legendFormat": "Proposal Commit Rate",
                   "metric": "etcd_server_proposals_committed_total",
@@ -996,7 +996,7 @@ data:
                   "step": 2
                 },
                 {
-                  "expr": "sum(rate(etcd_server_proposals_applied_total{job=\"$cluster\"}[5m]))",
+                  "expr": "sum(rate(etcd_server_proposals_applied_total{job=\"$cluster\"}[$__rate_interval]))",
                   "intervalFactor": 2,
                   "legendFormat": "Proposal Apply Rate",
                   "refId": "D",
@@ -1131,6 +1131,131 @@ data:
                   "show": true
                 }
               ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": 0,
+              "editable": true,
+              "error": false,
+              "fieldConfig": {
+                "defaults": {
+                  "custom": {
+
+                  }
+                },
+                "overrides": [
+
+                ]
+              },
+              "fill": 0,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 7,
+                "w": 12,
+                "x": 0,
+                "y": 28
+              },
+              "hiddenSeries": false,
+              "id": 42,
+              "isNew": true,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "options": {
+                "alertThreshold": true
+              },
+              "percentage": false,
+              "pluginVersion": "7.4.3",
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "histogram_quantile(0.99, sum by (instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{job=\"$cluster\"}[$__rate_interval])))",
+                  "interval": "",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} Peer round trip time",
+                  "metric": "etcd_network_peer_round_trip_time_seconds_bucket",
+                  "refId": "A",
+                  "step": 2
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeRegions": [
+
+              ],
+              "timeShift": null,
+              "title": "Peer round trip time",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "$$hashKey": "object:925",
+                  "decimals": null,
+                  "format": "s",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "$$hashKey": "object:926",
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ],
+              "yaxis": {
+                "align": false,
+                "alignLevel": null
+              }
             }
           ],
           "title": "New row"
@@ -1140,7 +1265,7 @@ data:
       "sharedCrosshair": false,
       "style": "dark",
       "tags": [
-
+        "etcd-mixin"
       ],
       "templating": {
         "list": [
@@ -1150,7 +1275,7 @@ data:
               "value": "Prometheus"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -1176,7 +1301,7 @@ data:
 
             ],
             "query": "label_values(etcd_server_has_leader, job)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 2,
             "tagValuesQuery": "",

addons/grafana/dashboards-k8s-network.yaml

@@ -0,0 +1,7644 @@
+apiVersion: v1
+data:
+  cluster-total.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "id": 3,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{namespace}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "id": 4,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{namespace}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "columns": [
+            {
+              "text": "Time",
+              "value": "Time"
+            },
+            {
+              "text": "Value #A",
+              "value": "Value #A"
+            },
+            {
+              "text": "Value #B",
+              "value": "Value #B"
+            },
+            {
+              "text": "Value #C",
+              "value": "Value #C"
+            },
+            {
+              "text": "Value #D",
+              "value": "Value #D"
+            },
+            {
+              "text": "Value #E",
+              "value": "Value #E"
+            },
+            {
+              "text": "Value #F",
+              "value": "Value #F"
+            },
+            {
+              "text": "Value #G",
+              "value": "Value #G"
+            },
+            {
+              "text": "Value #H",
+              "value": "Value #H"
+            },
+            {
+              "text": "namespace",
+              "value": "namespace"
+            }
+          ],
+          "datasource": "$datasource",
+          "fill": 1,
+          "fontSize": "90%",
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null as zero",
+          "renderer": "flot",
+          "scroll": true,
+          "showHeader": true,
+          "sort": {
+            "col": 0,
+            "desc": false
+          },
+          "spaceLength": 10,
+          "span": 24,
+          "styles": [
+            {
+              "alias": "Time",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Time",
+              "thresholds": [
+
+              ],
+              "type": "hidden",
+              "unit": "short"
+            },
+            {
+              "alias": "Current Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #A",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Current Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #B",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Average Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #C",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Average Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #D",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Rate of Received Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #E",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #F",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Received Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #G",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #H",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Namespace",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": true,
+              "linkTooltip": "Drill down",
+              "linkUrl": "d/8b7a8b326d7a6f1f04244066368c67af/kubernetes-networking-namespace-pods?orgId=1&refresh=30s&var-namespace=$__cell",
+              "pattern": "namespace",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "short"
+            }
+          ],
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "A",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "B",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "C",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "D",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "E",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "F",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "G",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "H",
+              "step": 10
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Status",
+          "type": "table"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 6,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 11
+              },
+              "id": 7,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Received",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 11
+              },
+              "id": 8,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Transmitted",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Average Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 11
+          },
+          "id": 9,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth History",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 12
+          },
+          "id": 10,
+          "legend": {
+            "alignAsTable": true,
+            "avg": true,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": true,
+            "min": true,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{namespace}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 21
+          },
+          "id": 11,
+          "legend": {
+            "alignAsTable": true,
+            "avg": true,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": true,
+            "min": true,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{namespace}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 30
+          },
+          "id": 12,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 31
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 40
+              },
+              "id": 14,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 31
+          },
+          "id": 15,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 50
+              },
+              "id": 16,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 59
+              },
+              "id": 17,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 59
+              },
+              "id": 18,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+                {
+                  "targetBlank": true,
+                  "title": "What is TCP Retransmit?",
+                  "url": "https://accedian.com/enterprises/blog/network-packet-loss-retransmissions-and-duplicate-acknowledgements/"
+                }
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(rate(node_netstat_Tcp_RetransSegs{cluster=\"$cluster\"}[$interval:$resolution]) / rate(node_netstat_Tcp_OutSegs{cluster=\"$cluster\"}[$interval:$resolution])) by (instance))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{instance}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of TCP Retransmits out of all sent segments",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 59
+              },
+              "id": 19,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+                {
+                  "targetBlank": true,
+                  "title": "Why monitor SYN retransmits?",
+                  "url": "https://github.com/prometheus/node_exporter/issues/1023#issuecomment-408128365"
+                }
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(rate(node_netstat_TcpExt_TCPSynRetrans{cluster=\"$cluster\"}[$interval:$resolution]) / rate(node_netstat_Tcp_RetransSegs{cluster=\"$cluster\"}[$interval:$resolution])) by (instance))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{instance}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of TCP SYN Retransmits out of all retransmits",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Cluster",
+      "uid": "ff635a025bcfea7bc3dd4f508990a3e9",
+      "version": 0
+    }
+  namespace-by-pod.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "cacheTimeout": null,
+          "colorBackground": false,
+          "colorValue": false,
+          "colors": [
+            "#299c46",
+            "rgba(237, 129, 40, 0.89)",
+            "#d44a3a"
+          ],
+          "datasource": "$datasource",
+          "decimals": 0,
+          "format": "time_series",
+          "gauge": {
+            "maxValue": 100,
+            "minValue": 0,
+            "show": false,
+            "thresholdLabels": false,
+            "thresholdMarkers": true
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "height": 9,
+          "id": 3,
+          "interval": null,
+          "links": [
+
+          ],
+          "mappingType": 1,
+          "mappingTypes": [
+            {
+              "name": "value to text",
+              "value": 1
+            },
+            {
+              "name": "range to text",
+              "value": 2
+            }
+          ],
+          "maxDataPoints": 100,
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "nullText": null,
+          "options": {
+            "fieldOptions": {
+              "calcs": [
+                "last"
+              ],
+              "defaults": {
+                "max": 10000000000,
+                "min": 0,
+                "title": "$namespace",
+                "unit": "Bps"
+              },
+              "mappings": [
+
+              ],
+              "override": {
+
+              },
+              "thresholds": [
+                {
+                  "color": "dark-green",
+                  "index": 0,
+                  "value": null
+                },
+                {
+                  "color": "dark-yellow",
+                  "index": 1,
+                  "value": 5000000000
+                },
+                {
+                  "color": "dark-red",
+                  "index": 2,
+                  "value": 7000000000
+                }
+              ],
+              "values": false
+            }
+          },
+          "postfix": "",
+          "postfixFontSize": "50%",
+          "prefix": "",
+          "prefixFontSize": "50%",
+          "rangeMaps": [
+            {
+              "from": "null",
+              "text": "N/A",
+              "to": "null"
+            }
+          ],
+          "span": 12,
+          "sparkline": {
+            "fillColor": "rgba(31, 118, 189, 0.18)",
+            "full": false,
+            "lineColor": "rgb(31, 120, 193)",
+            "show": false
+          },
+          "tableColumn": "",
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution]))",
+              "format": "time_series",
+              "instant": null,
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "thresholds": "",
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "type": "gauge",
+          "valueFontSize": "80%",
+          "valueMaps": [
+            {
+              "op": "=",
+              "text": "N/A",
+              "value": "null"
+            }
+          ],
+          "valueName": "current"
+        },
+        {
+          "cacheTimeout": null,
+          "colorBackground": false,
+          "colorValue": false,
+          "colors": [
+            "#299c46",
+            "rgba(237, 129, 40, 0.89)",
+            "#d44a3a"
+          ],
+          "datasource": "$datasource",
+          "decimals": 0,
+          "format": "time_series",
+          "gauge": {
+            "maxValue": 100,
+            "minValue": 0,
+            "show": false,
+            "thresholdLabels": false,
+            "thresholdMarkers": true
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "height": 9,
+          "id": 4,
+          "interval": null,
+          "links": [
+
+          ],
+          "mappingType": 1,
+          "mappingTypes": [
+            {
+              "name": "value to text",
+              "value": 1
+            },
+            {
+              "name": "range to text",
+              "value": 2
+            }
+          ],
+          "maxDataPoints": 100,
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "nullText": null,
+          "options": {
+            "fieldOptions": {
+              "calcs": [
+                "last"
+              ],
+              "defaults": {
+                "max": 10000000000,
+                "min": 0,
+                "title": "$namespace",
+                "unit": "Bps"
+              },
+              "mappings": [
+
+              ],
+              "override": {
+
+              },
+              "thresholds": [
+                {
+                  "color": "dark-green",
+                  "index": 0,
+                  "value": null
+                },
+                {
+                  "color": "dark-yellow",
+                  "index": 1,
+                  "value": 5000000000
+                },
+                {
+                  "color": "dark-red",
+                  "index": 2,
+                  "value": 7000000000
+                }
+              ],
+              "values": false
+            }
+          },
+          "postfix": "",
+          "postfixFontSize": "50%",
+          "prefix": "",
+          "prefixFontSize": "50%",
+          "rangeMaps": [
+            {
+              "from": "null",
+              "text": "N/A",
+              "to": "null"
+            }
+          ],
+          "span": 12,
+          "sparkline": {
+            "fillColor": "rgba(31, 118, 189, 0.18)",
+            "full": false,
+            "lineColor": "rgb(31, 120, 193)",
+            "show": false
+          },
+          "tableColumn": "",
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution]))",
+              "format": "time_series",
+              "instant": null,
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "thresholds": "",
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "type": "gauge",
+          "valueFontSize": "80%",
+          "valueMaps": [
+            {
+              "op": "=",
+              "text": "N/A",
+              "value": "null"
+            }
+          ],
+          "valueName": "current"
+        },
+        {
+          "columns": [
+            {
+              "text": "Time",
+              "value": "Time"
+            },
+            {
+              "text": "Value #A",
+              "value": "Value #A"
+            },
+            {
+              "text": "Value #B",
+              "value": "Value #B"
+            },
+            {
+              "text": "Value #C",
+              "value": "Value #C"
+            },
+            {
+              "text": "Value #D",
+              "value": "Value #D"
+            },
+            {
+              "text": "Value #E",
+              "value": "Value #E"
+            },
+            {
+              "text": "Value #F",
+              "value": "Value #F"
+            },
+            {
+              "text": "pod",
+              "value": "pod"
+            }
+          ],
+          "datasource": "$datasource",
+          "fill": 1,
+          "fontSize": "100%",
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null as zero",
+          "renderer": "flot",
+          "scroll": true,
+          "showHeader": true,
+          "sort": {
+            "col": 0,
+            "desc": false
+          },
+          "spaceLength": 10,
+          "span": 24,
+          "styles": [
+            {
+              "alias": "Time",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Time",
+              "thresholds": [
+
+              ],
+              "type": "hidden",
+              "unit": "short"
+            },
+            {
+              "alias": "Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #A",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #B",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Rate of Received Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #C",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #D",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Received Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #E",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #F",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Pod",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": true,
+              "linkTooltip": "Drill down",
+              "linkUrl": "d/7a18067ce943a40ae25454675c19ff5c/kubernetes-networking-pod?orgId=1&refresh=30s&var-namespace=$namespace&var-pod=$__cell",
+              "pattern": "pod",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "short"
+            }
+          ],
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "A",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "B",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "C",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "D",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "E",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "F",
+              "step": 10
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Status",
+          "type": "table"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 19
+          },
+          "id": 6,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 20
+          },
+          "id": 7,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 20
+          },
+          "id": 8,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 29
+          },
+          "id": 9,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 0,
+                "y": 30
+              },
+              "id": 10,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 12,
+                "y": 30
+              },
+              "id": 11,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 30
+          },
+          "id": 12,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 0,
+                "y": 40
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 12,
+                "y": 40
+              },
+              "id": 14,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": ".+",
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "kube-system",
+              "value": "kube-system"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "hide": 0,
+            "includeAll": true,
+            "label": null,
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Namespace (Pods)",
+      "uid": "8b7a8b326d7a6f1f04244066368c67af",
+      "version": 0
+    }
+  namespace-by-workload.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "id": 3,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{ workload }}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "id": 4,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{ workload }}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "columns": [
+            {
+              "text": "Time",
+              "value": "Time"
+            },
+            {
+              "text": "Value #A",
+              "value": "Value #A"
+            },
+            {
+              "text": "Value #B",
+              "value": "Value #B"
+            },
+            {
+              "text": "Value #C",
+              "value": "Value #C"
+            },
+            {
+              "text": "Value #D",
+              "value": "Value #D"
+            },
+            {
+              "text": "Value #E",
+              "value": "Value #E"
+            },
+            {
+              "text": "Value #F",
+              "value": "Value #F"
+            },
+            {
+              "text": "Value #G",
+              "value": "Value #G"
+            },
+            {
+              "text": "Value #H",
+              "value": "Value #H"
+            },
+            {
+              "text": "workload",
+              "value": "workload"
+            }
+          ],
+          "datasource": "$datasource",
+          "fill": 1,
+          "fontSize": "90%",
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null as zero",
+          "renderer": "flot",
+          "scroll": true,
+          "showHeader": true,
+          "sort": {
+            "col": 0,
+            "desc": false
+          },
+          "spaceLength": 10,
+          "span": 24,
+          "styles": [
+            {
+              "alias": "Time",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Time",
+              "thresholds": [
+
+              ],
+              "type": "hidden",
+              "unit": "short"
+            },
+            {
+              "alias": "Current Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #A",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Current Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #B",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Average Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #C",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Average Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #D",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Rate of Received Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #E",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #F",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Received Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #G",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #H",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Workload",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": true,
+              "linkTooltip": "Drill down",
+              "linkUrl": "d/728bf77cc1166d2f3133bf25846876cc/kubernetes-networking-workload?orgId=1&refresh=30s&var-namespace=$namespace&var-type=$type&var-workload=$__cell",
+              "pattern": "workload",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "short"
+            }
+          ],
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "A",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "B",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "C",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "D",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "E",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "F",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "G",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "H",
+              "step": 10
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Status",
+          "type": "table"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 19
+          },
+          "id": 6,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 20
+              },
+              "id": 7,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{ workload }}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Received",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 20
+              },
+              "id": 8,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{ workload }}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Transmitted",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Average Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 29
+          },
+          "id": 9,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth HIstory",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 38
+          },
+          "id": 10,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{workload}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 38
+          },
+          "id": 11,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{workload}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 39
+          },
+          "id": 12,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 40
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{workload}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 40
+              },
+              "id": 14,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{workload}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 40
+          },
+          "id": 15,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 41
+              },
+              "id": 16,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{workload}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 41
+              },
+              "id": 17,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{workload}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "kube-system",
+              "value": "kube-system"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "deployment",
+              "value": "deployment"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\"}, workload_type)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "type",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\"}, workload_type)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Namespace (Workload)",
+      "uid": "bbb2a765a623ae38130206c7d94a160f",
+      "version": 0
+    }
+  pod-total.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "cacheTimeout": null,
+          "colorBackground": false,
+          "colorValue": false,
+          "colors": [
+            "#299c46",
+            "rgba(237, 129, 40, 0.89)",
+            "#d44a3a"
+          ],
+          "datasource": "$datasource",
+          "decimals": 0,
+          "format": "time_series",
+          "gauge": {
+            "maxValue": 100,
+            "minValue": 0,
+            "show": false,
+            "thresholdLabels": false,
+            "thresholdMarkers": true
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "height": 9,
+          "id": 3,
+          "interval": null,
+          "links": [
+
+          ],
+          "mappingType": 1,
+          "mappingTypes": [
+            {
+              "name": "value to text",
+              "value": 1
+            },
+            {
+              "name": "range to text",
+              "value": 2
+            }
+          ],
+          "maxDataPoints": 100,
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "nullText": null,
+          "options": {
+            "fieldOptions": {
+              "calcs": [
+                "last"
+              ],
+              "defaults": {
+                "max": 10000000000,
+                "min": 0,
+                "title": "$namespace: $pod",
+                "unit": "Bps"
+              },
+              "mappings": [
+
+              ],
+              "override": {
+
+              },
+              "thresholds": [
+                {
+                  "color": "dark-green",
+                  "index": 0,
+                  "value": null
+                },
+                {
+                  "color": "dark-yellow",
+                  "index": 1,
+                  "value": 5000000000
+                },
+                {
+                  "color": "dark-red",
+                  "index": 2,
+                  "value": 7000000000
+                }
+              ],
+              "values": false
+            }
+          },
+          "postfix": "",
+          "postfixFontSize": "50%",
+          "prefix": "",
+          "prefixFontSize": "50%",
+          "rangeMaps": [
+            {
+              "from": "null",
+              "text": "N/A",
+              "to": "null"
+            }
+          ],
+          "span": 12,
+          "sparkline": {
+            "fillColor": "rgba(31, 118, 189, 0.18)",
+            "full": false,
+            "lineColor": "rgb(31, 120, 193)",
+            "show": false
+          },
+          "tableColumn": "",
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution]))",
+              "format": "time_series",
+              "instant": null,
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "thresholds": "",
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "type": "gauge",
+          "valueFontSize": "80%",
+          "valueMaps": [
+            {
+              "op": "=",
+              "text": "N/A",
+              "value": "null"
+            }
+          ],
+          "valueName": "current"
+        },
+        {
+          "cacheTimeout": null,
+          "colorBackground": false,
+          "colorValue": false,
+          "colors": [
+            "#299c46",
+            "rgba(237, 129, 40, 0.89)",
+            "#d44a3a"
+          ],
+          "datasource": "$datasource",
+          "decimals": 0,
+          "format": "time_series",
+          "gauge": {
+            "maxValue": 100,
+            "minValue": 0,
+            "show": false,
+            "thresholdLabels": false,
+            "thresholdMarkers": true
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "height": 9,
+          "id": 4,
+          "interval": null,
+          "links": [
+
+          ],
+          "mappingType": 1,
+          "mappingTypes": [
+            {
+              "name": "value to text",
+              "value": 1
+            },
+            {
+              "name": "range to text",
+              "value": 2
+            }
+          ],
+          "maxDataPoints": 100,
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "nullText": null,
+          "options": {
+            "fieldOptions": {
+              "calcs": [
+                "last"
+              ],
+              "defaults": {
+                "max": 10000000000,
+                "min": 0,
+                "title": "$namespace: $pod",
+                "unit": "Bps"
+              },
+              "mappings": [
+
+              ],
+              "override": {
+
+              },
+              "thresholds": [
+                {
+                  "color": "dark-green",
+                  "index": 0,
+                  "value": null
+                },
+                {
+                  "color": "dark-yellow",
+                  "index": 1,
+                  "value": 5000000000
+                },
+                {
+                  "color": "dark-red",
+                  "index": 2,
+                  "value": 7000000000
+                }
+              ],
+              "values": false
+            }
+          },
+          "postfix": "",
+          "postfixFontSize": "50%",
+          "prefix": "",
+          "prefixFontSize": "50%",
+          "rangeMaps": [
+            {
+              "from": "null",
+              "text": "N/A",
+              "to": "null"
+            }
+          ],
+          "span": 12,
+          "sparkline": {
+            "fillColor": "rgba(31, 118, 189, 0.18)",
+            "full": false,
+            "lineColor": "rgb(31, 120, 193)",
+            "show": false
+          },
+          "tableColumn": "",
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution]))",
+              "format": "time_series",
+              "instant": null,
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "thresholds": "",
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "type": "gauge",
+          "valueFontSize": "80%",
+          "valueMaps": [
+            {
+              "op": "=",
+              "text": "N/A",
+              "value": "null"
+            }
+          ],
+          "valueName": "current"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 11
+          },
+          "id": 6,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 11
+          },
+          "id": 7,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 20
+          },
+          "id": 8,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 0,
+                "y": 21
+              },
+              "id": 9,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 12,
+                "y": 21
+              },
+              "id": 10,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 21
+          },
+          "id": 11,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 0,
+                "y": 32
+              },
+              "id": 12,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 12,
+                "y": 32
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": ".+",
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "kube-system",
+              "value": "kube-system"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "hide": 0,
+            "includeAll": true,
+            "label": null,
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": ".+",
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "",
+              "value": ""
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}, pod)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "pod",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}, pod)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Pod",
+      "uid": "7a18067ce943a40ae25454675c19ff5c",
+      "version": 0
+    }
+  workload-total.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "id": 3,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{ pod }}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "id": 4,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{ pod }}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 11
+              },
+              "id": 6,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{ pod }}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Received",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 11
+              },
+              "id": 7,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{ pod }}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Transmitted",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Average Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 11
+          },
+          "id": 8,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth HIstory",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 12
+          },
+          "id": 9,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 12
+          },
+          "id": 10,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 21
+          },
+          "id": 11,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 22
+              },
+              "id": 12,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 22
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 22
+          },
+          "id": 14,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 23
+              },
+              "id": 15,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 23
+              },
+              "id": 16,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info{job=\"kube-state-metrics\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": ".+",
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "kube-system",
+              "value": "kube-system"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\"}, namespace)",
+            "hide": 0,
+            "includeAll": true,
+            "label": null,
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "",
+              "value": ""
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\"}, workload)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "workload",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\"}, workload)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "deployment",
+              "value": "deployment"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\"}, workload_type)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "type",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\"}, workload_type)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Workload",
+      "uid": "728bf77cc1166d2f3133bf25846876cc",
+      "version": 0
+    }
+kind: ConfigMap
+metadata:
+  name: grafana-dashboards-k8s-network
+  namespace: monitoring

addons/grafana/dashboards-k8s-nodes.yaml

@@ -21,155 +21,108 @@ data:
       "links": [
 
       ],
-      "refresh": "10s",
-      "rows": [
-        {
-          "collapse": false,
-          "collapsed": false,
       "panels": [
         {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
           "datasource": "$datasource",
-              "format": "none",
+          "fieldConfig": {
-              "gauge": {
+            "defaults": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 2,
-              "interval": null,
               "links": [
 
               ],
-              "mappingType": 1,
+              "mappings": [
-              "mappingTypes": [
+
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
               ],
-              "maxDataPoints": 100,
+              "thresholds": {
-              "nullPointMode": "connected",
+                "mode": "absolute",
-              "nullText": null,
+                "steps": [
-              "postfix": "",
+
-              "postfixFontSize": "50%",
+                ]
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
               },
-              "tableColumn": "",
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
           "targets": [
             {
-                  "expr": "sum(up{cluster=\"$cluster\", job=\"kubelet\"})",
+              "expr": "sum(kubelet_node_name{cluster=\"$cluster\", job=\"kubelet\"})",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "",
               "refId": "A"
             }
           ],
-              "thresholds": "",
+          "title": "Running Kubelets",
-              "title": "Up",
+          "transparent": false,
-              "tooltip": {
+          "type": "stat"
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
         },
         {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
           "datasource": "$datasource",
-              "format": "none",
+          "fieldConfig": {
-              "gauge": {
+            "defaults": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 3,
-              "interval": null,
               "links": [
 
               ],
-              "mappingType": 1,
+              "mappings": [
-              "mappingTypes": [
+
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
               ],
-              "maxDataPoints": 100,
+              "thresholds": {
-              "nullPointMode": "connected",
+                "mode": "absolute",
-              "nullText": null,
+                "steps": [
-              "postfix": "",
+
-              "postfixFontSize": "50%",
+                ]
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
               },
-              "tableColumn": "",
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 4,
+            "y": 0
+          },
+          "id": 3,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
           "targets": [
             {
               "expr": "sum(kubelet_running_pods{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}) OR sum(kubelet_running_pod_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"})",
@@ -179,81 +132,54 @@ data:
               "refId": "A"
             }
           ],
-              "thresholds": "",
           "title": "Running Pods",
-              "tooltip": {
+          "transparent": false,
-                "shared": false
+          "type": "stat"
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
         },
         {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
           "datasource": "$datasource",
-              "format": "none",
+          "fieldConfig": {
-              "gauge": {
+            "defaults": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 4,
-              "interval": null,
               "links": [
 
               ],
-              "mappingType": 1,
+              "mappings": [
-              "mappingTypes": [
+
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
               ],
-              "maxDataPoints": 100,
+              "thresholds": {
-              "nullPointMode": "connected",
+                "mode": "absolute",
-              "nullText": null,
+                "steps": [
-              "postfix": "",
+
-              "postfixFontSize": "50%",
+                ]
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
               },
-              "tableColumn": "",
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 8,
+            "y": 0
+          },
+          "id": 4,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
           "targets": [
             {
               "expr": "sum(kubelet_running_containers{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}) OR sum(kubelet_running_container_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"})",
@@ -263,81 +189,54 @@ data:
               "refId": "A"
             }
           ],
-              "thresholds": "",
+          "title": "Running Containers",
-              "title": "Running Container",
+          "transparent": false,
-              "tooltip": {
+          "type": "stat"
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
         },
         {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
           "datasource": "$datasource",
-              "format": "none",
+          "fieldConfig": {
-              "gauge": {
+            "defaults": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 5,
-              "interval": null,
               "links": [
 
               ],
-              "mappingType": 1,
+              "mappings": [
-              "mappingTypes": [
+
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
               ],
-              "maxDataPoints": 100,
+              "thresholds": {
-              "nullPointMode": "connected",
+                "mode": "absolute",
-              "nullText": null,
+                "steps": [
-              "postfix": "",
+
-              "postfixFontSize": "50%",
+                ]
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
               },
-              "tableColumn": "",
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 12,
+            "y": 0
+          },
+          "id": 5,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
           "targets": [
             {
               "expr": "sum(volume_manager_total_volumes{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\", state=\"actual_state_of_world\"})",
@@ -347,81 +246,54 @@ data:
               "refId": "A"
             }
           ],
-              "thresholds": "",
           "title": "Actual Volume Count",
-              "tooltip": {
+          "transparent": false,
-                "shared": false
+          "type": "stat"
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
         },
         {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
           "datasource": "$datasource",
-              "format": "none",
+          "fieldConfig": {
-              "gauge": {
+            "defaults": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 6,
-              "interval": null,
               "links": [
 
               ],
-              "mappingType": 1,
+              "mappings": [
-              "mappingTypes": [
+
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
               ],
-              "maxDataPoints": 100,
+              "thresholds": {
-              "nullPointMode": "connected",
+                "mode": "absolute",
-              "nullText": null,
+                "steps": [
-              "postfix": "",
+
-              "postfixFontSize": "50%",
+                ]
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
               },
-              "tableColumn": "",
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 16,
+            "y": 0
+          },
+          "id": 6,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
           "targets": [
             {
               "expr": "sum(volume_manager_total_volumes{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\",state=\"desired_state_of_world\"})",
@@ -431,119 +303,67 @@ data:
               "refId": "A"
             }
           ],
-              "thresholds": "",
           "title": "Desired Volume Count",
-              "tooltip": {
+          "transparent": false,
-                "shared": false
+          "type": "stat"
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
         },
         {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
           "datasource": "$datasource",
-              "format": "none",
+          "fieldConfig": {
-              "gauge": {
+            "defaults": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 7,
-              "interval": null,
               "links": [
 
               ],
-              "mappingType": 1,
+              "mappings": [
-              "mappingTypes": [
+
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
               ],
-              "maxDataPoints": 100,
+              "thresholds": {
-              "nullPointMode": "connected",
+                "mode": "absolute",
-              "nullText": null,
+                "steps": [
-              "postfix": "",
+
-              "postfixFontSize": "50%",
+                ]
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
               },
-              "tableColumn": "",
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 20,
+            "y": 0
+          },
+          "id": 7,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
           "targets": [
             {
-                  "expr": "sum(rate(kubelet_node_config_error{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m]))",
+              "expr": "sum(rate(kubelet_node_config_error{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval]))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}}",
               "refId": "A"
             }
           ],
-              "thresholds": "",
           "title": "Config Error Count",
-              "tooltip": {
+          "transparent": false,
-                "shared": false
+          "type": "stat"
         },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -555,7 +375,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 7
           },
           "id": 8,
           "legend": {
@@ -585,12 +408,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "sum(rate(kubelet_runtime_operations_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (operation_type, instance)",
+              "expr": "sum(rate(kubelet_runtime_operations_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (operation_type, instance)",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} {{operation_type}}",
@@ -604,7 +426,7 @@ data:
           "timeShift": null,
           "title": "Operation Rate",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -648,7 +470,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 7
           },
           "id": 9,
           "legend": {
@@ -678,12 +503,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "sum(rate(kubelet_runtime_operations_errors_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, operation_type)",
+              "expr": "sum(rate(kubelet_runtime_operations_errors_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_type)",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} {{operation_type}}",
@@ -697,7 +521,7 @@ data:
           "timeShift": null,
           "title": "Operation Error Rate",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -717,7 +541,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -725,24 +549,11 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -754,7 +565,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 14
           },
           "id": 10,
           "legend": {
@@ -784,12 +598,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 12,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_runtime_operations_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, operation_type, le))",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_runtime_operations_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_type, le))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} {{operation_type}}",
@@ -803,7 +616,7 @@ data:
           "timeShift": null,
           "title": "Operation duration 99th quantile",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -835,20 +648,7 @@ data:
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -860,7 +660,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 21
           },
           "id": 11,
           "legend": {
@@ -890,19 +693,18 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance)",
+              "expr": "sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance)",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} pod",
               "refId": "A"
             },
             {
-                  "expr": "sum(rate(kubelet_pod_worker_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance)",
+              "expr": "sum(rate(kubelet_pod_worker_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance)",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} worker",
@@ -916,7 +718,7 @@ data:
           "timeShift": null,
           "title": "Pod Start Rate",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -936,7 +738,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -944,7 +746,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
@@ -960,7 +762,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 21
           },
           "id": 12,
           "legend": {
@@ -990,19 +795,18 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, le))",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} pod",
               "refId": "A"
             },
             {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, le))",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} worker",
@@ -1016,7 +820,7 @@ data:
           "timeShift": null,
           "title": "Pod Start Duration",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1036,7 +840,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1044,24 +848,11 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -1073,7 +864,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 28
           },
           "id": 13,
           "legend": {
@@ -1105,12 +899,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "sum(rate(storage_operation_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, operation_name, volume_plugin)",
+              "expr": "sum(rate(storage_operation_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_name, volume_plugin)",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}",
@@ -1124,7 +917,7 @@ data:
           "timeShift": null,
           "title": "Storage Operation Rate",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1144,7 +937,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1152,7 +945,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
@@ -1168,7 +961,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 28
           },
           "id": 14,
           "legend": {
@@ -1200,12 +996,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "sum(rate(storage_operation_errors_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, operation_name, volume_plugin)",
+              "expr": "sum(rate(storage_operation_errors_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_name, volume_plugin)",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}",
@@ -1219,7 +1014,7 @@ data:
           "timeShift": null,
           "title": "Storage Operation Error Rate",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1239,7 +1034,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1247,24 +1042,11 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -1276,7 +1058,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 35
           },
           "id": 15,
           "legend": {
@@ -1308,12 +1093,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 12,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "histogram_quantile(0.99, sum(rate(storage_operation_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance, operation_name, volume_plugin, le))",
+              "expr": "histogram_quantile(0.99, sum(rate(storage_operation_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_name, volume_plugin, le))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}",
@@ -1327,7 +1111,7 @@ data:
           "timeShift": null,
           "title": "Storage Operation Duration 99th quantile",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1347,7 +1131,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1355,24 +1139,11 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -1384,7 +1155,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 42
           },
           "id": 16,
           "legend": {
@@ -1414,12 +1188,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "sum(rate(kubelet_cgroup_manager_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance, operation_type)",
+              "expr": "sum(rate(kubelet_cgroup_manager_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_type)",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{operation_type}}",
@@ -1433,7 +1206,7 @@ data:
           "timeShift": null,
           "title": "Cgroup manager operation rate",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1453,7 +1226,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1461,7 +1234,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
@@ -1477,7 +1250,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 42
           },
           "id": 17,
           "legend": {
@@ -1507,12 +1283,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_cgroup_manager_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance, operation_type, le))",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_cgroup_manager_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_type, le))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} {{operation_type}}",
@@ -1526,7 +1301,7 @@ data:
           "timeShift": null,
           "title": "Cgroup manager 99th quantile",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1546,7 +1321,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1554,24 +1329,11 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -1584,7 +1346,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 49
           },
           "id": 18,
           "legend": {
@@ -1614,12 +1379,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "sum(rate(kubelet_pleg_relist_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance)",
+              "expr": "sum(rate(kubelet_pleg_relist_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance)",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}}",
@@ -1633,7 +1397,7 @@ data:
           "timeShift": null,
           "title": "PLEG relist rate",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1653,7 +1417,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1661,7 +1425,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
@@ -1677,7 +1441,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 49
           },
           "id": 19,
           "legend": {
@@ -1707,12 +1474,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 6,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_interval_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, le))",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_interval_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}}",
@@ -1726,7 +1492,7 @@ data:
           "timeShift": null,
           "title": "PLEG relist interval",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1746,7 +1512,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1754,24 +1520,11 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -1783,7 +1536,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 56
           },
           "id": 20,
           "legend": {
@@ -1813,12 +1569,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 12,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, le))",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}}",
@@ -1832,7 +1587,7 @@ data:
           "timeShift": null,
           "title": "PLEG relist duration",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1852,7 +1607,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1860,24 +1615,11 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -1889,7 +1631,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 63
           },
           "id": 21,
           "legend": {
@@ -1919,33 +1664,32 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 12,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"2..\"}[5m]))",
+              "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"2..\"}[$__rate_interval]))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "2xx",
               "refId": "A"
             },
             {
-                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"3..\"}[5m]))",
+              "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"3..\"}[$__rate_interval]))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "3xx",
               "refId": "B"
             },
             {
-                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"4..\"}[5m]))",
+              "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"4..\"}[$__rate_interval]))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "4xx",
               "refId": "C"
             },
             {
-                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"5..\"}[5m]))",
+              "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"5..\"}[$__rate_interval]))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "5xx",
@@ -1959,7 +1703,7 @@ data:
           "timeShift": null,
           "title": "RPC Rate",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -1979,7 +1723,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -1987,24 +1731,11 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -2016,7 +1747,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 70
           },
           "id": 22,
           "legend": {
@@ -2046,12 +1780,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 12,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance, verb, url, le))",
+              "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance, verb, url, le))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}} {{verb}} {{url}}",
@@ -2065,7 +1798,7 @@ data:
           "timeShift": null,
           "title": "Request duration 99th quantile",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -2085,7 +1818,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -2093,24 +1826,11 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
         },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
         {
           "aliasColors": {
 
@@ -2122,7 +1842,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 8,
+            "x": 0,
+            "y": 77
           },
           "id": 23,
           "legend": {
@@ -2152,7 +1875,6 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 4,
           "stack": false,
           "steppedLine": false,
           "targets": [
@@ -2171,7 +1893,7 @@ data:
           "timeShift": null,
           "title": "Memory",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -2215,7 +1937,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 8,
+            "x": 8,
+            "y": 77
           },
           "id": 24,
           "legend": {
@@ -2245,12 +1970,11 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 4,
           "stack": false,
           "steppedLine": false,
           "targets": [
             {
-                  "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])",
+              "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "{{instance}}",
@@ -2264,7 +1988,7 @@ data:
           "timeShift": null,
           "title": "CPU usage",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -2284,7 +2008,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             },
             {
@@ -2292,7 +2016,7 @@ data:
               "label": null,
               "logBase": 1,
               "max": null,
-                  "min": 0,
+              "min": null,
               "show": true
             }
           ]
@@ -2308,7 +2032,10 @@ data:
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-
+            "h": 7,
+            "w": 8,
+            "x": 16,
+            "y": 77
           },
           "id": 25,
           "legend": {
@@ -2338,7 +2065,6 @@ data:
 
           ],
           "spaceLength": 10,
-              "span": 4,
           "stack": false,
           "steppedLine": false,
           "targets": [
@@ -2357,7 +2083,7 @@ data:
           "timeShift": null,
           "title": "Goroutines",
           "tooltip": {
-                "shared": false,
+            "shared": true,
             "sort": 0,
             "value_type": "individual"
           },
@@ -2391,14 +2117,9 @@ data:
           ]
         }
       ],
-          "repeat": null,
+      "refresh": "10s",
-          "repeatIteration": null,
+      "rows": [
-          "repeatRowId": null,
+
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        }
       ],
       "schemaVersion": 14,
       "style": "dark",
@@ -2413,7 +2134,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -2437,7 +2158,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info, cluster)",
+            "query": "label_values(up{job=\"kubelet\"}, cluster)",
             "refresh": 2,
             "regex": "",
             "sort": 1,
@@ -2457,13 +2178,13 @@ data:
             "datasource": "$datasource",
             "hide": 0,
             "includeAll": true,
-            "label": null,
+            "label": "instance",
             "multi": false,
             "name": "instance",
             "options": [
 
             ],
-            "query": "label_values(kubelet_runtime_operations_total{cluster=\"$cluster\", job=\"kubelet\"}, instance)",
+            "query": "label_values(up{job=\"kubelet\",cluster=\"$cluster\"}, instance)",
             "refresh": 2,
             "regex": "",
             "sort": 1,
@@ -2560,7 +2281,11 @@ data:
 
               },
               "id": 2,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
               "links": [
 
               ],
@@ -2599,7 +2324,7 @@ data:
               "tableColumn": "",
               "targets": [
                 {
-                  "expr": "sum(up{job=\"kube-proxy\"})",
+                  "expr": "sum(up{cluster=\"$cluster\", job=\"kube-proxy\"})",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "",
@@ -2636,13 +2361,14 @@ data:
 
               },
               "id": 3,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -2668,7 +2394,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(kubeproxy_sync_proxy_rules_duration_seconds_count{job=\"kube-proxy\", instance=~\"$instance\"}[5m]))",
+                  "expr": "sum(rate(kubeproxy_sync_proxy_rules_duration_seconds_count{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "rate",
@@ -2729,6 +2455,7 @@ data:
 
               },
               "id": 4,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -2761,7 +2488,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99,rate(kubeproxy_sync_proxy_rules_duration_seconds_bucket{job=\"kube-proxy\", instance=~\"$instance\"}[5m]))",
+                  "expr": "histogram_quantile(0.99,rate(kubeproxy_sync_proxy_rules_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -2835,13 +2562,14 @@ data:
 
               },
               "id": 5,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -2867,7 +2595,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(kubeproxy_network_programming_duration_seconds_count{job=\"kube-proxy\", instance=~\"$instance\"}[5m]))",
+                  "expr": "sum(rate(kubeproxy_network_programming_duration_seconds_count{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "rate",
@@ -2928,6 +2656,7 @@ data:
 
               },
               "id": 6,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -2960,7 +2689,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubeproxy_network_programming_duration_seconds_bucket{job=\"kube-proxy\", instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(kubeproxy_network_programming_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -3034,13 +2763,14 @@ data:
 
               },
               "id": 7,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -3066,28 +2796,28 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-proxy\", instance=~\"$instance\",code=~\"2..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\",code=~\"2..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "2xx",
                   "refId": "A"
                 },
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-proxy\", instance=~\"$instance\",code=~\"3..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\",code=~\"3..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "3xx",
                   "refId": "B"
                 },
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-proxy\", instance=~\"$instance\",code=~\"4..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\",code=~\"4..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "4xx",
                   "refId": "C"
                 },
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-proxy\", instance=~\"$instance\",code=~\"5..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\",code=~\"5..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "5xx",
@@ -3148,13 +2878,14 @@ data:
 
               },
               "id": 8,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -3180,7 +2911,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{job=\"kube-proxy\",instance=~\"$instance\",verb=\"POST\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-proxy\",instance=~\"$instance\",verb=\"POST\"}[$__rate_interval])) by (verb, url, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{verb}} {{url}}",
@@ -3254,6 +2985,7 @@ data:
 
               },
               "id": 9,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -3286,7 +3018,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{job=\"kube-proxy\", instance=~\"$instance\", verb=\"GET\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\", verb=\"GET\"}[$__rate_interval])) by (verb, url, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{verb}} {{url}}",
@@ -3360,13 +3092,14 @@ data:
 
               },
               "id": 10,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -3392,7 +3125,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "process_resident_memory_bytes{job=\"kube-proxy\",instance=~\"$instance\"}",
+                  "expr": "process_resident_memory_bytes{cluster=\"$cluster\", job=\"kube-proxy\",instance=~\"$instance\"}",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -3453,13 +3186,14 @@ data:
 
               },
               "id": 11,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -3485,7 +3219,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(process_cpu_seconds_total{job=\"kube-proxy\",instance=~\"$instance\"}[5m])",
+                  "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", job=\"kube-proxy\",instance=~\"$instance\"}[$__rate_interval])",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -3546,13 +3280,14 @@ data:
 
               },
               "id": 12,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -3578,7 +3313,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "go_goroutines{job=\"kube-proxy\",instance=~\"$instance\"}",
+                  "expr": "go_goroutines{cluster=\"$cluster\", job=\"kube-proxy\",instance=~\"$instance\"}",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -3648,7 +3383,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -3662,6 +3397,32 @@ data:
             "allValue": null,
             "current": {
 
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kube-proxy\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
             },
             "datasource": "$datasource",
             "hide": 0,
@@ -3672,7 +3433,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kubeproxy_network_programming_duration_seconds_bucket{job=\"kube-proxy\"}, instance)",
+            "query": "label_values(up{job=\"kube-proxy\", cluster=\"$cluster\", job=\"kube-proxy\"}, instance)",
             "refresh": 2,
             "regex": "",
             "sort": 1,

addons/grafana/dashboards-k8s-resources-1.yaml

@@ -33,10 +33,12 @@ data:
               "id": 1,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -60,7 +62,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "1 - avg(rate(node_cpu_seconds_total{mode=\"idle\", cluster=\"$cluster\"}[$__rate_interval]))",
+                  "expr": "cluster:node_cpu:ratio_rate5m{cluster=\"$cluster\"}",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -73,7 +75,7 @@ data:
               "title": "CPU Utilisation",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -116,11 +118,14 @@ data:
               "fill": 1,
               "format": "percentunit",
               "id": 2,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -144,7 +149,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_cpu_cores{cluster=\"$cluster\"})",
+                  "expr": "sum(namespace_cpu:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable{job=\"kube-state-metrics\",resource=\"cpu\",cluster=\"$cluster\"})",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -157,7 +162,7 @@ data:
               "title": "CPU Requests Commitment",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -200,11 +205,14 @@ data:
               "fill": 1,
               "format": "percentunit",
               "id": 3,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -228,7 +236,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_cpu_cores{cluster=\"$cluster\"})",
+                  "expr": "sum(namespace_cpu:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable{job=\"kube-state-metrics\",resource=\"cpu\",cluster=\"$cluster\"})",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -241,7 +249,7 @@ data:
               "title": "CPU Limits Commitment",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -284,11 +292,14 @@ data:
               "fill": 1,
               "format": "percentunit",
               "id": 4,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -312,7 +323,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "1 - sum(:node_memory_MemAvailable_bytes:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_memory_bytes{cluster=\"$cluster\"})",
+                  "expr": "1 - sum(:node_memory_MemAvailable_bytes:sum{cluster=\"$cluster\"}) / sum(node_memory_MemTotal_bytes{job=\"node-exporter\",cluster=\"$cluster\"})",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -325,7 +336,7 @@ data:
               "title": "Memory Utilisation",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -368,11 +379,14 @@ data:
               "fill": 1,
               "format": "percentunit",
               "id": 5,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -396,7 +410,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_memory_bytes{cluster=\"$cluster\"})",
+                  "expr": "sum(namespace_memory:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable{job=\"kube-state-metrics\",resource=\"memory\",cluster=\"$cluster\"})",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -409,7 +423,7 @@ data:
               "title": "Memory Requests Commitment",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -452,11 +466,14 @@ data:
               "fill": 1,
               "format": "percentunit",
               "id": 6,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -480,7 +497,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_memory_bytes{cluster=\"$cluster\"})",
+                  "expr": "sum(namespace_memory:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable{job=\"kube-state-metrics\",resource=\"memory\",cluster=\"$cluster\"})",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -493,7 +510,7 @@ data:
               "title": "Memory Limits Commitment",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -547,11 +564,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 7,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -575,7 +595,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\"}) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -591,7 +611,7 @@ data:
               "title": "CPU Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -645,11 +665,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 8,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -689,7 +712,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
                   "pattern": "Value #A",
                   "thresholds": [
 
@@ -708,7 +731,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down to workloads",
-                  "linkUrl": "./d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "linkUrl": "/d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
                   "pattern": "Value #B",
                   "thresholds": [
 
@@ -822,7 +845,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
                   "pattern": "namespace",
                   "thresholds": [
 
@@ -848,7 +871,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(kube_pod_owner{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(kube_pod_owner{job=\"kube-state-metrics\", cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -866,7 +889,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -875,7 +898,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(namespace_cpu:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -884,7 +907,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\"}) by (namespace) / sum(namespace_cpu:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -893,7 +916,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(namespace_cpu:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -902,7 +925,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\"}) by (namespace) / sum(namespace_cpu:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -919,7 +942,7 @@ data:
               "title": "CPU Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -974,11 +997,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 9,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1002,7 +1028,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container!=\"\"}) by (namespace)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\"}) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -1018,7 +1044,7 @@ data:
               "title": "Memory Usage (w/o cache)",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1072,11 +1098,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 10,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1116,7 +1145,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
                   "pattern": "Value #A",
                   "thresholds": [
 
@@ -1135,7 +1164,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down to workloads",
-                  "linkUrl": "./d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "linkUrl": "/d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
                   "pattern": "Value #B",
                   "thresholds": [
 
@@ -1249,7 +1278,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
                   "pattern": "namespace",
                   "thresholds": [
 
@@ -1275,7 +1304,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(kube_pod_owner{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(kube_pod_owner{job=\"kube-state-metrics\", cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1293,7 +1322,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container!=\"\"}) by (namespace)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1302,7 +1331,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(namespace_memory:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1311,7 +1340,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container!=\"\"}) by (namespace) / sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\"}) by (namespace) / sum(namespace_memory:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1320,7 +1349,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(namespace_memory:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1329,7 +1358,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container!=\"\"}) by (namespace) / sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\"}) by (namespace) / sum(namespace_memory:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1346,7 +1375,7 @@ data:
               "title": "Requests by Namespace",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -1403,10 +1432,12 @@ data:
               "id": 11,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1560,7 +1591,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
                   "pattern": "namespace",
                   "thresholds": [
 
@@ -1586,7 +1617,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1595,7 +1626,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1604,7 +1635,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1613,7 +1644,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1622,7 +1653,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1631,7 +1662,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -1648,7 +1679,7 @@ data:
               "title": "Current Network Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -1686,7 +1717,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Current Network Usage",
           "titleSize": "h6"
         },
         {
@@ -1703,11 +1734,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 12,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1726,12 +1760,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -1747,7 +1781,7 @@ data:
               "title": "Receive Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1778,19 +1812,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -1801,11 +1823,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 13,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1824,12 +1849,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -1845,7 +1870,7 @@ data:
               "title": "Transmit Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1882,7 +1907,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
           "titleSize": "h6"
         },
         {
@@ -1899,11 +1924,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 14,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1922,12 +1950,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "avg(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -1943,7 +1971,7 @@ data:
               "title": "Average Container Bandwidth by Namespace: Received",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1974,19 +2002,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -1997,11 +2013,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 15,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2020,12 +2039,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "avg(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -2041,7 +2060,7 @@ data:
               "title": "Average Container Bandwidth by Namespace: Transmitted",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2078,7 +2097,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Average Container Bandwidth by Namespace",
           "titleSize": "h6"
         },
         {
@@ -2095,11 +2114,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 16,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2118,12 +2140,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -2139,7 +2161,7 @@ data:
               "title": "Rate of Received Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2154,7 +2176,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -2170,19 +2192,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -2193,11 +2203,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 17,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2216,12 +2229,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -2237,7 +2250,7 @@ data:
               "title": "Rate of Transmitted Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2252,7 +2265,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -2274,7 +2287,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
           "titleSize": "h6"
         },
         {
@@ -2291,11 +2304,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 18,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2314,12 +2330,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -2335,7 +2351,7 @@ data:
               "title": "Rate of Received Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2350,7 +2366,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -2366,19 +2382,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -2389,11 +2393,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 19,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2412,12 +2419,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{namespace}}",
@@ -2433,7 +2440,198 @@ data:
               "title": "Rate of Transmitted Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Rate of Packets Dropped",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": -1,
+              "fill": 10,
+              "id": 20,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "ceil(sum by(namespace) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{namespace}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "IOPS(Reads+Writes)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 21,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{namespace}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "ThroughPut(Read+Write)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2470,7 +2668,315 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Storage IO",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 22,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "sort": {
+                "col": 4,
+                "desc": true
+              },
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "IOPS(Reads)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Reads + Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Throughput(Read)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Read + Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Namespace",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down to pods",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "pattern": "namespace",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Current Storage IO",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Storage IO - Distribution",
           "titleSize": "h6"
         }
       ],
@@ -2487,7 +2993,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -2512,7 +3018,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(node_cpu_seconds_total, cluster)",
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
             "refresh": 2,
             "regex": "",
             "sort": 1,
@@ -2591,11 +3097,14 @@ data:
               "fill": 1,
               "format": "percentunit",
               "id": 1,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2619,7 +3128,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"})",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) / sum(kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"})",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2632,7 +3141,7 @@ data:
               "title": "CPU Utilisation (from requests)",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -2675,11 +3184,14 @@ data:
               "fill": 1,
               "format": "percentunit",
               "id": 2,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2703,7 +3215,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"})",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) / sum(kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"})",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2716,7 +3228,7 @@ data:
               "title": "CPU Utilisation (from limits)",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -2759,11 +3271,14 @@ data:
               "fill": 1,
               "format": "percentunit",
               "id": 3,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2787,7 +3302,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\"})",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) / sum(kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"})",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2797,10 +3312,10 @@ data:
               "thresholds": "70,80",
               "timeFrom": null,
               "timeShift": null,
-              "title": "Memory Utilization (from requests)",
+              "title": "Memory Utilisation (from requests)",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -2843,11 +3358,14 @@ data:
               "fill": 1,
               "format": "percentunit",
               "id": 4,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2871,7 +3389,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\"})",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) / sum(kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"})",
                   "format": "time_series",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2884,7 +3402,7 @@ data:
               "title": "Memory Utilisation (from limits)",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "singlestat",
@@ -2938,11 +3456,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 5,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2963,8 +3484,9 @@ data:
                   "color": "#F2495C",
                   "dashes": true,
                   "fill": 0,
+                  "hiddenSeries": true,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 },
@@ -2973,8 +3495,9 @@ data:
                   "color": "#FF9830",
                   "dashes": true,
                   "fill": 0,
+                  "hiddenSeries": true,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 }
@@ -2985,7 +3508,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -3017,7 +3540,7 @@ data:
               "title": "CPU Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3071,11 +3594,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 6,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3210,7 +3736,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                   "pattern": "pod",
                   "thresholds": [
 
@@ -3236,7 +3762,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3245,7 +3771,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3254,7 +3780,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3263,7 +3789,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3272,7 +3798,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3289,7 +3815,7 @@ data:
               "title": "CPU Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -3344,11 +3870,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 7,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3369,8 +3898,9 @@ data:
                   "color": "#F2495C",
                   "dashes": true,
                   "fill": 0,
+                  "hiddenSeries": true,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 },
@@ -3379,8 +3909,9 @@ data:
                   "color": "#FF9830",
                   "dashes": true,
                   "fill": 0,
+                  "hiddenSeries": true,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 }
@@ -3391,7 +3922,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -3423,7 +3954,7 @@ data:
               "title": "Memory Usage (w/o cache)",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3477,11 +4008,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 8,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3673,7 +4207,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                   "pattern": "pod",
                   "thresholds": [
 
@@ -3699,7 +4233,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3708,7 +4242,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3717,7 +4251,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) by (pod) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) by (pod) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3726,7 +4260,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3735,7 +4269,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) by (pod) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) by (pod) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3744,7 +4278,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3753,7 +4287,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_cache{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_cache{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3762,7 +4296,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_swap{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_swap{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -3779,7 +4313,7 @@ data:
               "title": "Memory Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -3836,10 +4370,12 @@ data:
               "id": 9,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3993,7 +4529,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                   "pattern": "pod",
                   "thresholds": [
 
@@ -4019,7 +4555,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4028,7 +4564,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4037,7 +4573,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4046,7 +4582,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4055,7 +4591,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4064,7 +4600,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4081,7 +4617,7 @@ data:
               "title": "Current Network Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -4119,7 +4655,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Current Network Usage",
           "titleSize": "h6"
         },
         {
@@ -4136,11 +4672,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 10,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4159,12 +4698,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -4180,7 +4719,7 @@ data:
               "title": "Receive Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -4211,19 +4750,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -4234,11 +4761,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 11,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4257,12 +4787,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -4278,7 +4808,7 @@ data:
               "title": "Transmit Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -4315,7 +4845,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
           "titleSize": "h6"
         },
         {
@@ -4332,11 +4862,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 12,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4355,12 +4888,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -4376,7 +4909,7 @@ data:
               "title": "Rate of Received Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -4391,7 +4924,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -4407,19 +4940,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -4430,11 +4951,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 13,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4453,12 +4977,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -4474,7 +4998,7 @@ data:
               "title": "Rate of Transmitted Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -4489,7 +5013,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -4511,7 +5035,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
           "titleSize": "h6"
         },
         {
@@ -4528,11 +5052,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 14,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4551,12 +5078,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -4572,7 +5099,7 @@ data:
               "title": "Rate of Received Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -4587,7 +5114,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -4603,19 +5130,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -4626,11 +5141,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 15,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4649,12 +5167,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -4670,7 +5188,198 @@ data:
               "title": "Rate of Transmitted Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Rate of Packets Dropped",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": -1,
+              "fill": 10,
+              "id": 16,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "ceil(sum by(pod) (rate(container_fs_reads_total{container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]) + rate(container_fs_writes_total{container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{pod}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "IOPS(Reads+Writes)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 17,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_bytes_total{container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{pod}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "ThroughPut(Read+Write)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -4707,7 +5416,315 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Storage IO",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 18,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "sort": {
+                "col": 4,
+                "desc": true
+              },
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "IOPS(Reads)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Reads + Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Throughput(Read)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Read + Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Pod",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down to pods",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "pattern": "pod",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Current Storage IO",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Storage IO - Distribution",
           "titleSize": "h6"
         }
       ],
@@ -4724,7 +5741,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -4749,8 +5766,8 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info, cluster)",
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
@@ -4776,8 +5793,8 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
+            "query": "label_values(kube_namespace_status_phase{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
@@ -4854,11 +5871,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 1,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4874,7 +5894,17 @@ data:
               "points": false,
               "renderer": "flot",
               "seriesOverrides": [
-
+                {
+                  "alias": "max capacity",
+                  "color": "#F2495C",
+                  "dashes": true,
+                  "fill": 0,
+                  "hiddenSeries": true,
+                  "hideTooltip": true,
+                  "legend": true,
+                  "linewidth": 2,
+                  "stack": false
+                }
               ],
               "spaceLength": 10,
               "span": 12,
@@ -4882,7 +5912,15 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
+                  "expr": "sum(kube_node_status_capacity{cluster=\"$cluster\", node=~\"$node\", resource=\"cpu\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "max capacity",
+                  "legendLink": null,
+                  "step": 10
+                },
+                {
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -4898,7 +5936,7 @@ data:
               "title": "CPU Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -4952,11 +5990,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 2,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5117,7 +6158,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -5126,7 +6167,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -5135,7 +6176,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", node=~\"$node\"}) by (pod) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", node=~\"$node\"}) by (pod) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -5144,7 +6185,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -5153,7 +6194,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", node=~\"$node\"}) by (pod) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", node=~\"$node\"}) by (pod) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -5170,7 +6211,7 @@ data:
               "title": "CPU Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -5225,11 +6266,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 3,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5245,13 +6289,31 @@ data:
               "points": false,
               "renderer": "flot",
               "seriesOverrides": [
-
+                {
+                  "alias": "max capacity",
+                  "color": "#F2495C",
+                  "dashes": true,
+                  "fill": 0,
+                  "hiddenSeries": true,
+                  "hideTooltip": true,
+                  "legend": true,
+                  "linewidth": 2,
+                  "stack": false
+                }
               ],
               "spaceLength": 10,
               "span": 12,
               "stack": true,
               "steppedLine": false,
               "targets": [
+                {
+                  "expr": "sum(kube_node_status_capacity{cluster=\"$cluster\", node=~\"$node\", resource=\"memory\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "max capacity",
+                  "legendLink": null,
+                  "step": 10
+                },
                 {
                   "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=~\"$node\", container!=\"\"}) by (pod)",
                   "format": "time_series",
@@ -5269,7 +6331,7 @@ data:
               "title": "Memory Usage (w/o cache)",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -5323,11 +6385,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 4,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5554,7 +6619,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -5563,7 +6628,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod) / sum(kube_pod_container_resource_requests_memory_bytes{node=~\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -5572,7 +6637,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -5581,7 +6646,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod) / sum(kube_pod_container_resource_limits_memory_bytes{node=~\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -5625,7 +6690,7 @@ data:
               "title": "Memory Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -5680,7 +6745,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -5705,8 +6770,8 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info, cluster)",
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
@@ -5732,8 +6797,8 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, node)",
+            "query": "label_values(kube_node_info{cluster=\"$cluster\"}, node)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",

addons/grafana/dashboards-k8s-resources-2.yaml

@@ -30,11 +30,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 1,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -75,7 +78,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", cluster=\"$cluster\"}) by (container)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{namespace=\"$namespace\", pod=\"$pod\", cluster=\"$cluster\"}) by (container)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{container}}",
@@ -83,7 +86,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"})\n",
+                  "expr": "sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"cpu\"}\n)\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "requests",
@@ -91,7 +94,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"})\n",
+                  "expr": "sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"cpu\"}\n)\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "limits",
@@ -107,7 +110,7 @@ data:
               "title": "CPU Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -161,11 +164,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 2,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": true,
                 "max": true,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -189,7 +195,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(increase(container_cpu_cfs_throttled_periods_total{namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", container!=\"\", cluster=\"$cluster\"}[5m])) by (container) /sum(increase(container_cpu_cfs_periods_total{namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", container!=\"\", cluster=\"$cluster\"}[5m])) by (container)",
+                  "expr": "sum(increase(container_cpu_cfs_throttled_periods_total{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", cluster=\"$cluster\"}[$__rate_interval])) by (container) /sum(increase(container_cpu_cfs_periods_total{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", cluster=\"$cluster\"}[$__rate_interval])) by (container)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{container}}",
@@ -212,7 +218,7 @@ data:
               "title": "CPU Throttling",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -266,11 +272,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 3,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -431,7 +440,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\"}) by (container)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -440,7 +449,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -449,7 +458,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -458,7 +467,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -467,7 +476,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -484,7 +493,7 @@ data:
               "title": "CPU Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -539,11 +548,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 4,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -565,7 +577,7 @@ data:
                   "dashes": true,
                   "fill": 0,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 },
@@ -575,7 +587,7 @@ data:
                   "dashes": true,
                   "fill": 0,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 }
@@ -586,7 +598,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", container!=\"\", image!=\"\"}) by (container)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", image!=\"\"}) by (container)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{container}}",
@@ -594,7 +606,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"})\n",
+                  "expr": "sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"memory\"}\n)\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "requests",
@@ -602,7 +614,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"})\n",
+                  "expr": "sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"memory\"}\n)\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "limits",
@@ -615,10 +627,10 @@ data:
               ],
               "timeFrom": null,
               "timeShift": null,
-              "title": "Memory Usage",
+              "title": "Memory Usage (WSS)",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -672,11 +684,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 5,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -706,7 +721,7 @@ data:
                   "type": "hidden"
                 },
                 {
-                  "alias": "Memory Usage",
+                  "alias": "Memory Usage (WSS)",
                   "colorMode": null,
                   "colors": [
 
@@ -894,7 +909,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", container!=\"\", image!=\"\"}) by (container)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", image!=\"\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -903,7 +918,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -912,7 +927,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", image!=\"\"}) by (container) / sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", image!=\"\"}) by (container) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -921,7 +936,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\"}) by (container)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -930,7 +945,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", image!=\"\"}) by (container) / sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", image!=\"\"}) by (container) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -939,7 +954,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -948,7 +963,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_cache{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
+                  "expr": "sum(container_memory_cache{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -957,7 +972,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(container_memory_swap{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
+                  "expr": "sum(container_memory_swap{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -974,7 +989,7 @@ data:
               "title": "Memory Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -1031,10 +1046,12 @@ data:
               "id": 6,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1053,12 +1070,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -1074,7 +1091,7 @@ data:
               "title": "Receive Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1105,19 +1122,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -1130,10 +1135,12 @@ data:
               "id": 7,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1152,12 +1159,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -1173,7 +1180,7 @@ data:
               "title": "Transmit Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1210,7 +1217,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
           "titleSize": "h6"
         },
         {
@@ -1229,10 +1236,12 @@ data:
               "id": 8,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1251,12 +1260,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -1272,7 +1281,7 @@ data:
               "title": "Rate of Received Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1287,7 +1296,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -1303,19 +1312,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -1328,10 +1325,12 @@ data:
               "id": 9,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1350,12 +1349,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -1371,7 +1370,7 @@ data:
               "title": "Rate of Transmitted Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1386,7 +1385,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -1408,7 +1407,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
           "titleSize": "h6"
         },
         {
@@ -1427,10 +1426,12 @@ data:
               "id": 10,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1449,12 +1450,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -1470,7 +1471,7 @@ data:
               "title": "Rate of Received Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1485,7 +1486,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -1501,19 +1502,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -1526,10 +1515,12 @@ data:
               "id": 11,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1548,12 +1539,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -1569,7 +1560,214 @@ data:
               "title": "Rate of Transmitted Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Rate of Packets Dropped",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": -1,
+              "fill": 10,
+              "id": 12,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "ceil(sum by(pod) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Reads",
+                  "legendLink": null,
+                  "step": 10
+                },
+                {
+                  "expr": "ceil(sum by(pod) (rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\",namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Writes",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "IOPS",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 13,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Reads",
+                  "legendLink": null,
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Writes",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "ThroughPut",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1606,7 +1804,506 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Storage IO - Distribution(Pod - Read & Writes)",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": -1,
+              "fill": 10,
+              "id": 14,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "ceil(sum by(container) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{container}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "IOPS(Reads+Writes)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 15,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{container}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "ThroughPut(Read+Write)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Storage IO - Distribution(Containers)",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 16,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "sort": {
+                "col": 4,
+                "desc": true
+              },
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "IOPS(Reads)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Reads + Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Throughput(Read)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Read + Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Container",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "container",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_writes_total{job=\"kubernetes-cadvisor\",device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Current Storage IO",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Storage IO - Distribution",
           "titleSize": "h6"
         }
       ],
@@ -1623,7 +2320,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -1648,8 +2345,8 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info, cluster)",
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
@@ -1675,8 +2372,8 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
+            "query": "label_values(kube_namespace_status_phase{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
@@ -1702,7 +2399,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\", namespace=\"$namespace\"}, pod)",
+            "query": "label_values(kube_pod_info{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\"}, pod)",
             "refresh": 2,
             "regex": "",
             "sort": 1,
@@ -1780,11 +2477,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 1,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -1808,7 +2508,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -1824,7 +2524,7 @@ data:
               "title": "CPU Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -1878,11 +2578,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 2,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2017,7 +2720,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                   "pattern": "pod",
                   "thresholds": [
 
@@ -2043,7 +2746,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2052,7 +2755,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2061,7 +2764,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2070,7 +2773,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2079,7 +2782,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2096,7 +2799,7 @@ data:
               "title": "CPU Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -2151,11 +2854,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 3,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2195,7 +2901,7 @@ data:
               "title": "Memory Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2249,11 +2955,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 4,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2388,7 +3097,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                   "pattern": "pod",
                   "thresholds": [
 
@@ -2423,7 +3132,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2432,7 +3141,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2441,7 +3150,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2450,7 +3159,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2467,7 +3176,7 @@ data:
               "title": "Memory Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -2524,10 +3233,12 @@ data:
               "id": 5,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2681,7 +3392,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                   "pattern": "pod",
                   "thresholds": [
 
@@ -2707,7 +3418,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2716,7 +3427,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2725,7 +3436,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2734,7 +3445,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2743,7 +3454,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2752,7 +3463,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -2769,7 +3480,7 @@ data:
               "title": "Current Network Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -2807,7 +3518,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Current Network Usage",
           "titleSize": "h6"
         },
         {
@@ -2824,11 +3535,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 6,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2847,12 +3561,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -2868,7 +3582,7 @@ data:
               "title": "Receive Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2899,19 +3613,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -2922,11 +3624,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 7,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -2945,12 +3650,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -2966,7 +3671,7 @@ data:
               "title": "Transmit Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3003,7 +3708,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
           "titleSize": "h6"
         },
         {
@@ -3020,11 +3725,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 8,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3043,12 +3751,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(avg(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -3064,7 +3772,7 @@ data:
               "title": "Average Container Bandwidth by Pod: Received",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3095,19 +3803,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -3118,11 +3814,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 9,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3141,12 +3840,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(avg(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -3162,7 +3861,7 @@ data:
               "title": "Average Container Bandwidth by Pod: Transmitted",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3199,7 +3898,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Average Container Bandwidth by Pod",
           "titleSize": "h6"
         },
         {
@@ -3216,11 +3915,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 10,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3239,12 +3941,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -3260,7 +3962,7 @@ data:
               "title": "Rate of Received Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3275,7 +3977,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -3291,19 +3993,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -3314,11 +4004,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 11,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3337,12 +4030,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -3358,7 +4051,7 @@ data:
               "title": "Rate of Transmitted Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3373,7 +4066,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -3395,7 +4088,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
           "titleSize": "h6"
         },
         {
@@ -3412,11 +4105,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 12,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3435,12 +4131,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -3456,7 +4152,7 @@ data:
               "title": "Rate of Received Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3471,7 +4167,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -3487,19 +4183,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -3510,11 +4194,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 13,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3533,12 +4220,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{pod}}",
@@ -3554,7 +4241,7 @@ data:
               "title": "Rate of Transmitted Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3569,7 +4256,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -3591,7 +4278,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets Dropped",
           "titleSize": "h6"
         }
       ],
@@ -3608,7 +4295,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -3633,8 +4320,8 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info, cluster)",
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
@@ -3660,35 +4347,8 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
+            "query": "label_values(kube_namespace_status_phase{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)",
-            "refresh": 1,
+            "refresh": 2,
-            "regex": "",
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          },
-          {
-            "allValue": null,
-            "current": {
-              "text": "",
-              "value": ""
-            },
-            "datasource": "$datasource",
-            "hide": 0,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "workload",
-            "options": [
-
-            ],
-            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\"}, workload)",
-            "refresh": 1,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
@@ -3714,8 +4374,35 @@ data:
             "options": [
 
             ],
-            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\"}, workload_type)",
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\"}, workload_type)",
-            "refresh": 1,
+            "refresh": 2,
+            "regex": "",
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "",
+              "value": ""
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "workload",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}, workload)",
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
@@ -3792,11 +4479,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 1,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -3817,8 +4507,9 @@ data:
                   "color": "#F2495C",
                   "dashes": true,
                   "fill": 0,
+                  "hiddenSeries": true,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 },
@@ -3827,8 +4518,9 @@ data:
                   "color": "#FF9830",
                   "dashes": true,
                   "fill": 0,
+                  "hiddenSeries": true,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 }
@@ -3839,7 +4531,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}} - {{workload_type}}",
@@ -3871,7 +4563,7 @@ data:
               "title": "CPU Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -3925,11 +4617,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 2,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4083,7 +4778,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down",
-                  "linkUrl": "./d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
+                  "linkUrl": "/d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
                   "pattern": "workload",
                   "thresholds": [
 
@@ -4137,7 +4832,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4146,7 +4841,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n  kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4155,7 +4850,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4164,7 +4859,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n  kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4173,7 +4868,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4190,7 +4885,7 @@ data:
               "title": "CPU Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -4245,11 +4940,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 3,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4270,8 +4968,9 @@ data:
                   "color": "#F2495C",
                   "dashes": true,
                   "fill": 0,
+                  "hiddenSeries": true,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 },
@@ -4280,8 +4979,9 @@ data:
                   "color": "#FF9830",
                   "dashes": true,
                   "fill": 0,
+                  "hiddenSeries": true,
                   "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                   "linewidth": 2,
                   "stack": false
                 }
@@ -4292,7 +4992,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}} - {{workload_type}}",
@@ -4324,7 +5024,7 @@ data:
               "title": "Memory Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -4378,11 +5078,14 @@ data:
               "datasource": "$datasource",
               "fill": 1,
               "id": 4,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4536,7 +5239,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down",
-                  "linkUrl": "./d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
+                  "linkUrl": "/d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
                   "pattern": "workload",
                   "thresholds": [
 
@@ -4590,7 +5293,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4599,7 +5302,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n  kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4608,7 +5311,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4617,7 +5320,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n  kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4626,7 +5329,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4643,7 +5346,7 @@ data:
               "title": "Memory Quota",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -4700,10 +5403,12 @@ data:
               "id": 5,
               "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -4857,7 +5562,7 @@ data:
                   "link": true,
                   "linkTargetBlank": false,
                   "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$type",
+                  "linkUrl": "/d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$type",
                   "pattern": "workload",
                   "thresholds": [
 
@@ -4902,7 +5607,7 @@ data:
               ],
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4911,7 +5616,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4920,7 +5625,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4929,7 +5634,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4938,7 +5643,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4947,7 +5652,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "table",
                   "instant": true,
                   "intervalFactor": 2,
@@ -4964,7 +5669,7 @@ data:
               "title": "Current Network Usage",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -5002,7 +5707,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Current Network Usage",
           "titleSize": "h6"
         },
         {
@@ -5019,11 +5724,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 6,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5042,12 +5750,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}}",
@@ -5063,7 +5771,7 @@ data:
               "title": "Receive Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -5094,19 +5802,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -5117,11 +5813,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 7,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5140,12 +5839,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}}",
@@ -5161,7 +5860,7 @@ data:
               "title": "Transmit Bandwidth",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -5198,7 +5897,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
           "titleSize": "h6"
         },
         {
@@ -5215,11 +5914,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 8,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5238,12 +5940,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(avg(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}}",
@@ -5259,7 +5961,7 @@ data:
               "title": "Average Container Bandwidth by Workload: Received",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -5290,19 +5992,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -5313,11 +6003,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 9,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5336,12 +6029,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(avg(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}}",
@@ -5357,7 +6050,7 @@ data:
               "title": "Average Container Bandwidth by Workload: Transmitted",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -5394,7 +6087,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Average Container Bandwidth by Workload",
           "titleSize": "h6"
         },
         {
@@ -5411,11 +6104,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 10,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5434,12 +6130,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}}",
@@ -5455,7 +6151,7 @@ data:
               "title": "Rate of Received Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -5470,7 +6166,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -5486,19 +6182,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -5509,11 +6193,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 11,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5532,12 +6219,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}}",
@@ -5553,7 +6240,7 @@ data:
               "title": "Rate of Transmitted Packets",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -5568,7 +6255,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -5590,7 +6277,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
           "titleSize": "h6"
         },
         {
@@ -5607,11 +6294,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 12,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5630,12 +6320,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}}",
@@ -5651,7 +6341,7 @@ data:
               "title": "Rate of Received Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -5666,7 +6356,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -5682,19 +6372,7 @@ data:
                   "show": false
                 }
               ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
             },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
             {
               "aliasColors": {
 
@@ -5705,11 +6383,14 @@ data:
               "datasource": "$datasource",
               "fill": 10,
               "id": 13,
+              "interval": "1m",
               "legend": {
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
+                "rightSide": true,
                 "show": true,
                 "total": false,
                 "values": false
@@ -5728,12 +6409,12 @@ data:
 
               ],
               "spaceLength": 10,
-              "span": 12,
+              "span": 6,
               "stack": true,
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{workload}}",
@@ -5749,7 +6430,7 @@ data:
               "title": "Rate of Transmitted Packets Dropped",
               "tooltip": {
                 "shared": false,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -5764,7 +6445,7 @@ data:
               },
               "yaxes": [
                 {
-                  "format": "Bps",
+                  "format": "pps",
                   "label": null,
                   "logBase": 1,
                   "max": null,
@@ -5786,7 +6467,7 @@ data:
           "repeatIteration": null,
           "repeatRowId": null,
           "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets Dropped",
           "titleSize": "h6"
         }
       ],
@@ -5803,7 +6484,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -5813,38 +6494,6 @@ data:
             "regex": "",
             "type": "datasource"
           },
-          {
-            "allValue": null,
-            "auto": false,
-            "auto_count": 30,
-            "auto_min": "10s",
-            "current": {
-              "text": "deployment",
-              "value": "deployment"
-            },
-            "datasource": "$datasource",
-            "definition": "label_values(namespace_workload_pod:kube_pod_owner:relabel{namespace=~\"$namespace\", workload=~\".+\"}, workload_type)",
-            "hide": 0,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "type",
-            "options": [
-
-            ],
-            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{namespace=~\"$namespace\", workload=~\".+\"}, workload_type)",
-            "refresh": 1,
-            "regex": "",
-            "skipUrlSync": false,
-            "sort": 0,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          },
           {
             "allValue": null,
             "current": {
@@ -5860,8 +6509,8 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info, cluster)",
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
@@ -5887,13 +6536,45 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
+            "query": "label_values(kube_pod_info{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)",
-            "refresh": 1,
+            "refresh": 2,
             "regex": "",
             "sort": 1,
             "tagValuesQuery": "",
             "tags": [
 
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "deployment",
+              "value": "deployment"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\"}, workload_type)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "type",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\"}, workload_type)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
             ],
             "tagsQuery": "",
             "type": "query",

addons/grafana/dashboards-k8s.yaml

@@ -69,7 +69,11 @@ data:
 
               },
               "id": 3,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
               "links": [
 
               ],
@@ -147,13 +151,14 @@ data:
 
               },
               "id": 4,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -265,7 +270,11 @@ data:
 
               },
               "id": 5,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
               "links": [
 
               ],
@@ -342,13 +351,14 @@ data:
 
               },
               "id": 6,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -451,13 +461,14 @@ data:
 
               },
               "id": 7,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -545,13 +556,14 @@ data:
 
               },
               "id": 8,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -661,7 +673,11 @@ data:
 
               },
               "id": 9,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
               "links": [
 
               ],
@@ -738,13 +754,14 @@ data:
 
               },
               "id": 10,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -847,13 +864,14 @@ data:
 
               },
               "id": 11,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -941,13 +959,14 @@ data:
 
               },
               "id": 12,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -1047,13 +1066,14 @@ data:
 
               },
               "id": 13,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": false,
                 "sideWidth": null,
                 "total": false,
@@ -1079,7 +1099,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(workqueue_adds_total{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, name)",
+                  "expr": "sum(rate(workqueue_adds_total{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[$__rate_interval])) by (instance, name)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}} {{name}}",
@@ -1140,13 +1160,14 @@ data:
 
               },
               "id": 14,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": false,
                 "sideWidth": null,
                 "total": false,
@@ -1172,7 +1193,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(workqueue_depth{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, name)",
+                  "expr": "sum(rate(workqueue_depth{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[$__rate_interval])) by (instance, name)",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}} {{name}}",
@@ -1233,6 +1254,7 @@ data:
 
               },
               "id": 15,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -1265,7 +1287,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, name, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[$__rate_interval])) by (instance, name, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}} {{name}}",
@@ -1339,13 +1361,14 @@ data:
 
               },
               "id": 16,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -1432,13 +1455,14 @@ data:
 
               },
               "id": 17,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -1464,7 +1488,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(process_cpu_seconds_total{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])",
+                  "expr": "rate(process_cpu_seconds_total{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[$__rate_interval])",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -1525,13 +1549,14 @@ data:
 
               },
               "id": 18,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -1627,7 +1652,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -1651,7 +1676,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(apiserver_request_total, cluster)",
+            "query": "label_values(up{job=\"apiserver\"}, cluster)",
             "refresh": 2,
             "regex": "",
             "sort": 1,
@@ -1677,7 +1702,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(apiserver_request_total{job=\"apiserver\", cluster=\"$cluster\"}, instance)",
+            "query": "label_values(up{job=\"apiserver\", cluster=\"$cluster\"}, instance)",
             "refresh": 2,
             "regex": "",
             "sort": 1,
@@ -1774,7 +1799,11 @@ data:
 
               },
               "id": 2,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
               "links": [
 
               ],
@@ -1813,7 +1842,7 @@ data:
               "tableColumn": "",
               "targets": [
                 {
-                  "expr": "sum(up{job=\"kube-controller-manager\"})",
+                  "expr": "sum(up{cluster=\"$cluster\", job=\"kube-controller-manager\"})",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "",
@@ -1850,6 +1879,7 @@ data:
 
               },
               "id": 3,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -1882,10 +1912,10 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(workqueue_adds_total{job=\"kube-controller-manager\", instance=~\"$instance\"}[5m])) by (instance, name)",
+                  "expr": "sum(rate(workqueue_adds_total{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, name)",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{name}}",
+                  "legendFormat": "{{cluster}} {{instance}} {{name}}",
                   "refId": "A"
                 }
               ],
@@ -1956,6 +1986,7 @@ data:
 
               },
               "id": 4,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -1988,10 +2019,10 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(workqueue_depth{job=\"kube-controller-manager\", instance=~\"$instance\"}[5m])) by (instance, name)",
+                  "expr": "sum(rate(workqueue_depth{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, name)",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{name}}",
+                  "legendFormat": "{{cluster}} {{instance}} {{name}}",
                   "refId": "A"
                 }
               ],
@@ -2062,6 +2093,7 @@ data:
 
               },
               "id": 5,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -2094,10 +2126,10 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{job=\"kube-controller-manager\", instance=~\"$instance\"}[5m])) by (instance, name, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, name, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{name}}",
+                  "legendFormat": "{{cluster}} {{instance}} {{name}}",
                   "refId": "A"
                 }
               ],
@@ -2168,13 +2200,14 @@ data:
 
               },
               "id": 6,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -2200,28 +2233,28 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"2..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"2..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "2xx",
                   "refId": "A"
                 },
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"3..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"3..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "3xx",
                   "refId": "B"
                 },
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"4..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"4..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "4xx",
                   "refId": "C"
                 },
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"5..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"5..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "5xx",
@@ -2282,13 +2315,14 @@ data:
 
               },
               "id": 7,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -2314,7 +2348,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{job=\"kube-controller-manager\", instance=~\"$instance\", verb=\"POST\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\", verb=\"POST\"}[$__rate_interval])) by (verb, url, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{verb}} {{url}}",
@@ -2388,6 +2422,7 @@ data:
 
               },
               "id": 8,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -2420,7 +2455,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{job=\"kube-controller-manager\", instance=~\"$instance\", verb=\"GET\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\", verb=\"GET\"}[$__rate_interval])) by (verb, url, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{verb}} {{url}}",
@@ -2494,13 +2529,14 @@ data:
 
               },
               "id": 9,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -2526,7 +2562,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "process_resident_memory_bytes{job=\"kube-controller-manager\",instance=~\"$instance\"}",
+                  "expr": "process_resident_memory_bytes{cluster=\"$cluster\", job=\"kube-controller-manager\",instance=~\"$instance\"}",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -2587,13 +2623,14 @@ data:
 
               },
               "id": 10,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -2619,7 +2656,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(process_cpu_seconds_total{job=\"kube-controller-manager\",instance=~\"$instance\"}[5m])",
+                  "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", job=\"kube-controller-manager\",instance=~\"$instance\"}[$__rate_interval])",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -2680,13 +2717,14 @@ data:
 
               },
               "id": 11,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -2712,7 +2750,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "go_goroutines{job=\"kube-controller-manager\",instance=~\"$instance\"}",
+                  "expr": "go_goroutines{cluster=\"$cluster\", job=\"kube-controller-manager\",instance=~\"$instance\"}",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -2782,7 +2820,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -2796,6 +2834,32 @@ data:
             "allValue": null,
             "current": {
 
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kube-controller-manager\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
             },
             "datasource": "$datasource",
             "hide": 0,
@@ -2806,7 +2870,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(process_cpu_seconds_total{job=\"kube-controller-manager\"}, instance)",
+            "query": "label_values(up{cluster=\"$cluster\", job=\"kube-controller-manager\"}, instance)",
             "refresh": 2,
             "regex": "",
             "sort": 1,
@@ -2895,13 +2959,14 @@ data:
 
               },
               "id": 2,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": true,
                 "current": true,
                 "max": true,
                 "min": true,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -2927,14 +2992,14 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(\n  sum without(instance, node) (kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n  -\n  sum without(instance, node) (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n)\n",
+                  "expr": "(\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n  -\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n)\n",
                   "format": "time_series",
                   "intervalFactor": 1,
                   "legendFormat": "Used Space",
                   "refId": "A"
                 },
                 {
-                  "expr": "sum without(instance, node) (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n",
+                  "expr": "sum without(instance, node) (topk(1, (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n",
                   "format": "time_series",
                   "intervalFactor": 1,
                   "legendFormat": "Free Space",
@@ -3003,7 +3068,11 @@ data:
 
               },
               "id": 3,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
               "links": [
 
               ],
@@ -3042,7 +3111,7 @@ data:
               "tableColumn": "",
               "targets": [
                 {
-                  "expr": "max without(instance,node) (\n(\n  kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n  -\n  kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n)\n/\nkubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n* 100)\n",
+                  "expr": "max without(instance,node) (\n(\n  topk(1, kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n  -\n  topk(1, kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n)\n/\ntopk(1, kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n* 100)\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "",
@@ -3092,13 +3161,14 @@ data:
 
               },
               "id": 4,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": true,
                 "current": true,
                 "max": true,
                 "min": true,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -3124,14 +3194,14 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum without(instance, node) (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n",
+                  "expr": "sum without(instance, node) (topk(1, (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n",
                   "format": "time_series",
                   "intervalFactor": 1,
                   "legendFormat": "Used inodes",
                   "refId": "A"
                 },
                 {
-                  "expr": "(\n  sum without(instance, node) (kubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n  -\n  sum without(instance, node) (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n)\n",
+                  "expr": "(\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n  -\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n)\n",
                   "format": "time_series",
                   "intervalFactor": 1,
                   "legendFormat": " Free inodes",
@@ -3200,7 +3270,11 @@ data:
 
               },
               "id": 5,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
               "links": [
 
               ],
@@ -3239,7 +3313,7 @@ data:
               "tableColumn": "",
               "targets": [
                 {
-                  "expr": "max without(instance,node) (\nkubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n/\nkubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n* 100)\n",
+                  "expr": "max without(instance,node) (\ntopk(1, kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n/\ntopk(1, kubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n* 100)\n",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "",
@@ -3285,7 +3359,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -3309,7 +3383,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(kubelet_volume_stats_capacity_bytes, cluster)",
+            "query": "label_values(kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}, cluster)",
             "refresh": 2,
             "regex": "",
             "sort": 1,
@@ -3458,7 +3532,11 @@ data:
 
               },
               "id": 2,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
               "links": [
 
               ],
@@ -3497,7 +3575,7 @@ data:
               "tableColumn": "",
               "targets": [
                 {
-                  "expr": "sum(up{job=\"kube-scheduler\"})",
+                  "expr": "sum(up{cluster=\"$cluster\", job=\"kube-scheduler\"})",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "",
@@ -3534,6 +3612,7 @@ data:
 
               },
               "id": 3,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -3566,31 +3645,31 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(scheduler_e2e_scheduling_duration_seconds_count{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])) by (instance)",
+                  "expr": "sum(rate(scheduler_e2e_scheduling_duration_seconds_count{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance)",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} e2e",
+                  "legendFormat": "{{cluster}} {{instance}} e2e",
                   "refId": "A"
                 },
                 {
-                  "expr": "sum(rate(scheduler_binding_duration_seconds_count{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])) by (instance)",
+                  "expr": "sum(rate(scheduler_binding_duration_seconds_count{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance)",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} binding",
+                  "legendFormat": "{{cluster}} {{instance}} binding",
                   "refId": "B"
                 },
                 {
-                  "expr": "sum(rate(scheduler_scheduling_algorithm_duration_seconds_count{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])) by (instance)",
+                  "expr": "sum(rate(scheduler_scheduling_algorithm_duration_seconds_count{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance)",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} scheduling algorithm",
+                  "legendFormat": "{{cluster}} {{instance}} scheduling algorithm",
                   "refId": "C"
                 },
                 {
-                  "expr": "sum(rate(scheduler_volume_scheduling_duration_seconds_count{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])) by (instance)",
+                  "expr": "sum(rate(scheduler_volume_scheduling_duration_seconds_count{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance)",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} volume",
+                  "legendFormat": "{{cluster}} {{instance}} volume",
                   "refId": "D"
                 }
               ],
@@ -3648,6 +3727,7 @@ data:
 
               },
               "id": 4,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -3680,31 +3760,31 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\"kube-scheduler\",instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_e2e_scheduling_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} e2e",
+                  "legendFormat": "{{cluster}} {{instance}} e2e",
                   "refId": "A"
                 },
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_binding_duration_seconds_bucket{job=\"kube-scheduler\",instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_binding_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} binding",
+                  "legendFormat": "{{cluster}} {{instance}} binding",
                   "refId": "B"
                 },
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\"kube-scheduler\",instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_scheduling_algorithm_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} scheduling algorithm",
+                  "legendFormat": "{{cluster}} {{instance}} scheduling algorithm",
                   "refId": "C"
                 },
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_volume_scheduling_duration_seconds_bucket{job=\"kube-scheduler\",instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_volume_scheduling_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
-                  "legendFormat": "{{instance}} volume",
+                  "legendFormat": "{{cluster}} {{instance}} volume",
                   "refId": "D"
                 }
               ],
@@ -3775,13 +3855,14 @@ data:
 
               },
               "id": 5,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -3807,28 +3888,28 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-scheduler\", instance=~\"$instance\",code=~\"2..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\",code=~\"2..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "2xx",
                   "refId": "A"
                 },
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-scheduler\", instance=~\"$instance\",code=~\"3..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\",code=~\"3..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "3xx",
                   "refId": "B"
                 },
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-scheduler\", instance=~\"$instance\",code=~\"4..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\",code=~\"4..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "4xx",
                   "refId": "C"
                 },
                 {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-scheduler\", instance=~\"$instance\",code=~\"5..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\",code=~\"5..\"}[$__rate_interval]))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "5xx",
@@ -3889,13 +3970,14 @@ data:
 
               },
               "id": 6,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -3921,7 +4003,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{job=\"kube-scheduler\", instance=~\"$instance\", verb=\"POST\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\", verb=\"POST\"}[$__rate_interval])) by (verb, url, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{verb}} {{url}}",
@@ -3995,6 +4077,7 @@ data:
 
               },
               "id": 7,
+              "interval": "1m",
               "legend": {
                 "alignAsTable": true,
                 "avg": false,
@@ -4027,7 +4110,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{job=\"kube-scheduler\", instance=~\"$instance\", verb=\"GET\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\", verb=\"GET\"}[$__rate_interval])) by (verb, url, le))",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{verb}} {{url}}",
@@ -4101,13 +4184,14 @@ data:
 
               },
               "id": 8,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -4133,7 +4217,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "process_resident_memory_bytes{job=\"kube-scheduler\", instance=~\"$instance\"}",
+                  "expr": "process_resident_memory_bytes{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -4194,13 +4278,14 @@ data:
 
               },
               "id": 9,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -4226,7 +4311,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(process_cpu_seconds_total{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])",
+                  "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -4287,13 +4372,14 @@ data:
 
               },
               "id": 10,
+              "interval": "1m",
               "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                 "avg": false,
                 "current": false,
                 "max": false,
                 "min": false,
-                "rightSide": false,
+                "rightSide": true,
                 "show": true,
                 "sideWidth": null,
                 "total": false,
@@ -4319,7 +4405,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "go_goroutines{job=\"kube-scheduler\",instance=~\"$instance\"}",
+                  "expr": "go_goroutines{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "legendFormat": "{{instance}}",
@@ -4389,7 +4475,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -4403,6 +4489,32 @@ data:
             "allValue": null,
             "current": {
 
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kube-scheduler\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
             },
             "datasource": "$datasource",
             "hide": 0,
@@ -4413,7 +4525,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(process_cpu_seconds_total{job=\"kube-scheduler\"}, instance)",
+            "query": "label_values(up{job=\"kube-scheduler\", cluster=\"$cluster\"}, instance)",
             "refresh": 2,
             "regex": "",
             "sort": 1,
@@ -4461,912 +4573,6 @@ data:
       "uid": "2e6b6a3b4bddf1427b3a55aa1311c656",
       "version": 0
     }
-  statefulset.json: |-
-    {
-      "__inputs": [
-
-      ],
-      "__requires": [
-
-      ],
-      "annotations": {
-        "list": [
-
-        ]
-      },
-      "editable": false,
-      "gnetId": null,
-      "graphTooltip": 0,
-      "hideControls": false,
-      "id": null,
-      "links": [
-
-      ],
-      "refresh": "",
-      "rows": [
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 2,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "cores",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 4,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "lineColor": "rgb(31, 120, 193)",
-                "show": true
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "sum(rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\", namespace=\"$namespace\", pod=~\"$statefulset.*\"}[3m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "CPU",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "0",
-                  "value": "null"
-                }
-              ],
-              "valueName": "current"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 3,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "GB",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 4,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "lineColor": "rgb(31, 120, 193)",
-                "show": true
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "sum(container_memory_usage_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\", namespace=\"$namespace\", pod=~\"$statefulset.*\"}) / 1024^3",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Memory",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "0",
-                  "value": "null"
-                }
-              ],
-              "valueName": "current"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 4,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "Bps",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 4,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "lineColor": "rgb(31, 120, 193)",
-                "show": true
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "sum(rate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$statefulset.*\"}[3m])) + sum(rate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",pod=~\"$statefulset.*\"}[3m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Network",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "0",
-                  "value": "null"
-                }
-              ],
-              "valueName": "current"
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "height": "100px",
-          "panels": [
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 5,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 3,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "max(kube_statefulset_replicas{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Desired Replicas",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "0",
-                  "value": "null"
-                }
-              ],
-              "valueName": "current"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 6,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 3,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "min(kube_statefulset_status_replicas_current{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Replicas of current version",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "0",
-                  "value": "null"
-                }
-              ],
-              "valueName": "current"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 7,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 3,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "max(kube_statefulset_status_observed_generation{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Observed Generation",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "0",
-                  "value": "null"
-                }
-              ],
-              "valueName": "current"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 8,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 3,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "max(kube_statefulset_metadata_generation{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Metadata Generation",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "0",
-                  "value": "null"
-                }
-              ],
-              "valueName": "current"
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "fillGradient": 0,
-              "gridPos": {
-
-              },
-              "id": 9,
-              "legend": {
-                "alignAsTable": false,
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "rightSide": false,
-                "show": true,
-                "sideWidth": null,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "max(kube_statefulset_replicas{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "replicas specified",
-                  "refId": "A"
-                },
-                {
-                  "expr": "max(kube_statefulset_status_replicas{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "replicas created",
-                  "refId": "B"
-                },
-                {
-                  "expr": "min(kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "ready",
-                  "refId": "C"
-                },
-                {
-                  "expr": "min(kube_statefulset_status_replicas_current{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "replicas of current version",
-                  "refId": "D"
-                },
-                {
-                  "expr": "min(kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "updated",
-                  "refId": "E"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Replicas",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        }
-      ],
-      "schemaVersion": 14,
-      "style": "dark",
-      "tags": [
-        "kubernetes-mixin"
-      ],
-      "templating": {
-        "list": [
-          {
-            "current": {
-              "text": "default",
-              "value": "default"
-            },
-            "hide": 0,
-            "label": null,
-            "name": "datasource",
-            "options": [
-
-            ],
-            "query": "prometheus",
-            "refresh": 1,
-            "regex": "",
-            "type": "datasource"
-          },
-          {
-            "allValue": null,
-            "current": {
-
-            },
-            "datasource": "$datasource",
-            "hide": 2,
-            "includeAll": false,
-            "label": "cluster",
-            "multi": false,
-            "name": "cluster",
-            "options": [
-
-            ],
-            "query": "label_values(kube_statefulset_metadata_generation, cluster)",
-            "refresh": 2,
-            "regex": "",
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          },
-          {
-            "allValue": null,
-            "current": {
-
-            },
-            "datasource": "$datasource",
-            "hide": 0,
-            "includeAll": false,
-            "label": "Namespace",
-            "multi": false,
-            "name": "namespace",
-            "options": [
-
-            ],
-            "query": "label_values(kube_statefulset_metadata_generation{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)",
-            "refresh": 2,
-            "regex": "",
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          },
-          {
-            "allValue": null,
-            "current": {
-
-            },
-            "datasource": "$datasource",
-            "hide": 0,
-            "includeAll": false,
-            "label": "Name",
-            "multi": false,
-            "name": "statefulset",
-            "options": [
-
-            ],
-            "query": "label_values(kube_statefulset_metadata_generation{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\"}, statefulset)",
-            "refresh": 2,
-            "regex": "",
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          }
-        ]
-      },
-      "time": {
-        "from": "now-1h",
-        "to": "now"
-      },
-      "timepicker": {
-        "refresh_intervals": [
-          "5s",
-          "10s",
-          "30s",
-          "1m",
-          "5m",
-          "15m",
-          "30m",
-          "1h",
-          "2h",
-          "1d"
-        ],
-        "time_options": [
-          "5m",
-          "15m",
-          "1h",
-          "6h",
-          "12h",
-          "24h",
-          "2d",
-          "7d",
-          "30d"
-        ]
-      },
-      "timezone": "UTC",
-      "title": "Kubernetes / StatefulSets",
-      "uid": "a31c1f46e6f727cb37c0d731a7245005",
-      "version": 0
-    }
 kind: ConfigMap
 metadata:
   name: grafana-dashboards-k8s

addons/grafana/dashboards-node-exporter.yaml

@@ -15,13 +15,13 @@ data:
       },
       "editable": false,
       "gnetId": null,
-      "graphTooltip": 0,
+      "graphTooltip": 1,
       "hideControls": false,
       "id": null,
       "links": [
 
       ],
-      "refresh": "",
+      "refresh": "30s",
       "rows": [
         {
           "collapse": false,
@@ -73,9 +73,8 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "(\n  (1 - rate(node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\", instance=\"$instance\"}[$__interval]))\n/ ignoring(cpu) group_left\n  count without (cpu)( node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\", instance=\"$instance\"})\n)\n",
+                  "expr": "(\n  (1 - sum without (mode) (rate(node_cpu_seconds_total{job=\"node-exporter\", mode=~\"idle|iowait|steal\", instance=\"$instance\"}[$__rate_interval])))\n/ ignoring(cpu) group_left\n  count without (cpu, mode) (node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\", instance=\"$instance\"})\n)\n",
                   "format": "time_series",
-                  "interval": "1m",
                   "intervalFactor": 5,
                   "legendFormat": "{{cpu}}",
                   "refId": "A"
@@ -509,25 +508,22 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(node_disk_read_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__interval])",
+                  "expr": "rate(node_disk_read_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__rate_interval])",
                   "format": "time_series",
-                  "interval": "1m",
                   "intervalFactor": 2,
                   "legendFormat": "{{device}} read",
                   "refId": "A"
                 },
                 {
-                  "expr": "rate(node_disk_written_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__interval])",
+                  "expr": "rate(node_disk_written_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__rate_interval])",
                   "format": "time_series",
-                  "interval": "1m",
                   "intervalFactor": 2,
                   "legendFormat": "{{device}} written",
                   "refId": "B"
                 },
                 {
-                  "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__interval])",
+                  "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__rate_interval])",
                   "format": "time_series",
-                  "interval": "1m",
                   "intervalFactor": 2,
                   "legendFormat": "{{device}} io time",
                   "refId": "C"
@@ -739,9 +735,8 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(node_network_receive_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__interval])",
+                  "expr": "rate(node_network_receive_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__rate_interval])",
                   "format": "time_series",
-                  "interval": "1m",
                   "intervalFactor": 2,
                   "legendFormat": "{{device}}",
                   "refId": "A"
@@ -833,9 +828,8 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(node_network_transmit_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__interval])",
+                  "expr": "rate(node_network_transmit_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__rate_interval])",
                   "format": "time_series",
-                  "interval": "1m",
                   "intervalFactor": 2,
                   "legendFormat": "{{device}}",
                   "refId": "A"
@@ -894,17 +888,17 @@ data:
       "schemaVersion": 14,
       "style": "dark",
       "tags": [
-
+        "node-exporter-mixin"
       ],
       "templating": {
         "list": [
           {
             "current": {
-              "text": "Prometheus",
+              "text": "default",
-              "value": "Prometheus"
+              "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -972,7 +966,7 @@ data:
         ]
       },
       "timezone": "",
-      "title": "Nodes",
+      "title": "Node Exporter / Nodes",
       "uid": "fa49a4706d07a042595b664c87fb33ea",
       "version": 0
     }

addons/grafana/dashboards-prom.yaml

@@ -1536,11 +1536,11 @@ data:
             "includeAll": true,
             "label": null,
             "multi": false,
-            "name": "instance",
+            "name": "cluster",
             "options": [
 
             ],
-            "query": "label_values(prometheus_build_info, instance)",
+            "query": "label_values(kube_pod_container_info{image=~\".*prometheus.*\"}, cluster)",
             "refresh": 2,
             "regex": "",
             "sort": 0,
@@ -1571,11 +1571,11 @@ data:
             "includeAll": true,
             "label": null,
             "multi": false,
-            "name": "cluster",
+            "name": "instance",
             "options": [
 
             ],
-            "query": "label_values(kube_pod_container_info{image=~\".*prometheus.*\"}, cluster)",
+            "query": "label_values(prometheus_build_info{cluster=~\"$cluster\"}, instance)",
             "refresh": 2,
             "regex": "",
             "sort": 0,
@@ -1850,7 +1850,7 @@ data:
               "title": "Prometheus Stats",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "transform": "table",
@@ -1949,7 +1949,7 @@ data:
               "title": "Target Sync",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2035,7 +2035,7 @@ data:
               "title": "Targets",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2133,7 +2133,7 @@ data:
               "title": "Average Scrape Interval Duration",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2202,6 +2202,14 @@ data:
               "stack": true,
               "steppedLine": false,
               "targets": [
+                {
+                  "expr": "sum by (job) (rate(prometheus_target_scrapes_exceeded_body_size_limit_total[1m]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "exceeded body size limit: {{job}}",
+                  "legendLink": null,
+                  "step": 10
+                },
                 {
                   "expr": "sum by (job) (rate(prometheus_target_scrapes_exceeded_sample_limit_total[1m]))",
                   "format": "time_series",
@@ -2243,7 +2251,7 @@ data:
               "title": "Scrape failures",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2329,7 +2337,7 @@ data:
               "title": "Appended Samples",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2427,7 +2435,7 @@ data:
               "title": "Head Series",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2513,7 +2521,7 @@ data:
               "title": "Head Chunks",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2611,7 +2619,7 @@ data:
               "title": "Query Rate",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2697,7 +2705,7 @@ data:
               "title": "Stage Duration",
               "tooltip": {
                 "shared": true,
-                "sort": 0,
+                "sort": 2,
                 "value_type": "individual"
               },
               "type": "graph",
@@ -2751,7 +2759,7 @@ data:
               "value": "default"
             },
             "hide": 0,
-            "label": null,
+            "label": "Data Source",
             "name": "datasource",
             "options": [
 
@@ -2762,7 +2770,7 @@ data:
             "type": "datasource"
           },
           {
-            "allValue": null,
+            "allValue": ".+",
             "current": {
               "selected": true,
               "text": "All",
@@ -2777,7 +2785,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(prometheus_build_info, job)",
+            "query": "label_values(prometheus_build_info{job=\"prometheus\"}, job)",
             "refresh": 1,
             "regex": "",
             "sort": 2,
@@ -2790,7 +2798,7 @@ data:
             "useTags": false
           },
           {
-            "allValue": null,
+            "allValue": ".+",
             "current": {
               "selected": true,
               "text": "All",
@@ -2805,7 +2813,7 @@ data:
             "options": [
 
             ],
-            "query": "label_values(prometheus_build_info, instance)",
+            "query": "label_values(prometheus_build_info{job=~\"$job\"}, instance)",
             "refresh": 1,
             "regex": "",
             "sort": 2,

addons/grafana/deployment.yaml

@@ -24,7 +24,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: grafana
-          image: docker.io/grafana/grafana:8.3.4
+          image: docker.io/grafana/grafana:9.3.1
           env:
             - name: GF_PATHS_CONFIG
               value: "/etc/grafana/custom.ini"
@@ -32,15 +32,22 @@ spec:
             - name: http
               containerPort: 8080
           livenessProbe:
-            httpGet:
+            tcpSocket:
-              path: /metrics
               port: 8080
-            initialDelaySeconds: 10
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            timeoutSeconds: 1
+            failureThreshold: 5
+            successThreshold: 1
           readinessProbe:
             httpGet:
-              path: /api/health
+              scheme: HTTP
+              path: /robots.txt
               port: 8080
             initialDelaySeconds: 10
+            periodSeconds: 30
+            successThreshold: 1
+            timeoutSeconds: 5
           resources:
             requests:
               cpu: 100m
@@ -69,6 +76,8 @@ spec:
               mountPath: /etc/grafana/dashboards/k8s-resources-1
             - name: dashboards-k8s-resources-2
               mountPath: /etc/grafana/dashboards/k8s-resources-2
+            - name: dashboards-k8s-network
+              mountPath: /etc/grafana/dashboards/k8s-network
             - name: dashboards-coredns
               mountPath: /etc/grafana/dashboards/coredns
             - name: dashboards-nginx-ingress
@@ -101,6 +110,9 @@ spec:
         - name: dashboards-k8s-resources-1
           configMap:
             name: grafana-dashboards-k8s-resources-1
+        - name: dashboards-k8s-network
+          configMap:
+            name: grafana-dashboards-k8s-network
         - name: dashboards-k8s-resources-2
           configMap:
             name: grafana-dashboards-k8s-resources-2

addons/nginx-ingress/aws/class.yaml

@@ -3,4 +3,4 @@ kind: IngressClass
 metadata:
   name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public

addons/nginx-ingress/aws/deployment.yaml

@@ -23,9 +23,10 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
           args:
             - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
             - --ingress-class=public
           # use downward API
           env:

addons/nginx-ingress/aws/rbac/role.yaml

@@ -10,6 +10,7 @@ rules:
       - configmaps
       - pods
       - secrets
+      - endpoints
     verbs:
       - get
   - apiGroups:
@@ -37,3 +38,11 @@ rules:
       - endpoints
     verbs:
       - get
+  - apiGroups:
+      - "coordination.k8s.io"
+    resources:
+      - leases
+    verbs:
+      - create
+      - get
+      - update

addons/nginx-ingress/azure/class.yaml

@@ -3,4 +3,4 @@ kind: IngressClass
 metadata:
   name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public

addons/nginx-ingress/azure/deployment.yaml

@@ -23,9 +23,10 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
           args:
             - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
             - --ingress-class=public
           # use downward API
           env:

addons/nginx-ingress/azure/rbac/role.yaml

@@ -10,6 +10,7 @@ rules:
       - configmaps
       - pods
       - secrets
+      - endpoints
     verbs:
       - get
   - apiGroups:
@@ -32,8 +33,11 @@ rules:
     verbs:
       - create
   - apiGroups:
-      - ""
+      - "coordination.k8s.io"
     resources:
-      - endpoints
+      - leases
     verbs:
+      - create
       - get
+      - update
+

addons/nginx-ingress/bare-metal/class.yaml

@@ -3,4 +3,4 @@ kind: IngressClass
 metadata:
   name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -23,9 +23,10 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
           args:
             - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
             - --ingress-class=public
           # use downward API
           env:

addons/nginx-ingress/bare-metal/rbac/role.yaml

@@ -10,6 +10,7 @@ rules:
       - configmaps
       - pods
       - secrets
+      - endpoints
     verbs:
       - get
   - apiGroups:
@@ -32,8 +33,10 @@ rules:
     verbs:
       - create
   - apiGroups:
-      - ""
+      - "coordination.k8s.io"
     resources:
-      - endpoints
+      - leases
     verbs:
+      - create
       - get
+      - update

addons/nginx-ingress/digital-ocean/class.yaml

@@ -3,4 +3,4 @@ kind: IngressClass
 metadata:
   name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -23,9 +23,10 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
           args:
             - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
             - --ingress-class=public
           # use downward API
           env:

addons/nginx-ingress/digital-ocean/rbac/role.yaml

@@ -10,6 +10,7 @@ rules:
       - configmaps
       - pods
       - secrets
+      - endpoints
     verbs:
       - get
   - apiGroups:
@@ -32,8 +33,10 @@ rules:
     verbs:
       - create
   - apiGroups:
-      - ""
+      - "coordination.k8s.io"
     resources:
-      - endpoints
+      - leases
     verbs:
+      - create
       - get
+      - update

addons/nginx-ingress/google-cloud/class.yaml

@@ -3,4 +3,4 @@ kind: IngressClass
 metadata:
   name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -23,9 +23,10 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
           args:
             - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
             - --ingress-class=public
           # use downward API
           env:

addons/nginx-ingress/google-cloud/rbac/cluster-role.yaml

@@ -59,4 +59,11 @@ rules:
       - get
       - list
       - watch
-
+  - apiGroups: 
+      - discovery.k8s.io
+    resources:
+      - "endpointslices"
+    verbs: 
+      - get
+      - list
+      - watch

addons/nginx-ingress/google-cloud/rbac/role.yaml

@@ -10,6 +10,7 @@ rules:
       - configmaps
       - pods
       - secrets
+      - endpoints
     verbs:
       - get
   - apiGroups:
@@ -32,8 +33,10 @@ rules:
     verbs:
       - create
   - apiGroups:
-      - ""
+      - "coordination.k8s.io"
     resources:
-      - endpoints
+      - leases
     verbs:
+      - create
       - get
+      - update

addons/prometheus/deployment.yaml

@@ -21,7 +21,7 @@ spec:
       serviceAccountName: prometheus
       containers:
         - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.32.1
+          image: quay.io/prometheus/prometheus:v2.40.5
           args:
             - --web.listen-address=0.0.0.0:9090
             - --config.file=/etc/prometheus/prometheus.yaml

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -25,7 +25,7 @@ spec:
       serviceAccountName: kube-state-metrics
       containers:
       - name: kube-state-metrics
-        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0
+        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.7.0
         ports:
           - name: metrics
             containerPort: 8080

addons/prometheus/exporters/node-exporter/daemonset.yaml

@@ -22,19 +22,19 @@ spec:
       securityContext:
         runAsNonRoot: true
         runAsUser: 65534
+        runAsGroup: 65534
+        fsGroup: 65534
         seccompProfile:
           type: RuntimeDefault
       hostNetwork: true
       hostPID: true
       containers:
       - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v1.3.1
+        image: quay.io/prometheus/node-exporter:v1.5.0
         args:
           - --path.procfs=/host/proc
           - --path.sysfs=/host/sys
           - --path.rootfs=/host/root
-          - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)($|/)
-          - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
         ports:
           - name: metrics
             containerPort: 9100
@@ -46,6 +46,9 @@ spec:
           limits:
             cpu: 200m
             memory: 100Mi
+        securityContext:
+          seLinuxOptions:
+            type: spc_t
         volumeMounts:
           - name: proc
             mountPath: /host/proc
@@ -55,9 +58,12 @@ spec:
             readOnly: true
           - name: root
             mountPath: /host/root
+            mountPropagation: HostToContainer
             readOnly: true
       tolerations:
-        - key: node-role.kubernetes.io/master
+        - key: node-role.kubernetes.io/controller
+          operator: Exists
+        - key: node-role.kubernetes.io/control-plane
           operator: Exists
         - key: node.kubernetes.io/not-ready
           operator: Exists

addons/prometheus/rules.yaml

@@ -57,10 +57,10 @@ data:
             {
               "alert": "etcdGRPCRequestsSlow",
               "annotations": {
-                "description": "etcd cluster \"{{ $labels.job }}\": gRPC requests to {{ $labels.grpc_method }} are taking {{ $value }}s on etcd instance {{ $labels.instance }}.",
+                "description": "etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.",
                 "summary": "etcd grpc requests are slow"
               },
-              "expr": "histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~\".*etcd.*\", grpc_type=\"unary\"}[5m])) without(grpc_type))\n> 0.15\n",
+              "expr": "histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~\".*etcd.*\", grpc_method!=\"Defragment\", grpc_type=\"unary\"}[5m])) without(grpc_type))\n> 0.15\n",
               "for": "10m",
               "labels": {
                 "severity": "critical"
@@ -105,7 +105,8 @@ data:
             {
               "alert": "etcdHighFsyncDurations",
               "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}."
+                "description": "etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.",
+                "summary": "etcd cluster 99th percentile fsync durations are too high."
               },
               "expr": "histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 1\n",
               "for": "10m",
@@ -125,46 +126,11 @@ data:
                 "severity": "warning"
               }
             },
-            {
-              "alert": "etcdHighNumberOfFailedHTTPRequests",
-              "annotations": {
-                "description": "{{ $value }}% of requests for {{ $labels.method }} failed on etcd instance {{ $labels.instance }}",
-                "summary": "etcd has high number of failed HTTP requests."
-              },
-              "expr": "sum(rate(etcd_http_failed_total{job=~\".*etcd.*\", code!=\"404\"}[5m])) without (code) / sum(rate(etcd_http_received_total{job=~\".*etcd.*\"}[5m]))\nwithout (code) > 0.01\n",
-              "for": "10m",
-              "labels": {
-                "severity": "warning"
-              }
-            },
-            {
-              "alert": "etcdHighNumberOfFailedHTTPRequests",
-              "annotations": {
-                "description": "{{ $value }}% of requests for {{ $labels.method }} failed on etcd instance {{ $labels.instance }}.",
-                "summary": "etcd has high number of failed HTTP requests."
-              },
-              "expr": "sum(rate(etcd_http_failed_total{job=~\".*etcd.*\", code!=\"404\"}[5m])) without (code) / sum(rate(etcd_http_received_total{job=~\".*etcd.*\"}[5m]))\nwithout (code) > 0.05\n",
-              "for": "10m",
-              "labels": {
-                "severity": "critical"
-              }
-            },
-            {
-              "alert": "etcdHTTPRequestsSlow",
-              "annotations": {
-                "description": "etcd instance {{ $labels.instance }} HTTP requests to {{ $labels.method }} are slow.",
-                "summary": "etcd instance HTTP requests are slow."
-              },
-              "expr": "histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[5m]))\n> 0.15\n",
-              "for": "10m",
-              "labels": {
-                "severity": "warning"
-              }
-            },
             {
               "alert": "etcdBackendQuotaLowSpace",
               "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full."
+                "description": "etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.",
+                "summary": "etcd cluster database is running full."
               },
               "expr": "(etcd_mvcc_db_total_size_in_bytes/etcd_server_quota_backend_bytes)*100 > 95\n",
               "for": "10m",
@@ -175,7 +141,8 @@ data:
             {
               "alert": "etcdExcessiveDatabaseGrowth",
               "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": Observed surge in etcd writes leading to 50% increase in database size over the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive."
+                "description": "etcd cluster \"{{ $labels.job }}\": Observed surge in etcd writes leading to 50% increase in database size over the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.",
+                "summary": "etcd cluster database growing very fast."
               },
               "expr": "increase(((etcd_mvcc_db_total_size_in_bytes/etcd_server_quota_backend_bytes)*100)[240m:1m]) > 50\n",
               "for": "10m",
@@ -191,122 +158,113 @@ data:
     {
       "groups": [
         {
-          "name": "kube-apiserver.rules",
+          "name": "kube-apiserver-burnrate.rules",
           "rules": [
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\"}[1d]))\n    -\n    (\n      (\n        sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=~\"resource|\",le=\"0.1\"}[1d]))\n        or\n        vector(0)\n      )\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"namespace\",le=\"0.5\"}[1d]))\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"cluster\",le=\"5\"}[1d]))\n    )\n  )\n  +\n  # errors\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[1d]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1d]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[1d]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[1d]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[1d]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[1d]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[1d]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1d]))\n",
               "labels": {
                 "verb": "read"
               },
               "record": "apiserver_request:burnrate1d"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))\n    -\n    (\n      (\n        sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=~\"resource|\",le=\"0.1\"}[1h]))\n        or\n        vector(0)\n      )\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"namespace\",le=\"0.5\"}[1h]))\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"cluster\",le=\"5\"}[1h]))\n    )\n  )\n  +\n  # errors\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[1h]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[1h]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[1h]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[1h]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[1h]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[1h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))\n",
               "labels": {
                 "verb": "read"
               },
               "record": "apiserver_request:burnrate1h"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\"}[2h]))\n    -\n    (\n      (\n        sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=~\"resource|\",le=\"0.1\"}[2h]))\n        or\n        vector(0)\n      )\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"namespace\",le=\"0.5\"}[2h]))\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"cluster\",le=\"5\"}[2h]))\n    )\n  )\n  +\n  # errors\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[2h]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[2h]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[2h]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[2h]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[2h]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[2h]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[2h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[2h]))\n",
               "labels": {
                 "verb": "read"
               },
               "record": "apiserver_request:burnrate2h"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))\n    -\n    (\n      (\n        sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=~\"resource|\",le=\"0.1\"}[30m]))\n        or\n        vector(0)\n      )\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"namespace\",le=\"0.5\"}[30m]))\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"cluster\",le=\"5\"}[30m]))\n    )\n  )\n  +\n  # errors\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[30m]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[30m]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[30m]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[30m]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[30m]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[30m]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))\n",
               "labels": {
                 "verb": "read"
               },
               "record": "apiserver_request:burnrate30m"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\"}[3d]))\n    -\n    (\n      (\n        sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=~\"resource|\",le=\"0.1\"}[3d]))\n        or\n        vector(0)\n      )\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"namespace\",le=\"0.5\"}[3d]))\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"cluster\",le=\"5\"}[3d]))\n    )\n  )\n  +\n  # errors\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[3d]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[3d]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[3d]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[3d]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[3d]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[3d]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[3d]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[3d]))\n",
               "labels": {
                 "verb": "read"
               },
               "record": "apiserver_request:burnrate3d"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))\n    -\n    (\n      (\n        sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=~\"resource|\",le=\"0.1\"}[5m]))\n        or\n        vector(0)\n      )\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"namespace\",le=\"0.5\"}[5m]))\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"cluster\",le=\"5\"}[5m]))\n    )\n  )\n  +\n  # errors\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[5m]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[5m]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[5m]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[5m]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[5m]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[5m]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))\n",
               "labels": {
                 "verb": "read"
               },
               "record": "apiserver_request:burnrate5m"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))\n    -\n    (\n      (\n        sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=~\"resource|\",le=\"0.1\"}[6h]))\n        or\n        vector(0)\n      )\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"namespace\",le=\"0.5\"}[6h]))\n      +\n      sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"cluster\",le=\"5\"}[6h]))\n    )\n  )\n  +\n  # errors\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[6h]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[6h]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[6h]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[6h]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[6h]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[6h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))\n",
               "labels": {
                 "verb": "read"
               },
               "record": "apiserver_request:burnrate6h"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1d]))\n    -\n    sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"}[1d]))\n  )\n  +\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[1d]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1d]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[1d]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[1d]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[1d]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1d]))\n",
               "labels": {
                 "verb": "write"
               },
               "record": "apiserver_request:burnrate1d"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))\n    -\n    sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"}[1h]))\n  )\n  +\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[1h]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[1h]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[1h]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[1h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))\n",
               "labels": {
                 "verb": "write"
               },
               "record": "apiserver_request:burnrate1h"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[2h]))\n    -\n    sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"}[2h]))\n  )\n  +\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[2h]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[2h]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[2h]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[2h]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[2h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[2h]))\n",
               "labels": {
                 "verb": "write"
               },
               "record": "apiserver_request:burnrate2h"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))\n    -\n    sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"}[30m]))\n  )\n  +\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[30m]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[30m]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[30m]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[30m]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))\n",
               "labels": {
                 "verb": "write"
               },
               "record": "apiserver_request:burnrate30m"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[3d]))\n    -\n    sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"}[3d]))\n  )\n  +\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[3d]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[3d]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[3d]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[3d]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[3d]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[3d]))\n",
               "labels": {
                 "verb": "write"
               },
               "record": "apiserver_request:burnrate3d"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))\n    -\n    sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"}[5m]))\n  )\n  +\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[5m]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[5m]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[5m]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[5m]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))\n",
               "labels": {
                 "verb": "write"
               },
               "record": "apiserver_request:burnrate5m"
             },
             {
-              "expr": "(\n  (\n    # too slow\n    sum(rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))\n    -\n    sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"}[6h]))\n  )\n  +\n  sum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[6h]))\n)\n/\nsum(rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[6h]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[6h]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[6h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))\n",
               "labels": {
                 "verb": "write"
               },
               "record": "apiserver_request:burnrate6h"
+            }
+          ]
         },
         {
-              "expr": "sum by (code,resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))\n",
+          "name": "kube-apiserver-histogram.rules",
-              "labels": {
+          "rules": [
-                "verb": "read"
-              },
-              "record": "code_resource:apiserver_request_total:rate5m"
-            },
             {
-              "expr": "sum by (code,resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))\n",
+              "expr": "histogram_quantile(0.99, sum by (cluster, le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[5m]))) > 0\n",
-              "labels": {
-                "verb": "write"
-              },
-              "record": "code_resource:apiserver_request_total:rate5m"
-            },
-            {
-              "expr": "histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) > 0\n",
               "labels": {
                 "quantile": "0.99",
                 "verb": "read"
@@ -314,33 +272,12 @@ data:
               "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
             },
             {
-              "expr": "histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) > 0\n",
+              "expr": "histogram_quantile(0.99, sum by (cluster, le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[5m]))) > 0\n",
               "labels": {
                 "quantile": "0.99",
                 "verb": "write"
               },
               "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
-            },
-            {
-              "expr": "histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
-              "labels": {
-                "quantile": "0.99"
-              },
-              "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
-            },
-            {
-              "expr": "histogram_quantile(0.9, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
-              "labels": {
-                "quantile": "0.9"
-              },
-              "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
-            },
-            {
-              "expr": "histogram_quantile(0.5, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
-              "labels": {
-                "quantile": "0.5"
-              },
-              "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
             }
           ]
         },
@@ -349,135 +286,89 @@ data:
           "name": "kube-apiserver-availability.rules",
           "rules": [
             {
-              "expr": "1 - (\n  (\n    # write too slow\n    sum(increase(apiserver_request_duration_seconds_count{verb=~\"POST|PUT|PATCH|DELETE\"}[30d]))\n    -\n    sum(increase(apiserver_request_duration_seconds_bucket{verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"}[30d]))\n  ) +\n  (\n    # read too slow\n    sum(increase(apiserver_request_duration_seconds_count{verb=~\"LIST|GET\"}[30d]))\n    -\n    (\n      (\n        sum(increase(apiserver_request_duration_seconds_bucket{verb=~\"LIST|GET\",scope=~\"resource|\",le=\"0.1\"}[30d]))\n        or\n        vector(0)\n      )\n      +\n      sum(increase(apiserver_request_duration_seconds_bucket{verb=~\"LIST|GET\",scope=\"namespace\",le=\"0.5\"}[30d]))\n      +\n      sum(increase(apiserver_request_duration_seconds_bucket{verb=~\"LIST|GET\",scope=\"cluster\",le=\"5\"}[30d]))\n    )\n  ) +\n  # errors\n  sum(code:apiserver_request_total:increase30d{code=~\"5..\"} or vector(0))\n)\n/\nsum(code:apiserver_request_total:increase30d)\n",
+              "expr": "avg_over_time(code_verb:apiserver_request_total:increase1h[30d]) * 24 * 30\n",
+              "record": "code_verb:apiserver_request_total:increase30d"
+            },
+            {
+              "expr": "sum by (cluster, code) (code_verb:apiserver_request_total:increase30d{verb=~\"LIST|GET\"})\n",
+              "labels": {
+                "verb": "read"
+              },
+              "record": "code:apiserver_request_total:increase30d"
+            },
+            {
+              "expr": "sum by (cluster, code) (code_verb:apiserver_request_total:increase30d{verb=~\"POST|PUT|PATCH|DELETE\"})\n",
+              "labels": {
+                "verb": "write"
+              },
+              "record": "code:apiserver_request_total:increase30d"
+            },
+            {
+              "expr": "sum by (cluster, verb, scope) (increase(apiserver_request_duration_seconds_count[1h]))\n",
+              "record": "cluster_verb_scope:apiserver_request_duration_seconds_count:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, verb, scope) (avg_over_time(cluster_verb_scope:apiserver_request_duration_seconds_count:increase1h[30d]) * 24 * 30)\n",
+              "record": "cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d"
+            },
+            {
+              "expr": "sum by (cluster, verb, scope, le) (increase(apiserver_request_duration_seconds_bucket[1h]))\n",
+              "record": "cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, verb, scope, le) (avg_over_time(cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase1h[30d]) * 24 * 30)\n",
+              "record": "cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d"
+            },
+            {
+              "expr": "1 - (\n  (\n    # write too slow\n    sum by (cluster) (cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d{verb=~\"POST|PUT|PATCH|DELETE\"})\n    -\n    sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"})\n  ) +\n  (\n    # read too slow\n    sum by (cluster) (cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d{verb=~\"LIST|GET\"})\n    -\n    (\n      (\n        sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=~\"resource|\",le=\"1\"})\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=\"namespace\",le=\"5\"})\n      +\n      sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=\"cluster\",le=\"30\"})\n    )\n  ) +\n  # errors\n  sum by (cluster) (code:apiserver_request_total:increase30d{code=~\"5..\"} or vector(0))\n)\n/\nsum by (cluster) (code:apiserver_request_total:increase30d)\n",
               "labels": {
                 "verb": "all"
               },
               "record": "apiserver_request:availability30d"
             },
             {
-              "expr": "1 - (\n  sum(increase(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\"}[30d]))\n  -\n  (\n    # too slow\n    (\n      sum(increase(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=~\"resource|\",le=\"0.1\"}[30d]))\n      or\n      vector(0)\n    )\n    +\n    sum(increase(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"namespace\",le=\"0.5\"}[30d]))\n    +\n    sum(increase(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",scope=\"cluster\",le=\"5\"}[30d]))\n  )\n  +\n  # errors\n  sum(code:apiserver_request_total:increase30d{verb=\"read\",code=~\"5..\"} or vector(0))\n)\n/\nsum(code:apiserver_request_total:increase30d{verb=\"read\"})\n",
+              "expr": "1 - (\n  sum by (cluster) (cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d{verb=~\"LIST|GET\"})\n  -\n  (\n    # too slow\n    (\n      sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=~\"resource|\",le=\"1\"})\n      or\n      vector(0)\n    )\n    +\n    sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=\"namespace\",le=\"5\"})\n    +\n    sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=\"cluster\",le=\"30\"})\n  )\n  +\n  # errors\n  sum by (cluster) (code:apiserver_request_total:increase30d{verb=\"read\",code=~\"5..\"} or vector(0))\n)\n/\nsum by (cluster) (code:apiserver_request_total:increase30d{verb=\"read\"})\n",
               "labels": {
                 "verb": "read"
               },
               "record": "apiserver_request:availability30d"
             },
             {
-              "expr": "1 - (\n  (\n    # too slow\n    sum(increase(apiserver_request_duration_seconds_count{verb=~\"POST|PUT|PATCH|DELETE\"}[30d]))\n    -\n    sum(increase(apiserver_request_duration_seconds_bucket{verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"}[30d]))\n  )\n  +\n  # errors\n  sum(code:apiserver_request_total:increase30d{verb=\"write\",code=~\"5..\"} or vector(0))\n)\n/\nsum(code:apiserver_request_total:increase30d{verb=\"write\"})\n",
+              "expr": "1 - (\n  (\n    # too slow\n    sum by (cluster) (cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d{verb=~\"POST|PUT|PATCH|DELETE\"})\n    -\n    sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"})\n  )\n  +\n  # errors\n  sum by (cluster) (code:apiserver_request_total:increase30d{verb=\"write\",code=~\"5..\"} or vector(0))\n)\n/\nsum by (cluster) (code:apiserver_request_total:increase30d{verb=\"write\"})\n",
               "labels": {
                 "verb": "write"
               },
               "record": "apiserver_request:availability30d"
             },
             {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"LIST\",code=~\"2..\"}[30d]))\n",
+              "expr": "sum by (cluster,code,resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"GET\",code=~\"2..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"POST\",code=~\"2..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"PUT\",code=~\"2..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"PATCH\",code=~\"2..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"DELETE\",code=~\"2..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"LIST\",code=~\"3..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"GET\",code=~\"3..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"POST\",code=~\"3..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"PUT\",code=~\"3..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"PATCH\",code=~\"3..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"DELETE\",code=~\"3..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"LIST\",code=~\"4..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"GET\",code=~\"4..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"POST\",code=~\"4..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"PUT\",code=~\"4..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"PATCH\",code=~\"4..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"DELETE\",code=~\"4..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"LIST\",code=~\"5..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"GET\",code=~\"5..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"POST\",code=~\"5..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"PUT\",code=~\"5..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"PATCH\",code=~\"5..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=\"DELETE\",code=~\"5..\"}[30d]))\n",
-              "record": "code_verb:apiserver_request_total:increase30d"
-            },
-            {
-              "expr": "sum by (code) (code_verb:apiserver_request_total:increase30d{verb=~\"LIST|GET\"})\n",
               "labels": {
                 "verb": "read"
               },
-              "record": "code:apiserver_request_total:increase30d"
+              "record": "code_resource:apiserver_request_total:rate5m"
             },
             {
-              "expr": "sum by (code) (code_verb:apiserver_request_total:increase30d{verb=~\"POST|PUT|PATCH|DELETE\"})\n",
+              "expr": "sum by (cluster,code,resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))\n",
               "labels": {
                 "verb": "write"
               },
-              "record": "code:apiserver_request_total:increase30d"
+              "record": "code_resource:apiserver_request_total:rate5m"
+            },
+            {
+              "expr": "sum by (cluster, code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET|POST|PUT|PATCH|DELETE\",code=~\"2..\"}[1h]))\n",
+              "record": "code_verb:apiserver_request_total:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET|POST|PUT|PATCH|DELETE\",code=~\"3..\"}[1h]))\n",
+              "record": "code_verb:apiserver_request_total:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET|POST|PUT|PATCH|DELETE\",code=~\"4..\"}[1h]))\n",
+              "record": "code_verb:apiserver_request_total:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET|POST|PUT|PATCH|DELETE\",code=~\"5..\"}[1h]))\n",
+              "record": "code_verb:apiserver_request_total:increase1h"
             }
           ]
         },
@@ -485,8 +376,8 @@ data:
           "name": "k8s.rules",
           "rules": [
             {
-              "expr": "sum by (cluster, namespace, pod, container) (\n  rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\", container!=\"POD\"}[5m])\n) * on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (\n  1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=\"\"})\n)\n",
+              "expr": "sum by (cluster, namespace, pod, container) (\n  irate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\"}[5m])\n) * on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (\n  1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=\"\"})\n)\n",
-              "record": "node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate"
+              "record": "node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate"
             },
             {
               "expr": "container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", image!=\"\"}\n* on (namespace, pod) group_left(node) topk by(namespace, pod) (1,\n  max by(namespace, pod, node) (kube_pod_info{node!=\"\"})\n)\n",
@@ -505,12 +396,36 @@ data:
               "record": "node_namespace_pod_container:container_memory_swap"
             },
             {
-              "expr": "sum by (namespace) (\n    sum by (namespace, pod) (\n        max by (namespace, pod, container) (\n            kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"}\n        ) * on(namespace, pod) group_left() max by (namespace, pod) (\n            kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
+              "expr": "kube_pod_container_resource_requests{resource=\"memory\",job=\"kube-state-metrics\"}  * on (namespace, pod, cluster)\ngroup_left() max by (namespace, pod, cluster) (\n  (kube_pod_status_phase{phase=~\"Pending|Running\"} == 1)\n)\n",
-              "record": "namespace:kube_pod_container_resource_requests_memory_bytes:sum"
+              "record": "cluster:namespace:pod_memory:active:kube_pod_container_resource_requests"
             },
             {
-              "expr": "sum by (namespace) (\n    sum by (namespace, pod) (\n        max by (namespace, pod, container) (\n            kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n        ) * on(namespace, pod) group_left() max by (namespace, pod) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
+              "expr": "sum by (namespace, cluster) (\n    sum by (namespace, pod, cluster) (\n        max by (namespace, pod, container, cluster) (\n          kube_pod_container_resource_requests{resource=\"memory\",job=\"kube-state-metrics\"}\n        ) * on(namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
-              "record": "namespace:kube_pod_container_resource_requests_cpu_cores:sum"
+              "record": "namespace_memory:kube_pod_container_resource_requests:sum"
+            },
+            {
+              "expr": "kube_pod_container_resource_requests{resource=\"cpu\",job=\"kube-state-metrics\"}  * on (namespace, pod, cluster)\ngroup_left() max by (namespace, pod, cluster) (\n  (kube_pod_status_phase{phase=~\"Pending|Running\"} == 1)\n)\n",
+              "record": "cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests"
+            },
+            {
+              "expr": "sum by (namespace, cluster) (\n    sum by (namespace, pod, cluster) (\n        max by (namespace, pod, container, cluster) (\n          kube_pod_container_resource_requests{resource=\"cpu\",job=\"kube-state-metrics\"}\n        ) * on(namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
+              "record": "namespace_cpu:kube_pod_container_resource_requests:sum"
+            },
+            {
+              "expr": "kube_pod_container_resource_limits{resource=\"memory\",job=\"kube-state-metrics\"}  * on (namespace, pod, cluster)\ngroup_left() max by (namespace, pod, cluster) (\n  (kube_pod_status_phase{phase=~\"Pending|Running\"} == 1)\n)\n",
+              "record": "cluster:namespace:pod_memory:active:kube_pod_container_resource_limits"
+            },
+            {
+              "expr": "sum by (namespace, cluster) (\n    sum by (namespace, pod, cluster) (\n        max by (namespace, pod, container, cluster) (\n          kube_pod_container_resource_limits{resource=\"memory\",job=\"kube-state-metrics\"}\n        ) * on(namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
+              "record": "namespace_memory:kube_pod_container_resource_limits:sum"
+            },
+            {
+              "expr": "kube_pod_container_resource_limits{resource=\"cpu\",job=\"kube-state-metrics\"}  * on (namespace, pod, cluster)\ngroup_left() max by (namespace, pod, cluster) (\n (kube_pod_status_phase{phase=~\"Pending|Running\"} == 1)\n )\n",
+              "record": "cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits"
+            },
+            {
+              "expr": "sum by (namespace, cluster) (\n    sum by (namespace, pod, cluster) (\n        max by (namespace, pod, container, cluster) (\n          kube_pod_container_resource_limits{resource=\"cpu\",job=\"kube-state-metrics\"}\n        ) * on(namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
+              "record": "namespace_cpu:kube_pod_container_resource_limits:sum"
             },
             {
               "expr": "max by (cluster, namespace, workload, pod) (\n  label_replace(\n    label_replace(\n      kube_pod_owner{job=\"kube-state-metrics\", owner_kind=\"ReplicaSet\"},\n      \"replicaset\", \"$1\", \"owner_name\", \"(.*)\"\n    ) * on(replicaset, namespace) group_left(owner_name) topk by(replicaset, namespace) (\n      1, max by (replicaset, namespace, owner_name) (\n        kube_replicaset_owner{job=\"kube-state-metrics\"}\n      )\n    ),\n    \"workload\", \"$1\", \"owner_name\", \"(.*)\"\n  )\n)\n",
@@ -532,6 +447,13 @@ data:
                 "workload_type": "statefulset"
               },
               "record": "namespace_workload_pod:kube_pod_owner:relabel"
+            },
+            {
+              "expr": "max by (cluster, namespace, workload, pod) (\n  label_replace(\n    kube_pod_owner{job=\"kube-state-metrics\", owner_kind=\"Job\"},\n    \"workload\", \"$1\", \"owner_name\", \"(.*)\"\n  )\n)\n",
+              "labels": {
+                "workload_type": "job"
+              },
+              "record": "namespace_workload_pod:kube_pod_owner:relabel"
             }
           ]
         },
@@ -611,12 +533,16 @@ data:
               "record": "node_namespace_pod:kube_pod_info:"
             },
             {
-              "expr": "count by (cluster, node) (sum by (node, cpu) (\n  node_cpu_seconds_total{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n))\n",
+              "expr": "count by (cluster, node) (sum by (node, cpu) (\n  node_cpu_seconds_total{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  topk by(namespace, pod) (1, node_namespace_pod:kube_pod_info:)\n))\n",
               "record": "node:node_num_cpu:sum"
             },
             {
               "expr": "sum(\n  node_memory_MemAvailable_bytes{job=\"node-exporter\"} or\n  (\n    node_memory_Buffers_bytes{job=\"node-exporter\"} +\n    node_memory_Cached_bytes{job=\"node-exporter\"} +\n    node_memory_MemFree_bytes{job=\"node-exporter\"} +\n    node_memory_Slab_bytes{job=\"node-exporter\"}\n  )\n) by (cluster)\n",
               "record": ":node_memory_MemAvailable_bytes:sum"
+            },
+            {
+              "expr": "sum(rate(node_cpu_seconds_total{job=\"node-exporter\",mode!=\"idle\",mode!=\"iowait\",mode!=\"steal\"}[5m])) /\ncount(sum(node_cpu_seconds_total{job=\"node-exporter\"}) by (cluster, instance, cpu))\n",
+              "record": "cluster:node_cpu:ratio_rate5m"
             }
           ]
         },
@@ -624,21 +550,21 @@ data:
           "name": "kubelet.rules",
           "rules": [
             {
-              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (instance, le) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (cluster, instance, le) * on(cluster, instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
               "labels": {
                 "quantile": "0.99"
               },
               "record": "node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile"
             },
             {
-              "expr": "histogram_quantile(0.9, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (instance, le) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
+              "expr": "histogram_quantile(0.9, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (cluster, instance, le) * on(cluster, instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
               "labels": {
                 "quantile": "0.9"
               },
               "record": "node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile"
             },
             {
-              "expr": "histogram_quantile(0.5, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (instance, le) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
+              "expr": "histogram_quantile(0.5, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (cluster, instance, le) * on(cluster, instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
               "labels": {
                 "quantile": "0.5"
               },
@@ -652,11 +578,11 @@ data:
             {
               "alert": "KubePodCrashLooping",
               "annotations": {
-                "description": "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is restarting {{ printf \"%.2f\" $value }} times / 5 minutes.",
+                "description": "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is in waiting state (reason: \"CrashLoopBackOff\").",
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping",
                 "summary": "Pod is crash looping."
               },
-              "expr": "rate(kube_pod_container_status_restarts_total{job=\"kube-state-metrics\"}[5m]) * 60 * 5 > 0\n",
+              "expr": "max_over_time(kube_pod_container_status_waiting_reason{reason=\"CrashLoopBackOff\", job=\"kube-state-metrics\"}[5m]) >= 1\n",
               "for": "15m",
               "labels": {
                 "severity": "warning"
@@ -695,7 +621,7 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch",
                 "summary": "Deployment has not matched the expected number of replicas."
               },
-              "expr": "(\n  kube_deployment_spec_replicas{job=\"kube-state-metrics\"}\n    !=\n  kube_deployment_status_replicas_available{job=\"kube-state-metrics\"}\n) and (\n  changes(kube_deployment_status_replicas_updated{job=\"kube-state-metrics\"}[5m])\n    ==\n  0\n)\n",
+              "expr": "(\n  kube_deployment_spec_replicas{job=\"kube-state-metrics\"}\n    >\n  kube_deployment_status_replicas_available{job=\"kube-state-metrics\"}\n) and (\n  changes(kube_deployment_status_replicas_updated{job=\"kube-state-metrics\"}[10m])\n    ==\n  0\n)\n",
               "for": "15m",
               "labels": {
                 "severity": "warning"
@@ -708,7 +634,7 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetreplicasmismatch",
                 "summary": "Deployment has not matched the expected number of replicas."
               },
-              "expr": "(\n  kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\"}\n    !=\n  kube_statefulset_status_replicas{job=\"kube-state-metrics\"}\n) and (\n  changes(kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\"}[5m])\n    ==\n  0\n)\n",
+              "expr": "(\n  kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\"}\n    !=\n  kube_statefulset_status_replicas{job=\"kube-state-metrics\"}\n) and (\n  changes(kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\"}[10m])\n    ==\n  0\n)\n",
               "for": "15m",
               "labels": {
                 "severity": "warning"
@@ -747,7 +673,7 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetrolloutstuck",
                 "summary": "DaemonSet rollout is stuck."
               },
-              "expr": "(\n  (\n    kube_daemonset_status_current_number_scheduled{job=\"kube-state-metrics\"}\n     !=\n    kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  ) or (\n    kube_daemonset_status_number_misscheduled{job=\"kube-state-metrics\"}\n     !=\n    0\n  ) or (\n    kube_daemonset_updated_number_scheduled{job=\"kube-state-metrics\"}\n     !=\n    kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  ) or (\n    kube_daemonset_status_number_available{job=\"kube-state-metrics\"}\n     !=\n    kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  )\n) and (\n  changes(kube_daemonset_updated_number_scheduled{job=\"kube-state-metrics\"}[5m])\n    ==\n  0\n)\n",
+              "expr": "(\n  (\n    kube_daemonset_status_current_number_scheduled{job=\"kube-state-metrics\"}\n     !=\n    kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  ) or (\n    kube_daemonset_status_number_misscheduled{job=\"kube-state-metrics\"}\n     !=\n    0\n  ) or (\n    kube_daemonset_status_updated_number_scheduled{job=\"kube-state-metrics\"}\n     !=\n    kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  ) or (\n    kube_daemonset_status_number_available{job=\"kube-state-metrics\"}\n     !=\n    kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  )\n) and (\n  changes(kube_daemonset_status_updated_number_scheduled{job=\"kube-state-metrics\"}[5m])\n    ==\n  0\n)\n",
               "for": "15m",
               "labels": {
                 "severity": "warning"
@@ -756,7 +682,7 @@ data:
             {
               "alert": "KubeContainerWaiting",
               "annotations": {
-                "description": "Pod {{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container}} has been in waiting state for longer than 1 hour.",
+                "description": "pod/{{ $labels.pod }} in namespace {{ $labels.namespace }} on container {{ $labels.container}} has been in waiting state for longer than 1 hour.",
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontainerwaiting",
                 "summary": "Pod container waiting longer than 1 hour"
               },
@@ -821,11 +747,11 @@ data:
             {
               "alert": "KubeHpaReplicasMismatch",
               "annotations": {
-                "description": "HPA {{ $labels.namespace }}/{{ $labels.hpa }} has not matched the desired number of replicas for longer than 15 minutes.",
+                "description": "HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has not matched the desired number of replicas for longer than 15 minutes.",
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubehpareplicasmismatch",
                 "summary": "HPA has not matched descired number of replicas."
               },
-              "expr": "(kube_hpa_status_desired_replicas{job=\"kube-state-metrics\"}\n  !=\nkube_hpa_status_current_replicas{job=\"kube-state-metrics\"})\n  and\n(kube_hpa_status_current_replicas{job=\"kube-state-metrics\"}\n  >\nkube_hpa_spec_min_replicas{job=\"kube-state-metrics\"})\n  and\n(kube_hpa_status_current_replicas{job=\"kube-state-metrics\"}\n  <\nkube_hpa_spec_max_replicas{job=\"kube-state-metrics\"})\n  and\nchanges(kube_hpa_status_current_replicas[15m]) == 0\n",
+              "expr": "(kube_horizontalpodautoscaler_status_desired_replicas{job=\"kube-state-metrics\"}\n  !=\nkube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"})\n  and\n(kube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"}\n  >\nkube_horizontalpodautoscaler_spec_min_replicas{job=\"kube-state-metrics\"})\n  and\n(kube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"}\n  <\nkube_horizontalpodautoscaler_spec_max_replicas{job=\"kube-state-metrics\"})\n  and\nchanges(kube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"}[15m]) == 0\n",
               "for": "15m",
               "labels": {
                 "severity": "warning"
@@ -834,11 +760,11 @@ data:
             {
               "alert": "KubeHpaMaxedOut",
               "annotations": {
-                "description": "HPA {{ $labels.namespace }}/{{ $labels.hpa }} has been running at max replicas for longer than 15 minutes.",
+                "description": "HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has been running at max replicas for longer than 15 minutes.",
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubehpamaxedout",
                 "summary": "HPA is running at max replicas"
               },
-              "expr": "kube_hpa_status_current_replicas{job=\"kube-state-metrics\"}\n  ==\nkube_hpa_spec_max_replicas{job=\"kube-state-metrics\"}\n",
+              "expr": "kube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"}\n  ==\nkube_horizontalpodautoscaler_spec_max_replicas{job=\"kube-state-metrics\"}\n",
               "for": "15m",
               "labels": {
                 "severity": "warning"
@@ -852,12 +778,12 @@ data:
             {
               "alert": "KubeCPUOvercommit",
               "annotations": {
-                "description": "Cluster has overcommitted CPU resource requests for Pods and cannot tolerate node failure.",
+                "description": "Cluster has overcommitted CPU resource requests for Pods by {{ $value }} CPU shares and cannot tolerate node failure.",
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit",
                 "summary": "Cluster has overcommitted CPU resource requests."
               },
-              "expr": "sum(namespace:kube_pod_container_resource_requests_cpu_cores:sum{})\n  /\nsum(kube_node_status_allocatable_cpu_cores)\n  >\n(count(kube_node_status_allocatable_cpu_cores)-1) / count(kube_node_status_allocatable_cpu_cores)\n",
+              "expr": "sum(namespace_cpu:kube_pod_container_resource_requests:sum{}) - (sum(kube_node_status_allocatable{resource=\"cpu\"}) - max(kube_node_status_allocatable{resource=\"cpu\"})) > 0\nand\n(sum(kube_node_status_allocatable{resource=\"cpu\"}) - max(kube_node_status_allocatable{resource=\"cpu\"})) > 0\n",
-              "for": "5m",
+              "for": "10m",
               "labels": {
                 "severity": "warning"
               }
@@ -865,12 +791,12 @@ data:
             {
               "alert": "KubeMemoryOvercommit",
               "annotations": {
-                "description": "Cluster has overcommitted memory resource requests for Pods and cannot tolerate node failure.",
+                "description": "Cluster has overcommitted memory resource requests for Pods by {{ $value | humanize }} bytes and cannot tolerate node failure.",
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememoryovercommit",
                 "summary": "Cluster has overcommitted memory resource requests."
               },
-              "expr": "sum(namespace:kube_pod_container_resource_requests_memory_bytes:sum{})\n  /\nsum(kube_node_status_allocatable_memory_bytes)\n  >\n(count(kube_node_status_allocatable_memory_bytes)-1)\n  /\ncount(kube_node_status_allocatable_memory_bytes)\n",
+              "expr": "sum(namespace_memory:kube_pod_container_resource_requests:sum{}) - (sum(kube_node_status_allocatable{resource=\"memory\"}) - max(kube_node_status_allocatable{resource=\"memory\"})) > 0\nand\n(sum(kube_node_status_allocatable{resource=\"memory\"}) - max(kube_node_status_allocatable{resource=\"memory\"})) > 0\n",
-              "for": "5m",
+              "for": "10m",
               "labels": {
                 "severity": "warning"
               }
@@ -882,7 +808,7 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuquotaovercommit",
                 "summary": "Cluster has overcommitted CPU resource requests."
               },
-              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"cpu\"})\n  /\nsum(kube_node_status_allocatable_cpu_cores)\n  > 1.5\n",
+              "expr": "sum(min without(resource) (kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=~\"(cpu|requests.cpu)\"}))\n  /\nsum(kube_node_status_allocatable{resource=\"cpu\", job=\"kube-state-metrics\"})\n  > 1.5\n",
               "for": "5m",
               "labels": {
                 "severity": "warning"
@@ -895,7 +821,7 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememoryquotaovercommit",
                 "summary": "Cluster has overcommitted memory resource requests."
               },
-              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"memory\"})\n  /\nsum(kube_node_status_allocatable_memory_bytes{job=\"kube-state-metrics\"})\n  > 1.5\n",
+              "expr": "sum(min without(resource) (kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=~\"(memory|requests.memory)\"}))\n  /\nsum(kube_node_status_allocatable{resource=\"memory\", job=\"kube-state-metrics\"})\n  > 1.5\n",
               "for": "5m",
               "labels": {
                 "severity": "warning"
@@ -965,7 +891,7 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumefillingup",
                 "summary": "PersistentVolume is filling up."
               },
-              "expr": "kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n  /\nkubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n  < 0.03\n",
+              "expr": "(\n  kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n    /\n  kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n) < 0.03\nand\nkubelet_volume_stats_used_bytes{job=\"kubelet\"} > 0\nunless on(namespace, persistentvolumeclaim)\nkube_persistentvolumeclaim_access_mode{ access_mode=\"ReadOnlyMany\"} == 1\nunless on(namespace, persistentvolumeclaim)\nkube_persistentvolumeclaim_labels{label_excluded_from_alerts=\"true\"} == 1\n",
               "for": "1m",
               "labels": {
                 "severity": "critical"
@@ -978,7 +904,7 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumefillingup",
                 "summary": "PersistentVolume is filling up."
               },
-              "expr": "(\n  kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n    /\n  kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n) < 0.15\nand\npredict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\"}[6h], 4 * 24 * 3600) < 0\n",
+              "expr": "(\n  kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n    /\n  kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n) < 0.15\nand\nkubelet_volume_stats_used_bytes{job=\"kubelet\"} > 0\nand\npredict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\"}[6h], 4 * 24 * 3600) < 0\nunless on(namespace, persistentvolumeclaim)\nkube_persistentvolumeclaim_access_mode{ access_mode=\"ReadOnlyMany\"} == 1\nunless on(namespace, persistentvolumeclaim)\nkube_persistentvolumeclaim_labels{label_excluded_from_alerts=\"true\"} == 1\n",
               "for": "1h",
               "labels": {
                 "severity": "warning"
@@ -1009,7 +935,7 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeversionmismatch",
                 "summary": "Different semantic versions of Kubernetes components running."
               },
-              "expr": "count(count by (gitVersion) (label_replace(kubernetes_build_info{job!~\"kube-dns|coredns\"},\"gitVersion\",\"$1\",\"gitVersion\",\"(v[0-9]*.[0-9]*).*\"))) > 1\n",
+              "expr": "count(count by (git_version) (label_replace(kubernetes_build_info{job!~\"kube-dns|coredns\"},\"git_version\",\"$1\",\"git_version\",\"(v[0-9]*.[0-9]*).*\"))) > 1\n",
               "for": "15m",
               "labels": {
                 "severity": "warning"
@@ -1022,7 +948,7 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclienterrors",
                 "summary": "Kubernetes API server client is experiencing errors."
               },
-              "expr": "(sum(rate(rest_client_requests_total{code=~\"5..\"}[5m])) by (instance, job)\n  /\nsum(rate(rest_client_requests_total[5m])) by (instance, job))\n> 0.01\n",
+              "expr": "(sum(rate(rest_client_requests_total{code=~\"5..\"}[5m])) by (instance, job, namespace)\n  /\nsum(rate(rest_client_requests_total[5m])) by (instance, job, namespace))\n> 0.01\n",
               "for": "15m",
               "labels": {
                 "severity": "warning"
@@ -1101,7 +1027,7 @@ data:
             {
               "alert": "KubeClientCertificateExpiration",
               "annotations": {
-                "description": "A client certificate used to authenticate to the apiserver is expiring in less than 1.0 hours.",
+                "description": "A client certificate used to authenticate to kubernetes apiserver is expiring in less than 1.0 hours.",
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration",
                 "summary": "Client certificate is about to expire."
               },
@@ -1113,7 +1039,7 @@ data:
             {
               "alert": "KubeClientCertificateExpiration",
               "annotations": {
-                "description": "A client certificate used to authenticate to the apiserver is expiring in less than 0.1 hours.",
+                "description": "A client certificate used to authenticate to kubernetes apiserver is expiring in less than 0.1 hours.",
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration",
                 "summary": "Client certificate is about to expire."
               },
@@ -1123,23 +1049,23 @@ data:
               }
             },
             {
-              "alert": "AggregatedAPIErrors",
+              "alert": "KubeAggregatedAPIErrors",
               "annotations": {
-                "description": "An aggregated API {{ $labels.name }}/{{ $labels.namespace }} has reported errors. It has appeared unavailable {{ $value | humanize }} times averaged over the past 10m.",
+                "description": "Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has reported errors. It has appeared unavailable {{ $value | humanize }} times averaged over the past 10m.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-aggregatedapierrors",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeaggregatedapierrors",
-                "summary": "An aggregated API has reported errors."
+                "summary": "Kubernetes aggregated API has reported errors."
               },
-              "expr": "sum by(name, namespace)(increase(aggregator_unavailable_apiservice_count[10m])) > 4\n",
+              "expr": "sum by(name, namespace)(increase(aggregator_unavailable_apiservice_total[10m])) > 4\n",
               "labels": {
                 "severity": "warning"
               }
             },
             {
-              "alert": "AggregatedAPIDown",
+              "alert": "KubeAggregatedAPIDown",
               "annotations": {
-                "description": "An aggregated API {{ $labels.name }}/{{ $labels.namespace }} has been only {{ $value | humanize }}% available over the last 10m.",
+                "description": "Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has been only {{ $value | humanize }}% available over the last 10m.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-aggregatedapidown",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeaggregatedapidown",
-                "summary": "An aggregated API is down."
+                "summary": "Kubernetes aggregated API is down."
               },
               "expr": "(1 - max by(name, namespace)(avg_over_time(aggregator_unavailable_apiservice[10m]))) * 100 < 85\n",
               "for": "5m",
@@ -1159,6 +1085,19 @@ data:
               "labels": {
                 "severity": "critical"
               }
+            },
+            {
+              "alert": "KubeAPITerminatedRequests",
+              "annotations": {
+                "description": "The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapiterminatedrequests",
+                "summary": "The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests."
+              },
+              "expr": "sum(rate(apiserver_request_terminations_total{job=\"apiserver\"}[10m]))  / (  sum(rate(apiserver_request_total{job=\"apiserver\"}[10m])) + sum(rate(apiserver_request_terminations_total{job=\"apiserver\"}[10m])) ) > 0.20\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
             }
           ]
         },
@@ -1198,10 +1137,10 @@ data:
                 "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods",
                 "summary": "Kubelet is running at capacity."
               },
-              "expr": "count by(node) (\n  (kube_pod_status_phase{job=\"kube-state-metrics\",phase=\"Running\"} == 1) * on(instance,pod,namespace,cluster) group_left(node) topk by(instance,pod,namespace,cluster) (1, kube_pod_info{job=\"kube-state-metrics\"})\n)\n/\nmax by(node) (\n  kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1\n) > 0.95\n",
+              "expr": "count by(node) (\n  (kube_pod_status_phase{job=\"kube-state-metrics\",phase=\"Running\"} == 1) * on(instance,pod,namespace,cluster) group_left(node) topk by(instance,pod,namespace,cluster) (1, kube_pod_info{job=\"kube-state-metrics\"})\n)\n/\nmax by(node) (\n  kube_node_status_capacity{job=\"kube-state-metrics\",resource=\"pods\"} != 1\n) > 0.95\n",
               "for": "15m",
               "labels": {
-                "severity": "warning"
+                "severity": "info"
               }
             },
             {
@@ -1367,6 +1306,24 @@ data:
               }
             }
           ]
+        },
+        {
+          "name": "kubernetes-system-kube-proxy",
+          "rules": [
+            {
+              "alert": "KubeProxyDown",
+              "annotations": {
+                "description": "KubeProxy has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeproxydown",
+                "summary": "Target disappeared from Prometheus target discovery."
+              },
+              "expr": "absent(up{job=\"kube-proxy\"} == 1)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
         }
       ]
     }
@@ -1377,48 +1334,48 @@ data:
           "name": "node-exporter.rules",
           "rules": [
             {
-              "expr": "count without (cpu) (\n  count without (mode) (\n    node_cpu_seconds_total{job=\"node-exporter\"}\n  )\n)\n",
+              "expr": "count without (cpu, mode) (\n  node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}\n)\n",
               "record": "instance:node_num_cpu:sum"
             },
             {
-              "expr": "1 - avg without (cpu, mode) (\n  rate(node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\"}[1m])\n)\n",
+              "expr": "1 - avg without (cpu) (\n  sum without (mode) (rate(node_cpu_seconds_total{job=\"node-exporter\", mode=~\"idle|iowait|steal\"}[5m]))\n)\n",
-              "record": "instance:node_cpu_utilisation:rate1m"
+              "record": "instance:node_cpu_utilisation:rate5m"
             },
             {
               "expr": "(\n  node_load1{job=\"node-exporter\"}\n/\n  instance:node_num_cpu:sum{job=\"node-exporter\"}\n)\n",
               "record": "instance:node_load1_per_cpu:ratio"
             },
             {
-              "expr": "1 - (\n  node_memory_MemAvailable_bytes{job=\"node-exporter\"}\n/\n  node_memory_MemTotal_bytes{job=\"node-exporter\"}\n)\n",
+              "expr": "1 - (\n  (\n    node_memory_MemAvailable_bytes{job=\"node-exporter\"}\n    or\n    (\n      node_memory_Buffers_bytes{job=\"node-exporter\"}\n      +\n      node_memory_Cached_bytes{job=\"node-exporter\"}\n      +\n      node_memory_MemFree_bytes{job=\"node-exporter\"}\n      +\n      node_memory_Slab_bytes{job=\"node-exporter\"}\n    )\n  )\n/\n  node_memory_MemTotal_bytes{job=\"node-exporter\"}\n)\n",
               "record": "instance:node_memory_utilisation:ratio"
             },
             {
-              "expr": "rate(node_vmstat_pgmajfault{job=\"node-exporter\"}[1m])\n",
+              "expr": "rate(node_vmstat_pgmajfault{job=\"node-exporter\"}[5m])\n",
-              "record": "instance:node_vmstat_pgmajfault:rate1m"
+              "record": "instance:node_vmstat_pgmajfault:rate5m"
             },
             {
-              "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[1m])\n",
+              "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[5m])\n",
-              "record": "instance_device:node_disk_io_time_seconds:rate1m"
+              "record": "instance_device:node_disk_io_time_seconds:rate5m"
             },
             {
-              "expr": "rate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[1m])\n",
+              "expr": "rate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[5m])\n",
-              "record": "instance_device:node_disk_io_time_weighted_seconds:rate1m"
+              "record": "instance_device:node_disk_io_time_weighted_seconds:rate5m"
             },
             {
-              "expr": "sum without (device) (\n  rate(node_network_receive_bytes_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
+              "expr": "sum without (device) (\n  rate(node_network_receive_bytes_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n",
-              "record": "instance:node_network_receive_bytes_excluding_lo:rate1m"
+              "record": "instance:node_network_receive_bytes_excluding_lo:rate5m"
             },
             {
-              "expr": "sum without (device) (\n  rate(node_network_transmit_bytes_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
+              "expr": "sum without (device) (\n  rate(node_network_transmit_bytes_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n",
-              "record": "instance:node_network_transmit_bytes_excluding_lo:rate1m"
+              "record": "instance:node_network_transmit_bytes_excluding_lo:rate5m"
             },
             {
-              "expr": "sum without (device) (\n  rate(node_network_receive_drop_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
+              "expr": "sum without (device) (\n  rate(node_network_receive_drop_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n",
-              "record": "instance:node_network_receive_drop_excluding_lo:rate1m"
+              "record": "instance:node_network_receive_drop_excluding_lo:rate5m"
             },
             {
-              "expr": "sum without (device) (\n  rate(node_network_transmit_drop_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
+              "expr": "sum without (device) (\n  rate(node_network_transmit_drop_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n",
-              "record": "instance:node_network_transmit_drop_excluding_lo:rate1m"
+              "record": "instance:node_network_transmit_drop_excluding_lo:rate5m"
             }
           ]
         },
@@ -1456,7 +1413,7 @@ data:
                 "summary": "Filesystem has less than 5% space left."
               },
               "expr": "(\n  node_filesystem_avail_bytes{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} / node_filesystem_size_bytes{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} * 100 < 5\nand\n  node_filesystem_readonly{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} == 0\n)\n",
-              "for": "1h",
+              "for": "30m",
               "labels": {
                 "severity": "warning"
               }
@@ -1468,7 +1425,7 @@ data:
                 "summary": "Filesystem has less than 3% space left."
               },
               "expr": "(\n  node_filesystem_avail_bytes{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} / node_filesystem_size_bytes{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} * 100 < 3\nand\n  node_filesystem_readonly{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} == 0\n)\n",
-              "for": "1h",
+              "for": "30m",
               "labels": {
                 "severity": "critical"
               }
@@ -1570,7 +1527,7 @@ data:
             {
               "alert": "NodeClockSkewDetected",
               "annotations": {
-                "message": "Clock on {{ $labels.instance }} is out of sync by more than 300s. Ensure NTP is configured correctly on this host.",
+                "description": "Clock on {{ $labels.instance }} is out of sync by more than 300s. Ensure NTP is configured correctly on this host.",
                 "summary": "Clock skew detected."
               },
               "expr": "(\n  node_timex_offset_seconds > 0.05\nand\n  deriv(node_timex_offset_seconds[5m]) >= 0\n)\nor\n(\n  node_timex_offset_seconds < -0.05\nand\n  deriv(node_timex_offset_seconds[5m]) <= 0\n)\n",
@@ -1582,7 +1539,7 @@ data:
             {
               "alert": "NodeClockNotSynchronising",
               "annotations": {
-                "message": "Clock on {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.",
+                "description": "Clock on {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.",
                 "summary": "Clock not synchronising."
               },
               "expr": "min_over_time(node_timex_sync_status[5m]) == 0\nand\nnode_timex_maxerror_seconds >= 16\n",
@@ -1609,10 +1566,34 @@ data:
                 "description": "At least one device in RAID array on {{ $labels.instance }} failed. Array '{{ $labels.device }}' needs attention and possibly a disk swap.",
                 "summary": "Failed device in RAID array"
               },
-              "expr": "node_md_disks{state=\"fail\"} > 0\n",
+              "expr": "node_md_disks{state=\"failed\"} > 0\n",
               "labels": {
                 "severity": "warning"
               }
+            },
+            {
+              "alert": "NodeFileDescriptorLimit",
+              "annotations": {
+                "description": "File descriptors limit at {{ $labels.instance }} is currently at {{ printf \"%.2f\" $value }}%.",
+                "summary": "Kernel is predicted to exhaust file descriptors limit soon."
+              },
+              "expr": "(\n  node_filefd_allocated{job=\"node-exporter\"} * 100 / node_filefd_maximum{job=\"node-exporter\"} > 70\n)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "NodeFileDescriptorLimit",
+              "annotations": {
+                "description": "File descriptors limit at {{ $labels.instance }} is currently at {{ printf \"%.2f\" $value }}%.",
+                "summary": "Kernel is predicted to exhaust file descriptors limit soon."
+              },
+              "expr": "(\n  node_filefd_allocated{job=\"node-exporter\"} * 100 / node_filefd_maximum{job=\"node-exporter\"} > 90\n)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
             }
           ]
         }
@@ -1738,7 +1719,7 @@ data:
                 "description": "Prometheus {{$labels.instance}} failed to send {{ printf \"%.1f\" $value }}% of the samples to {{ $labels.remote_name}}:{{ $labels.url }}",
                 "summary": "Prometheus fails to send samples to remote storage."
               },
-              "expr": "(\n  rate(prometheus_remote_storage_failed_samples_total{job=\"prometheus\"}[5m])\n/\n  (\n    rate(prometheus_remote_storage_failed_samples_total{job=\"prometheus\"}[5m])\n  +\n    rate(prometheus_remote_storage_succeeded_samples_total{job=\"prometheus\"}[5m])\n  )\n)\n* 100\n> 1\n",
+              "expr": "(\n  (rate(prometheus_remote_storage_failed_samples_total{job=\"prometheus\"}[5m]) or rate(prometheus_remote_storage_samples_failed_total{job=\"prometheus\"}[5m]))\n/\n  (\n    (rate(prometheus_remote_storage_failed_samples_total{job=\"prometheus\"}[5m]) or rate(prometheus_remote_storage_samples_failed_total{job=\"prometheus\"}[5m]))\n  +\n    (rate(prometheus_remote_storage_succeeded_samples_total{job=\"prometheus\"}[5m]) or rate(prometheus_remote_storage_samples_total{job=\"prometheus\"}[5m]))\n  )\n)\n* 100\n> 1\n",
               "for": "15m",
               "labels": {
                 "severity": "critical"
@@ -1804,6 +1785,30 @@ data:
                 "severity": "warning"
               }
             },
+            {
+              "alert": "PrometheusLabelLimitHit",
+              "annotations": {
+                "description": "Prometheus {{$labels.instance}} has dropped {{ printf \"%.0f\" $value }} targets because some samples exceeded the configured label_limit, label_name_length_limit or label_value_length_limit.",
+                "summary": "Prometheus has dropped targets because some scrape configs have exceeded the labels limit."
+              },
+              "expr": "increase(prometheus_target_scrape_pool_exceeded_label_limits_total{job=\"prometheus\"}[5m]) > 0\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTargetSyncFailure",
+              "annotations": {
+                "description": "{{ printf \"%.0f\" $value }} targets in Prometheus {{$labels.instance}} have failed to sync because invalid configuration was supplied.",
+                "summary": "Prometheus has failed to sync targets."
+              },
+              "expr": "increase(prometheus_target_sync_failed_total{job=\"prometheus\"}[30m]) > 0\n",
+              "for": "5m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
             {
               "alert": "PrometheusErrorSendingAlertsToAnyAlertmanager",
               "annotations": {

aws/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.23.2 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/fedora-coreos/kubernetes/ami.tf

@@ -19,7 +19,7 @@ data "aws_ami" "fedora-coreos" {
 }
 
 data "aws_ami" "fedora-coreos-arm" {
-  count = var.arch == "arm64" ? 1 : 0
+  count = var.controller_arch == "arm64" ? 1 : 0
 
   most_recent = true
   owners      = ["125523088429"]

aws/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -9,9 +9,7 @@ module "bootstrap" {
   network_mtu           = var.network_mtu
   pod_cidr              = var.pod_cidr
   service_cidr          = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
+  components            = var.components
 }
 

aws/fedora-coreos/kubernetes/butane/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.4.0
+version: 1.5.0
 systemd:
   units:
     - name: etcd-member.service
@@ -9,15 +9,16 @@ systemd:
         [Unit]
         Description=etcd (System Container)
         Documentation=https://github.com/etcd-io/etcd
-        Wants=network-online.target network.target
+        Wants=network-online.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.13
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
         ExecStart=/usr/bin/podman run --name etcd \
           --env-file /etc/etcd/etcd.env \
+          --log-driver k8s-file \
           --network host \
           --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
           --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
@@ -56,7 +57,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -66,15 +67,19 @@ systemd:
         ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
         ExecStartPre=-/usr/bin/podman rm kubelet
         ExecStart=/usr/bin/podman run --name kubelet \
+          --log-driver k8s-file \
           --privileged \
           --pid host \
           --network host \
           --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /etc/machine-id:/etc/machine-id:ro \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
           --volume /sys/fs/cgroup:/sys/fs/cgroup \
+          --volume /etc/selinux:/etc/selinux \
+          --volume /sys/fs/selinux:/sys/fs/selinux \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/containerd:/var/lib/containerd \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -82,28 +87,13 @@ systemd:
           --volume /var/run/lock:/var/run/lock:z \
           --volume /opt/cni/bin:/opt/cni/bin:z \
           $${KUBELET_IMAGE} \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=systemd \
+          --config=/etc/kubernetes/kubelet.yaml \
-          --cgroups-per-qos=true \
-          --container-runtime=remote \
           --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
-          --enforce-node-allocatable=pods \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
           --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
-          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule
-          --resolv-conf=/run/systemd/resolve/resolv.conf \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/podman stop kubelet
         Delegate=yes
         Restart=always
@@ -126,7 +116,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.23.2
+            quay.io/poseidon/kubelet:v1.31.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -141,12 +131,39 @@ storage:
       contents:
         inline: |
           ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: cluster.local
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
     - path: /opt/bootstrap/layout
       mode: 0544
       contents:
         inline: |
           #!/bin/bash -e
-          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          mkdir -p -- auth tls/{etcd,k8s} static-manifests manifests/{coredns,kube-proxy,network}
           awk '/#####/ {filename=$2; next} {print > filename}' assets
           mkdir -p /etc/ssl/etcd/etcd
           mkdir -p /etc/kubernetes/pki
@@ -160,8 +177,7 @@ storage:
           mv static-manifests/* /etc/kubernetes/manifests/
           mkdir -p /opt/bootstrap/assets
           mv manifests /opt/bootstrap/assets/manifests
-          mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          rm -rf assets auth static-manifests tls manifests
-          rm -rf assets auth static-manifests tls manifests-networking
           chcon -R -u system_u -t container_file_t /etc/kubernetes/pki
     - path: /opt/bootstrap/apply
       mode: 0544
@@ -177,6 +193,11 @@ storage:
              echo "Retry applying manifests"
              sleep 5
           done
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
     - path: /etc/sysctl.d/max-user-watches.conf
       contents:
         inline: |
@@ -221,7 +242,6 @@ storage:
           ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
-    - path: /etc/fedora-coreos/iptables-legacy.stamp
     - path: /etc/containerd/config.toml
       overwrite: true
       contents:

aws/fedora-coreos/kubernetes/controllers.tf

@@ -20,17 +20,18 @@ resource "aws_instance" "controllers" {
   tags = {
     Name = "${var.cluster_name}-controller-${count.index}"
   }
-
   instance_type = var.controller_type
-  ami           = var.arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
+  ami           = var.controller_arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
-  user_data     = data.ct_config.controller-ignitions.*.rendered[count.index]
 
   # storage
   root_block_device {
-    volume_type = var.disk_type
+    volume_type = var.controller_disk_type
-    volume_size = var.disk_size
+    volume_size = var.controller_disk_size
-    iops        = var.disk_iops
+    iops        = var.controller_disk_iops
     encrypted   = true
+    tags = {
+      Name = "${var.cluster_name}-controller-${count.index}"
+    }
   }
 
   # network
@@ -38,6 +39,14 @@ resource "aws_instance" "controllers" {
   subnet_id                   = element(aws_subnet.public.*.id, count.index)
   vpc_security_group_ids      = [aws_security_group.controller.id]
 
+  # boot
+  user_data = data.ct_config.controllers.*.rendered[count.index]
+
+  # cost
+  credit_specification {
+    cpu_credits = var.controller_cpu_credits
+  }
+
   lifecycle {
     ignore_changes = [
       ami,
@@ -46,41 +55,21 @@ resource "aws_instance" "controllers" {
   }
 }
 
-# Controller Ignition configs
+# Fedora CoreOS controllers
-data "ct_config" "controller-ignitions" {
+data "ct_config" "controllers" {
   count = var.controller_count
-  content  = data.template_file.controller-configs.*.rendered[count.index]
+  content = templatefile("${path.module}/butane/controller.yaml", {
-  strict   = true
-  snippets = var.controller_snippets
-}
-
-# Controller Fedora CoreOS configs
-data "template_file" "controller-configs" {
-  count = var.controller_count
-
-  template = file("${path.module}/fcc/controller.yaml")
-
-  vars = {
     # Cannot use cyclic dependencies on controllers or their DNS records
     etcd_name   = "etcd${count.index}"
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    etcd_initial_cluster = join(",", [
+      for i in range(var.controller_count) : "etcd${i}=https://${var.cluster_name}-etcd${i}.${var.dns_zone}:2380"
+    ])
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
+  })
+  strict   = true
+  snippets = var.controller_snippets
 }
-}
-
-data "template_file" "etcds" {
-  count    = var.controller_count
-  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
-
-  vars = {
-    index        = count.index
-    cluster_name = var.cluster_name
-    dns_zone     = var.dns_zone
-  }
-}
-

aws/fedora-coreos/kubernetes/network.tf

@@ -47,17 +47,25 @@ resource "aws_route" "egress-ipv6" {
 resource "aws_subnet" "public" {
   count = length(data.aws_availability_zones.all.names)
 
-  vpc_id            = aws_vpc.network.id
-  availability_zone = data.aws_availability_zones.all.names[count.index]
-
-  cidr_block                      = cidrsubnet(var.host_cidr, 4, count.index)
-  ipv6_cidr_block                 = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
-  map_public_ip_on_launch         = true
-  assign_ipv6_address_on_creation = true
-
   tags = {
     "Name" = "${var.cluster_name}-public-${count.index}"
   }
+  vpc_id            = aws_vpc.network.id
+  availability_zone = data.aws_availability_zones.all.names[count.index]
+
+  # IPv4 and IPv6 CIDR blocks
+  cidr_block      = cidrsubnet(var.host_cidr, 4, count.index)
+  ipv6_cidr_block = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
+
+  # Assign IPv4 and IPv6 addresses to instances
+  map_public_ip_on_launch         = true
+  assign_ipv6_address_on_creation = true
+
+  # Hostnames assigned to instances
+  # resource-name: <ec2-instance-id>.region.compute.internal
+  private_dns_hostname_type_on_launch            = "resource-name"
+  enable_resource_name_dns_a_record_on_launch    = true
+  enable_resource_name_dns_aaaa_record_on_launch = true
 }
 
 resource "aws_route_table_association" "public" {

aws/fedora-coreos/kubernetes/security.tf

@@ -92,6 +92,30 @@ resource "aws_security_group_rule" "controller-cilium-health-self" {
   self      = true
 }
 
+resource "aws_security_group_rule" "controller-cilium-metrics" {
+  count = var.networking == "cilium" ? 1 : 0
+
+  security_group_id = aws_security_group.controller.id
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 9962
+  to_port                  = 9965
+  source_security_group_id = aws_security_group.worker.id
+}
+
+resource "aws_security_group_rule" "controller-cilium-metrics-self" {
+  count = var.networking == "cilium" ? 1 : 0
+
+  security_group_id = aws_security_group.controller.id
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 9962
+  to_port   = 9965
+  self      = true
+}
+
 # IANA VXLAN default
 resource "aws_security_group_rule" "controller-vxlan" {
   count = var.networking == "flannel" ? 1 : 0
@@ -379,6 +403,30 @@ resource "aws_security_group_rule" "worker-cilium-health-self" {
   self      = true
 }
 
+resource "aws_security_group_rule" "worker-cilium-metrics" {
+  count = var.networking == "cilium" ? 1 : 0
+
+  security_group_id = aws_security_group.worker.id
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 9962
+  to_port                  = 9965
+  source_security_group_id = aws_security_group.controller.id
+}
+
+resource "aws_security_group_rule" "worker-cilium-metrics-self" {
+  count = var.networking == "cilium" ? 1 : 0
+
+  security_group_id = aws_security_group.worker.id
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 9962
+  to_port   = 9965
+  self      = true
+}
+
 # IANA VXLAN default
 resource "aws_security_group_rule" "worker-vxlan" {
   count = var.networking == "flannel" ? 1 : 0

aws/fedora-coreos/kubernetes/variables.tf

@@ -17,30 +17,6 @@ variable "dns_zone_id" {
 
 # instances
 
-variable "controller_count" {
-  type        = number
-  description = "Number of controllers (i.e. masters)"
-  default     = 1
-}
-
-variable "worker_count" {
-  type        = number
-  description = "Number of workers"
-  default     = 1
-}
-
-variable "controller_type" {
-  type        = string
-  description = "EC2 instance type for controllers"
-  default     = "t3.small"
-}
-
-variable "worker_type" {
-  type        = string
-  description = "EC2 instance type for workers"
-  default     = "t3.small"
-}
-
 variable "os_stream" {
   type        = string
   description = "Fedora CoreOS image stream for instances (e.g. stable, testing, next)"
@@ -52,24 +28,78 @@ variable "os_stream" {
   }
 }
 
-variable "disk_size" {
+variable "controller_count" {
+  type        = number
+  description = "Number of controllers (i.e. masters)"
+  default     = 1
+}
+
+variable "controller_type" {
+  type        = string
+  description = "EC2 instance type for controllers"
+  default     = "t3.small"
+}
+
+variable "controller_disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
   default     = 30
 }
 
-variable "disk_type" {
+variable "controller_disk_type" {
   type        = string
   description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
   default     = "gp3"
 }
 
-variable "disk_iops" {
+variable "controller_disk_iops" {
   type        = number
   description = "IOPS of the EBS volume (e.g. 3000)"
   default     = 3000
 }
 
+variable "controller_cpu_credits" {
+  type        = string
+  description = "CPU credits mode (if using a burstable instance type)"
+  default     = null
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
+variable "worker_type" {
+  type        = string
+  description = "EC2 instance type for workers"
+  default     = "t3.small"
+}
+
+variable "worker_disk_size" {
+  type        = number
+  description = "Size of the EBS volume in GB"
+  default     = 30
+}
+
+variable "worker_disk_type" {
+  type        = string
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
+}
+
+variable "worker_disk_iops" {
+  type        = number
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
+}
+
+variable "worker_cpu_credits" {
+  type        = string
+  description = "CPU credits mode (if using a burstable instance type)"
+  default     = null
+}
+
 variable "worker_price" {
   type        = number
   description = "Spot price in USD for worker instances or 0 to use on-demand instances"
@@ -103,8 +133,8 @@ variable "ssh_authorized_key" {
 
 variable "networking" {
   type        = string
-  description = "Choice of networking provider (calico or flannel)"
+  description = "Choice of networking provider (flannel, calico, or cilium)"
-  default     = "calico"
+  default     = "cilium"
 }
 
 variable "network_mtu" {
@@ -134,40 +164,31 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "enable_reporting" {
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
 variable "worker_node_labels" {
   type        = list(string)
   description = "List of initial worker node labels"
   default     = []
 }
 
-# unofficial, undocumented, unsupported
+# advanced
 
-variable "cluster_domain_suffix" {
+variable "controller_arch" {
   type        = string
-  description = "Queries for domains with the suffix will be answered by CoreDNS. Default is cluster.local (e.g. foo.default.svc.cluster.local)"
+  description = "Controller node(s) architecture (amd64 or arm64)"
-  default     = "cluster.local"
+  default     = "amd64"
+  validation {
+    condition     = contains(["amd64", "arm64"], var.controller_arch)
+    error_message = "The controller_arch must be amd64 or arm64."
+  }
 }
 
-variable "arch" {
+variable "worker_arch" {
   type        = string
-  description = "Container architecture (amd64 or arm64)"
+  description = "Worker node(s) architecture (amd64 or arm64)"
   default     = "amd64"
-
   validation {
-    condition     = var.arch == "amd64" || var.arch == "arm64"
+    condition     = contains(["amd64", "arm64"], var.worker_arch)
-    error_message = "The arch must be amd64 or arm64."
+    error_message = "The worker_arch must be amd64 or arm64."
   }
 }
 
@@ -176,3 +197,19 @@ variable "daemonset_tolerations" {
   description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
   default     = []
 }
+
+variable "components" {
+  description = "Configure pre-installed cluster components"
+  # Component configs are passed through to terraform-render-bootstrap,
+  # which handles type enforcement and defines defaults
+  # https://github.com/poseidon/terraform-render-bootstrap/blob/main/variables.tf#L95
+  type = object({
+    enable     = optional(bool)
+    coredns    = optional(map(any))
+    kube_proxy = optional(map(any))
+    flannel    = optional(map(any))
+    calico     = optional(map(any))
+    cilium     = optional(map(any))
+  })
+  default = null
+}

aws/fedora-coreos/kubernetes/versions.tf

@@ -3,13 +3,11 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    aws      = ">= 2.23, <= 4.0"
+    aws  = ">= 2.23, <= 6.0"
-    template = "~> 2.2"
     null = ">= 2.1"
-
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.9"
+      version = "~> 0.13"
     }
   }
 }

aws/fedora-coreos/kubernetes/workers.tf

@@ -6,11 +6,16 @@ module "workers" {
   vpc_id          = aws_vpc.network.id
   subnet_ids      = aws_subnet.public.*.id
   security_groups = [aws_security_group.worker.id]
+
+  # instances
+  os_stream     = var.os_stream
   worker_count  = var.worker_count
   instance_type = var.worker_type
-  os_stream       = var.os_stream
+  arch          = var.worker_arch
-  arch            = var.arch
+  disk_type     = var.worker_disk_type
-  disk_size       = var.disk_size
+  disk_size     = var.worker_disk_size
+  disk_iops     = var.worker_disk_iops
+  cpu_credits   = var.worker_cpu_credits
   spot_price    = var.worker_price
   target_groups = var.worker_target_groups
 
@@ -18,7 +23,6 @@ module "workers" {
   kubeconfig         = module.bootstrap.kubeconfig-kubelet
   ssh_authorized_key = var.ssh_authorized_key
   service_cidr       = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
   snippets           = var.worker_snippets
   node_labels        = var.worker_node_labels
 }

aws/fedora-coreos/kubernetes/workers/ami.tf

@@ -1,3 +1,7 @@
+locals {
+  ami_id = var.arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
+}
+
 data "aws_ami" "fedora-coreos" {
   most_recent = true
   owners      = ["125523088429"]

aws/fedora-coreos/kubernetes/workers/butane/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.4.0
+version: 1.5.0
 systemd:
   units:
     - name: containerd.service
@@ -29,7 +29,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -39,15 +39,19 @@ systemd:
         ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
         ExecStartPre=-/usr/bin/podman rm kubelet
         ExecStart=/usr/bin/podman run --name kubelet \
+          --log-driver k8s-file \
           --privileged \
           --pid host \
           --network host \
           --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /etc/machine-id:/etc/machine-id:ro \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
           --volume /sys/fs/cgroup:/sys/fs/cgroup \
+          --volume /etc/selinux:/etc/selinux \
+          --volume /sys/fs/selinux:/sys/fs/selinux \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/containerd:/var/lib/containerd \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -55,19 +59,9 @@ systemd:
           --volume /var/run/lock:/var/run/lock:z \
           --volume /opt/cni/bin:/opt/cni/bin:z \
           $${KUBELET_IMAGE} \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=systemd \
+          --config=/etc/kubernetes/kubelet.yaml \
-          --cgroups-per-qos=true \
-          --container-runtime=remote \
           --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
-          --enforce-node-allocatable=pods \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --node-labels=node.kubernetes.io/node \
           %{~ for label in split(",", node_labels) ~}
@@ -76,31 +70,13 @@ systemd:
           %{~ for taint in split(",", node_taints) ~}
           --register-with-taints=${taint} \
           %{~ endfor ~}
-          --pod-manifest-path=/etc/kubernetes/manifests \
+          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID}
-          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
-          --read-only-port=0 \
-          --resolv-conf=/run/systemd/resolve/resolv.conf \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/podman stop kubelet
         Delegate=yes
         Restart=always
         RestartSec=10
         [Install]
         WantedBy=multi-user.target
-    - name: delete-node.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Delete Kubernetes node on shutdown
-        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /var/lib/kubelet:/var/lib/kubelet:ro,z --entrypoint /usr/local/bin/kubectl $${KUBELET_IMAGE} --kubeconfig=/var/lib/kubelet/kubeconfig delete node $HOSTNAME'
-        [Install]
-        WantedBy=multi-user.target
 storage:
   directories:
     - path: /etc/kubernetes
@@ -110,6 +86,38 @@ storage:
       contents:
         inline: |
           ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: cluster.local
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
     - path: /etc/sysctl.d/max-user-watches.conf
       contents:
         inline: |
@@ -133,7 +141,6 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
-    - path: /etc/fedora-coreos/iptables-legacy.stamp
     - path: /etc/containerd/config.toml
       overwrite: true
       contents:

aws/fedora-coreos/kubernetes/workers/variables.tf

@@ -69,6 +69,12 @@ variable "spot_price" {
   default     = 0
 }
 
+variable "cpu_credits" {
+  type        = string
+  description = "CPU burst credits mode (if applicable)"
+  default     = null
+}
+
 variable "target_groups" {
   type        = list(string)
   description = "Additional target group ARNs to which instances should be added"
@@ -102,12 +108,6 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
-
 variable "node_labels" {
   type        = list(string)
   description = "List of initial node labels"
@@ -120,15 +120,14 @@ variable "node_taints" {
   default     = []
 }
 
-# unofficial, undocumented, unsupported
+# advanced
 
 variable "arch" {
   type        = string
   description = "Container architecture (amd64 or arm64)"
   default     = "amd64"
-
   validation {
-    condition     = var.arch == "amd64" || var.arch == "arm64"
+    condition     = contains(["amd64", "arm64"], var.arch)
     error_message = "The arch must be amd64 or arm64."
   }
 }

aws/fedora-coreos/kubernetes/workers/versions.tf

@@ -3,12 +3,10 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    aws      = ">= 2.23, <= 4.0"
+    aws = ">= 2.23, <= 6.0"
-    template = "~> 2.2"
-
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.9"
+      version = "~> 0.13"
     }
   }
 }

aws/fedora-coreos/kubernetes/workers/workers.tf

@@ -1,19 +1,20 @@
 # Workers AutoScaling Group
 resource "aws_autoscaling_group" "workers" {
-  name = "${var.name}-worker ${aws_launch_configuration.worker.name}"
+  name = "${var.name}-worker"
 
   # count
   desired_capacity = var.worker_count
   min_size         = var.worker_count
   max_size         = var.worker_count + 2
-  default_cooldown          = 30
-  health_check_grace_period = 30
 
   # network
   vpc_zone_identifier = var.subnet_ids
 
-  # template
+  # instance template
-  launch_configuration = aws_launch_configuration.worker.name
+  launch_template {
+    id      = aws_launch_template.worker.id
+    version = aws_launch_template.worker.latest_version
+  }
 
   # target groups to which instances should be added
   target_group_arns = flatten([
@@ -22,6 +23,19 @@ resource "aws_autoscaling_group" "workers" {
     var.target_groups,
   ])
 
+  instance_refresh {
+    strategy = "Rolling"
+    preferences {
+      instance_warmup        = 120
+      min_healthy_percentage = 90
+    }
+  }
+  # Grace period before checking new instance's health
+  health_check_grace_period = 30
+  # Cooldown period between scaling activities
+  default_cooldown = 30
+
+
   lifecycle {
     # override the default destroy and replace update behavior
     create_before_destroy = true
@@ -33,34 +47,62 @@ resource "aws_autoscaling_group" "workers" {
   # used. Disable wait to avoid issues and align with other clouds.
   wait_for_capacity_timeout = "0"
 
-  tags = [
+  tag {
-    {
     key                 = "Name"
     value               = "${var.name}-worker"
     propagate_at_launch = true
-    },
+  }
-  ]
 }
 
 # Worker template
-resource "aws_launch_configuration" "worker" {
+resource "aws_launch_template" "worker" {
-  image_id          = var.arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
+  name_prefix   = "${var.name}-worker"
+  image_id      = local.ami_id
   instance_type = var.instance_type
-  spot_price        = var.spot_price > 0 ? var.spot_price : null
-  enable_monitoring = false
-
-  user_data = data.ct_config.worker-ignition.rendered
 
   # storage
-  root_block_device {
+  ebs_optimized = true
+  block_device_mappings {
+    device_name = "/dev/xvda"
+    ebs {
       volume_type           = var.disk_type
       volume_size           = var.disk_size
       iops                  = var.disk_iops
       encrypted             = true
+      delete_on_termination = true
+    }
   }
 
   # network
+  network_interfaces {
+    associate_public_ip_address = true
     security_groups             = var.security_groups
+  }
+
+  # boot
+  user_data = sensitive(base64encode(data.ct_config.worker.rendered))
+
+  # metadata
+  metadata_options {
+    http_tokens = "optional"
+  }
+  monitoring {
+    enabled = false
+  }
+
+  # cost
+  credit_specification {
+    cpu_credits = var.cpu_credits
+  }
+  dynamic "instance_market_options" {
+    for_each = var.spot_price > 0 ? [1] : []
+    content {
+      market_type = "spot"
+      spot_options {
+        max_price = var.spot_price
+      }
+    }
+  }
 
   lifecycle {
     // Override the default destroy and replace update behavior
@@ -69,24 +111,15 @@ resource "aws_launch_configuration" "worker" {
   }
 }
 
-# Worker Ignition config
+# Fedora CoreOS worker
-data "ct_config" "worker-ignition" {
+data "ct_config" "worker" {
-  content  = data.template_file.worker-config.rendered
+  content = templatefile("${path.module}/butane/worker.yaml", {
-  strict   = true
-  snippets = var.snippets
-}
-
-# Worker Fedora CoreOS config
-data "template_file" "worker-config" {
-  template = file("${path.module}/fcc/worker.yaml")
-
-  vars = {
     kubeconfig             = indent(10, var.kubeconfig)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
     node_taints            = join(",", var.node_taints)
+  })
+  strict   = true
+  snippets = var.snippets
 }
-}
-

aws/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.23.2 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/flatcar-linux/kubernetes/ami.tf

@@ -1,7 +1,7 @@
 locals {
   # Pick a Flatcar Linux AMI
   # flatcar-stable -> Flatcar Linux AMI
-  ami_id  = var.arch == "arm64" ? data.aws_ami.flatcar-arm64[0].image_id : data.aws_ami.flatcar.image_id
+  ami_id  = var.controller_arch == "arm64" ? data.aws_ami.flatcar-arm64[0].image_id : data.aws_ami.flatcar.image_id
   channel = split("-", var.os_image)[1]
 }
 
@@ -26,7 +26,7 @@ data "aws_ami" "flatcar" {
 }
 
 data "aws_ami" "flatcar-arm64" {
-  count = var.arch == "arm64" ? 1 : 0
+  count = var.controller_arch == "arm64" ? 1 : 0
 
   most_recent = true
   owners      = ["075585003325"]

aws/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -9,9 +9,7 @@ module "bootstrap" {
   network_mtu           = var.network_mtu
   pod_cidr              = var.pod_cidr
   service_cidr          = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
+  components            = var.components
 }
 

aws/flatcar-linux/kubernetes/butane/controller.yaml

@@ -1,4 +1,5 @@
----
+variant: flatcar
+version: 1.0.0
 systemd:
   units:
     - name: etcd-member.service
@@ -10,7 +11,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.13
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -57,7 +58,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -83,26 +84,13 @@ systemd:
           -v /var/log:/var/log \
           -v /opt/cni/bin:/opt/cni/bin \
           $${KUBELET_IMAGE} \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=systemd \
+          --config=/etc/kubernetes/kubelet.yaml \
-          --container-runtime=remote \
           --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
           --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
-          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule
-          --resolv-conf=/run/systemd/resolve/resolv.conf \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStart=docker logs -f kubelet
         ExecStop=docker stop kubelet
         ExecStopPost=docker rm kubelet
@@ -121,7 +109,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \
@@ -134,23 +122,47 @@ systemd:
 storage:
   directories:
     - path: /var/lib/etcd
-      filesystem: root
       mode: 0700
       overwrite: true
   files:
     - path: /etc/kubernetes/kubeconfig
-      filesystem: root
       mode: 0644
       contents:
         inline: |
           ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: cluster.local
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
     - path: /opt/bootstrap/layout
-      filesystem: root
       mode: 0544
       contents:
         inline: |
           #!/bin/bash -e
-          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          mkdir -p -- auth tls/{etcd,k8s} static-manifests manifests/{coredns,kube-proxy,network}
           awk '/#####/ {filename=$2; next} {print > filename}' assets
           mkdir -p /etc/ssl/etcd/etcd
           mkdir -p /etc/kubernetes/pki
@@ -165,10 +177,8 @@ storage:
           mv static-manifests/* /etc/kubernetes/manifests/
           mkdir -p /opt/bootstrap/assets
           mv manifests /opt/bootstrap/assets/manifests
-          mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          rm -rf assets auth static-manifests tls manifests
-          rm -rf assets auth static-manifests tls manifests-networking
     - path: /opt/bootstrap/apply
-      filesystem: root
       mode: 0544
       contents:
         inline: |
@@ -182,14 +192,17 @@ storage:
              echo "Retry applying manifests"
              sleep 5
           done
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
     - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
       mode: 0644
       contents:
         inline: |
           fs.inotify.max_user_watches=16184
     - path: /etc/etcd/etcd.env
-      filesystem: root
       mode: 0644
       contents:
           inline: |

aws/flatcar-linux/kubernetes/controllers.tf

@@ -20,18 +20,18 @@ resource "aws_instance" "controllers" {
   tags = {
     Name = "${var.cluster_name}-controller-${count.index}"
   }
-
   instance_type = var.controller_type
-
   ami           = local.ami_id
-  user_data = data.ct_config.controller-ignitions.*.rendered[count.index]
 
   # storage
   root_block_device {
-    volume_type = var.disk_type
+    volume_type = var.controller_disk_type
-    volume_size = var.disk_size
+    volume_size = var.controller_disk_size
-    iops        = var.disk_iops
+    iops        = var.controller_disk_iops
     encrypted   = true
+    tags = {
+      Name = "${var.cluster_name}-controller-${count.index}"
+    }
   }
 
   # network
@@ -39,6 +39,14 @@ resource "aws_instance" "controllers" {
   subnet_id                   = element(aws_subnet.public.*.id, count.index)
   vpc_security_group_ids      = [aws_security_group.controller.id]
 
+  # boot
+  user_data = data.ct_config.controllers.*.rendered[count.index]
+
+  # cost
+  credit_specification {
+    cpu_credits = var.controller_cpu_credits
+  }
+
   lifecycle {
     ignore_changes = [
       ami,
@@ -47,41 +55,21 @@ resource "aws_instance" "controllers" {
   }
 }
 
-# Controller Ignition configs
+# Flatcar Linux controllers
-data "ct_config" "controller-ignitions" {
+data "ct_config" "controllers" {
   count = var.controller_count
-  content  = data.template_file.controller-configs.*.rendered[count.index]
+  content = templatefile("${path.module}/butane/controller.yaml", {
-  strict   = true
-  snippets = var.controller_snippets
-}
-
-# Controller Container Linux configs
-data "template_file" "controller-configs" {
-  count = var.controller_count
-
-  template = file("${path.module}/cl/controller.yaml")
-
-  vars = {
     # Cannot use cyclic dependencies on controllers or their DNS records
     etcd_name   = "etcd${count.index}"
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    etcd_initial_cluster = join(",", [
+      for i in range(var.controller_count) : "etcd${i}=https://${var.cluster_name}-etcd${i}.${var.dns_zone}:2380"
+    ])
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
+  })
+  strict   = true
+  snippets = var.controller_snippets
 }
-}
-
-data "template_file" "etcds" {
-  count    = var.controller_count
-  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
-
-  vars = {
-    index        = count.index
-    cluster_name = var.cluster_name
-    dns_zone     = var.dns_zone
-  }
-}
-

aws/flatcar-linux/kubernetes/network.tf

@@ -47,17 +47,25 @@ resource "aws_route" "egress-ipv6" {
 resource "aws_subnet" "public" {
   count = length(data.aws_availability_zones.all.names)
 
-  vpc_id            = aws_vpc.network.id
-  availability_zone = data.aws_availability_zones.all.names[count.index]
-
-  cidr_block                      = cidrsubnet(var.host_cidr, 4, count.index)
-  ipv6_cidr_block                 = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
-  map_public_ip_on_launch         = true
-  assign_ipv6_address_on_creation = true
-
   tags = {
     "Name" = "${var.cluster_name}-public-${count.index}"
   }
+  vpc_id            = aws_vpc.network.id
+  availability_zone = data.aws_availability_zones.all.names[count.index]
+
+  # IPv4 and IPv6 CIDR blocks
+  cidr_block      = cidrsubnet(var.host_cidr, 4, count.index)
+  ipv6_cidr_block = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
+
+  # Assign IPv4 and IPv6 addresses to instances
+  map_public_ip_on_launch         = true
+  assign_ipv6_address_on_creation = true
+
+  # Hostnames assigned to instances
+  # resource-name: <ec2-instance-id>.region.compute.internal
+  private_dns_hostname_type_on_launch            = "resource-name"
+  enable_resource_name_dns_a_record_on_launch    = true
+  enable_resource_name_dns_aaaa_record_on_launch = true
 }
 
 resource "aws_route_table_association" "public" {

aws/flatcar-linux/kubernetes/security.tf

@@ -92,6 +92,30 @@ resource "aws_security_group_rule" "controller-cilium-health-self" {
   self      = true
 }
 
+resource "aws_security_group_rule" "controller-cilium-metrics" {
+  count = var.networking == "cilium" ? 1 : 0
+
+  security_group_id = aws_security_group.controller.id
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 9962
+  to_port                  = 9965
+  source_security_group_id = aws_security_group.worker.id
+}
+
+resource "aws_security_group_rule" "controller-cilium-metrics-self" {
+  count = var.networking == "cilium" ? 1 : 0
+
+  security_group_id = aws_security_group.controller.id
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 9962
+  to_port   = 9965
+  self      = true
+}
+
 # IANA VXLAN default
 resource "aws_security_group_rule" "controller-vxlan" {
   count = var.networking == "flannel" ? 1 : 0
@@ -379,6 +403,30 @@ resource "aws_security_group_rule" "worker-cilium-health-self" {
   self      = true
 }
 
+resource "aws_security_group_rule" "worker-cilium-metrics" {
+  count = var.networking == "cilium" ? 1 : 0
+
+  security_group_id = aws_security_group.worker.id
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 9962
+  to_port                  = 9965
+  source_security_group_id = aws_security_group.controller.id
+}
+
+resource "aws_security_group_rule" "worker-cilium-metrics-self" {
+  count = var.networking == "cilium" ? 1 : 0
+
+  security_group_id = aws_security_group.worker.id
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 9962
+  to_port   = 9965
+  self      = true
+}
+
 # IANA VXLAN default
 resource "aws_security_group_rule" "worker-vxlan" {
   count = var.networking == "flannel" ? 1 : 0

aws/flatcar-linux/kubernetes/variables.tf

@@ -17,30 +17,6 @@ variable "dns_zone_id" {
 
 # instances
 
-variable "controller_count" {
-  type        = number
-  description = "Number of controllers (i.e. masters)"
-  default     = 1
-}
-
-variable "worker_count" {
-  type        = number
-  description = "Number of workers"
-  default     = 1
-}
-
-variable "controller_type" {
-  type        = string
-  description = "EC2 instance type for controllers"
-  default     = "t3.small"
-}
-
-variable "worker_type" {
-  type        = string
-  description = "EC2 instance type for workers"
-  default     = "t3.small"
-}
-
 variable "os_image" {
   type        = string
   description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha)"
@@ -52,24 +28,78 @@ variable "os_image" {
   }
 }
 
-variable "disk_size" {
+variable "controller_count" {
+  type        = number
+  description = "Number of controllers (i.e. masters)"
+  default     = 1
+}
+
+variable "controller_type" {
+  type        = string
+  description = "EC2 instance type for controllers"
+  default     = "t3.small"
+}
+
+variable "controller_disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
   default     = 30
 }
 
-variable "disk_type" {
+variable "controller_disk_type" {
   type        = string
   description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
   default     = "gp3"
 }
 
-variable "disk_iops" {
+variable "controller_disk_iops" {
   type        = number
   description = "IOPS of the EBS volume (e.g. 3000)"
   default     = 3000
 }
 
+variable "controller_cpu_credits" {
+  type        = string
+  description = "CPU credits mode (if using a burstable instance type)"
+  default     = null
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
+variable "worker_type" {
+  type        = string
+  description = "EC2 instance type for workers"
+  default     = "t3.small"
+}
+
+variable "worker_disk_size" {
+  type        = number
+  description = "Size of the EBS volume in GB"
+  default     = 30
+}
+
+variable "worker_disk_type" {
+  type        = string
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
+}
+
+variable "worker_disk_iops" {
+  type        = number
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
+}
+
+variable "worker_cpu_credits" {
+  type        = string
+  description = "CPU credits mode (if using a burstable instance type)"
+  default     = null
+}
+
 variable "worker_price" {
   type        = number
   description = "Spot price in USD for worker instances or 0 to use on-demand instances"
@@ -103,8 +133,8 @@ variable "ssh_authorized_key" {
 
 variable "networking" {
   type        = string
-  description = "Choice of networking provider (calico or flannel)"
+  description = "Choice of networking provider (flannel, calico, or cilium)"
-  default     = "calico"
+  default     = "cilium"
 }
 
 variable "network_mtu" {
@@ -134,40 +164,31 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "enable_reporting" {
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
 variable "worker_node_labels" {
   type        = list(string)
   description = "List of initial worker node labels"
   default     = []
 }
 
-# unofficial, undocumented, unsupported
+# advanced
 
-variable "cluster_domain_suffix" {
+variable "controller_arch" {
   type        = string
-  description = "Queries for domains with the suffix will be answered by CoreDNS. Default is cluster.local (e.g. foo.default.svc.cluster.local)"
+  description = "Controller node(s) architecture (amd64 or arm64)"
-  default     = "cluster.local"
+  default     = "amd64"
+  validation {
+    condition     = contains(["amd64", "arm64"], var.controller_arch)
+    error_message = "The controller_arch must be amd64 or arm64."
+  }
 }
 
-variable "arch" {
+variable "worker_arch" {
   type        = string
-  description = "Container architecture (amd64 or arm64)"
+  description = "Worker node(s) architecture (amd64 or arm64)"
   default     = "amd64"
-
   validation {
-    condition     = var.arch == "amd64" || var.arch == "arm64"
+    condition     = contains(["amd64", "arm64"], var.worker_arch)
-    error_message = "The arch must be amd64 or arm64."
+    error_message = "The worker_arch must be amd64 or arm64."
   }
 }
 
@@ -176,3 +197,19 @@ variable "daemonset_tolerations" {
   description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
   default     = []
 }
+
+variable "components" {
+  description = "Configure pre-installed cluster components"
+  # Component configs are passed through to terraform-render-bootstrap,
+  # which handles type enforcement and defines defaults
+  # https://github.com/poseidon/terraform-render-bootstrap/blob/main/variables.tf#L95
+  type = object({
+    enable     = optional(bool)
+    coredns    = optional(map(any))
+    kube_proxy = optional(map(any))
+    flannel    = optional(map(any))
+    calico     = optional(map(any))
+    cilium     = optional(map(any))
+  })
+  default = null
+}

aws/flatcar-linux/kubernetes/versions.tf

@@ -3,13 +3,11 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    aws      = ">= 2.23, <= 4.0"
+    aws  = ">= 2.23, <= 6.0"
-    template = "~> 2.2"
     null = ">= 2.1"
-
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.9"
+      version = "~> 0.13"
     }
   }
 }

aws/flatcar-linux/kubernetes/workers.tf

@@ -6,11 +6,15 @@ module "workers" {
   vpc_id          = aws_vpc.network.id
   subnet_ids      = aws_subnet.public.*.id
   security_groups = [aws_security_group.worker.id]
+
+  # instances
+  os_image      = var.os_image
   worker_count  = var.worker_count
   instance_type = var.worker_type
-  os_image        = var.os_image
+  arch          = var.worker_arch
-  arch            = var.arch
+  disk_type     = var.worker_disk_type
-  disk_size       = var.disk_size
+  disk_size     = var.worker_disk_size
+  disk_iops     = var.worker_disk_iops
   spot_price    = var.worker_price
   target_groups = var.worker_target_groups
 
@@ -18,7 +22,6 @@ module "workers" {
   kubeconfig         = module.bootstrap.kubeconfig-kubelet
   ssh_authorized_key = var.ssh_authorized_key
   service_cidr       = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
   snippets           = var.worker_snippets
   node_labels        = var.worker_node_labels
 }

aws/flatcar-linux/kubernetes/workers/butane/worker.yaml

@@ -1,4 +1,5 @@
----
+variant: flatcar
+version: 1.0.0
 systemd:
   units:
     - name: docker.service
@@ -29,7 +30,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -58,17 +59,9 @@ systemd:
           -v /var/log:/var/log \
           -v /opt/cni/bin:/opt/cni/bin \
           $${KUBELET_IMAGE} \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=systemd \
+          --config=/etc/kubernetes/kubelet.yaml \
-          --container-runtime=remote \
           --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --node-labels=node.kubernetes.io/node \
           %{~ for label in split(",", node_labels) ~}
@@ -77,12 +70,7 @@ systemd:
           %{~ for taint in split(",", node_taints) ~}
           --register-with-taints=${taint} \
           %{~ endfor ~}
-          --pod-manifest-path=/etc/kubernetes/manifests \
+          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID}
-          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
-          --read-only-port=0 \
-          --resolv-conf=/run/systemd/resolve/resolv.conf \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStart=docker logs -f kubelet
         ExecStop=docker stop kubelet
         ExecStopPost=docker rm kubelet
@@ -90,29 +78,46 @@ systemd:
         RestartSec=5
         [Install]
         WantedBy=multi-user.target
-    - name: delete-node.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Delete Kubernetes node on shutdown
-        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/docker run -v /var/lib/kubelet:/var/lib/kubelet:ro --entrypoint /usr/local/bin/kubectl $${KUBELET_IMAGE} --kubeconfig=/var/lib/kubelet/kubeconfig delete node $HOSTNAME'
-        [Install]
-        WantedBy=multi-user.target
 storage:
   files:
     - path: /etc/kubernetes/kubeconfig
-      filesystem: root
       mode: 0644
       contents:
         inline: |
           ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: cluster.local
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
     - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
       mode: 0644
       contents:
         inline: |

aws/flatcar-linux/kubernetes/workers/variables.tf

@@ -69,6 +69,12 @@ variable "spot_price" {
   default     = 0
 }
 
+variable "cpu_credits" {
+  type        = string
+  description = "CPU burst credits mode (if applicable)"
+  default     = null
+}
+
 variable "target_groups" {
   type        = list(string)
   description = "Additional target group ARNs to which instances should be added"
@@ -102,12 +108,6 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
-
 variable "node_labels" {
   type        = list(string)
   description = "List of initial node labels"
@@ -128,7 +128,7 @@ variable "arch" {
   default     = "amd64"
 
   validation {
-    condition     = var.arch == "amd64" || var.arch == "arm64"
+    condition     = contains(["amd64", "arm64"], var.arch)
     error_message = "The arch must be amd64 or arm64."
   }
 }

aws/flatcar-linux/kubernetes/workers/versions.tf

@@ -3,12 +3,10 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    aws      = ">= 2.23, <= 4.0"
+    aws = ">= 2.23, <= 6.0"
-    template = "~> 2.2"
-
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.9"
+      version = "~> 0.13"
     }
   }
 }

aws/flatcar-linux/kubernetes/workers/workers.tf

@@ -1,19 +1,20 @@
 # Workers AutoScaling Group
 resource "aws_autoscaling_group" "workers" {
-  name = "${var.name}-worker ${aws_launch_configuration.worker.name}"
+  name = "${var.name}-worker"
 
   # count
   desired_capacity = var.worker_count
   min_size         = var.worker_count
   max_size         = var.worker_count + 2
-  default_cooldown          = 30
-  health_check_grace_period = 30
 
   # network
   vpc_zone_identifier = var.subnet_ids
 
-  # template
+  # instance template
-  launch_configuration = aws_launch_configuration.worker.name
+  launch_template {
+    id      = aws_launch_template.worker.id
+    version = aws_launch_template.worker.latest_version
+  }
 
   # target groups to which instances should be added
   target_group_arns = flatten([
@@ -22,6 +23,18 @@ resource "aws_autoscaling_group" "workers" {
     var.target_groups,
   ])
 
+  instance_refresh {
+    strategy = "Rolling"
+    preferences {
+      instance_warmup        = 120
+      min_healthy_percentage = 90
+    }
+  }
+  # Grace period before checking new instance's health
+  health_check_grace_period = 30
+  # Cooldown period between scaling activities
+  default_cooldown = 30
+
   lifecycle {
     # override the default destroy and replace update behavior
     create_before_destroy = true
@@ -33,34 +46,62 @@ resource "aws_autoscaling_group" "workers" {
   # used. Disable wait to avoid issues and align with other clouds.
   wait_for_capacity_timeout = "0"
 
-  tags = [
+  tag {
-    {
     key                 = "Name"
     value               = "${var.name}-worker"
     propagate_at_launch = true
-    },
+  }
-  ]
 }
 
 # Worker template
-resource "aws_launch_configuration" "worker" {
+resource "aws_launch_template" "worker" {
+  name_prefix   = "${var.name}-worker"
   image_id      = local.ami_id
   instance_type = var.instance_type
-  spot_price        = var.spot_price > 0 ? var.spot_price : null
-  enable_monitoring = false
-
-  user_data = data.ct_config.worker-ignition.rendered
 
   # storage
-  root_block_device {
+  ebs_optimized = true
+  block_device_mappings {
+    device_name = "/dev/xvda"
+    ebs {
       volume_type           = var.disk_type
       volume_size           = var.disk_size
       iops                  = var.disk_iops
       encrypted             = true
+      delete_on_termination = true
+    }
   }
 
   # network
+  network_interfaces {
+    associate_public_ip_address = true
     security_groups             = var.security_groups
+  }
+
+  # boot
+  user_data = sensitive(base64encode(data.ct_config.worker.rendered))
+
+  # metadata
+  metadata_options {
+    http_tokens = "optional"
+  }
+  monitoring {
+    enabled = false
+  }
+
+  # cost
+  credit_specification {
+    cpu_credits = var.cpu_credits
+  }
+  dynamic "instance_market_options" {
+    for_each = var.spot_price > 0 ? [1] : []
+    content {
+      market_type = "spot"
+      spot_options {
+        max_price = var.spot_price
+      }
+    }
+  }
 
   lifecycle {
     // Override the default destroy and replace update behavior
@@ -69,24 +110,15 @@ resource "aws_launch_configuration" "worker" {
   }
 }
 
-# Worker Ignition config
+# Flatcar Linux worker
-data "ct_config" "worker-ignition" {
+data "ct_config" "worker" {
-  content  = data.template_file.worker-config.rendered
+  content = templatefile("${path.module}/butane/worker.yaml", {
-  strict   = true
-  snippets = var.snippets
-}
-
-# Worker Container Linux config
-data "template_file" "worker-config" {
-  template = file("${path.module}/cl/worker.yaml")
-
-  vars = {
     kubeconfig             = indent(10, var.kubeconfig)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
     node_taints            = join(",", var.node_taints)
+  })
+  strict   = true
+  snippets = var.snippets
 }
-}
-

azure/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.23.2 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,13 +1,12 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
   etcd_servers = formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)
 
   networking = var.networking
-
   # only effective with Calico networking
   # we should be able to use 1450 MTU, but in practice, 1410 was needed
   network_encapsulation = "vxlan"
@@ -15,9 +14,7 @@ module "bootstrap" {
 
   pod_cidr              = var.pod_cidr
   service_cidr          = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
+  components            = var.components
 }
 

azure/fedora-coreos/kubernetes/butane/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.4.0
+version: 1.5.0
 systemd:
   units:
     - name: etcd-member.service
@@ -9,15 +9,16 @@ systemd:
         [Unit]
         Description=etcd (System Container)
         Documentation=https://github.com/etcd-io/etcd
-        Wants=network-online.target network.target
+        Wants=network-online.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.13
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
         ExecStart=/usr/bin/podman run --name etcd \
           --env-file /etc/etcd/etcd.env \
+          --log-driver k8s-file \
           --network host \
           --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
           --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
@@ -53,7 +54,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -62,15 +63,19 @@ systemd:
         ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
         ExecStartPre=-/usr/bin/podman rm kubelet
         ExecStart=/usr/bin/podman run --name kubelet \
+          --log-driver k8s-file \
           --privileged \
           --pid host \
           --network host \
           --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /etc/machine-id:/etc/machine-id:ro \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
           --volume /sys/fs/cgroup:/sys/fs/cgroup \
+          --volume /etc/selinux:/etc/selinux \
+          --volume /sys/fs/selinux:/sys/fs/selinux \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/containerd:/var/lib/containerd \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -78,27 +83,12 @@ systemd:
           --volume /var/run/lock:/var/run/lock:z \
           --volume /opt/cni/bin:/opt/cni/bin:z \
           $${KUBELET_IMAGE} \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=systemd \
+          --config=/etc/kubernetes/kubelet.yaml \
-          --cgroups-per-qos=true \
-          --container-runtime=remote \
           --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
-          --enforce-node-allocatable=pods \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule
-          --read-only-port=0 \
-          --resolv-conf=/run/systemd/resolve/resolv.conf \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/podman stop kubelet
         Delegate=yes
         Restart=always
@@ -121,7 +111,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.23.2
+            quay.io/poseidon/kubelet:v1.31.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -136,12 +126,39 @@ storage:
       contents:
         inline: |
           ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: cluster.local
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
     - path: /opt/bootstrap/layout
       mode: 0544
       contents:
         inline: |
           #!/bin/bash -e
-          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          mkdir -p -- auth tls/{etcd,k8s} static-manifests manifests/{coredns,kube-proxy,network}
           awk '/#####/ {filename=$2; next} {print > filename}' assets
           mkdir -p /etc/ssl/etcd/etcd
           mkdir -p /etc/kubernetes/pki
@@ -155,8 +172,7 @@ storage:
           mv static-manifests/* /etc/kubernetes/manifests/
           mkdir -p /opt/bootstrap/assets
           mv manifests /opt/bootstrap/assets/manifests
-          mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          rm -rf assets auth static-manifests tls manifests
-          rm -rf assets auth static-manifests tls manifests-networking
           chcon -R -u system_u -t container_file_t /etc/kubernetes/pki
     - path: /opt/bootstrap/apply
       mode: 0544
@@ -172,6 +188,11 @@ storage:
              echo "Retry applying manifests"
              sleep 5
           done
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
     - path: /etc/sysctl.d/max-user-watches.conf
       contents:
         inline: |
@@ -216,7 +237,6 @@ storage:
           ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
-    - path: /etc/fedora-coreos/iptables-legacy.stamp
     - path: /etc/containerd/config.toml
       overwrite: true
       contents:
@@ -241,3 +261,4 @@ passwd:
     - name: core
       ssh_authorized_keys:
         - ${ssh_authorized_key}
+

azure/fedora-coreos/kubernetes/controllers.tf

@@ -1,25 +1,30 @@
+locals {
+  # Typhoon ssh_authorized_key supports RSA or a newer formats (e.g. ed25519).
+  # However, Azure requires an older RSA key to pass validations. To use a
+  # newer key format, pass a dummy RSA key as the azure_authorized_key and
+  # delete the associated private key so it's never used.
+  azure_authorized_key = var.azure_authorized_key == "" ? var.ssh_authorized_key : var.azure_authorized_key
+}
+
 # Discrete DNS records for each controller's private IPv4 for etcd usage
 resource "azurerm_dns_a_record" "etcds" {
   count = var.controller_count
-  resource_group_name = var.dns_zone_group
 
   # DNS Zone name where record should be created
   zone_name           = var.dns_zone
-
+  resource_group_name = var.dns_zone_group
   # DNS record
   name = format("%s-etcd%d", var.cluster_name, count.index)
   ttl  = 300
-
   # private IPv4 address for etcd
-  records = [azurerm_network_interface.controllers.*.private_ip_address[count.index]]
+  records = [azurerm_network_interface.controllers[count.index].private_ip_address]
 }
 
 # Controller availability set to spread controllers
 resource "azurerm_availability_set" "controllers" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                         = "${var.cluster_name}-controllers"
-  location                     = var.region
+  resource_group_name          = azurerm_resource_group.cluster.name
+  location                     = var.location
   platform_fault_domain_count  = 2
   platform_update_domain_count = 4
   managed                      = true
@@ -28,34 +33,38 @@ resource "azurerm_availability_set" "controllers" {
 # Controller instances
 resource "azurerm_linux_virtual_machine" "controllers" {
   count = var.controller_count
-  resource_group_name = azurerm_resource_group.cluster.name
 
   name                = "${var.cluster_name}-controller-${count.index}"
-  location            = var.region
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = var.location
   availability_set_id = azurerm_availability_set.controllers.id
-
   size                = var.controller_type
-  custom_data = base64encode(data.ct_config.controller-ignitions.*.rendered[count.index])
 
   # storage
   source_image_id = var.os_image
   os_disk {
     name                 = "${var.cluster_name}-controller-${count.index}"
+    storage_account_type = var.controller_disk_type
+    disk_size_gb         = var.controller_disk_size
     caching              = "None"
-    disk_size_gb         = var.disk_size
-    storage_account_type = "Premium_LRS"
   }
 
   # network
   network_interface_ids = [
-    azurerm_network_interface.controllers.*.id[count.index]
+    azurerm_network_interface.controllers[count.index].id
   ]
 
-  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
+  # boot
+  custom_data = base64encode(data.ct_config.controllers[count.index].rendered)
+  boot_diagnostics {
+    # defaults to a managed storage account
+  }
+
+  # Azure requires an RSA admin_ssh_key
   admin_username = "core"
   admin_ssh_key {
     username   = "core"
-    public_key = var.ssh_authorized_key
+    public_key = local.azure_authorized_key
   }
 
   lifecycle {
@@ -66,31 +75,52 @@ resource "azurerm_linux_virtual_machine" "controllers" {
   }
 }
 
-# Controller public IPv4 addresses
+# Controller node public IPv4 addresses
-resource "azurerm_public_ip" "controllers" {
+resource "azurerm_public_ip" "controllers-ipv4" {
   count = var.controller_count
-  resource_group_name = azurerm_resource_group.cluster.name
 
-  name              = "${var.cluster_name}-controller-${count.index}"
+  name                = "${var.cluster_name}-controller-${count.index}-ipv4"
+  resource_group_name = azurerm_resource_group.cluster.name
   location            = azurerm_resource_group.cluster.location
+  ip_version          = "IPv4"
   sku                 = "Standard"
   allocation_method   = "Static"
 }
 
-# Controller NICs with public and private IPv4
+# Controller node public IPv6 addresses
+resource "azurerm_public_ip" "controllers-ipv6" {
+  count = var.controller_count
+
+  name                = "${var.cluster_name}-controller-${count.index}-ipv6"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = azurerm_resource_group.cluster.location
+  ip_version          = "IPv6"
+  sku                 = "Standard"
+  allocation_method   = "Static"
+}
+
+# Controllers' network interfaces
 resource "azurerm_network_interface" "controllers" {
   count = var.controller_count
-  resource_group_name = azurerm_resource_group.cluster.name
 
   name                = "${var.cluster_name}-controller-${count.index}"
+  resource_group_name = azurerm_resource_group.cluster.name
   location            = azurerm_resource_group.cluster.location
 
   ip_configuration {
-    name                          = "ip0"
+    name                          = "ipv4"
+    primary                       = true
     subnet_id                     = azurerm_subnet.controller.id
     private_ip_address_allocation = "Dynamic"
-    # instance public IPv4
+    private_ip_address_version    = "IPv4"
-    public_ip_address_id = azurerm_public_ip.controllers.*.id[count.index]
+    public_ip_address_id          = azurerm_public_ip.controllers-ipv4[count.index].id
+  }
+  ip_configuration {
+    name                          = "ipv6"
+    subnet_id                     = azurerm_subnet.controller.id
+    private_ip_address_allocation = "Dynamic"
+    private_ip_address_version    = "IPv6"
+    public_ip_address_id          = azurerm_public_ip.controllers-ipv6[count.index].id
   }
 }
 
@@ -103,49 +133,37 @@ resource "azurerm_network_interface_security_group_association" "controllers" {
 }
 
 # Associate controller network interface with controller backend address pool
-resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
+resource "azurerm_network_interface_backend_address_pool_association" "controllers-ipv4" {
   count = var.controller_count
 
   network_interface_id    = azurerm_network_interface.controllers[count.index].id
-  ip_configuration_name   = "ip0"
+  ip_configuration_name   = "ipv4"
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller-ipv4.id
 }
 
-# Controller Ignition configs
+resource "azurerm_network_interface_backend_address_pool_association" "controllers-ipv6" {
-data "ct_config" "controller-ignitions" {
   count = var.controller_count
-  content  = data.template_file.controller-configs.*.rendered[count.index]
+
-  strict   = true
+  network_interface_id    = azurerm_network_interface.controllers[count.index].id
-  snippets = var.controller_snippets
+  ip_configuration_name   = "ipv6"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller-ipv6.id
 }
 
-# Controller Fedora CoreOS configs
+# Fedora CoreOS controllers
-data "template_file" "controller-configs" {
+data "ct_config" "controllers" {
   count = var.controller_count
-
+  content = templatefile("${path.module}/butane/controller.yaml", {
-  template = file("${path.module}/fcc/controller.yaml")
-
-  vars = {
     # Cannot use cyclic dependencies on controllers or their DNS records
     etcd_name   = "etcd${count.index}"
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    etcd_initial_cluster = join(",", [
+      for i in range(var.controller_count) : "etcd${i}=https://${var.cluster_name}-etcd${i}.${var.dns_zone}:2380"
+    ])
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
+  })
+  strict   = true
+  snippets = var.controller_snippets
 }
-}
-
-data "template_file" "etcds" {
-  count    = var.controller_count
-  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
-
-  vars = {
-    index        = count.index
-    cluster_name = var.cluster_name
-    dns_zone     = var.dns_zone
-  }
-}
-

azure/fedora-coreos/kubernetes/lb.tf

@@ -1,124 +1,164 @@
-# DNS record for the apiserver load balancer
+# DNS A record for the apiserver load balancer
 resource "azurerm_dns_a_record" "apiserver" {
-  resource_group_name = var.dns_zone_group
-
   # DNS Zone name where record should be created
   zone_name           = var.dns_zone
-
+  resource_group_name = var.dns_zone_group
   # DNS record
   name = var.cluster_name
   ttl  = 300
-
   # IPv4 address of apiserver load balancer
-  records = [azurerm_public_ip.apiserver-ipv4.ip_address]
+  records = [azurerm_public_ip.frontend-ipv4.ip_address]
 }
 
-# Static IPv4 address for the apiserver frontend
+# DNS AAAA record for the apiserver load balancer
-resource "azurerm_public_ip" "apiserver-ipv4" {
+resource "azurerm_dns_aaaa_record" "apiserver" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  # DNS Zone name where record should be created
+  zone_name           = var.dns_zone
+  resource_group_name = var.dns_zone_group
+  # DNS record
+  name = var.cluster_name
+  ttl  = 300
+  # IPv4 address of apiserver load balancer
+  records = [azurerm_public_ip.frontend-ipv6.ip_address]
+}
 
-  name              = "${var.cluster_name}-apiserver-ipv4"
+# Static IPv4 address for the load balancer
-  location          = var.region
+resource "azurerm_public_ip" "frontend-ipv4" {
+  name                = "${var.cluster_name}-frontend-ipv4"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = var.location
+  ip_version          = "IPv4"
   sku                 = "Standard"
   allocation_method   = "Static"
 }
 
-# Static IPv4 address for the ingress frontend
+# Static IPv6 address for the load balancer
-resource "azurerm_public_ip" "ingress-ipv4" {
+resource "azurerm_public_ip" "frontend-ipv6" {
+  name                = "${var.cluster_name}-frontend-ipv6"
   resource_group_name = azurerm_resource_group.cluster.name
-
+  location            = var.location
-  name              = "${var.cluster_name}-ingress-ipv4"
+  ip_version          = "IPv6"
-  location          = var.region
   sku                 = "Standard"
   allocation_method   = "Static"
 }
 
 # Network Load Balancer for apiservers and ingress
 resource "azurerm_lb" "cluster" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                = var.cluster_name
-  location = var.region
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = var.location
   sku                 = "Standard"
 
   frontend_ip_configuration {
-    name                 = "apiserver"
+    name                 = "frontend-ipv4"
-    public_ip_address_id = azurerm_public_ip.apiserver-ipv4.id
+    public_ip_address_id = azurerm_public_ip.frontend-ipv4.id
   }
 
   frontend_ip_configuration {
-    name                 = "ingress"
+    name                 = "frontend-ipv6"
-    public_ip_address_id = azurerm_public_ip.ingress-ipv4.id
+    public_ip_address_id = azurerm_public_ip.frontend-ipv6.id
   }
 }
 
-resource "azurerm_lb_rule" "apiserver" {
+resource "azurerm_lb_rule" "apiserver-ipv4" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  name                           = "apiserver-ipv4"
-
-  name                           = "apiserver"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration_name = "apiserver"
+  frontend_ip_configuration_name = "frontend-ipv4"
+  disable_outbound_snat          = true
 
   protocol                 = "Tcp"
   frontend_port            = 6443
   backend_port             = 6443
-  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller-ipv4.id]
   probe_id                 = azurerm_lb_probe.apiserver.id
 }
 
-resource "azurerm_lb_rule" "ingress-http" {
+resource "azurerm_lb_rule" "apiserver-ipv6" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  name                           = "apiserver-ipv6"
-
-  name                           = "ingress-http"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration_name = "ingress"
+  frontend_ip_configuration_name = "frontend-ipv6"
+  disable_outbound_snat          = true
+
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller-ipv6.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
+}
+
+resource "azurerm_lb_rule" "ingress-http-ipv4" {
+  name                           = "ingress-http-ipv4"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "frontend-ipv4"
   disable_outbound_snat          = true
 
   protocol                 = "Tcp"
   frontend_port            = 80
   backend_port             = 80
-  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv4.id]
   probe_id                 = azurerm_lb_probe.ingress.id
 }
 
-resource "azurerm_lb_rule" "ingress-https" {
+resource "azurerm_lb_rule" "ingress-https-ipv4" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  name                           = "ingress-https-ipv4"
-
-  name                           = "ingress-https"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration_name = "ingress"
+  frontend_ip_configuration_name = "frontend-ipv4"
   disable_outbound_snat          = true
 
   protocol                 = "Tcp"
   frontend_port            = 443
   backend_port             = 443
-  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv4.id]
   probe_id                 = azurerm_lb_probe.ingress.id
 }
 
-# Worker outbound TCP/UDP SNAT
+resource "azurerm_lb_rule" "ingress-http-ipv6" {
-resource "azurerm_lb_outbound_rule" "worker-outbound" {
+  name                           = "ingress-http-ipv6"
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name            = "worker"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration {
+  frontend_ip_configuration_name = "frontend-ipv6"
-    name = "ingress"
+  disable_outbound_snat          = true
+
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }
 
-  protocol                = "All"
+resource "azurerm_lb_rule" "ingress-https-ipv6" {
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  name                           = "ingress-https-ipv6"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "frontend-ipv6"
+  disable_outbound_snat          = true
+
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }
 
+# Backend Address Pools
+
 # Address pool of controllers
-resource "azurerm_lb_backend_address_pool" "controller" {
+resource "azurerm_lb_backend_address_pool" "controller-ipv4" {
-  name            = "controller"
+  name            = "controller-ipv4"
+  loadbalancer_id = azurerm_lb.cluster.id
+}
+
+resource "azurerm_lb_backend_address_pool" "controller-ipv6" {
+  name            = "controller-ipv6"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
 # Address pool of workers
-resource "azurerm_lb_backend_address_pool" "worker" {
+resource "azurerm_lb_backend_address_pool" "worker-ipv4" {
-  name            = "worker"
+  name            = "worker-ipv4"
+  loadbalancer_id = azurerm_lb.cluster.id
+}
+
+resource "azurerm_lb_backend_address_pool" "worker-ipv6" {
+  name            = "worker-ipv6"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
@@ -126,32 +166,45 @@ resource "azurerm_lb_backend_address_pool" "worker" {
 
 # TCP health check for apiserver
 resource "azurerm_lb_probe" "apiserver" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "apiserver"
   loadbalancer_id = azurerm_lb.cluster.id
   protocol        = "Tcp"
   port            = 6443
-
   # unhealthy threshold
   number_of_probes    = 3
-
   interval_in_seconds = 5
 }
 
 # HTTP health check for ingress
 resource "azurerm_lb_probe" "ingress" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "ingress"
   loadbalancer_id = azurerm_lb.cluster.id
   protocol        = "Http"
   port            = 10254
   request_path    = "/healthz"
-
   # unhealthy threshold
   number_of_probes    = 3
-
   interval_in_seconds = 5
 }
 
+# Outbound SNAT
+
+resource "azurerm_lb_outbound_rule" "outbound-ipv4" {
+  name                    = "outbound-ipv4"
+  protocol                = "All"
+  loadbalancer_id         = azurerm_lb.cluster.id
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker-ipv4.id
+  frontend_ip_configuration {
+    name = "frontend-ipv4"
+  }
+}
+
+resource "azurerm_lb_outbound_rule" "outbound-ipv6" {
+  name                    = "outbound-ipv6"
+  protocol                = "All"
+  loadbalancer_id         = azurerm_lb.cluster.id
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker-ipv6.id
+  frontend_ip_configuration {
+    name = "frontend-ipv6"
+  }
+}

azure/fedora-coreos/kubernetes/locals.tf

@@ -0,0 +1,6 @@
+locals {
+  backend_address_pool_ids = {
+    ipv4 = [azurerm_lb_backend_address_pool.worker-ipv4.id]
+    ipv6 = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  }
+}

azure/fedora-coreos/kubernetes/network.tf

@@ -1,27 +1,64 @@
+# Choose an IPv6 ULA subnet at random
+# https://datatracker.ietf.org/doc/html/rfc4193
+resource "random_id" "ula-netnum" {
+  byte_length = 5 # 40 bits
+}
+
+locals {
+  # fd00::/8 -> shift 40 -> 2^40 possible /48 subnets
+  ula-range = cidrsubnet("fd00::/8", 40, random_id.ula-netnum.dec)
+  network_cidr = {
+    ipv4 = var.network_cidr.ipv4
+    ipv6 = length(var.network_cidr.ipv6) > 0 ? var.network_cidr.ipv6 : [local.ula-range]
+  }
+
+  # Subdivide the virtual network into subnets
+  # - controllers use netnum 0
+  # - workers use netnum 1
+  controller_subnets = {
+    ipv4 = [for i, cidr in local.network_cidr.ipv4 : cidrsubnet(cidr, 1, 0)]
+    ipv6 = [for i, cidr in local.network_cidr.ipv6 : cidrsubnet(cidr, 16, 0)]
+  }
+  worker_subnets = {
+    ipv4 = [for i, cidr in local.network_cidr.ipv4 : cidrsubnet(cidr, 1, 1)]
+    ipv6 = [for i, cidr in local.network_cidr.ipv6 : cidrsubnet(cidr, 16, 1)]
+  }
+  cluster_subnets = {
+    ipv4 = concat(local.controller_subnets.ipv4, local.worker_subnets.ipv4)
+    ipv6 = concat(local.controller_subnets.ipv6, local.worker_subnets.ipv6)
+  }
+}
+
 # Organize cluster into a resource group
 resource "azurerm_resource_group" "cluster" {
   name     = var.cluster_name
-  location = var.region
+  location = var.location
 }
 
 resource "azurerm_virtual_network" "network" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                = var.cluster_name
+  resource_group_name = azurerm_resource_group.cluster.name
   location            = azurerm_resource_group.cluster.location
-  address_space = [var.host_cidr]
+  address_space = concat(
+    local.network_cidr.ipv4,
+    local.network_cidr.ipv6
+  )
 }
 
-# Subnets - separate subnets for controller and workers because Azure
+# Subnets - separate subnets for controllers and workers because Azure
-# network security groups are based on IPv4 CIDR rather than instance
+# network security groups are oriented around address prefixes rather
-# tags like GCP or security group membership like AWS
+# than instance tags (GCP) or security group membership (AWS)
 
 resource "azurerm_subnet" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                 = "controller"
+  resource_group_name  = azurerm_resource_group.cluster.name
   virtual_network_name = azurerm_virtual_network.network.name
-  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 0)]
+  address_prefixes = concat(
+    local.controller_subnets.ipv4,
+    local.controller_subnets.ipv6,
+  )
+  default_outbound_access_enabled = false
+
 }
 
 resource "azurerm_subnet_network_security_group_association" "controller" {
@@ -30,15 +67,17 @@ resource "azurerm_subnet_network_security_group_association" "controller" {
 }
 
 resource "azurerm_subnet" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                 = "worker"
+  resource_group_name  = azurerm_resource_group.cluster.name
   virtual_network_name = azurerm_virtual_network.network.name
-  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 1)]
+  address_prefixes = concat(
+    local.worker_subnets.ipv4,
+    local.worker_subnets.ipv6,
+  )
+  default_outbound_access_enabled = false
 }
 
 resource "azurerm_subnet_network_security_group_association" "worker" {
   subnet_id                 = azurerm_subnet.worker.id
   network_security_group_id = azurerm_network_security_group.worker.id
 }
-

azure/fedora-coreos/kubernetes/outputs.tf

@@ -6,13 +6,18 @@ output "kubeconfig-admin" {
 # Outputs for Kubernetes Ingress
 
 output "ingress_static_ipv4" {
-  value       = azurerm_public_ip.ingress-ipv4.ip_address
+  value       = azurerm_public_ip.frontend-ipv4.ip_address
   description = "IPv4 address of the load balancer for distributing traffic to Ingress controllers"
 }
 
+output "ingress_static_ipv6" {
+  value       = azurerm_public_ip.frontend-ipv6.ip_address
+  description = "IPv6 address of the load balancer for distributing traffic to Ingress controllers"
+}
+
 # Outputs for worker pools
 
-output "region" {
+output "location" {
   value = azurerm_resource_group.cluster.location
 }
 
@@ -39,13 +44,24 @@ output "kubeconfig" {
 
 # Outputs for custom firewalling
 
+output "controller_security_group_name" {
+  description = "Network Security Group for controller nodes"
+  value       = azurerm_network_security_group.controller.name
+}
+
 output "worker_security_group_name" {
+  description = "Network Security Group for worker nodes"
   value       = azurerm_network_security_group.worker.name
 }
 
-output "worker_address_prefix" {
+output "controller_address_prefixes" {
-  description = "Worker network subnet CIDR address (for source/destination)"
+  description = "Controller network subnet CIDR addresses (for source/destination)"
-  value       = azurerm_subnet.worker.address_prefix
+  value       = local.controller_subnets
+}
+
+output "worker_address_prefixes" {
+  description = "Worker network subnet CIDR addresses (for source/destination)"
+  value       = local.worker_subnets
 }
 
 # Outputs for custom load balancing
@@ -55,9 +71,12 @@ output "loadbalancer_id" {
   value       = azurerm_lb.cluster.id
 }
 
-output "backend_address_pool_id" {
+output "backend_address_pool_ids" {
-  description = "ID of the worker backend address pool"
+  description = "IDs of the worker backend address pools"
-  value       = azurerm_lb_backend_address_pool.worker.id
+  value = {
+    ipv4 = [azurerm_lb_backend_address_pool.worker-ipv4.id]
+    ipv6 = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  }
 }
 
 # Outputs for debug

azure/fedora-coreos/kubernetes/security.tf

@@ -1,198 +1,223 @@
 # Controller security group
 
 resource "azurerm_network_security_group" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                = "${var.cluster_name}-controller"
+  resource_group_name = azurerm_resource_group.cluster.name
   location            = azurerm_resource_group.cluster.location
 }
 
 resource "azurerm_network_security_rule" "controller-icmp" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-icmp"
+  name                         = "allow-icmp-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "1995"
+  priority                     = 1995 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Icmp"
   source_port_range            = "*"
   destination_port_range       = "*"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-ssh" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-ssh"
+  name                         = "allow-ssh-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2000"
+  priority                     = 2000 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "22"
   source_address_prefix        = "*"
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-etcd" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-etcd"
+  name                         = "allow-etcd-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2005"
+  priority                     = 2005 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "2379-2380"
-  source_address_prefix       = azurerm_subnet.controller.address_prefix
+  source_address_prefixes      = local.controller_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape etcd metrics
 resource "azurerm_network_security_rule" "controller-etcd-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-etcd-metrics"
+  name                         = "allow-etcd-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2010"
+  priority                     = 2010 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "2381"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  source_address_prefixes      = local.worker_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape kube-proxy metrics
 resource "azurerm_network_security_rule" "controller-kube-proxy" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-kube-proxy-metrics"
+  name                         = "allow-kube-proxy-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2011"
+  priority                     = 2012 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "10249"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  source_address_prefixes      = local.worker_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape kube-scheduler and kube-controller-manager metrics
 resource "azurerm_network_security_rule" "controller-kube-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-kube-metrics"
+  name                         = "allow-kube-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2012"
+  priority                     = 2014 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "10257-10259"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  source_address_prefixes      = local.worker_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-apiserver" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-apiserver"
+  name                         = "allow-apiserver-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2015"
+  priority                     = 2016 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "6443"
   source_address_prefix        = "*"
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-cilium-health" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = var.networking == "cilium" ? local.controller_subnets : {}
-  count               = var.networking == "cilium" ? 1 : 0
 
-  name                        = "allow-cilium-health"
+  name                         = "allow-cilium-health-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2019"
+  priority                     = 2018 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "4240"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
+}
+
+resource "azurerm_network_security_rule" "controller-cilium-metrics" {
+  for_each = var.networking == "cilium" ? local.controller_subnets : {}
+
+  name                         = "allow-cilium-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = 2035 + (each.key == "ipv4" ? 0 : 1)
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "9962-9965"
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-vxlan"
+  name                         = "allow-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2020"
+  priority                     = 2020 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "4789"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-linux-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-linux-vxlan"
+  name                         = "allow-linux-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2021"
+  priority                     = 2022 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "8472"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "controller-node-exporter" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-node-exporter"
+  name                         = "allow-node-exporter-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                    = "2025"
+  priority                     = 2025 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9100"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  source_address_prefixes      = local.worker_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "controller-kubelet" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-kubelet"
+  name                        = "allow-kubelet-${each.key}"
+  resource_group_name         = azurerm_resource_group.cluster.name
   network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2030"
+  priority                    = 2030 + (each.key == "ipv4" ? 0 : 1)
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "10250"
-
   # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix = azurerm_subnet.controller.address_prefix
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
@@ -231,166 +256,189 @@ resource "azurerm_network_security_rule" "controller-deny-all" {
 # Worker security group
 
 resource "azurerm_network_security_group" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                = "${var.cluster_name}-worker"
+  resource_group_name = azurerm_resource_group.cluster.name
   location            = azurerm_resource_group.cluster.location
 }
 
 resource "azurerm_network_security_rule" "worker-icmp" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-icmp"
+  name                         = "allow-icmp-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                    = "1995"
+  priority                     = 1995 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Icmp"
   source_port_range            = "*"
   destination_port_range       = "*"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-ssh" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-ssh"
+  name                         = "allow-ssh-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                    = "2000"
+  priority                     = 2000 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "22"
-  source_address_prefix       = azurerm_subnet.controller.address_prefix
+  source_address_prefixes      = local.controller_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-http" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-http"
+  name                         = "allow-http-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                    = "2005"
+  priority                     = 2005 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "80"
   source_address_prefix        = "*"
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-https" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-https"
+  name                         = "allow-https-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                    = "2010"
+  priority                     = 2010 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "443"
   source_address_prefix        = "*"
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-cilium-health" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = var.networking == "cilium" ? local.worker_subnets : {}
-  count               = var.networking == "cilium" ? 1 : 0
 
-  name                        = "allow-cilium-health"
+  name                         = "allow-cilium-health-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                    = "2014"
+  priority                     = 2012 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "4240"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
+}
+
+resource "azurerm_network_security_rule" "worker-cilium-metrics" {
+  for_each = var.networking == "cilium" ? local.worker_subnets : {}
+
+  name                         = "allow-cilium-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = 2014 + (each.key == "ipv4" ? 0 : 1)
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "9962-9965"
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-vxlan"
+  name                         = "allow-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                    = "2015"
+  priority                     = 2016 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "4789"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-linux-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-linux-vxlan"
+  name                         = "allow-linux-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                    = "2016"
+  priority                     = 2018 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "8472"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "worker-node-exporter" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-node-exporter"
+  name                         = "allow-node-exporter-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                    = "2020"
+  priority                     = 2020 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9100"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  source_address_prefixes      = local.worker_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Allow Prometheus to scrape kube-proxy
 resource "azurerm_network_security_rule" "worker-kube-proxy" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-kube-proxy"
+  name                         = "allow-kube-proxy-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                    = "2024"
+  priority                     = 2024 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "10249"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  source_address_prefixes      = local.worker_subnets[each.key]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "worker-kubelet" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-kubelet"
+  name                        = "allow-kubelet-${each.key}"
+  resource_group_name         = azurerm_resource_group.cluster.name
   network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2025"
+  priority                    = 2026 + (each.key == "ipv4" ? 0 : 1)
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "10250"
-
   # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  source_address_prefixes      = local.cluster_subnets[each.key]
-  destination_address_prefix = azurerm_subnet.worker.address_prefix
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound

azure/fedora-coreos/kubernetes/ssh.tf

@@ -18,7 +18,7 @@ resource "null_resource" "copy-controller-secrets" {
 
   connection {
     type    = "ssh"
-    host    = azurerm_public_ip.controllers.*.ip_address[count.index]
+    host    = azurerm_public_ip.controllers-ipv4[count.index].ip_address
     user    = "core"
     timeout = "15m"
   }
@@ -45,7 +45,7 @@ resource "null_resource" "bootstrap" {
 
   connection {
     type    = "ssh"
-    host    = azurerm_public_ip.controllers.*.ip_address[0]
+    host    = azurerm_public_ip.controllers-ipv4[0].ip_address
     user    = "core"
     timeout = "15m"
   }

azure/fedora-coreos/kubernetes/variables.tf

@@ -5,9 +5,9 @@ variable "cluster_name" {
 
 # Azure
 
-variable "region" {
+variable "location" {
   type        = string
-  description = "Azure Region (e.g. centralus , see `az account list-locations --output table`)"
+  description = "Azure location (e.g. centralus , see `az account list-locations --output table`)"
 }
 
 variable "dns_zone" {
@@ -22,41 +22,65 @@ variable "dns_zone_group" {
 
 # instances
 
+variable "os_image" {
+  type        = string
+  description = "Fedora CoreOS image for instances"
+}
+
 variable "controller_count" {
   type        = number
   description = "Number of controllers (i.e. masters)"
   default     = 1
 }
 
-variable "worker_count" {
-  type        = number
-  description = "Number of workers"
-  default     = 1
-}
-
 variable "controller_type" {
   type        = string
   description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
   default     = "Standard_B2s"
 }
 
+variable "controller_disk_type" {
+  type        = string
+  description = "Type of managed disk for controller node(s)"
+  default     = "Premium_LRS"
+}
+
+variable "controller_disk_size" {
+  type        = number
+  description = "Size of the managed disk in GB for controller node(s)"
+  default     = 30
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
 variable "worker_type" {
   type        = string
   description = "Machine type for workers (see `az vm list-skus --location centralus`)"
-  default     = "Standard_DS1_v2"
+  default     = "Standard_D2as_v5"
 }
 
-variable "os_image" {
+variable "worker_disk_type" {
   type        = string
-  description = "Fedora CoreOS image for instances"
+  description = "Type of managed disk for worker nodes"
+  default     = "Standard_LRS"
 }
 
-variable "disk_size" {
+variable "worker_disk_size" {
   type        = number
-  description = "Size of the disk in GB"
+  description = "Size of the managed disk in GB for worker nodes"
   default     = 30
 }
 
+variable "worker_ephemeral_disk" {
+  type        = bool
+  description = "Use ephemeral local disk instead of managed disk (requires vm_type with local storage)"
+  default     = false
+}
+
 variable "worker_priority" {
   type        = string
   description = "Set worker priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
@@ -82,16 +106,27 @@ variable "ssh_authorized_key" {
   description = "SSH public key for user 'core'"
 }
 
-variable "networking" {
+variable "azure_authorized_key" {
   type        = string
-  description = "Choice of networking provider (flannel or calico)"
+  description = "Optionally, pass a dummy RSA key to satisfy Azure validations (then use an ed25519 key set above)"
-  default     = "calico"
+  default     = ""
 }
 
-variable "host_cidr" {
+variable "networking" {
   type        = string
-  description = "CIDR IPv4 range to assign to instances"
+  description = "Choice of networking provider (flannel, calico, or cilium)"
-  default     = "10.0.0.0/16"
+  default     = "cilium"
+}
+
+variable "network_cidr" {
+  type = object({
+    ipv4 = list(string)
+    ipv6 = optional(list(string), [])
+  })
+  description = "Virtual network CIDR ranges"
+  default = {
+    ipv4 = ["10.0.0.0/16"]
+  }
 }
 
 variable "pod_cidr" {
@@ -109,34 +144,32 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "enable_reporting" {
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
 variable "worker_node_labels" {
   type        = list(string)
   description = "List of initial worker node labels"
   default     = []
 }
 
-# unofficial, undocumented, unsupported
+# advanced
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
 
 variable "daemonset_tolerations" {
   type        = list(string)
   description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
   default     = []
 }
+
+variable "components" {
+  description = "Configure pre-installed cluster components"
+  # Component configs are passed through to terraform-render-bootstrap,
+  # which handles type enforcement and defines defaults
+  # https://github.com/poseidon/terraform-render-bootstrap/blob/main/variables.tf#L95
+  type = object({
+    enable     = optional(bool)
+    coredns    = optional(map(any))
+    kube_proxy = optional(map(any))
+    flannel    = optional(map(any))
+    calico     = optional(map(any))
+    cilium     = optional(map(any))
+  })
+  default = null
+}

azure/fedora-coreos/kubernetes/versions.tf

@@ -3,13 +3,11 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    azurerm  = "~> 2.8"
+    azurerm = ">= 2.8"
-    template = "~> 2.2"
     null    = ">= 2.1"
-
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.9"
+      version = "~> 0.13"
     }
   }
 }

azure/fedora-coreos/kubernetes/workers.tf

@@ -4,21 +4,25 @@ module "workers" {
 
   # Azure
   resource_group_name      = azurerm_resource_group.cluster.name
-  region                  = azurerm_resource_group.cluster.location
+  location                 = azurerm_resource_group.cluster.location
   subnet_id                = azurerm_subnet.worker.id
   security_group_id        = azurerm_network_security_group.worker.id
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  backend_address_pool_ids = local.backend_address_pool_ids
 
+  # instances
+  os_image       = var.os_image
   worker_count   = var.worker_count
   vm_type        = var.worker_type
-  os_image     = var.os_image
+  disk_type      = var.worker_disk_type
+  disk_size      = var.worker_disk_size
+  ephemeral_disk = var.worker_ephemeral_disk
   priority       = var.worker_priority
 
   # configuration
   kubeconfig           = module.bootstrap.kubeconfig-kubelet
   ssh_authorized_key   = var.ssh_authorized_key
+  azure_authorized_key = var.azure_authorized_key
   service_cidr         = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
   snippets             = var.worker_snippets
   node_labels          = var.worker_node_labels
 }