Add links and clarifications in CHANGES for release

* Minimum versions of Terraform provider plugins are enforced in
each module already. Its better to provide examples with newer
versions. Some folks don't update them
* Previously, tutorials showed the minimum viable version of each
terraform provider that might be used

CHANGES.md

@@ -4,6 +4,45 @@ Notable changes between versions.
 
 ## Latest
 
+## v1.13.4
+
+* Kubernetes [v1.13.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1134)
+* Update etcd from v3.3.11 to [v3.3.12](https://github.com/etcd-io/etcd/releases/tag/v3.3.12)
+* Update Calico from v3.5.0 to [v3.5.2](https://docs.projectcalico.org/v3.5/releases/)
+* Assign priorityClassNames to critical cluster and node components ([#406](https://github.com/poseidon/typhoon/pull/406))
+  * Inform node out-of-resource eviction and scheduler preemption and ordering
+* Add CoreDNS readiness probe ([#410](https://github.com/poseidon/typhoon/pull/410))
+
+#### Bare-Metal
+
+* Recommend updating [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin from v0.2.2 to [v0.2.3](https://github.com/coreos/terraform-provider-matchbox/releases/tag/v0.2.3) ([#402](https://github.com/poseidon/typhoon/pull/402))
+* Improve docs on using Ubiquiti EdgeOS with bare-metal clusters ([#413](https://github.com/poseidon/typhoon/pull/413))
+
+#### Google Cloud
+
+* Support `terraform-provider-google` v2.0+ ([#407](https://github.com/poseidon/typhoon/pull/407))
+  * Require `terraform-provider-google` v1.19+ (**action required**)
+* Set the minimum CPU platform to Intel Haswell ([#405](https://github.com/poseidon/typhoon/pull/405))
+  * Haswell or better is available in every zone (no price change)
+  * A few zones still default to Sandy/Ivy Bridge (shifts in April 2019)
+
+#### Addons
+
+* Modernize Prometheus rules and alerts ([#404](https://github.com/poseidon/typhoon/pull/404))
+  * Drop extraneous metrics ([#397](https://github.com/poseidon/typhoon/pull/397))
+  * Add `pod` name label to metrics discovered via service endpoints
+  * Rename `kubernetes_namespace` label to `namespace`
+* Modernize Grafana and dashboards, see [docs](https://typhoon.psdn.io/addons/grafana/) ([#403](https://github.com/poseidon/typhoon/pull/403), [#404](https://github.com/poseidon/typhoon/pull/404))
+  * Upgrade Grafana from v5.4.3 to [v6.0.0](https://github.com/grafana/grafana/releases/tag/v6.0.0)!
+  * Enable Grafana [Explore](http://docs.grafana.org/guides/whats-new-in-v6-0/#explore) UI as a Viewer (inspect/edit without saving)
+* Update nginx-ingress from v0.22.0 to v0.23.0
+  * Raise nginx-ingress liveness/readiness timeout to 5 seconds
+  * Remove nginx-ingess default-backend ([#401](https://github.com/poseidon/typhoon/pull/401))
+
+#### Fedora Atomic
+
+* Build Kubelet [system container](https://github.com/poseidon/system-containers) with buildah. The image is an OCI format and slightly larger.
+
 ## v1.13.3
 
 * Kubernetes [v1.13.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1133)

README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
@@ -50,7 +50,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.4"
   
   providers = {
     google   = "google.default"
@@ -91,9 +91,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                       ROLES              STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.3
-yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.3
-yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.3
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.4
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.4
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.4
 ```
 
 List the pods.

addons/grafana/config.yaml

@@ -0,0 +1,36 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-config
+  namespace: monitoring
+data:
+  custom.ini: |+
+    [server]
+    http_port = 8080
+
+    [paths]
+    data    = /var/lib/grafana
+    plugins = /var/lib/grafana/plugins
+    provisioning = /etc/grafana/provisioning
+
+    [users]
+    allow_sign_up    = false
+    allow_org_create = false
+    # viewers can edit/inspect, but not save
+    viewers_can_edit = true
+
+    # Disable login form, since Grafana always creates an admin user
+    [auth]
+    disable_login_form = true
+
+    # Disable the user/pass login system
+    [auth.basic]
+    enabled = false
+
+    # Allow anonymous authentication with view-only authorization
+    [auth.anonymous]
+    enabled = true
+    org_role = Viewer
+
+    [analytics]
+    reporting_enabled = false

addons/grafana/datasources.yaml

@@ -10,7 +10,15 @@ data:
     - name: prometheus
       type: prometheus
       access: proxy
-      orgId: 1
       url: http://prometheus.monitoring.svc.cluster.local
       version: 1
       editable: false
+  loki.yaml: |+
+    apiVersion: 1
+    datasources:
+    - name: loki
+      type: loki
+      access: proxy
+      url: http://loki.monitoring.svc.cluster.local
+      version: 1
+      editable: false

addons/grafana/deployment.yaml

@@ -23,20 +23,10 @@ spec:
     spec:
       containers:
         - name: grafana
-          image: grafana/grafana:5.4.3
+          image: grafana/grafana:6.0.0
           env:
-            - name: GF_SERVER_HTTP_PORT
-              value: "8080"
-            - name: GF_AUTH_BASIC_ENABLED
-              value: "false"
-            - name: GF_AUTH_DISABLE_LOGIN_FORM
-              value: "true"
-            - name: GF_AUTH_ANONYMOUS_ENABLED
-              value: "true"
-            - name: GF_AUTH_ANONYMOUS_ORG_ROLE
-              value: Viewer
-            - name: GF_ANALYTICS_REPORTING_ENABLED
-              value: "false"
+            - name: GF_PATHS_CONFIG
+              value: "/etc/grafana/custom.ini"
           ports:
             - name: http
               containerPort: 8080
@@ -48,19 +38,24 @@ spec:
               memory: 200Mi
               cpu: 200m
           volumeMounts:
+            - name: config
+              mountPath: /etc/grafana
             - name: datasources
               mountPath: /etc/grafana/provisioning/datasources
-            - name: dashboard-providers
+            - name: providers
               mountPath: /etc/grafana/provisioning/dashboards
             - name: dashboards
-              mountPath: /var/lib/grafana/dashboards
+              mountPath: /etc/grafana/dashboards
       volumes:
+        - name: config
+          configMap:
+            name: grafana-config
         - name: datasources
           configMap:
             name: grafana-datasources
-        - name: dashboard-providers
+        - name: providers
           configMap:
-            name: grafana-dashboard-providers
+            name: grafana-providers
         - name: dashboards
           configMap:
             name: grafana-dashboards

addons/grafana/providers.yaml

@@ -1,10 +1,10 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: grafana-dashboard-providers
+  name: grafana-providers
   namespace: monitoring
 data:
-  dashboard-providers.yaml: |+
+  providers.yaml: |+
     apiVersion: 1
     providers:
     - name: 'default'
@@ -12,4 +12,4 @@ data:
       folder: ''
       type: file
       options:
-        path: /var/lib/grafana/dashboards
+        path: /etc/grafana/dashboards

addons/nginx-ingress/aws/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/aws/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/aws/deployment.yaml

@@ -24,10 +24,9 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -58,7 +57,7 @@ spec:
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             failureThreshold: 3
             httpGet:
@@ -67,7 +66,7 @@ spec:
               scheme: HTTP
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/nginx-ingress/azure/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/azure/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/azure/deployment.yaml

@@ -24,10 +24,9 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -58,7 +57,7 @@ spec:
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             failureThreshold: 3
             httpGet:
@@ -67,7 +66,7 @@ spec:
               scheme: HTTP
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/nginx-ingress/bare-metal/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/bare-metal/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -22,10 +22,9 @@ spec:
     spec:
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -53,7 +52,7 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             failureThreshold: 3
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             httpGet:
               path: /healthz
@@ -62,7 +61,7 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             failureThreshold: 3
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -24,10 +24,9 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -58,7 +57,7 @@ spec:
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             failureThreshold: 3
             httpGet:
@@ -67,7 +66,7 @@ spec:
               scheme: HTTP
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/nginx-ingress/digital-ocean/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/digital-ocean/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/google-cloud/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/google-cloud/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -24,10 +24,9 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -58,7 +57,7 @@ spec:
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             failureThreshold: 3
             httpGet:
@@ -67,7 +66,7 @@ spec:
               scheme: HTTP
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/prometheus/config.yaml

@@ -55,6 +55,17 @@ data:
         action: replace
         target_label: job
 
+      metric_relabel_configs:
+      - source_labels: [__name__]
+        action: drop
+        regex: etcd_(debugging|disk|request|server).*
+      - source_labels: [__name__]
+        action: drop
+        regex: apiserver_admission_controller_admission_latencies_seconds_.*
+      - source_labels: [__name__]
+        action: drop
+        regex: apiserver_admission_step_admission_latencies_seconds_.*
+
     # Scrape config for node (i.e. kubelet) /metrics (e.g. 'kubelet_'). Explore
     # metrics from a node by scraping kubelet (127.0.0.1:10250/metrics).
     - job_name: 'kubelet'
@@ -89,6 +100,13 @@ data:
       relabel_configs:
       - action: labelmap
         regex: __meta_kubernetes_node_label_(.+)
+      metric_relabel_configs:
+      - source_labels: [__name__, image]
+        action: drop
+        regex: container_([a-z_]+);
+      - source_labels: [__name__]
+        action: drop
+        regex: container_(network_tcp_usage_total|network_udp_usage_total|tasks_state|cpu_load_average_10s)
 
 
     # Scrap etcd metrics from controllers via listen-metrics-urls
@@ -119,10 +137,10 @@ data:
     # * `prometheus.io/port`: If the metrics are exposed on a different port to the
     # service then set this appropriately.
     - job_name: 'kubernetes-service-endpoints'
-
       kubernetes_sd_configs:
       - role: endpoints
 
+      honor_labels: true
       relabel_configs:
       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
         action: keep
@@ -144,10 +162,18 @@ data:
         regex: __meta_kubernetes_service_label_(.+)
       - source_labels: [__meta_kubernetes_namespace]
         action: replace
-        target_label: kubernetes_namespace
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_pod_name]
+        action: replace
+        target_label: pod
       - source_labels: [__meta_kubernetes_service_name]
         action: replace
         target_label: job
+      
+      metric_relabel_configs:
+      - source_labels: [__name__]
+        action: drop
+        regex: etcd_(debugging|disk|request|server).*
 
     # Example scrape config for probing services via the Blackbox Exporter.
     #
@@ -177,7 +203,7 @@ data:
       - action: labelmap
         regex: __meta_kubernetes_service_label_(.+)
       - source_labels: [__meta_kubernetes_namespace]
-        target_label: kubernetes_namespace
+        target_label: namespace
       - source_labels: [__meta_kubernetes_service_name]
         target_label: job
 

addons/prometheus/rules.yaml

@@ -4,582 +4,1089 @@ metadata:
   name: prometheus-rules
   namespace: monitoring
 data:
-  alertmanager.rules.yaml: |
-    groups:
-    - name: alertmanager.rules
-      rules:
-      - alert: AlertmanagerConfigInconsistent
-        expr: count_values("config_hash", alertmanager_config_hash) BY (service) / ON(service)
-          GROUP_LEFT() label_replace(prometheus_operator_alertmanager_spec_replicas, "service",
-          "alertmanager-$1", "alertmanager", "(.*)") != 1
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          description: The configuration of the instances of the Alertmanager cluster
-            `{{$labels.service}}` are out of sync.
-      - alert: AlertmanagerDownOrMissing
-        expr: label_replace(prometheus_operator_alertmanager_spec_replicas, "job", "alertmanager-$1",
-          "alertmanager", "(.*)") / ON(job) GROUP_RIGHT() sum(up) BY (job) != 1
-        for: 5m
-        labels:
-          severity: warning
-        annotations:
-          description: An unexpected number of Alertmanagers are scraped or Alertmanagers
-            disappeared from discovery.
-      - alert: AlertmanagerFailedReload
-        expr: alertmanager_config_last_reload_successful == 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Reloading Alertmanager's configuration has failed for {{ $labels.namespace
-            }}/{{ $labels.pod}}.
-  etcd3.rules.yaml: |
-    groups:
-    - name: ./etcd3.rules
-      rules:
-      - alert: InsufficientMembers
-        expr: count(up{job="etcd"} == 0) > (count(up{job="etcd"}) / 2 - 1)
-        for: 3m
-        labels:
-          severity: critical
-        annotations:
-          description: If one more etcd member goes down the cluster will be unavailable
-          summary: etcd cluster insufficient members
-      - alert: NoLeader
-        expr: etcd_server_has_leader{job="etcd"} == 0
-        for: 1m
-        labels:
-          severity: critical
-        annotations:
-          description: etcd member {{ $labels.instance }} has no leader
-          summary: etcd member has no leader
-      - alert: HighNumberOfLeaderChanges
-        expr: increase(etcd_server_leader_changes_seen_total{job="etcd"}[1h]) > 3
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} has seen {{ $value }} leader
-            changes within the last hour
-          summary: a high number of leader changes within the etcd cluster are happening
-      - alert: GRPCRequestsSlow
-        expr: histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job="etcd",grpc_type="unary"}[5m])) by (grpc_service, grpc_method, le))
-          > 0.15
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: on etcd instance {{ $labels.instance }} gRPC requests to {{ $labels.grpc_method
-            }} are slow
-          summary: slow gRPC requests
-      - alert: HighNumberOfFailedHTTPRequests
-        expr: sum(rate(etcd_http_failed_total{job="etcd"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job="etcd"}[5m]))
-          BY (method) > 0.01
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: '{{ $value }}% of requests for {{ $labels.method }} failed on etcd
-            instance {{ $labels.instance }}'
-          summary: a high number of HTTP requests are failing
-      - alert: HighNumberOfFailedHTTPRequests
-        expr: sum(rate(etcd_http_failed_total{job="etcd"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job="etcd"}[5m]))
-          BY (method) > 0.05
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          description: '{{ $value }}% of requests for {{ $labels.method }} failed on etcd
-            instance {{ $labels.instance }}'
-          summary: a high number of HTTP requests are failing
-      - alert: HTTPRequestsSlow
-        expr: histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[5m]))
-          > 0.15
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: on etcd instance {{ $labels.instance }} HTTP requests to {{ $labels.method
-            }} are slow
-          summary: slow HTTP requests
-      - alert: EtcdMemberCommunicationSlow
-        expr: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[5m]))
-          > 0.15
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} member communication with
-            {{ $labels.To }} is slow
-          summary: etcd member communication is slow
-      - alert: HighNumberOfFailedProposals
-        expr: increase(etcd_server_proposals_failed_total{job="etcd"}[1h]) > 5
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} has seen {{ $value }} proposal
-            failures within the last hour
-          summary: a high number of proposals within the etcd cluster are failing
-      - alert: HighFsyncDurations
-        expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))
-          > 0.5
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} fync durations are high
-          summary: high fsync durations
-      - alert: HighCommitDurations
-        expr: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m]))
-          > 0.25
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} commit durations are high
-          summary: high commit durations
-  general.rules.yaml: |
-    groups:
-    - name: general.rules
-      rules:
-      - alert: TargetDown
-        expr: 100 * (count(up == 0) BY (job) / count(up) BY (job)) > 10
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: '{{ $value }}% of {{ $labels.job }} targets are down.'
-          summary: Targets are down
-      - record: fd_utilization
-        expr: process_open_fds / process_max_fds
-      - alert: FdExhaustionClose
-        expr: predict_linear(fd_utilization[1h], 3600 * 4) > 1
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: '{{ $labels.job }}: {{ $labels.namespace }}/{{ $labels.pod }} instance
-            will exhaust in file/socket descriptors within the next 4 hours'
-          summary: file descriptors soon exhausted
-      - alert: FdExhaustionClose
-        expr: predict_linear(fd_utilization[10m], 3600) > 1
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: '{{ $labels.job }}: {{ $labels.namespace }}/{{ $labels.pod }} instance
-            will exhaust in file/socket descriptors within the next hour'
-          summary: file descriptors soon exhausted
-  kube-controller-manager.rules.yaml: |
-    groups:
-    - name: kube-controller-manager.rules
-      rules:
-      - alert: K8SControllerManagerDown
-        expr: absent(up{job="kube-controller-manager"} == 1)
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          description: There is no running K8S controller manager. Deployments and replication
-            controllers are not making progress.
-          summary: Controller manager is down
-  kube-scheduler.rules.yaml: |
-    groups:
-    - name: kube-scheduler.rules
-      rules:
-      - record: cluster:scheduler_e2e_scheduling_latency_seconds:quantile
-        expr: histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.99"
-      - record: cluster:scheduler_e2e_scheduling_latency_seconds:quantile
-        expr: histogram_quantile(0.9, sum(scheduler_e2e_scheduling_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.9"
-      - record: cluster:scheduler_e2e_scheduling_latency_seconds:quantile
-        expr: histogram_quantile(0.5, sum(scheduler_e2e_scheduling_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.5"
-      - record: cluster:scheduler_scheduling_algorithm_latency_seconds:quantile
-        expr: histogram_quantile(0.99, sum(scheduler_scheduling_algorithm_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.99"
-      - record: cluster:scheduler_scheduling_algorithm_latency_seconds:quantile
-        expr: histogram_quantile(0.9, sum(scheduler_scheduling_algorithm_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.9"
-      - record: cluster:scheduler_scheduling_algorithm_latency_seconds:quantile
-        expr: histogram_quantile(0.5, sum(scheduler_scheduling_algorithm_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.5"
-      - record: cluster:scheduler_binding_latency_seconds:quantile
-        expr: histogram_quantile(0.99, sum(scheduler_binding_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.99"
-      - record: cluster:scheduler_binding_latency_seconds:quantile
-        expr: histogram_quantile(0.9, sum(scheduler_binding_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.9"
-      - record: cluster:scheduler_binding_latency_seconds:quantile
-        expr: histogram_quantile(0.5, sum(scheduler_binding_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.5"
-      - alert: K8SSchedulerDown
-        expr: absent(up{job="kube-scheduler"} == 1)
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          description: There is no running K8S scheduler. New pods are not being assigned
-            to nodes.
-          summary: Scheduler is down
-  kube-state-metrics.rules.yaml: |
-    groups:
-    - name: kube-state-metrics.rules
-      rules:
-      - alert: DeploymentGenerationMismatch
-        expr: kube_deployment_status_observed_generation != kube_deployment_metadata_generation
-        for: 15m
-        labels:
-          severity: warning
-        annotations:
-          description: Observed deployment generation does not match expected one for
-            deployment {{$labels.namespaces}}/{{$labels.deployment}}
-          summary: Deployment is outdated
-      - alert: DeploymentReplicasNotUpdated
-        expr: ((kube_deployment_status_replicas_updated != kube_deployment_spec_replicas)
-          or (kube_deployment_status_replicas_available != kube_deployment_spec_replicas))
-          unless (kube_deployment_spec_paused == 1)
-        for: 15m
-        labels:
-          severity: warning
-        annotations:
-          description: Replicas are not updated and available for deployment {{$labels.namespaces}}/{{$labels.deployment}}
-          summary: Deployment replicas are outdated
-      - alert: DaemonSetRolloutStuck
-        expr: kube_daemonset_status_number_ready / kube_daemonset_status_desired_number_scheduled
-          * 100 < 100
-        for: 15m
-        labels:
-          severity: warning
-        annotations:
-          description: Only {{$value}}% of desired pods scheduled and ready for daemon
-            set {{$labels.namespaces}}/{{$labels.daemonset}}
-          summary: DaemonSet is missing pods
-      - alert: K8SDaemonSetsNotScheduled
-        expr: kube_daemonset_status_desired_number_scheduled - kube_daemonset_status_current_number_scheduled
-          > 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: A number of daemonsets are not scheduled.
-          summary: Daemonsets are not scheduled correctly
-      - alert: DaemonSetsMissScheduled
-        expr: kube_daemonset_status_number_misscheduled > 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: A number of daemonsets are running where they are not supposed
-            to run.
-          summary: Daemonsets are not scheduled correctly
-      - alert: PodFrequentlyRestarting
-        expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Pod {{$labels.namespaces}}/{{$labels.pod}} restarted {{$value}}
-            times within the last hour
-          summary: Pod is restarting frequently
-  kubelet.rules.yaml: |
-    groups:
-    - name: kubelet.rules
-      rules:
-      - alert: K8SNodeNotReady
-        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
-        for: 1h
-        labels:
-          severity: warning
-        annotations:
-          description: The Kubelet on {{ $labels.node }} has not checked in with the API,
-            or has set itself to NotReady, for more than an hour
-          summary: Node status is NotReady
-      - alert: K8SManyNodesNotReady
-        expr: count(kube_node_status_condition{condition="Ready",status="true"} == 0)
-          > 1 and (count(kube_node_status_condition{condition="Ready",status="true"} ==
-          0) / count(kube_node_status_condition{condition="Ready",status="true"})) > 0.2
-        for: 1m
-        labels:
-          severity: critical
-        annotations:
-          description: '{{ $value }}% of Kubernetes nodes are not ready'
-      - alert: K8SKubeletDown
-        expr: count(up{job="kubelet"} == 0) / count(up{job="kubelet"}) * 100 > 3
-        for: 1h
-        labels:
-          severity: warning
-        annotations:
-          description: Prometheus failed to scrape {{ $value }}% of kubelets.
-      - alert: K8SKubeletDown
-        expr: (absent(up{job="kubelet"} == 1) or count(up{job="kubelet"} == 0) / count(up{job="kubelet"}))
-          * 100 > 10
-        for: 1h
-        labels:
-          severity: critical
-        annotations:
-          description: Prometheus failed to scrape {{ $value }}% of kubelets, or all Kubelets
-            have disappeared from service discovery.
-          summary: Many Kubelets cannot be scraped
-      - alert: K8SKubeletTooManyPods
-        expr: kubelet_running_pod_count > 100
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Kubelet {{$labels.instance}} is running {{$value}} pods, close
-            to the limit of 110
-          summary: Kubelet is close to pod limit
-  kubernetes.rules.yaml: |
-    groups:
-    - name: kubernetes.rules
-      rules:
-      - record: pod_name:container_memory_usage_bytes:sum
-        expr: sum(container_memory_usage_bytes{container_name!="POD",pod_name!=""}) BY
-          (pod_name)
-      - record: pod_name:container_spec_cpu_shares:sum
-        expr: sum(container_spec_cpu_shares{container_name!="POD",pod_name!=""}) BY (pod_name)
-      - record: pod_name:container_cpu_usage:sum
-        expr: sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m]))
-          BY (pod_name)
-      - record: pod_name:container_fs_usage_bytes:sum
-        expr: sum(container_fs_usage_bytes{container_name!="POD",pod_name!=""}) BY (pod_name)
-      - record: namespace:container_memory_usage_bytes:sum
-        expr: sum(container_memory_usage_bytes{container_name!=""}) BY (namespace)
-      - record: namespace:container_spec_cpu_shares:sum
-        expr: sum(container_spec_cpu_shares{container_name!=""}) BY (namespace)
-      - record: namespace:container_cpu_usage:sum
-        expr: sum(rate(container_cpu_usage_seconds_total{container_name!="POD"}[5m]))
-          BY (namespace)
-      - record: cluster:memory_usage:ratio
-        expr: sum(container_memory_usage_bytes{container_name!="POD",pod_name!=""}) BY
-          (cluster) / sum(machine_memory_bytes) BY (cluster)
-      - record: cluster:container_spec_cpu_shares:ratio
-        expr: sum(container_spec_cpu_shares{container_name!="POD",pod_name!=""}) / 1000
-          / sum(machine_cpu_cores)
-      - record: cluster:container_cpu_usage:ratio
-        expr: sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m]))
-          / sum(machine_cpu_cores)
-      - record: apiserver_latency_seconds:quantile
-        expr: histogram_quantile(0.99, rate(apiserver_request_latencies_bucket[5m])) /
-          1e+06
-        labels:
-          quantile: "0.99"
-      - record: apiserver_latency:quantile_seconds
-        expr: histogram_quantile(0.9, rate(apiserver_request_latencies_bucket[5m])) /
-          1e+06
-        labels:
-          quantile: "0.9"
-      - record: apiserver_latency_seconds:quantile
-        expr: histogram_quantile(0.5, rate(apiserver_request_latencies_bucket[5m])) /
-          1e+06
-        labels:
-          quantile: "0.5"
-      - alert: APIServerLatencyHigh
-        expr: apiserver_latency_seconds:quantile{quantile="0.99",subresource!="log",verb!~"^(?:WATCH|WATCHLIST|PROXY|CONNECT)$"}
-          > 1
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: the API server has a 99th percentile latency of {{ $value }} seconds
-            for {{$labels.verb}} {{$labels.resource}}
-      - alert: APIServerLatencyHigh
-        expr: apiserver_latency_seconds:quantile{quantile="0.99",subresource!="log",verb!~"^(?:WATCH|WATCHLIST|PROXY|CONNECT)$"}
-          > 4
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: the API server has a 99th percentile latency of {{ $value }} seconds
-            for {{$labels.verb}} {{$labels.resource}}
-      - alert: APIServerErrorsHigh
-        expr: rate(apiserver_request_count{code=~"^(?:5..)$"}[5m]) / rate(apiserver_request_count[5m])
-          * 100 > 2
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: API server returns errors for {{ $value }}% of requests
-      - alert: APIServerErrorsHigh
-        expr: rate(apiserver_request_count{code=~"^(?:5..)$"}[5m]) / rate(apiserver_request_count[5m])
-          * 100 > 5
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: API server returns errors for {{ $value }}% of requests
-      - alert: K8SApiserverDown
-        expr: absent(up{job="apiserver"} == 1)
-        for: 20m
-        labels:
-          severity: critical
-        annotations:
-          description: No API servers are reachable or all have disappeared from service
-            discovery
-
-      - alert: K8sCertificateExpirationNotice
-        labels:
-          severity: warning
-        annotations:
-          description: Kubernetes API Certificate is expiring soon (less than 7 days)
-        expr: sum(apiserver_client_certificate_expiration_seconds_bucket{le="604800"}) > 0
-
-      - alert: K8sCertificateExpirationNotice
-        labels:
-          severity: critical
-        annotations:
-          description: Kubernetes API Certificate is expiring in less than 1 day
-        expr: sum(apiserver_client_certificate_expiration_seconds_bucket{le="86400"}) > 0
-  node.rules.yaml: |
-    groups:
-    - name: node.rules
-      rules:
-      - record: instance:node_cpu:rate:sum
-        expr: sum(rate(node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!~"^(?:guest.*)$"}[3m]))
-          BY (instance)
-      - record: instance:node_filesystem_usage:sum
-        expr: sum((node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_free_bytes{mountpoint="/"}))
-          BY (instance)
-      - record: instance:node_network_receive_bytes:rate:sum
-        expr: sum(rate(node_network_receive_bytes_total[3m])) BY (instance)
-      - record: instance:node_network_transmit_bytes:rate:sum
-        expr: sum(rate(node_network_transmit_bytes_total[3m])) BY (instance)
-      - record: instance:node_cpu:ratio
-        expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) WITHOUT (cpu, mode) / ON(instance)
-          GROUP_LEFT() count(sum(node_cpu_seconds_total) BY (instance, cpu)) BY (instance)
-      - record: cluster:node_cpu:sum_rate5m
-        expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m]))
-      - record: cluster:node_cpu:ratio
-        expr: cluster:node_cpu:sum_rate5m / count(sum(node_cpu_seconds_total) BY (instance, cpu))
-      - alert: NodeExporterDown
-        expr: absent(up{job="node-exporter"} == 1)
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Prometheus could not scrape a node-exporter for more than 10m,
-            or node-exporters have disappeared from discovery
-      - alert: NodeDiskRunningFull
-        expr: predict_linear(node_filesystem_free_bytes[6h], 3600 * 24) < 0
-        for: 30m
-        labels:
-          severity: warning
-        annotations:
-          description: device {{$labels.device}} on node {{$labels.instance}} is running
-            full within the next 24 hours (mounted at {{$labels.mountpoint}})
-      - alert: NodeDiskRunningFull
-        expr: predict_linear(node_filesystem_free_bytes[30m], 3600 * 2) < 0
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: device {{$labels.device}} on node {{$labels.instance}} is running
-            full within the next 2 hours (mounted at {{$labels.mountpoint}})
-      - alert: InactiveRAIDDisk
-        expr: node_md_disks - node_md_disks_active > 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: '{{$value}} RAID disk(s) on node {{$labels.instance}} are inactive'
-  prometheus.rules.yaml: |
-    groups:
-    - name: prometheus.rules
-      rules:
-      - alert: PrometheusConfigReloadFailed
-        expr: prometheus_config_last_reload_successful == 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Reloading Prometheus' configuration has failed for {{$labels.namespace}}/{{$labels.pod}}
-      - alert: PrometheusNotificationQueueRunningFull
-        expr: predict_linear(prometheus_notifications_queue_length[5m], 60 * 30) > prometheus_notifications_queue_capacity
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Prometheus' alert notification queue is running full for {{$labels.namespace}}/{{
-            $labels.pod}}
-      - alert: PrometheusErrorSendingAlerts
-        expr: rate(prometheus_notifications_errors_total[5m]) / rate(prometheus_notifications_sent_total[5m])
-          > 0.01
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Errors while sending alerts from Prometheus {{$labels.namespace}}/{{
-            $labels.pod}} to Alertmanager {{$labels.Alertmanager}}
-      - alert: PrometheusErrorSendingAlerts
-        expr: rate(prometheus_notifications_errors_total[5m]) / rate(prometheus_notifications_sent_total[5m])
-          > 0.03
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: Errors while sending alerts from Prometheus {{$labels.namespace}}/{{
-            $labels.pod}} to Alertmanager {{$labels.Alertmanager}}
-      - alert: PrometheusNotConnectedToAlertmanagers
-        expr: prometheus_notifications_alertmanagers_discovered < 1
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Prometheus {{ $labels.namespace }}/{{ $labels.pod}} is not connected
-            to any Alertmanagers
-      - alert: PrometheusTSDBReloadsFailing
-        expr: increase(prometheus_tsdb_reloads_failures_total[2h]) > 0
-        for: 12h
-        labels:
-          severity: warning
-        annotations:
-          description: '{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}}
-            reload failures over the last four hours.'
-          summary: Prometheus has issues reloading data blocks from disk
-      - alert: PrometheusTSDBCompactionsFailing
-        expr: increase(prometheus_tsdb_compactions_failed_total[2h]) > 0
-        for: 12h
-        labels:
-          severity: warning
-        annotations:
-          description: '{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}}
-            compaction failures over the last four hours.'
-          summary: Prometheus has issues compacting sample blocks
-      - alert: PrometheusTSDBWALCorruptions
-        expr: tsdb_wal_corruptions_total > 0
-        for: 4h
-        labels:
-          severity: warning
-        annotations:
-          description: '{{$labels.job}} at {{$labels.instance}} has a corrupted write-ahead
-            log (WAL).'
-          summary: Prometheus write-ahead log is corrupted
-      - alert: PrometheusNotIngestingSamples
-        expr: rate(prometheus_tsdb_head_samples_appended_total[5m]) <= 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: "Prometheus {{ $labels.namespace }}/{{ $labels.pod}} isn't ingesting samples."
-          summary: "Prometheus isn't ingesting samples"
+  etcd.yaml: |-
+    {
+      "groups": [
+        {
+          "name": "etcd",
+          "rules": [
+            {
+              "alert": "etcdInsufficientMembers",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": insufficient members ({{ $value }})."
+              },
+              "expr": "sum(up{job=~\".*etcd.*\"} == bool 1) by (job) < ((count(up{job=~\".*etcd.*\"}) by (job) + 1) / 2)\n",
+              "for": "3m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "etcdNoLeader",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader."
+              },
+              "expr": "etcd_server_has_leader{job=~\".*etcd.*\"} == 0\n",
+              "for": "1m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "etcdHighNumberOfLeaderChanges",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": instance {{ $labels.instance }} has seen {{ $value }} leader changes within the last 30 minutes."
+              },
+              "expr": "rate(etcd_server_leader_changes_seen_total{job=~\".*etcd.*\"}[15m]) > 3\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdGRPCRequestsSlow",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": gRPC requests to {{ $labels.grpc_method }} are taking {{ $value }}s on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~\".*etcd.*\", grpc_type=\"unary\"}[5m])) by (job, instance, grpc_service, grpc_method, le))\n> 0.15\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "etcdMemberCommunicationSlow",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 0.15\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighNumberOfFailedProposals",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) > 5\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighFsyncDurations",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": 99th percentile fync durations are {{ $value }}s on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 0.5\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighCommitDurations",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 0.25\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighNumberOfFailedHTTPRequests",
+              "annotations": {
+                "message": "{{ $value }}% of requests for {{ $labels.method }} failed on etcd instance {{ $labels.instance }}"
+              },
+              "expr": "sum(rate(etcd_http_failed_total{job=~\".*etcd.*\", code!=\"404\"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job=~\".*etcd.*\"}[5m]))\nBY (method) > 0.01\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighNumberOfFailedHTTPRequests",
+              "annotations": {
+                "message": "{{ $value }}% of requests for {{ $labels.method }} failed on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "sum(rate(etcd_http_failed_total{job=~\".*etcd.*\", code!=\"404\"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job=~\".*etcd.*\"}[5m]))\nBY (method) > 0.05\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "etcdHTTPRequestsSlow",
+              "annotations": {
+                "message": "etcd instance {{ $labels.instance }} HTTP requests to {{ $labels.method }} are slow."
+              },
+              "expr": "histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[5m]))\n> 0.15\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        }
+      ]
+    }
+  extra.yaml: |-
+    {
+      "groups": [
+        {
+          "name": "extra.rules",
+          "rules": [
+            {
+              "alert": "InactiveRAIDDisk",
+              "annotations": {
+                "message": "{{ $value }} RAID disk(s) on node {{ $labels.instance }} are inactive."
+              },
+              "expr": "node_md_disks - node_md_disks_active > 0",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        }
+      ]
+    }
+  kube.yaml: |-
+    {
+      "groups": [
+        {
+          "name": "k8s.rules",
+          "rules": [
+            {
+              "expr": "sum(rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\", container_name!=\"\"}[5m])) by (namespace)\n",
+              "record": "namespace:container_cpu_usage_seconds_total:sum_rate"
+            },
+            {
+              "expr": "sum by (namespace, pod_name, container_name) (\n  rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\", container_name!=\"\"}[5m])\n)\n",
+              "record": "namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate"
+            },
+            {
+              "expr": "sum(container_memory_usage_bytes{job=\"kubernetes-cadvisor\", image!=\"\", container_name!=\"\"}) by (namespace)\n",
+              "record": "namespace:container_memory_usage_bytes:sum"
+            },
+            {
+              "expr": "sum by (namespace, label_name) (\n   sum(rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\", container_name!=\"\"}[5m])) by (namespace, pod_name)\n * on (namespace, pod_name) group_left(label_name)\n   label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "record": "namespace_name:container_cpu_usage_seconds_total:sum_rate"
+            },
+            {
+              "expr": "sum by (namespace, label_name) (\n  sum(container_memory_usage_bytes{job=\"kubernetes-cadvisor\",image!=\"\", container_name!=\"\"}) by (pod_name, namespace)\n* on (namespace, pod_name) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "record": "namespace_name:container_memory_usage_bytes:sum"
+            },
+            {
+              "expr": "sum by (namespace, label_name) (\n  sum(kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"}) by (namespace, pod)\n* on (namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "record": "namespace_name:kube_pod_container_resource_requests_memory_bytes:sum"
+            },
+            {
+              "expr": "sum by (namespace, label_name) (\n  sum(kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"} and on(pod) kube_pod_status_scheduled{condition=\"true\"}) by (namespace, pod)\n* on (namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "record": "namespace_name:kube_pod_container_resource_requests_cpu_cores:sum"
+            }
+          ]
+        },
+        {
+          "name": "kube-scheduler.rules",
+          "rules": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(scheduler_e2e_scheduling_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.99"
+              },
+              "record": "cluster_quantile:scheduler_e2e_scheduling_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(scheduler_scheduling_algorithm_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.99"
+              },
+              "record": "cluster_quantile:scheduler_scheduling_algorithm_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(scheduler_binding_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.99"
+              },
+              "record": "cluster_quantile:scheduler_binding_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.9, sum(rate(scheduler_e2e_scheduling_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.9"
+              },
+              "record": "cluster_quantile:scheduler_e2e_scheduling_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.9, sum(rate(scheduler_scheduling_algorithm_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.9"
+              },
+              "record": "cluster_quantile:scheduler_scheduling_algorithm_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.9, sum(rate(scheduler_binding_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.9"
+              },
+              "record": "cluster_quantile:scheduler_binding_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.5, sum(rate(scheduler_e2e_scheduling_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.5"
+              },
+              "record": "cluster_quantile:scheduler_e2e_scheduling_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.5, sum(rate(scheduler_scheduling_algorithm_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.5"
+              },
+              "record": "cluster_quantile:scheduler_scheduling_algorithm_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.5, sum(rate(scheduler_binding_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.5"
+              },
+              "record": "cluster_quantile:scheduler_binding_latency:histogram_quantile"
+            }
+          ]
+        },
+        {
+          "name": "kube-apiserver.rules",
+          "rules": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(apiserver_request_latencies_bucket{job=\"apiserver\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.99"
+              },
+              "record": "cluster_quantile:apiserver_request_latencies:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.9, sum(rate(apiserver_request_latencies_bucket{job=\"apiserver\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.9"
+              },
+              "record": "cluster_quantile:apiserver_request_latencies:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.5, sum(rate(apiserver_request_latencies_bucket{job=\"apiserver\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.5"
+              },
+              "record": "cluster_quantile:apiserver_request_latencies:histogram_quantile"
+            }
+          ]
+        },
+        {
+          "name": "node.rules",
+          "rules": [
+            {
+              "expr": "sum(min(kube_pod_info) by (node))",
+              "record": ":kube_pod_info_node_count:"
+            },
+            {
+              "expr": "max(label_replace(kube_pod_info{job=\"kube-state-metrics\"}, \"pod\", \"$1\", \"pod\", \"(.*)\")) by (node, namespace, pod)\n",
+              "record": "node_namespace_pod:kube_pod_info:"
+            },
+            {
+              "expr": "count by (node) (sum by (node, cpu) (\n  node_cpu_seconds_total{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n))\n",
+              "record": "node:node_num_cpu:sum"
+            },
+            {
+              "expr": "1 - avg(rate(node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}[1m]))\n",
+              "record": ":node_cpu_utilisation:avg1m"
+            },
+            {
+              "expr": "1 - avg by (node) (\n  rate(node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}[1m])\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:)\n",
+              "record": "node:node_cpu_utilisation:avg1m"
+            },
+            {
+              "expr": "node:node_cpu_utilisation:avg1m\n  *\nnode:node_num_cpu:sum\n  /\nscalar(sum(node:node_num_cpu:sum))\n",
+              "record": "node:cluster_cpu_utilisation:ratio"
+            },
+            {
+              "expr": "sum(node_load1{job=\"node-exporter\"})\n/\nsum(node:node_num_cpu:sum)\n",
+              "record": ":node_cpu_saturation_load1:"
+            },
+            {
+              "expr": "sum by (node) (\n  node_load1{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n/\nnode:node_num_cpu:sum\n",
+              "record": "node:node_cpu_saturation_load1:"
+            },
+            {
+              "expr": "1 -\nsum(node_memory_MemFree_bytes{job=\"node-exporter\"} + node_memory_Cached_bytes{job=\"node-exporter\"} + node_memory_Buffers_bytes{job=\"node-exporter\"})\n/\nsum(node_memory_MemTotal_bytes{job=\"node-exporter\"})\n",
+              "record": ":node_memory_utilisation:"
+            },
+            {
+              "expr": "sum(node_memory_MemFree_bytes{job=\"node-exporter\"} + node_memory_Cached_bytes{job=\"node-exporter\"} + node_memory_Buffers_bytes{job=\"node-exporter\"})\n",
+              "record": ":node_memory_MemFreeCachedBuffers_bytes:sum"
+            },
+            {
+              "expr": "sum(node_memory_MemTotal_bytes{job=\"node-exporter\"})\n",
+              "record": ":node_memory_MemTotal_bytes:sum"
+            },
+            {
+              "expr": "sum by (node) (\n  (node_memory_MemFree_bytes{job=\"node-exporter\"} + node_memory_Cached_bytes{job=\"node-exporter\"} + node_memory_Buffers_bytes{job=\"node-exporter\"})\n  * on (namespace, pod) group_left(node)\n    node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_memory_bytes_available:sum"
+            },
+            {
+              "expr": "sum by (node) (\n  node_memory_MemTotal_bytes{job=\"node-exporter\"}\n  * on (namespace, pod) group_left(node)\n    node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_memory_bytes_total:sum"
+            },
+            {
+              "expr": "(node:node_memory_bytes_total:sum - node:node_memory_bytes_available:sum)\n/\nnode:node_memory_bytes_total:sum\n",
+              "record": "node:node_memory_utilisation:ratio"
+            },
+            {
+              "expr": "(node:node_memory_bytes_total:sum - node:node_memory_bytes_available:sum)\n/\nscalar(sum(node:node_memory_bytes_total:sum))\n",
+              "record": "node:cluster_memory_utilisation:ratio"
+            },
+            {
+              "expr": "1e3 * sum(\n  (rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m])\n + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n)\n",
+              "record": ":node_memory_swap_io_bytes:sum_rate"
+            },
+            {
+              "expr": "1 -\nsum by (node) (\n  (node_memory_MemFree_bytes{job=\"node-exporter\"} + node_memory_Cached_bytes{job=\"node-exporter\"} + node_memory_Buffers_bytes{job=\"node-exporter\"})\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n/\nsum by (node) (\n  node_memory_MemTotal_bytes{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_memory_utilisation:"
+            },
+            {
+              "expr": "1 - (node:node_memory_bytes_available:sum / node:node_memory_bytes_total:sum)\n",
+              "record": "node:node_memory_utilisation_2:"
+            },
+            {
+              "expr": "1e3 * sum by (node) (\n  (rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m])\n + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n * on (namespace, pod) group_left(node)\n   node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_memory_swap_io_bytes:sum_rate"
+            },
+            {
+              "expr": "avg(irate(node_disk_io_time_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m]))\n",
+              "record": ":node_disk_utilisation:avg_irate"
+            },
+            {
+              "expr": "avg by (node) (\n  irate(node_disk_io_time_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m])\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_disk_utilisation:avg_irate"
+            },
+            {
+              "expr": "avg(irate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m]) / 1e3)\n",
+              "record": ":node_disk_saturation:avg_irate"
+            },
+            {
+              "expr": "avg by (node) (\n  irate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m]) / 1e3\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_disk_saturation:avg_irate"
+            },
+            {
+              "expr": "max by (namespace, pod, device) ((node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"}\n- node_filesystem_avail_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"})\n/ node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"})\n",
+              "record": "node:node_filesystem_usage:"
+            },
+            {
+              "expr": "max by (namespace, pod, device) (node_filesystem_avail_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"} / node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"})\n",
+              "record": "node:node_filesystem_avail:"
+            },
+            {
+              "expr": "sum(irate(node_network_receive_bytes_total{job=\"node-exporter\",device!~\"veth.+\"}[1m])) +\nsum(irate(node_network_transmit_bytes_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]))\n",
+              "record": ":node_net_utilisation:sum_irate"
+            },
+            {
+              "expr": "sum by (node) (\n  (irate(node_network_receive_bytes_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]) +\n  irate(node_network_transmit_bytes_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]))\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_net_utilisation:sum_irate"
+            },
+            {
+              "expr": "sum(irate(node_network_receive_drop_total{job=\"node-exporter\",device!~\"veth.+\"}[1m])) +\nsum(irate(node_network_transmit_drop_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]))\n",
+              "record": ":node_net_saturation:sum_irate"
+            },
+            {
+              "expr": "sum by (node) (\n  (irate(node_network_receive_drop_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]) +\n  irate(node_network_transmit_drop_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]))\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_net_saturation:sum_irate"
+            },
+            {
+              "expr": "max(\n  max(\n    kube_pod_info{job=\"kube-state-metrics\", host_ip!=\"\"}\n  ) by (node, host_ip)\n  * on (host_ip) group_right (node)\n  label_replace(\n    (max(node_filesystem_files{job=\"node-exporter\", mountpoint=\"/\"}) by (instance)), \"host_ip\", \"$1\", \"instance\", \"(.*):.*\"\n  )\n) by (node)\n",
+              "record": "node:node_inodes_total:"
+            },
+            {
+              "expr": "max(\n  max(\n    kube_pod_info{job=\"kube-state-metrics\", host_ip!=\"\"}\n  ) by (node, host_ip)\n  * on (host_ip) group_right (node)\n  label_replace(\n    (max(node_filesystem_files_free{job=\"node-exporter\", mountpoint=\"/\"}) by (instance)), \"host_ip\", \"$1\", \"instance\", \"(.*):.*\"\n  )\n) by (node)\n",
+              "record": "node:node_inodes_free:"
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-absent",
+          "rules": [
+            {
+              "alert": "KubeAPIDown",
+              "annotations": {
+                "message": "KubeAPI has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapidown"
+              },
+              "expr": "absent(up{job=\"apiserver\"} == 1)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeControllerManagerDown",
+              "annotations": {
+                "message": "KubeControllerManager has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontrollermanagerdown"
+              },
+              "expr": "absent(up{job=\"kube-controller-manager\"} == 1)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeSchedulerDown",
+              "annotations": {
+                "message": "KubeScheduler has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeschedulerdown"
+              },
+              "expr": "absent(up{job=\"kube-scheduler\"} == 1)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeletDown",
+              "annotations": {
+                "message": "Kubelet has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletdown"
+              },
+              "expr": "absent(up{job=\"kubelet\"} == 1)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-apps",
+          "rules": [
+            {
+              "alert": "KubePodCrashLooping",
+              "annotations": {
+                "message": "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is restarting {{ printf \"%.2f\" $value }} times / 5 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping"
+              },
+              "expr": "rate(kube_pod_container_status_restarts_total{job=\"kube-state-metrics\"}[15m]) * 60 * 5 > 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubePodNotReady",
+              "annotations": {
+                "message": "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than an hour.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready"
+              },
+              "expr": "sum by (namespace, pod) (kube_pod_status_phase{job=\"kube-state-metrics\", phase=~\"Pending|Unknown\"}) > 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeDeploymentGenerationMismatch",
+              "annotations": {
+                "message": "Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentgenerationmismatch"
+              },
+              "expr": "kube_deployment_status_observed_generation{job=\"kube-state-metrics\"}\n  !=\nkube_deployment_metadata_generation{job=\"kube-state-metrics\"}\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeDeploymentReplicasMismatch",
+              "annotations": {
+                "message": "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than an hour.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch"
+              },
+              "expr": "kube_deployment_spec_replicas{job=\"kube-state-metrics\"}\n  !=\nkube_deployment_status_replicas_available{job=\"kube-state-metrics\"}\n",
+              "for": "1h",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeStatefulSetReplicasMismatch",
+              "annotations": {
+                "message": "StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetreplicasmismatch"
+              },
+              "expr": "kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\"}\n  !=\nkube_statefulset_status_replicas{job=\"kube-state-metrics\"}\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeStatefulSetGenerationMismatch",
+              "annotations": {
+                "message": "StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetgenerationmismatch"
+              },
+              "expr": "kube_statefulset_status_observed_generation{job=\"kube-state-metrics\"}\n  !=\nkube_statefulset_metadata_generation{job=\"kube-state-metrics\"}\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeStatefulSetUpdateNotRolledOut",
+              "annotations": {
+                "message": "StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetupdatenotrolledout"
+              },
+              "expr": "max without (revision) (\n  kube_statefulset_status_current_revision{job=\"kube-state-metrics\"}\n    unless\n  kube_statefulset_status_update_revision{job=\"kube-state-metrics\"}\n)\n  *\n(\n  kube_statefulset_replicas{job=\"kube-state-metrics\"}\n    !=\n  kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\"}\n)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeDaemonSetRolloutStuck",
+              "annotations": {
+                "message": "Only {{ $value }}% of the desired Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are scheduled and ready.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetrolloutstuck"
+              },
+              "expr": "kube_daemonset_status_number_ready{job=\"kube-state-metrics\"}\n  /\nkube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"} * 100 < 100\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeDaemonSetNotScheduled",
+              "annotations": {
+                "message": "{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetnotscheduled"
+              },
+              "expr": "kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  -\nkube_daemonset_status_current_number_scheduled{job=\"kube-state-metrics\"} > 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeDaemonSetMisScheduled",
+              "annotations": {
+                "message": "{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetmisscheduled"
+              },
+              "expr": "kube_daemonset_status_number_misscheduled{job=\"kube-state-metrics\"} > 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeCronJobRunning",
+              "annotations": {
+                "message": "CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} is taking more than 1h to complete.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecronjobrunning"
+              },
+              "expr": "time() - kube_cronjob_next_schedule_time{job=\"kube-state-metrics\"} > 3600\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeJobCompletion",
+              "annotations": {
+                "message": "Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than one hour to complete.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobcompletion"
+              },
+              "expr": "kube_job_spec_completions{job=\"kube-state-metrics\"} - kube_job_status_succeeded{job=\"kube-state-metrics\"}  > 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeJobFailed",
+              "annotations": {
+                "message": "Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobfailed"
+              },
+              "expr": "kube_job_status_failed{job=\"kube-state-metrics\"}  > 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-resources",
+          "rules": [
+            {
+              "alert": "KubeCPUOvercommit",
+              "annotations": {
+                "message": "Cluster has overcommitted CPU resource requests for Pods and cannot tolerate node failure.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit"
+              },
+              "expr": "sum(namespace_name:kube_pod_container_resource_requests_cpu_cores:sum)\n  /\nsum(node:node_num_cpu:sum)\n  >\n(count(node:node_num_cpu:sum)-1) / count(node:node_num_cpu:sum)\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeMemOvercommit",
+              "annotations": {
+                "message": "Cluster has overcommitted memory resource requests for Pods and cannot tolerate node failure.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememovercommit"
+              },
+              "expr": "sum(namespace_name:kube_pod_container_resource_requests_memory_bytes:sum)\n  /\nsum(node_memory_MemTotal_bytes)\n  >\n(count(node:node_num_cpu:sum)-1)\n  /\ncount(node:node_num_cpu:sum)\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeCPUOvercommit",
+              "annotations": {
+                "message": "Cluster has overcommitted CPU resource requests for Namespaces.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit"
+              },
+              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"requests.cpu\"})\n  /\nsum(node:node_num_cpu:sum)\n  > 1.5\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeMemOvercommit",
+              "annotations": {
+                "message": "Cluster has overcommitted memory resource requests for Namespaces.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememovercommit"
+              },
+              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"requests.memory\"})\n  /\nsum(node_memory_MemTotal_bytes{job=\"node-exporter\"})\n  > 1.5\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeQuotaExceeded",
+              "annotations": {
+                "message": "Namespace {{ $labels.namespace }} is using {{ printf \"%0.0f\" $value }}% of its {{ $labels.resource }} quota.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubequotaexceeded"
+              },
+              "expr": "100 * kube_resourcequota{job=\"kube-state-metrics\", type=\"used\"}\n  / ignoring(instance, job, type)\n(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\"} > 0)\n  > 90\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "CPUThrottlingHigh",
+              "annotations": {
+                "message": "{{ printf \"%0.0f\" $value }}% throttling of CPU in namespace {{ $labels.namespace }} for container {{ $labels.container_name }} in pod {{ $labels.pod_name }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-cputhrottlinghigh"
+              },
+              "expr": "100 * sum(increase(container_cpu_cfs_throttled_periods_total{container_name!=\"\", }[5m])) by (container_name, pod_name, namespace)\n  /\nsum(increase(container_cpu_cfs_periods_total{}[5m])) by (container_name, pod_name, namespace)\n  > 100 \n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-storage",
+          "rules": [
+            {
+              "alert": "KubePersistentVolumeUsageCritical",
+              "annotations": {
+                "message": "The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ printf \"%0.2f\" $value }}% free.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumeusagecritical"
+              },
+              "expr": "100 * kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n  /\nkubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n  < 3\n",
+              "for": "1m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubePersistentVolumeFullInFourDays",
+              "annotations": {
+                "message": "Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to fill up within four days. Currently {{ printf \"%0.2f\" $value }}% is available.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumefullinfourdays"
+              },
+              "expr": "100 * (\n  kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n    /\n  kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n) < 15\nand\npredict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\"}[6h], 4 * 24 * 3600) < 0\n",
+              "for": "5m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubePersistentVolumeErrors",
+              "annotations": {
+                "message": "The persistent volume {{ $labels.persistentvolume }} has status {{ $labels.phase }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumeerrors"
+              },
+              "expr": "kube_persistentvolume_status_phase{phase=~\"Failed|Pending\",job=\"kube-state-metrics\"} > 0\n",
+              "for": "5m",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-system",
+          "rules": [
+            {
+              "alert": "KubeNodeNotReady",
+              "annotations": {
+                "message": "{{ $labels.node }} has been unready for more than an hour.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubenodenotready"
+              },
+              "expr": "kube_node_status_condition{job=\"kube-state-metrics\",condition=\"Ready\",status=\"true\"} == 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeVersionMismatch",
+              "annotations": {
+                "message": "There are {{ $value }} different semantic versions of Kubernetes components running.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeversionmismatch"
+              },
+              "expr": "count(count by (gitVersion) (label_replace(kubernetes_build_info{job!=\"coredns\"},\"gitVersion\",\"$1\",\"gitVersion\",\"(v[0-9]*.[0-9]*.[0-9]*).*\"))) > 1\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeClientErrors",
+              "annotations": {
+                "message": "Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ printf \"%0.0f\" $value }}% errors.'",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclienterrors"
+              },
+              "expr": "(sum(rate(rest_client_requests_total{code=~\"5..\"}[5m])) by (instance, job)\n  /\nsum(rate(rest_client_requests_total[5m])) by (instance, job))\n* 100 > 1\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeClientErrors",
+              "annotations": {
+                "message": "Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ printf \"%0.0f\" $value }} errors / second.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclienterrors"
+              },
+              "expr": "sum(rate(ksm_scrape_error_total{job=\"kube-state-metrics\"}[5m])) by (instance, job) > 0.1\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeletTooManyPods",
+              "annotations": {
+                "message": "Kubelet {{ $labels.instance }} is running {{ $value }} Pods, close to the limit of 110.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods"
+              },
+              "expr": "kubelet_running_pod_count{job=\"kubelet\"} > 110 * 0.9\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeAPILatencyHigh",
+              "annotations": {
+                "message": "The API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh"
+              },
+              "expr": "cluster_quantile:apiserver_request_latencies:histogram_quantile{job=\"apiserver\",quantile=\"0.99\",subresource!=\"log\",verb!~\"^(?:LIST|WATCH|WATCHLIST|PROXY|CONNECT)$\"} > 1\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeAPILatencyHigh",
+              "annotations": {
+                "message": "The API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh"
+              },
+              "expr": "cluster_quantile:apiserver_request_latencies:histogram_quantile{job=\"apiserver\",quantile=\"0.99\",subresource!=\"log\",verb!~\"^(?:LIST|WATCH|WATCHLIST|PROXY|CONNECT)$\"} > 4\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeAPIErrorsHigh",
+              "annotations": {
+                "message": "API server is returning errors for {{ $value }}% of requests.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
+              },
+              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m])) without(instance, pod)\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) without(instance, pod) * 100 > 10\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeAPIErrorsHigh",
+              "annotations": {
+                "message": "API server is returning errors for {{ $value }}% of requests.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
+              },
+              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m])) without(instance, pod)\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) without(instance, pod) * 100 > 5\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeClientCertificateExpiration",
+              "annotations": {
+                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 7 days.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
+              },
+              "expr": "histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 604800\n",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeClientCertificateExpiration",
+              "annotations": {
+                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 24 hours.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
+              },
+              "expr": "histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 86400\n",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
+        }
+      ]
+    }
+  kubeprom.yaml: |-
+    {
+      "groups": [
+        {
+          "name": "kube-prometheus-node-recording.rules",
+          "rules": [
+            {
+              "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\",mode!=\"iowait\"}[3m])) BY (instance)",
+              "record": "instance:node_cpu:rate:sum"
+            },
+            {
+              "expr": "sum((node_filesystem_size_bytes{mountpoint=\"/\"} - node_filesystem_free_bytes{mountpoint=\"/\"})) BY (instance)",
+              "record": "instance:node_filesystem_usage:sum"
+            },
+            {
+              "expr": "sum(rate(node_network_receive_bytes_total[3m])) BY (instance)",
+              "record": "instance:node_network_receive_bytes:rate:sum"
+            },
+            {
+              "expr": "sum(rate(node_network_transmit_bytes_total[3m])) BY (instance)",
+              "record": "instance:node_network_transmit_bytes:rate:sum"
+            },
+            {
+              "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\",mode!=\"iowait\"}[5m])) WITHOUT (cpu, mode) / ON(instance) GROUP_LEFT() count(sum(node_cpu_seconds_total) BY (instance, cpu)) BY (instance)",
+              "record": "instance:node_cpu:ratio"
+            },
+            {
+              "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\",mode!=\"iowait\"}[5m]))",
+              "record": "cluster:node_cpu:sum_rate5m"
+            },
+            {
+              "expr": "cluster:node_cpu_seconds_total:rate5m / count(sum(node_cpu_seconds_total) BY (instance, cpu))",
+              "record": "cluster:node_cpu:ratio"
+            }
+          ]
+        },
+        {
+          "name": "kube-prometheus-node-alerting.rules",
+          "rules": [
+            {
+              "alert": "NodeDiskRunningFull",
+              "annotations": {
+                "message": "Device {{ $labels.device }} of node-exporter {{ $labels.namespace }}/{{ $labels.pod }} will be full within the next 24 hours."
+              },
+              "expr": "(node:node_filesystem_usage: > 0.85) and (predict_linear(node:node_filesystem_avail:[6h], 3600 * 24) < 0)\n",
+              "for": "30m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "NodeDiskRunningFull",
+              "annotations": {
+                "message": "Device {{ $labels.device }} of node-exporter {{ $labels.namespace }}/{{ $labels.pod }} will be full within the next 2 hours."
+              },
+              "expr": "(node:node_filesystem_usage: > 0.85) and (predict_linear(node:node_filesystem_avail:[30m], 3600 * 2) < 0)\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
+        },
+        {
+          "name": "prometheus.rules",
+          "rules": [
+            {
+              "alert": "PrometheusConfigReloadFailed",
+              "annotations": {
+                "description": "Reloading Prometheus' configuration has failed for {{$labels.namespace}}/{{$labels.pod}}",
+                "summary": "Reloading Prometheus' configuration failed"
+              },
+              "expr": "prometheus_config_last_reload_successful{job=\"prometheus\"} == 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusNotificationQueueRunningFull",
+              "annotations": {
+                "description": "Prometheus' alert notification queue is running full for {{$labels.namespace}}/{{ $labels.pod}}",
+                "summary": "Prometheus' alert notification queue is running full"
+              },
+              "expr": "predict_linear(prometheus_notifications_queue_length{job=\"prometheus\"}[5m], 60 * 30) > prometheus_notifications_queue_capacity{job=\"prometheus\"}\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusErrorSendingAlerts",
+              "annotations": {
+                "description": "Errors while sending alerts from Prometheus {{$labels.namespace}}/{{ $labels.pod}} to Alertmanager {{$labels.Alertmanager}}",
+                "summary": "Errors while sending alert from Prometheus"
+              },
+              "expr": "rate(prometheus_notifications_errors_total{job=\"prometheus\"}[5m]) / rate(prometheus_notifications_sent_total{job=\"prometheus\"}[5m]) > 0.01\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusErrorSendingAlerts",
+              "annotations": {
+                "description": "Errors while sending alerts from Prometheus {{$labels.namespace}}/{{ $labels.pod}} to Alertmanager {{$labels.Alertmanager}}",
+                "summary": "Errors while sending alerts from Prometheus"
+              },
+              "expr": "rate(prometheus_notifications_errors_total{job=\"prometheus\"}[5m]) / rate(prometheus_notifications_sent_total{job=\"prometheus\"}[5m]) > 0.03\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "PrometheusNotConnectedToAlertmanagers",
+              "annotations": {
+                "description": "Prometheus {{ $labels.namespace }}/{{ $labels.pod}} is not connected to any Alertmanagers",
+                "summary": "Prometheus is not connected to any Alertmanagers"
+              },
+              "expr": "prometheus_notifications_alertmanagers_discovered{job=\"prometheus\"} < 1\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTSDBReloadsFailing",
+              "annotations": {
+                "description": "{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}} reload failures over the last four hours.",
+                "summary": "Prometheus has issues reloading data blocks from disk"
+              },
+              "expr": "increase(prometheus_tsdb_reloads_failures_total{job=\"prometheus\"}[2h]) > 0\n",
+              "for": "12h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTSDBCompactionsFailing",
+              "annotations": {
+                "description": "{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}} compaction failures over the last four hours.",
+                "summary": "Prometheus has issues compacting sample blocks"
+              },
+              "expr": "increase(prometheus_tsdb_compactions_failed_total{job=\"prometheus\"}[2h]) > 0\n",
+              "for": "12h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTSDBWALCorruptions",
+              "annotations": {
+                "description": "{{$labels.job}} at {{$labels.instance}} has a corrupted write-ahead log (WAL).",
+                "summary": "Prometheus write-ahead log is corrupted"
+              },
+              "expr": "tsdb_wal_corruptions_total{job=\"prometheus\"} > 0\n",
+              "for": "4h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusNotIngestingSamples",
+              "annotations": {
+                "description": "Prometheus {{ $labels.namespace }}/{{ $labels.pod}} isn't ingesting samples.",
+                "summary": "Prometheus isn't ingesting samples"
+              },
+              "expr": "rate(prometheus_tsdb_head_samples_appended_total{job=\"prometheus\"}[5m]) <= 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTargetScrapesDuplicate",
+              "annotations": {
+                "description": "{{$labels.namespace}}/{{$labels.pod}} has many samples rejected due to duplicate timestamps but different values",
+                "summary": "Prometheus has many samples rejected"
+              },
+              "expr": "increase(prometheus_target_scrapes_sample_duplicate_timestamp_total{job=\"prometheus\"}[5m]) > 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "name": "general.rules",
+          "rules": [
+            {
+              "alert": "TargetDown",
+              "annotations": {
+                "message": "{{ $value }}% of the {{ $labels.job }} targets are down."
+              },
+              "expr": "100 * (count(up == 0) BY (job) / count(up) BY (job)) > 10",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        }
+      ]
+    }

aws/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

aws/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

aws/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.11"
+            Environment="ETCD_IMAGE_TAG=v3.3.12"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -123,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -93,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -111,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.13.3 \
+            docker://k8s.gcr.io/hyperkube:v1.13.4 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

aws/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [spot](https://typhoon.psdn.io/cl/aws/#spot) workers

aws/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -78,8 +78,8 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, start, --no-block, kubelet.service]

aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -54,7 +54,7 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default

azure/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

azure/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

azure/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.11"
+            Environment="ETCD_IMAGE_TAG=v3.3.12"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -123,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -93,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -111,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.13.3 \
+            docker://k8s.gcr.io/hyperkube:v1.13.4 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')

bare-metal/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

bare-metal/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name                    = "${var.cluster_name}"
   api_servers                     = ["${var.k8s_domain_name}"]

bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.11"
+            Environment="ETCD_IMAGE_TAG=v3.3.12"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
@@ -128,7 +128,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -89,7 +89,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

bare-metal/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${var.k8s_domain_name}"]

bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -84,8 +84,8 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, kubelet.path]

bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -60,7 +60,7 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

digital-ocean/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

digital-ocean/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.11"
+            Environment="ETCD_IMAGE_TAG=v3.3.12"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -125,7 +125,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -95,7 +95,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -113,7 +113,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.13.3 \
+            docker://k8s.gcr.io/hyperkube:v1.13.4 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

digital-ocean/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

digital-ocean/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -75,8 +75,8 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, kubelet.path]

digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -51,7 +51,7 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

docs/addons/grafana.md

@@ -14,7 +14,8 @@ kubectl port-forward grafana-POD-ID 8080 -n monitoring
 
 Visit [127.0.0.1:8080](http://127.0.0.1:8080) to view the bundled dashboards.
 
-![Grafana Capacity Planning](../img/grafana-capacity.png)
-![Grafana Control Plane](../img/grafana-control-plane.png)
-![Grafana Node View](../img/grafana-node.png)
+![Grafana etcd](../img/grafana-etcd.png)
+![Grafana resources cluster](../img/grafana-resources-cluster.png)
+![Grafana usage cluster](../img/grafana-usage-cluster.png)
+![Grafana usage node](../img/grafana-usage-node.png)
 

docs/advanced/worker-pools.md

@@ -16,7 +16,7 @@ Create a cluster following the AWS [tutorial](../cl/aws.md#cluster). Define a wo
 
 ```tf
 module "tempest-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.13.4"
   
   providers = {
     aws = "aws.default"
@@ -82,7 +82,7 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define
 
 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.13.4"
   
   providers = {
     azurerm = "azurerm.default"
@@ -152,7 +152,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu
 
 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.13.4"
 
   providers = {
     google = "google.default"
@@ -187,11 +187,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.13.3
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.13.3
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.13.3
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.13.3
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.13.3
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.13.4
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.13.4
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.13.4
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.13.4
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.13.4
 ```
 
 ### Variables

docs/atomic/aws.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.13.3 cluster on AWS with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on AWS with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -44,7 +44,7 @@ Configure the AWS provider to use your access key credentials in a `providers.tf
 
 ```tf
 provider "aws" {
-  version = "~> 1.13.0"
+  version = "~> 1.60.0"
   alias   = "default"
 
   region                  = "eu-central-1"
@@ -83,7 +83,7 @@ Define a Kubernetes cluster using the module `aws/fedora-atomic/kubernetes`.
 
 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.13.4"
 
   providers = {
     aws = "aws.default"
@@ -156,9 +156,9 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
 NAME           STATUS  ROLES              AGE  VERSION
-ip-10-0-3-155  Ready   controller,master  10m  v1.13.3
-ip-10-0-26-65  Ready   node               10m  v1.13.3
-ip-10-0-41-21  Ready   node               10m  v1.13.3
+ip-10-0-3-155  Ready   controller,master  10m  v1.13.4
+ip-10-0-26-65  Ready   node               10m  v1.13.4
+ip-10-0-41-21  Ready   node               10m  v1.13.4
 ```
 
 List the pods.

docs/atomic/bare-metal.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.13.3 cluster on bare-metal with Fedora Atomic.
+In this tutorial, we'll network boot and provision a Kubernetes v1.13.4 cluster on bare-metal with Fedora Atomic.
 
 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora Atomic via kickstart, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via cloud-init.
 
@@ -174,20 +174,12 @@ $ terraform version
 Terraform v0.11.7
 ```
 
-Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system.
+Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
 
 ```sh
-wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.2/terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-tar xzf terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-sudo mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  matchbox = "/usr/local/bin/terraform-provider-matchbox"
-}
+wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.3/terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
+tar xzf terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
+mv terraform-provider-matchbox-v0.2.3-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.3
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -202,6 +194,7 @@ Configure the Matchbox provider to use your Matchbox API endpoint and client cer
 
 ```tf
 provider "matchbox" {
+  version     = "0.2.3"
   endpoint    = "matchbox.example.com:8081"
   client_cert = "${file("~/.config/matchbox/client.crt")}"
   client_key  = "${file("~/.config/matchbox/client.key")}"
@@ -235,7 +228,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-atomic/kubernete
 
 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.13.4"
   
   providers = {
     local = "local.default"
@@ -361,9 +354,9 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
 NAME                STATUS  ROLES              AGE  VERSION
-node1.example.com   Ready   controller,master  10m  v1.13.3
-node2.example.com   Ready   node               10m  v1.13.3
-node3.example.com   Ready   node               10m  v1.13.3
+node1.example.com   Ready   controller,master  10m  v1.13.4
+node2.example.com   Ready   node               10m  v1.13.4
+node3.example.com   Ready   node               10m  v1.13.4
 ```
 
 List the pods.

docs/atomic/digital-ocean.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.13.3 cluster on DigitalOcean with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on DigitalOcean with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -45,7 +45,7 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.
 
 ```tf
 provider "digitalocean" {
-  version = "1.0.0"
+  version = "~> 1.1.0"
   token = "${chomp(file("~/.config/digital-ocean/token"))}"
   alias = "default"
 }
@@ -77,7 +77,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-atomic/kubern
 
 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.13.4"
   
   providers = {
     digitalocean = "digitalocean.default"
@@ -152,9 +152,9 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
 NAME               STATUS  ROLES              AGE  VERSION
-nemo-controller-0  Ready   controller,master  10m  v1.13.3
-nemo-worker-0      Ready   node               10m  v1.13.3
-nemo-worker-1      Ready   node               10m  v1.13.3
+nemo-controller-0  Ready   controller,master  10m  v1.13.4
+nemo-worker-0      Ready   node               10m  v1.13.4
+nemo-worker-1      Ready   node               10m  v1.13.4
 ```
 
 List the pods.

docs/atomic/google-cloud.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Fedora does not publish official images for Google Cloud so you must prepare them yourself. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.13.3 cluster on Google Compute Engine with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on Google Compute Engine with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -35,7 +35,7 @@ cd infra/clusters
 
 Login to your Google Console [API Manager](https://console.cloud.google.com/apis/dashboard) and select a project, or [signup](https://cloud.google.com/free/) if you don't have an account.
 
-Select "Credentials" and create a service account key. Choose the "Compute Engine Admin" role and save the JSON private key to a file that can be referenced in configs.
+Select "Credentials" and create a service account key. Choose the "Compute Engine Admin" and "DNS Administrator" roles and save the JSON private key to a file that can be referenced in configs.
 
 ```sh
 mv ~/Downloads/project-id-43048204.json ~/.config/google-cloud/terraform.json
@@ -45,7 +45,7 @@ Configure the Google Cloud provider to use your service account key, project-id,
 
 ```tf
 provider "google" {
-  version = "1.6"
+  version = "~> 2.1.0"
   alias   = "default"
 
   credentials = "${file("~/.config/google-cloud/terraform.json")}"
@@ -121,7 +121,7 @@ Define a Kubernetes cluster using the module `google-cloud/fedora-atomic/kuberne
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.13.4"
   
   providers = {
     google   = "google.default"
@@ -197,9 +197,9 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                       ROLES              STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.3
-yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.3
-yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.3
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.4
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.4
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.4
 ```
 
 List the pods.

docs/cl/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.13.3 cluster on AWS with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on AWS with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -49,7 +49,7 @@ Configure the AWS provider to use your access key credentials in a `providers.tf
 
 ```tf
 provider "aws" {
-  version = "~> 1.13.0"
+  version = "~> 1.60.0"
   alias   = "default"
 
   region                  = "eu-central-1"
@@ -92,7 +92,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.
 
 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.13.4"
 
   providers = {
     aws = "aws.default"
@@ -165,9 +165,9 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
 NAME           STATUS  ROLES              AGE  VERSION
-ip-10-0-3-155  Ready   controller,master  10m  v1.13.3
-ip-10-0-26-65  Ready   node               10m  v1.13.3
-ip-10-0-41-21  Ready   node               10m  v1.13.3
+ip-10-0-3-155  Ready   controller,master  10m  v1.13.4
+ip-10-0-26-65  Ready   node               10m  v1.13.4
+ip-10-0-41-21  Ready   node               10m  v1.13.4
 ```
 
 List the pods.

docs/cl/azure.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Azure is alpha. For production, use AWS, Google Cloud, or bare-metal. As Azure matures, check [errata](https://github.com/poseidon/typhoon/wiki/Errata) for known shortcomings.
 
-In this tutorial, we'll create a Kubernetes v1.13.3 cluster on Azure with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on Azure with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -50,7 +50,7 @@ Configure the Azure provider in a `providers.tf` file.
 
 ```tf
 provider "azurerm" {
-  version = "1.16.0"
+  version = "~> 1.22.1"
   alias   = "default"
 }
 
@@ -87,7 +87,7 @@ Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.
 
 ```tf
 module "azure-ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.13.4"
 
   providers = {
     azurerm  = "azurerm.default"
@@ -161,9 +161,9 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/ramius/auth/kubeconfig
 $ kubectl get nodes
 NAME                  STATUS  ROLES              AGE  VERSION
-ramius-controller-0   Ready   controller,master  24m  v1.13.3
-ramius-worker-000001  Ready   node               25m  v1.13.3
-ramius-worker-000002  Ready   node               24m  v1.13.3
+ramius-controller-0   Ready   controller,master  24m  v1.13.4
+ramius-worker-000001  Ready   node               25m  v1.13.4
+ramius-worker-000002  Ready   node               24m  v1.13.4
 ```
 
 List the pods.

docs/cl/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.13.3 cluster on bare-metal with Container Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.13.4 cluster on bare-metal with Container Linux.
 
 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -116,9 +116,9 @@ Terraform v0.11.7
 Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
 
 ```sh
-wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.2/terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-tar xzf terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.2
+wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.3/terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
+tar xzf terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
+mv terraform-provider-matchbox-v0.2.3-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.3
 ```
 
 Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
@@ -141,7 +141,7 @@ Configure the Matchbox provider to use your Matchbox API endpoint and client cer
 
 ```tf
 provider "matchbox" {
-  version = "0.2.2"
+  version     = "0.2.3"
   endpoint    = "matchbox.example.com:8081"
   client_cert = "${file("~/.config/matchbox/client.crt")}"
   client_key  = "${file("~/.config/matchbox/client.key")}"
@@ -179,7 +179,7 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne
 
 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.13.4"
   
   providers = {
     local = "local.default"
@@ -288,9 +288,9 @@ Apply complete! Resources: 55 added, 0 changed, 0 destroyed.
 To watch the install to disk (until machines reboot from disk), SSH to port 2222.
 
 ```
-# before v1.13.3
+# before v1.13.4
 $ ssh debug@node1.example.com
-# after v1.13.3
+# after v1.13.4
 $ ssh -p 2222 core@node1.example.com
 ```
 
@@ -315,9 +315,9 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
 NAME                STATUS  ROLES              AGE  VERSION
-node1.example.com   Ready   controller,master  10m  v1.13.3
-node2.example.com   Ready   node               10m  v1.13.3
-node3.example.com   Ready   node               10m  v1.13.3
+node1.example.com   Ready   controller,master  10m  v1.13.4
+node2.example.com   Ready   node               10m  v1.13.4
+node3.example.com   Ready   node               10m  v1.13.4
 ```
 
 List the pods.

docs/cl/digital-ocean.md

@@ -1,6 +1,6 @@
 # Digital Ocean
 
-In this tutorial, we'll create a Kubernetes v1.13.3 cluster on DigitalOcean with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on DigitalOcean with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -50,7 +50,7 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.
 
 ```tf
 provider "digitalocean" {
-  version = "1.0.0"
+  version = "~> 1.1.0"
   token = "${chomp(file("~/.config/digital-ocean/token"))}"
   alias = "default"
 }
@@ -86,7 +86,7 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube
 
 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.13.4"
   
   providers = {
     digitalocean = "digitalocean.default"
@@ -160,9 +160,9 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
 NAME               STATUS  ROLES              AGE  VERSION
-nemo-controller-0  Ready   controller,master  10m  v1.13.3
-nemo-worker-0      Ready   node               10m  v1.13.3
-nemo-worker-1      Ready   node               10m  v1.13.3
+nemo-controller-0  Ready   controller,master  10m  v1.13.4
+nemo-worker-0      Ready   node               10m  v1.13.4
+nemo-worker-1      Ready   node               10m  v1.13.4
 ```
 
 List the pods.

docs/cl/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.13.3 cluster on Google Compute Engine with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on Google Compute Engine with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -39,7 +39,7 @@ cd infra/clusters
 
 Login to your Google Console [API Manager](https://console.cloud.google.com/apis/dashboard) and select a project, or [signup](https://cloud.google.com/free/) if you don't have an account.
 
-Select "Credentials" and create a service account key. Choose the "Compute Engine Admin" role and save the JSON private key to a file that can be referenced in configs.
+Select "Credentials" and create a service account key. Choose the "Compute Engine Admin" and "DNS Administrator" roles and save the JSON private key to a file that can be referenced in configs.
 
 ```sh
 mv ~/Downloads/project-id-43048204.json ~/.config/google-cloud/terraform.json
@@ -49,7 +49,7 @@ Configure the Google Cloud provider to use your service account key, project-id,
 
 ```tf
 provider "google" {
-  version = "1.6"
+  version = "~> 2.1.0"
   alias   = "default"
 
   credentials = "${file("~/.config/google-cloud/terraform.json")}"
@@ -93,7 +93,7 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.4"
   
   providers = {
     google   = "google.default"
@@ -168,9 +168,9 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                       ROLES              STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.3
-yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.3
-yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.3
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.4
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.4
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.4
 ```
 
 List the pods.

docs/index.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](cl/google-cloud/#preemption) workers, and [snippets](advanced/customization/#container-linux) customization
@@ -49,7 +49,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.4"
   
   providers = {
     google   = "google.default"
@@ -90,9 +90,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                       ROLES              STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.3
-yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.3
-yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.3
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.4
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.4
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.4
 ```
 
 List the pods.

docs/topics/hardware.md

@@ -4,7 +4,7 @@ Typhoon ensures certain networking hardware integrates well with bare-metal Kube
 
 ## Ubiquiti
 
-Ubiquiti EdgeRouters work well with bare-metal Kubernetes clusters. Knowledge about how to setup an EdgeRouter and use the CLI is required.
+Ubiquiti EdgeRouters and EdgeOS work well with bare-metal Kubernetes clusters. Familiarity with EdgeRouter setup and CLI usage is required.
 
 ### PXE
 
@@ -12,7 +12,7 @@ Ubiquiti EdgeRouters can provide a PXE-enabled network boot environment for clie
 
 #### ISC DHCP
 
-Add a subnet parameter to the LAN DHCP server to include an ISC DHCP config file.
+With ISC DHCP, add a subnet parameter to the LAN DHCP server to include an ISC DHCP config file.
 
 ```
 configure
@@ -21,7 +21,7 @@ set service dhcp-server shared-network-name NAME subnet SUBNET subnet-parameters
 commit-confirm
 ```
 
-Switch to root (i.e. `sudo -i`) and write the ISC DHCP config `/config/scripts/ipxe.conf`. iPXE client machines will chainload to `matchbox.example.com`, while non-iPXE clients will chainload to `undionly.kpxe` (requires TFTP to be enabled).
+Switch to root (i.e. `sudo -i`) and write the ISC DHCP config `/config/scripts/ipxe.conf`. iPXE client machines will chainload to `matchbox.example.com`, while non-iPXE clients will chainload to `undionly.kpxe` (requires TFTP).
 
 ```
 allow bootp;
@@ -35,14 +35,23 @@ if exists user-class and option user-class = "iPXE" {
 }
 ```
 
+#### dnsmasq
+
+With dnsmasq for DHCP, add options to chainload PXE clients to iPXE `undionly.kpxe` (requires TFTP), tag iPXE clients, and chainload iPXE clients to `matchbox.example.com`.
+
+```
+set service dns forwarding options 'dhcp-userclass=set:ipxe,iPXE'
+set service dns forwarding options 'pxe-service=tag:#ipxe,x86PC,PXE chainload to iPXE,undionly.kpxe'
+set service dns forwarding options 'pxe-service=tag:ipxe,x86PC,iPXE,http://matchbox.example.com/boot.ipxe'
+```   
+
 ### TFTP
 
-Use `dnsmasq` as a TFTP server to serve [undionly.kpxe](http://boot.ipxe.org/undionly.kpxe).
+Use `dnsmasq` as a TFTP server to serve `undionly.kpxe`. Compiling from [source](https://github.com/ipxe/ipxe) with TLS support is recommended, but you may also download a [pre-compiled](http://boot.ipxe.org/undionly.kpxe) copy.
 
 ```
 sudo -i
-mkdir /var/lib/tftpboot
-cd /var/lib/tftpboot
+mkdir /config/tftpboot && cd /config/tftpboot
 curl http://boot.ipxe.org/undionly.kpxe -o undionly.kpxe
 ```
 
@@ -52,13 +61,10 @@ Add `dnsmasq` command line options to enable the TFTP file server.
 configure
 show service dns forwarding
 set service dns forwarding options enable-tftp
-set service dns forwarding options tftp-root=/var/lib/tftpboot
+set service dns forwarding options tftp-root=/config/tftpboot
 commit-confirm
 ```
 
-!!! warning
-    After firmware upgrades, the `/var/lib/tftpboot` directory will not exist and dnsmasq will not start properly. Repeat this process following an upgrade.
-
 ### DHCP
 
 Assign static IPs to clients with known MAC addresses. This is called a static mapping by EdgeOS. Configure the router with the commands based on region inventory.
@@ -106,6 +112,9 @@ set protocols static route 10.3.0.0/16 next-hop NODE_IP
 commit-confirm
 ```
 
+!!! note
+    Adding multiple next-hop nodes provides equal-cost multi-path (ECMP) routing. EdgeOS v2.0+ is required. The kernel in prior versions used flow-hash to balanced packets, whereas with v2.0, round-robin sessions are used. 
+
 ### Port Forwarding
 
 Expose the [Ingress Controller](/addons/ingress.md#bare-metal) by adding `port-forward` rules that DNAT a port on the router's WAN interface to an internal IP and port. By convention, a public Ingress controller is assigned a fixed service IP (e.g. 10.3.0.12).

docs/topics/maintenance.md

@@ -18,7 +18,7 @@ module "google-cloud-yavin" {
 }
 
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.13.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.13.4"
   ...
 }
 ```
@@ -156,9 +156,9 @@ mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct ~/.terraform.d
 If you use bare-metal, add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the versioned name.
 
 ```sh
-wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.2/terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-tar xzf terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.2
+wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.3/terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
+tar xzf terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
+mv terraform-provider-matchbox-v0.2.3-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.3
 ```
 
 Binary names are versioned. This enables the ability to upgrade different plugins and have clusters pin different versions.
@@ -168,7 +168,7 @@ $ tree ~/.terraform.d/
 /home/user/.terraform.d/
 └── plugins
     ├── terraform-provider-ct_v0.2.1
-    └── terraform-provider-matchbox_v0.2.2
+    └── terraform-provider-matchbox_v0.2.3
 ```
 
 In each Terraform working directory, set the version of each provider.
@@ -177,7 +177,7 @@ In each Terraform working directory, set the version of each provider.
 # providers.tf
 
 provider "matchbox" {
-  version = "0.2.2"
+  version = "0.2.3"
   ...
 }
 
@@ -215,7 +215,7 @@ $ tree ~/.terraform.d/
 └── plugins
     ├── terraform-provider-ct_v0.2.1
     ├── terraform-provider-ct_v0.3.0
-    └── terraform-provider-matchbox_v0.2.2
+    └── terraform-provider-matchbox_v0.2.3
 ```
 
 

docs/topics/performance.md

@@ -6,11 +6,11 @@ Provisioning times vary based on the operating system and platform. Sampling the
 
 | Platform      | Apply | Destroy |
 |---------------|-------|---------|
-| AWS           | 6 min | 4 min   |
-| Azure         | 7 min | 7 min   |
+| AWS           | 5 min | 3 min   |
+| Azure         | 10 min | 7 min   |
 | Bare-Metal    | 10-15 min | NA  |
 | Digital Ocean | 3 min 30 sec | 20 sec |
-| Google Cloud  | 7 min | 6 min   |
+| Google Cloud  | 8 min | 5 min   |
 
 Notes:
 

google-cloud/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

google-cloud/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

google-cloud/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.11"
+            Environment="ETCD_IMAGE_TAG=v3.3.12"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -124,7 +124,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

google-cloud/container-linux/kubernetes/controllers.tf

@@ -11,7 +11,7 @@ resource "google_dns_record_set" "etcds" {
   ttl  = 300
 
   # private IPv4 address for etcd
-  rrdatas = ["${element(google_compute_instance.controllers.*.network_interface.0.address, count.index)}"]
+  rrdatas = ["${element(google_compute_instance.controllers.*.network_interface.0.network_ip, count.index)}"]
 }
 
 # Zones in the region
@@ -24,16 +24,17 @@ locals {
   # controllers over up to 3 zones, since all GCP regions have at least 3.
   zones = "${slice(data.google_compute_zones.all.names, 0, 3)}"
 
-  controllers_ipv4_public = ["${google_compute_instance.controllers.*.network_interface.0.access_config.0.assigned_nat_ip}"]
+  controllers_ipv4_public = ["${google_compute_instance.controllers.*.network_interface.0.access_config.0.nat_ip}"]
 }
 
 # Controller instances
 resource "google_compute_instance" "controllers" {
   count = "${var.controller_count}"
 
-  name         = "${var.cluster_name}-controller-${count.index}"
-  zone         = "${element(local.zones, count.index)}"
-  machine_type = "${var.controller_type}"
+  name             = "${var.cluster_name}-controller-${count.index}"
+  zone             = "${element(local.zones, count.index)}"
+  machine_type     = "${var.controller_type}"
+  min_cpu_platform = "Intel Haswell"
 
   metadata {
     user-data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"

google-cloud/container-linux/kubernetes/require.tf

@@ -5,7 +5,7 @@ terraform {
 }
 
 provider "google" {
-  version = "~> 1.6"
+  version = ">= 1.19, < 3.0"
 }
 
 provider "local" {

google-cloud/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -94,7 +94,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -112,7 +112,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.13.3 \
+            docker://k8s.gcr.io/hyperkube:v1.13.4 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

google-cloud/container-linux/kubernetes/workers/workers.tf

@@ -23,9 +23,10 @@ resource "google_compute_region_instance_group_manager" "workers" {
 
 # Worker instance template
 resource "google_compute_instance_template" "worker" {
-  name_prefix  = "${var.name}-worker-"
-  description  = "Worker Instance template"
-  machine_type = "${var.machine_type}"
+  name_prefix      = "${var.name}-worker-"
+  description      = "Worker Instance template"
+  machine_type     = "${var.machine_type}"
+  min_cpu_platform = "Intel Haswell"
 
   metadata {
     user-data = "${data.ct_config.worker-ignition.rendered}"

google-cloud/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers

google-cloud/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

google-cloud/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -78,8 +78,8 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, start, --no-block, kubelet.service]

google-cloud/fedora-atomic/kubernetes/controllers.tf

@@ -11,7 +11,7 @@ resource "google_dns_record_set" "etcds" {
   ttl  = 300
 
   # private IPv4 address for etcd
-  rrdatas = ["${element(google_compute_instance.controllers.*.network_interface.0.address, count.index)}"]
+  rrdatas = ["${element(google_compute_instance.controllers.*.network_interface.0.network_ip, count.index)}"]
 }
 
 # Zones in the region
@@ -24,16 +24,17 @@ locals {
   # controllers over up to 3 zones, since all GCP regions have at least 3.
   zones = "${slice(data.google_compute_zones.all.names, 0, 3)}"
 
-  controllers_ipv4_public = ["${google_compute_instance.controllers.*.network_interface.0.access_config.0.assigned_nat_ip}"]
+  controllers_ipv4_public = ["${google_compute_instance.controllers.*.network_interface.0.access_config.0.nat_ip}"]
 }
 
 # Controller instances
 resource "google_compute_instance" "controllers" {
   count = "${var.controller_count}"
 
-  name         = "${var.cluster_name}-controller-${count.index}"
-  zone         = "${element(local.zones, count.index)}"
-  machine_type = "${var.controller_type}"
+  name             = "${var.cluster_name}-controller-${count.index}"
+  zone             = "${element(local.zones, count.index)}"
+  machine_type     = "${var.controller_type}"
+  min_cpu_platform = "Intel Haswell"
 
   metadata {
     user-data = "${element(data.template_file.controller-cloudinit.*.rendered, count.index)}"

google-cloud/fedora-atomic/kubernetes/require.tf

@@ -5,7 +5,7 @@ terraform {
 }
 
 provider "google" {
-  version = "~> 1.6"
+  version = ">= 1.19, < 3.0"
 }
 
 provider "local" {

google-cloud/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -54,7 +54,7 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default

google-cloud/fedora-atomic/kubernetes/workers/workers.tf

@@ -23,9 +23,10 @@ resource "google_compute_region_instance_group_manager" "workers" {
 
 # Worker instance template
 resource "google_compute_instance_template" "worker" {
-  name_prefix  = "${var.name}-worker-"
-  description  = "Worker Instance template"
-  machine_type = "${var.machine_type}"
+  name_prefix      = "${var.name}-worker-"
+  description      = "Worker Instance template"
+  machine_type     = "${var.machine_type}"
+  min_cpu_platform = "Intel Haswell"
 
   metadata {
     user-data = "${data.template_file.worker-cloudinit.rendered}"

requirements.txt

@@ -1,4 +1,4 @@
 mkdocs==1.0.4
-mkdocs-material==3.3.0
+mkdocs-material==4.0.1
 pygments==2.2.0
 pymdown-extensions==5.0.0