Add links and clarifications in CHANGES for release

* Minimum versions of Terraform provider plugins are enforced in
each module already. Its better to provide examples with newer
versions. Some folks don't update them
* Previously, tutorials showed the minimum viable version of each
terraform provider that might be used

.github/ISSUE_TEMPLATE.md

@@ -5,8 +5,8 @@
 ### Environment
 
 * Platform: aws, azure, bare-metal, google-cloud, digital-ocean
-* OS: container-linux, fedora-atomic
-* Ref: Release version or Git SHA (reporting latest is **not** helpful)
+* OS: container-linux, flatcar-linux, or fedora-atomic
+* Release: Typhoon version or Git SHA (reporting latest is **not** helpful)
 * Terraform: `terraform version` (reporting latest is **not** helpful)
 * Plugins: Provider plugin versions (reporting latest is **not** helpful)
 

CHANGES.md

@@ -4,6 +4,130 @@ Notable changes between versions.
 
 ## Latest
 
+## v1.13.4
+
+* Kubernetes [v1.13.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1134)
+* Update etcd from v3.3.11 to [v3.3.12](https://github.com/etcd-io/etcd/releases/tag/v3.3.12)
+* Update Calico from v3.5.0 to [v3.5.2](https://docs.projectcalico.org/v3.5/releases/)
+* Assign priorityClassNames to critical cluster and node components ([#406](https://github.com/poseidon/typhoon/pull/406))
+  * Inform node out-of-resource eviction and scheduler preemption and ordering
+* Add CoreDNS readiness probe ([#410](https://github.com/poseidon/typhoon/pull/410))
+
+#### Bare-Metal
+
+* Recommend updating [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin from v0.2.2 to [v0.2.3](https://github.com/coreos/terraform-provider-matchbox/releases/tag/v0.2.3) ([#402](https://github.com/poseidon/typhoon/pull/402))
+* Improve docs on using Ubiquiti EdgeOS with bare-metal clusters ([#413](https://github.com/poseidon/typhoon/pull/413))
+
+#### Google Cloud
+
+* Support `terraform-provider-google` v2.0+ ([#407](https://github.com/poseidon/typhoon/pull/407))
+  * Require `terraform-provider-google` v1.19+ (**action required**)
+* Set the minimum CPU platform to Intel Haswell ([#405](https://github.com/poseidon/typhoon/pull/405))
+  * Haswell or better is available in every zone (no price change)
+  * A few zones still default to Sandy/Ivy Bridge (shifts in April 2019)
+
+#### Addons
+
+* Modernize Prometheus rules and alerts ([#404](https://github.com/poseidon/typhoon/pull/404))
+  * Drop extraneous metrics ([#397](https://github.com/poseidon/typhoon/pull/397))
+  * Add `pod` name label to metrics discovered via service endpoints
+  * Rename `kubernetes_namespace` label to `namespace`
+* Modernize Grafana and dashboards, see [docs](https://typhoon.psdn.io/addons/grafana/) ([#403](https://github.com/poseidon/typhoon/pull/403), [#404](https://github.com/poseidon/typhoon/pull/404))
+  * Upgrade Grafana from v5.4.3 to [v6.0.0](https://github.com/grafana/grafana/releases/tag/v6.0.0)!
+  * Enable Grafana [Explore](http://docs.grafana.org/guides/whats-new-in-v6-0/#explore) UI as a Viewer (inspect/edit without saving)
+* Update nginx-ingress from v0.22.0 to v0.23.0
+  * Raise nginx-ingress liveness/readiness timeout to 5 seconds
+  * Remove nginx-ingess default-backend ([#401](https://github.com/poseidon/typhoon/pull/401))
+
+#### Fedora Atomic
+
+* Build Kubelet [system container](https://github.com/poseidon/system-containers) with buildah. The image is an OCI format and slightly larger.
+
+## v1.13.3
+
+* Kubernetes [v1.13.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1133)
+* Update etcd from v3.3.10 to [v3.3.11](https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3311-2019-1-11)
+* Update CoreDNS from v1.3.0 to [v1.3.1](https://coredns.io/2019/01/13/coredns-1.3.1-release/)
+  * Switch from the `proxy` plugin to the faster `forward` plugin for upsteam resolvers
+* Update Calico from v3.4.0 to [v3.5.0](https://docs.projectcalico.org/v3.5/releases/)
+* Update flannel from v0.10.0 to [v0.11.0](https://github.com/coreos/flannel/releases/tag/v0.11.0)
+* Reduce pod eviction timeout for deleting pods on unready nodes to 1 minute
+  * Respond more quickly to node preemption (previously 5 minutes)
+* Fix automatic worker deletion on shutdown for cloud platforms
+  * Lowering Kubelet privileges in [#372](https://github.com/poseidon/typhoon/pull/372) dropped a needed node deletion authorization. Scale-in due to manual terraform apply (any cloud), AWS spot termination, or Azure low priority deletion left old nodes registered, requiring manual deletion (`kubectl delete node name`)
+
+#### AWS
+
+* Add `ingress_zone_id` output with the NLB DNS name's Route53 zone for use in alias records ([#380](https://github.com/poseidon/typhoon/pull/380))
+
+#### Azure
+
+* Fix azure provider warning, `public_ip` `allocation_method` replaces `public_ip_address_allocation`
+  * Require `terraform-provider-azurerm` v1.21+ (action required)
+
+#### Addons
+
+* Update nginx-ingress from v0.21.0 to v0.22.0
+* Update Prometheus from v2.6.0 to v2.7.1
+* Update kube-state-metrics from v1.4.0 to v1.5.0
+  * Fix ClusterRole to collect and export PodDisruptionBudget metrics ([#383](https://github.com/poseidon/typhoon/pull/383))
+* Update node-exporter from v0.15.2 to v0.17.0
+* Update Grafana from v5.4.2 to v5.4.3
+
+## v1.13.2
+
+* Kubernetes [v1.13.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1132)
+* Add ServiceAccounts for `kube-apiserver` and `kube-scheduler` ([#370](https://github.com/poseidon/typhoon/pull/370))
+* Use lower-privilege TLS client certificates for Kubelets ([#372](https://github.com/poseidon/typhoon/pull/372))
+* Use HTTPS liveness probes for `kube-scheduler` and `kube-controller-manager` ([#377](https://github.com/poseidon/typhoon/pull/377))
+* Update CoreDNS from v1.2.6 to [v1.3.0](https://coredns.io/2018/12/15/coredns-1.3.0-release/)
+* Allow the `certificates.k8s.io` API to issue certificates signed by the cluster CA ([#376](https://github.com/poseidon/typhoon/pull/376))
+  * Configure controller manager to sign CSRs that are manually [approved](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster) by an administrator
+
+#### AWS
+
+* Change `controller_type` and `worker_type` default from t2.small to t3.small ([#365](https://github.com/poseidon/typhoon/pull/365))
+  * t3.small is cheaper, provides 2 vCPU (instead of 1), and 5 Gbps of pod-to-pod bandwidth!
+
+#### Bare-Metal
+
+* Remove the `kubeconfig` output variable
+
+#### Addons
+
+* Update Prometheus from v2.5.0 to v2.6.0
+
+## v1.13.1
+
+* Kubernetes [v1.13.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1131)
+* Update Calico from v3.3.2 to [v3.4.0](https://docs.projectcalico.org/v3.4/releases/) ([#362](https://github.com/poseidon/typhoon/pull/362))
+  * Install CNI plugins with an init container rather than a sidecar
+  * Improve the `calico-node` ClusterRole
+* Recommend updating `terraform-provider-ct` plugin from v0.2.1 to v0.3.0 ([#363](https://github.com/poseidon/typhoon/pull/363))
+  * [Migration](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct) instructions for upgrading `terraform-provider-ct` in-place for v1.12.2+ clusters (**action required**)
+  * [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-plugins-directory) switching from `~/.terraformrc` to the Terraform [third-party plugins](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) directory `~/.terraform.d/plugins/`
+  * Require Container Linux 1688.5.3 or newer
+
+#### Google Cloud
+
+* Increase TCP proxy apiserver backend service timeout from 1 minute to 5 minutes ([#361](https://github.com/poseidon/typhoon/pull/361))
+  * Align `port-forward` behavior closer to AWS/Azure (no timeout)
+
+#### Addons
+
+* Update Grafana from v5.4.0 to v5.4.2
+
+## v1.13.0
+
+* Kubernetes [v1.13.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1130)
+* Update Calico from v3.3.1 to [v3.3.2](https://docs.projectcalico.org/v3.3/releases/)
+
+#### Addons
+
+* Update Grafana from v5.3.4 to v5.4.0
+* Disable Grafana login form, since admin user can't be disabled ([#352](https://github.com/poseidon/typhoon/pull/352))
+  * Example manifests aim to provide a read-only dashboard view
+
 ## v1.12.3
 
 * Kubernetes [v1.12.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1123)

README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
@@ -50,7 +50,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.4"
   
   providers = {
     google   = "google.default"
@@ -91,9 +91,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                       ROLES              STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.12.3
-yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.12.3
-yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.12.3
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.4
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.4
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.4
 ```
 
 List the pods.

addons/grafana/config.yaml

@@ -0,0 +1,36 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-config
+  namespace: monitoring
+data:
+  custom.ini: |+
+    [server]
+    http_port = 8080
+
+    [paths]
+    data    = /var/lib/grafana
+    plugins = /var/lib/grafana/plugins
+    provisioning = /etc/grafana/provisioning
+
+    [users]
+    allow_sign_up    = false
+    allow_org_create = false
+    # viewers can edit/inspect, but not save
+    viewers_can_edit = true
+
+    # Disable login form, since Grafana always creates an admin user
+    [auth]
+    disable_login_form = true
+
+    # Disable the user/pass login system
+    [auth.basic]
+    enabled = false
+
+    # Allow anonymous authentication with view-only authorization
+    [auth.anonymous]
+    enabled = true
+    org_role = Viewer
+
+    [analytics]
+    reporting_enabled = false

addons/grafana/datasources.yaml

@@ -10,7 +10,15 @@ data:
     - name: prometheus
       type: prometheus
       access: proxy
-      orgId: 1
       url: http://prometheus.monitoring.svc.cluster.local
       version: 1
       editable: false
+  loki.yaml: |+
+    apiVersion: 1
+    datasources:
+    - name: loki
+      type: loki
+      access: proxy
+      url: http://loki.monitoring.svc.cluster.local
+      version: 1
+      editable: false

addons/grafana/deployment.yaml

@@ -23,18 +23,10 @@ spec:
     spec:
       containers:
         - name: grafana
-          image: grafana/grafana:5.3.4
+          image: grafana/grafana:6.0.0
           env:
-            - name: GF_SERVER_HTTP_PORT
-              value: "8080"
-            - name: GF_AUTH_BASIC_ENABLED
-              value: "false"
-            - name: GF_AUTH_ANONYMOUS_ENABLED
-              value: "true"
-            - name: GF_AUTH_ANONYMOUS_ORG_ROLE
-              value: Viewer
-            - name: GF_ANALYTICS_REPORTING_ENABLED
-              value: "false"
+            - name: GF_PATHS_CONFIG
+              value: "/etc/grafana/custom.ini"
           ports:
             - name: http
               containerPort: 8080
@@ -46,19 +38,24 @@ spec:
               memory: 200Mi
               cpu: 200m
           volumeMounts:
+            - name: config
+              mountPath: /etc/grafana
             - name: datasources
               mountPath: /etc/grafana/provisioning/datasources
-            - name: dashboard-providers
+            - name: providers
               mountPath: /etc/grafana/provisioning/dashboards
             - name: dashboards
-              mountPath: /var/lib/grafana/dashboards
+              mountPath: /etc/grafana/dashboards
       volumes:
+        - name: config
+          configMap:
+            name: grafana-config
         - name: datasources
           configMap:
             name: grafana-datasources
-        - name: dashboard-providers
+        - name: providers
           configMap:
-            name: grafana-dashboard-providers
+            name: grafana-providers
         - name: dashboards
           configMap:
             name: grafana-dashboards

addons/grafana/providers.yaml

@@ -1,10 +1,10 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: grafana-dashboard-providers
+  name: grafana-providers
   namespace: monitoring
 data:
-  dashboard-providers.yaml: |+
+  providers.yaml: |+
     apiVersion: 1
     providers:
     - name: 'default'
@@ -12,4 +12,4 @@ data:
       folder: ''
       type: file
       options:
-        path: /var/lib/grafana/dashboards
+        path: /etc/grafana/dashboards

addons/nginx-ingress/aws/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/aws/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/aws/deployment.yaml

@@ -24,10 +24,9 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -58,7 +57,7 @@ spec:
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             failureThreshold: 3
             httpGet:
@@ -67,7 +66,7 @@ spec:
               scheme: HTTP
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/nginx-ingress/azure/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/azure/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/azure/deployment.yaml

@@ -24,10 +24,9 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -58,7 +57,7 @@ spec:
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             failureThreshold: 3
             httpGet:
@@ -67,7 +66,7 @@ spec:
               scheme: HTTP
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/nginx-ingress/bare-metal/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/bare-metal/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -22,10 +22,9 @@ spec:
     spec:
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -53,7 +52,7 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             failureThreshold: 3
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             httpGet:
               path: /healthz
@@ -62,7 +61,7 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             failureThreshold: 3
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -24,10 +24,9 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -58,7 +57,7 @@ spec:
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             failureThreshold: 3
             httpGet:
@@ -67,7 +66,7 @@ spec:
               scheme: HTTP
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/nginx-ingress/digital-ocean/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/digital-ocean/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/google-cloud/default-backend/deployment.yaml

@@ -1,42 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: default-backend
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: default-backend
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      containers:
-        - name: default-backend
-          # Any image is permissable as long as:
-          # 1. It serves a 404 page at /
-          # 2. It serves 200 on a /healthz endpoint
-          image: k8s.gcr.io/defaultbackend:1.4
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: 10m
-              memory: 20Mi
-            requests:
-              cpu: 10m
-              memory: 20Mi
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-              scheme: HTTP
-            initialDelaySeconds: 30
-            timeoutSeconds: 5
-      terminationGracePeriodSeconds: 60

addons/nginx-ingress/google-cloud/default-backend/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: default-backend
-  namespace: ingress
-spec:
-  type: ClusterIP
-  selector:
-    name: default-backend
-    phase: prod
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -24,10 +24,9 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
           args:
             - /nginx-ingress-controller
-            - --default-backend-service=$(POD_NAMESPACE)/default-backend
             - --ingress-class=public
           # use downward API
           env:
@@ -58,7 +57,7 @@ spec:
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           readinessProbe:
             failureThreshold: 3
             httpGet:
@@ -67,7 +66,7 @@ spec:
               scheme: HTTP
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
+            timeoutSeconds: 5
           securityContext:
             capabilities:
               add:

addons/prometheus/config.yaml

@@ -55,6 +55,17 @@ data:
         action: replace
         target_label: job
 
+      metric_relabel_configs:
+      - source_labels: [__name__]
+        action: drop
+        regex: etcd_(debugging|disk|request|server).*
+      - source_labels: [__name__]
+        action: drop
+        regex: apiserver_admission_controller_admission_latencies_seconds_.*
+      - source_labels: [__name__]
+        action: drop
+        regex: apiserver_admission_step_admission_latencies_seconds_.*
+
     # Scrape config for node (i.e. kubelet) /metrics (e.g. 'kubelet_'). Explore
     # metrics from a node by scraping kubelet (127.0.0.1:10250/metrics).
     - job_name: 'kubelet'
@@ -89,6 +100,13 @@ data:
       relabel_configs:
       - action: labelmap
         regex: __meta_kubernetes_node_label_(.+)
+      metric_relabel_configs:
+      - source_labels: [__name__, image]
+        action: drop
+        regex: container_([a-z_]+);
+      - source_labels: [__name__]
+        action: drop
+        regex: container_(network_tcp_usage_total|network_udp_usage_total|tasks_state|cpu_load_average_10s)
 
 
     # Scrap etcd metrics from controllers via listen-metrics-urls
@@ -119,10 +137,10 @@ data:
     # * `prometheus.io/port`: If the metrics are exposed on a different port to the
     # service then set this appropriately.
     - job_name: 'kubernetes-service-endpoints'
-
       kubernetes_sd_configs:
       - role: endpoints
 
+      honor_labels: true
       relabel_configs:
       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
         action: keep
@@ -144,10 +162,18 @@ data:
         regex: __meta_kubernetes_service_label_(.+)
       - source_labels: [__meta_kubernetes_namespace]
         action: replace
-        target_label: kubernetes_namespace
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_pod_name]
+        action: replace
+        target_label: pod
       - source_labels: [__meta_kubernetes_service_name]
         action: replace
         target_label: job
+      
+      metric_relabel_configs:
+      - source_labels: [__name__]
+        action: drop
+        regex: etcd_(debugging|disk|request|server).*
 
     # Example scrape config for probing services via the Blackbox Exporter.
     #
@@ -177,7 +203,7 @@ data:
       - action: labelmap
         regex: __meta_kubernetes_service_label_(.+)
       - source_labels: [__meta_kubernetes_namespace]
-        target_label: kubernetes_namespace
+        target_label: namespace
       - source_labels: [__meta_kubernetes_service_name]
         target_label: job
 

addons/prometheus/deployment.yaml

@@ -20,7 +20,7 @@ spec:
       serviceAccountName: prometheus
       containers:
         - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.5.0
+          image: quay.io/prometheus/prometheus:v2.7.1
           args:
             - --web.listen-address=0.0.0.0:9090
             - --config.file=/etc/prometheus/prometheus.yaml

addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml

@@ -3,7 +3,8 @@ kind: ClusterRole
 metadata:
   name: kube-state-metrics
 rules:
-- apiGroups: [""]
+- apiGroups:
+  - ""
   resources:
   - configmaps
   - secrets
@@ -17,23 +18,47 @@ rules:
   - persistentvolumes
   - namespaces
   - endpoints
-  verbs: ["list", "watch"]
-- apiGroups: ["extensions"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - extensions
   resources:
   - daemonsets
   - deployments
   - replicasets
-  verbs: ["list", "watch"]
-- apiGroups: ["apps"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - apps
   resources:
   - statefulsets
-  verbs: ["list", "watch"]
-- apiGroups: ["batch"]
+  - daemonsets
+  - deployments
+  - replicasets
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - batch
   resources:
   - cronjobs
   - jobs
-  verbs: ["list", "watch"]
-- apiGroups: ["autoscaling"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - autoscaling
   resources:
   - horizontalpodautoscalers
-  verbs: ["list", "watch"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - policy
+  resources:
+  - poddisruptionbudgets
+  verbs:
+  - list
+  - watch

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -24,7 +24,7 @@ spec:
       serviceAccountName: kube-state-metrics
       containers:
       - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.4.0
+        image: quay.io/coreos/kube-state-metrics:v1.5.0
         ports:
           - name: metrics
             containerPort: 8080
@@ -35,7 +35,7 @@ spec:
           initialDelaySeconds: 5
           timeoutSeconds: 5
       - name: addon-resizer
-        image: k8s.gcr.io/addon-resizer:1.7
+        image: k8s.gcr.io/addon-resizer:1.8.4
         resources:
           limits:
             cpu: 100m

addons/prometheus/exporters/kube-state-metrics/resizer-role-binding.yaml

@@ -6,7 +6,7 @@ metadata:
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: Role
-  name: kube-state-metrics-resizer
+  name: kube-state-metrics
 subjects:
 - kind: ServiceAccount
   name: kube-state-metrics

addons/prometheus/exporters/kube-state-metrics/resizer-role.yaml

@@ -1,15 +1,31 @@
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
-  name: kube-state-metrics-resizer
+  name: kube-state-metrics
   namespace: monitoring
 rules:
-- apiGroups: [""]
+- apiGroups:
+  - ""
   resources:
   - pods
-  verbs: ["get"]
-- apiGroups: ["extensions"]
+  verbs:
+  - get
+- apiGroups:
+  - extensions
   resources:
   - deployments
-  resourceNames: ["kube-state-metrics"]
-  verbs: ["get", "update"]
+  resourceNames:
+  - kube-state-metrics
+  verbs:
+  - get
+  - update
+- apiGroups:
+  - apps
+  resources:
+  - deployments
+  resourceNames:
+  - kube-state-metrics
+  verbs:
+  - get
+  - update
+

addons/prometheus/exporters/node-exporter/daemonset.yaml

@@ -28,21 +28,24 @@ spec:
       hostPID: true
       containers:
       - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v0.15.2
+        image: quay.io/prometheus/node-exporter:v0.17.0
         args:
-          - "--path.procfs=/host/proc"
-          - "--path.sysfs=/host/sys"
+          - --path.procfs=/host/proc
+          - --path.sysfs=/host/sys
+          - --path.rootfs=/host/root
+          - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
+          - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
         ports:
           - name: metrics
             containerPort: 9100
             hostPort: 9100
         resources:
           requests:
-            memory: 30Mi
             cpu: 100m
-          limits:
             memory: 50Mi
+          limits:
             cpu: 200m
+            memory: 100Mi
         volumeMounts:
           - name: proc
             mountPath: /host/proc
@@ -50,6 +53,9 @@ spec:
           - name: sys
             mountPath: /host/sys
             readOnly: true
+          - name: root
+            mountPath: /host/root
+            readOnly: true
       tolerations:
         - effect: NoSchedule
           operator: Exists
@@ -60,3 +66,6 @@ spec:
         - name: sys
           hostPath:
             path: /sys
+        - name: root
+          hostPath:
+            path: /

addons/prometheus/rules.yaml

@@ -4,582 +4,1089 @@ metadata:
   name: prometheus-rules
   namespace: monitoring
 data:
-  alertmanager.rules.yaml: |
-    groups:
-    - name: alertmanager.rules
-      rules:
-      - alert: AlertmanagerConfigInconsistent
-        expr: count_values("config_hash", alertmanager_config_hash) BY (service) / ON(service)
-          GROUP_LEFT() label_replace(prometheus_operator_alertmanager_spec_replicas, "service",
-          "alertmanager-$1", "alertmanager", "(.*)") != 1
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          description: The configuration of the instances of the Alertmanager cluster
-            `{{$labels.service}}` are out of sync.
-      - alert: AlertmanagerDownOrMissing
-        expr: label_replace(prometheus_operator_alertmanager_spec_replicas, "job", "alertmanager-$1",
-          "alertmanager", "(.*)") / ON(job) GROUP_RIGHT() sum(up) BY (job) != 1
-        for: 5m
-        labels:
-          severity: warning
-        annotations:
-          description: An unexpected number of Alertmanagers are scraped or Alertmanagers
-            disappeared from discovery.
-      - alert: AlertmanagerFailedReload
-        expr: alertmanager_config_last_reload_successful == 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Reloading Alertmanager's configuration has failed for {{ $labels.namespace
-            }}/{{ $labels.pod}}.
-  etcd3.rules.yaml: |
-    groups:
-    - name: ./etcd3.rules
-      rules:
-      - alert: InsufficientMembers
-        expr: count(up{job="etcd"} == 0) > (count(up{job="etcd"}) / 2 - 1)
-        for: 3m
-        labels:
-          severity: critical
-        annotations:
-          description: If one more etcd member goes down the cluster will be unavailable
-          summary: etcd cluster insufficient members
-      - alert: NoLeader
-        expr: etcd_server_has_leader{job="etcd"} == 0
-        for: 1m
-        labels:
-          severity: critical
-        annotations:
-          description: etcd member {{ $labels.instance }} has no leader
-          summary: etcd member has no leader
-      - alert: HighNumberOfLeaderChanges
-        expr: increase(etcd_server_leader_changes_seen_total{job="etcd"}[1h]) > 3
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} has seen {{ $value }} leader
-            changes within the last hour
-          summary: a high number of leader changes within the etcd cluster are happening
-      - alert: GRPCRequestsSlow
-        expr: histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job="etcd",grpc_type="unary"}[5m])) by (grpc_service, grpc_method, le))
-          > 0.15
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: on etcd instance {{ $labels.instance }} gRPC requests to {{ $labels.grpc_method
-            }} are slow
-          summary: slow gRPC requests
-      - alert: HighNumberOfFailedHTTPRequests
-        expr: sum(rate(etcd_http_failed_total{job="etcd"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job="etcd"}[5m]))
-          BY (method) > 0.01
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: '{{ $value }}% of requests for {{ $labels.method }} failed on etcd
-            instance {{ $labels.instance }}'
-          summary: a high number of HTTP requests are failing
-      - alert: HighNumberOfFailedHTTPRequests
-        expr: sum(rate(etcd_http_failed_total{job="etcd"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job="etcd"}[5m]))
-          BY (method) > 0.05
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          description: '{{ $value }}% of requests for {{ $labels.method }} failed on etcd
-            instance {{ $labels.instance }}'
-          summary: a high number of HTTP requests are failing
-      - alert: HTTPRequestsSlow
-        expr: histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[5m]))
-          > 0.15
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: on etcd instance {{ $labels.instance }} HTTP requests to {{ $labels.method
-            }} are slow
-          summary: slow HTTP requests
-      - alert: EtcdMemberCommunicationSlow
-        expr: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[5m]))
-          > 0.15
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} member communication with
-            {{ $labels.To }} is slow
-          summary: etcd member communication is slow
-      - alert: HighNumberOfFailedProposals
-        expr: increase(etcd_server_proposals_failed_total{job="etcd"}[1h]) > 5
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} has seen {{ $value }} proposal
-            failures within the last hour
-          summary: a high number of proposals within the etcd cluster are failing
-      - alert: HighFsyncDurations
-        expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))
-          > 0.5
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} fync durations are high
-          summary: high fsync durations
-      - alert: HighCommitDurations
-        expr: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m]))
-          > 0.25
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: etcd instance {{ $labels.instance }} commit durations are high
-          summary: high commit durations
-  general.rules.yaml: |
-    groups:
-    - name: general.rules
-      rules:
-      - alert: TargetDown
-        expr: 100 * (count(up == 0) BY (job) / count(up) BY (job)) > 10
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: '{{ $value }}% of {{ $labels.job }} targets are down.'
-          summary: Targets are down
-      - record: fd_utilization
-        expr: process_open_fds / process_max_fds
-      - alert: FdExhaustionClose
-        expr: predict_linear(fd_utilization[1h], 3600 * 4) > 1
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: '{{ $labels.job }}: {{ $labels.namespace }}/{{ $labels.pod }} instance
-            will exhaust in file/socket descriptors within the next 4 hours'
-          summary: file descriptors soon exhausted
-      - alert: FdExhaustionClose
-        expr: predict_linear(fd_utilization[10m], 3600) > 1
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: '{{ $labels.job }}: {{ $labels.namespace }}/{{ $labels.pod }} instance
-            will exhaust in file/socket descriptors within the next hour'
-          summary: file descriptors soon exhausted
-  kube-controller-manager.rules.yaml: |
-    groups:
-    - name: kube-controller-manager.rules
-      rules:
-      - alert: K8SControllerManagerDown
-        expr: absent(up{job="kube-controller-manager"} == 1)
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          description: There is no running K8S controller manager. Deployments and replication
-            controllers are not making progress.
-          summary: Controller manager is down
-  kube-scheduler.rules.yaml: |
-    groups:
-    - name: kube-scheduler.rules
-      rules:
-      - record: cluster:scheduler_e2e_scheduling_latency_seconds:quantile
-        expr: histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.99"
-      - record: cluster:scheduler_e2e_scheduling_latency_seconds:quantile
-        expr: histogram_quantile(0.9, sum(scheduler_e2e_scheduling_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.9"
-      - record: cluster:scheduler_e2e_scheduling_latency_seconds:quantile
-        expr: histogram_quantile(0.5, sum(scheduler_e2e_scheduling_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.5"
-      - record: cluster:scheduler_scheduling_algorithm_latency_seconds:quantile
-        expr: histogram_quantile(0.99, sum(scheduler_scheduling_algorithm_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.99"
-      - record: cluster:scheduler_scheduling_algorithm_latency_seconds:quantile
-        expr: histogram_quantile(0.9, sum(scheduler_scheduling_algorithm_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.9"
-      - record: cluster:scheduler_scheduling_algorithm_latency_seconds:quantile
-        expr: histogram_quantile(0.5, sum(scheduler_scheduling_algorithm_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.5"
-      - record: cluster:scheduler_binding_latency_seconds:quantile
-        expr: histogram_quantile(0.99, sum(scheduler_binding_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.99"
-      - record: cluster:scheduler_binding_latency_seconds:quantile
-        expr: histogram_quantile(0.9, sum(scheduler_binding_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.9"
-      - record: cluster:scheduler_binding_latency_seconds:quantile
-        expr: histogram_quantile(0.5, sum(scheduler_binding_latency_microseconds_bucket)
-          BY (le, cluster)) / 1e+06
-        labels:
-          quantile: "0.5"
-      - alert: K8SSchedulerDown
-        expr: absent(up{job="kube-scheduler"} == 1)
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          description: There is no running K8S scheduler. New pods are not being assigned
-            to nodes.
-          summary: Scheduler is down
-  kube-state-metrics.rules.yaml: |
-    groups:
-    - name: kube-state-metrics.rules
-      rules:
-      - alert: DeploymentGenerationMismatch
-        expr: kube_deployment_status_observed_generation != kube_deployment_metadata_generation
-        for: 15m
-        labels:
-          severity: warning
-        annotations:
-          description: Observed deployment generation does not match expected one for
-            deployment {{$labels.namespaces}}/{{$labels.deployment}}
-          summary: Deployment is outdated
-      - alert: DeploymentReplicasNotUpdated
-        expr: ((kube_deployment_status_replicas_updated != kube_deployment_spec_replicas)
-          or (kube_deployment_status_replicas_available != kube_deployment_spec_replicas))
-          unless (kube_deployment_spec_paused == 1)
-        for: 15m
-        labels:
-          severity: warning
-        annotations:
-          description: Replicas are not updated and available for deployment {{$labels.namespaces}}/{{$labels.deployment}}
-          summary: Deployment replicas are outdated
-      - alert: DaemonSetRolloutStuck
-        expr: kube_daemonset_status_number_ready / kube_daemonset_status_desired_number_scheduled
-          * 100 < 100
-        for: 15m
-        labels:
-          severity: warning
-        annotations:
-          description: Only {{$value}}% of desired pods scheduled and ready for daemon
-            set {{$labels.namespaces}}/{{$labels.daemonset}}
-          summary: DaemonSet is missing pods
-      - alert: K8SDaemonSetsNotScheduled
-        expr: kube_daemonset_status_desired_number_scheduled - kube_daemonset_status_current_number_scheduled
-          > 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: A number of daemonsets are not scheduled.
-          summary: Daemonsets are not scheduled correctly
-      - alert: DaemonSetsMissScheduled
-        expr: kube_daemonset_status_number_misscheduled > 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: A number of daemonsets are running where they are not supposed
-            to run.
-          summary: Daemonsets are not scheduled correctly
-      - alert: PodFrequentlyRestarting
-        expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Pod {{$labels.namespaces}}/{{$labels.pod}} restarted {{$value}}
-            times within the last hour
-          summary: Pod is restarting frequently
-  kubelet.rules.yaml: |
-    groups:
-    - name: kubelet.rules
-      rules:
-      - alert: K8SNodeNotReady
-        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
-        for: 1h
-        labels:
-          severity: warning
-        annotations:
-          description: The Kubelet on {{ $labels.node }} has not checked in with the API,
-            or has set itself to NotReady, for more than an hour
-          summary: Node status is NotReady
-      - alert: K8SManyNodesNotReady
-        expr: count(kube_node_status_condition{condition="Ready",status="true"} == 0)
-          > 1 and (count(kube_node_status_condition{condition="Ready",status="true"} ==
-          0) / count(kube_node_status_condition{condition="Ready",status="true"})) > 0.2
-        for: 1m
-        labels:
-          severity: critical
-        annotations:
-          description: '{{ $value }}% of Kubernetes nodes are not ready'
-      - alert: K8SKubeletDown
-        expr: count(up{job="kubelet"} == 0) / count(up{job="kubelet"}) * 100 > 3
-        for: 1h
-        labels:
-          severity: warning
-        annotations:
-          description: Prometheus failed to scrape {{ $value }}% of kubelets.
-      - alert: K8SKubeletDown
-        expr: (absent(up{job="kubelet"} == 1) or count(up{job="kubelet"} == 0) / count(up{job="kubelet"}))
-          * 100 > 10
-        for: 1h
-        labels:
-          severity: critical
-        annotations:
-          description: Prometheus failed to scrape {{ $value }}% of kubelets, or all Kubelets
-            have disappeared from service discovery.
-          summary: Many Kubelets cannot be scraped
-      - alert: K8SKubeletTooManyPods
-        expr: kubelet_running_pod_count > 100
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Kubelet {{$labels.instance}} is running {{$value}} pods, close
-            to the limit of 110
-          summary: Kubelet is close to pod limit
-  kubernetes.rules.yaml: |
-    groups:
-    - name: kubernetes.rules
-      rules:
-      - record: pod_name:container_memory_usage_bytes:sum
-        expr: sum(container_memory_usage_bytes{container_name!="POD",pod_name!=""}) BY
-          (pod_name)
-      - record: pod_name:container_spec_cpu_shares:sum
-        expr: sum(container_spec_cpu_shares{container_name!="POD",pod_name!=""}) BY (pod_name)
-      - record: pod_name:container_cpu_usage:sum
-        expr: sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m]))
-          BY (pod_name)
-      - record: pod_name:container_fs_usage_bytes:sum
-        expr: sum(container_fs_usage_bytes{container_name!="POD",pod_name!=""}) BY (pod_name)
-      - record: namespace:container_memory_usage_bytes:sum
-        expr: sum(container_memory_usage_bytes{container_name!=""}) BY (namespace)
-      - record: namespace:container_spec_cpu_shares:sum
-        expr: sum(container_spec_cpu_shares{container_name!=""}) BY (namespace)
-      - record: namespace:container_cpu_usage:sum
-        expr: sum(rate(container_cpu_usage_seconds_total{container_name!="POD"}[5m]))
-          BY (namespace)
-      - record: cluster:memory_usage:ratio
-        expr: sum(container_memory_usage_bytes{container_name!="POD",pod_name!=""}) BY
-          (cluster) / sum(machine_memory_bytes) BY (cluster)
-      - record: cluster:container_spec_cpu_shares:ratio
-        expr: sum(container_spec_cpu_shares{container_name!="POD",pod_name!=""}) / 1000
-          / sum(machine_cpu_cores)
-      - record: cluster:container_cpu_usage:ratio
-        expr: sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m]))
-          / sum(machine_cpu_cores)
-      - record: apiserver_latency_seconds:quantile
-        expr: histogram_quantile(0.99, rate(apiserver_request_latencies_bucket[5m])) /
-          1e+06
-        labels:
-          quantile: "0.99"
-      - record: apiserver_latency:quantile_seconds
-        expr: histogram_quantile(0.9, rate(apiserver_request_latencies_bucket[5m])) /
-          1e+06
-        labels:
-          quantile: "0.9"
-      - record: apiserver_latency_seconds:quantile
-        expr: histogram_quantile(0.5, rate(apiserver_request_latencies_bucket[5m])) /
-          1e+06
-        labels:
-          quantile: "0.5"
-      - alert: APIServerLatencyHigh
-        expr: apiserver_latency_seconds:quantile{quantile="0.99",subresource!="log",verb!~"^(?:WATCH|WATCHLIST|PROXY|CONNECT)$"}
-          > 1
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: the API server has a 99th percentile latency of {{ $value }} seconds
-            for {{$labels.verb}} {{$labels.resource}}
-      - alert: APIServerLatencyHigh
-        expr: apiserver_latency_seconds:quantile{quantile="0.99",subresource!="log",verb!~"^(?:WATCH|WATCHLIST|PROXY|CONNECT)$"}
-          > 4
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: the API server has a 99th percentile latency of {{ $value }} seconds
-            for {{$labels.verb}} {{$labels.resource}}
-      - alert: APIServerErrorsHigh
-        expr: rate(apiserver_request_count{code=~"^(?:5..)$"}[5m]) / rate(apiserver_request_count[5m])
-          * 100 > 2
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: API server returns errors for {{ $value }}% of requests
-      - alert: APIServerErrorsHigh
-        expr: rate(apiserver_request_count{code=~"^(?:5..)$"}[5m]) / rate(apiserver_request_count[5m])
-          * 100 > 5
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: API server returns errors for {{ $value }}% of requests
-      - alert: K8SApiserverDown
-        expr: absent(up{job="apiserver"} == 1)
-        for: 20m
-        labels:
-          severity: critical
-        annotations:
-          description: No API servers are reachable or all have disappeared from service
-            discovery
-
-      - alert: K8sCertificateExpirationNotice
-        labels:
-          severity: warning
-        annotations:
-          description: Kubernetes API Certificate is expiring soon (less than 7 days)
-        expr: sum(apiserver_client_certificate_expiration_seconds_bucket{le="604800"}) > 0
-
-      - alert: K8sCertificateExpirationNotice
-        labels:
-          severity: critical
-        annotations:
-          description: Kubernetes API Certificate is expiring in less than 1 day
-        expr: sum(apiserver_client_certificate_expiration_seconds_bucket{le="86400"}) > 0
-  node.rules.yaml: |
-    groups:
-    - name: node.rules
-      rules:
-      - record: instance:node_cpu:rate:sum
-        expr: sum(rate(node_cpu{mode!="idle",mode!="iowait",mode!~"^(?:guest.*)$"}[3m]))
-          BY (instance)
-      - record: instance:node_filesystem_usage:sum
-        expr: sum((node_filesystem_size{mountpoint="/"} - node_filesystem_free{mountpoint="/"}))
-          BY (instance)
-      - record: instance:node_network_receive_bytes:rate:sum
-        expr: sum(rate(node_network_receive_bytes[3m])) BY (instance)
-      - record: instance:node_network_transmit_bytes:rate:sum
-        expr: sum(rate(node_network_transmit_bytes[3m])) BY (instance)
-      - record: instance:node_cpu:ratio
-        expr: sum(rate(node_cpu{mode!="idle"}[5m])) WITHOUT (cpu, mode) / ON(instance)
-          GROUP_LEFT() count(sum(node_cpu) BY (instance, cpu)) BY (instance)
-      - record: cluster:node_cpu:sum_rate5m
-        expr: sum(rate(node_cpu{mode!="idle"}[5m]))
-      - record: cluster:node_cpu:ratio
-        expr: cluster:node_cpu:rate5m / count(sum(node_cpu) BY (instance, cpu))
-      - alert: NodeExporterDown
-        expr: absent(up{job="node-exporter"} == 1)
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Prometheus could not scrape a node-exporter for more than 10m,
-            or node-exporters have disappeared from discovery
-      - alert: NodeDiskRunningFull
-        expr: predict_linear(node_filesystem_free[6h], 3600 * 24) < 0
-        for: 30m
-        labels:
-          severity: warning
-        annotations:
-          description: device {{$labels.device}} on node {{$labels.instance}} is running
-            full within the next 24 hours (mounted at {{$labels.mountpoint}})
-      - alert: NodeDiskRunningFull
-        expr: predict_linear(node_filesystem_free[30m], 3600 * 2) < 0
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: device {{$labels.device}} on node {{$labels.instance}} is running
-            full within the next 2 hours (mounted at {{$labels.mountpoint}})
-      - alert: InactiveRAIDDisk
-        expr: node_md_disks - node_md_disks_active > 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: '{{$value}} RAID disk(s) on node {{$labels.instance}} are inactive'
-  prometheus.rules.yaml: |
-    groups:
-    - name: prometheus.rules
-      rules:
-      - alert: PrometheusConfigReloadFailed
-        expr: prometheus_config_last_reload_successful == 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Reloading Prometheus' configuration has failed for {{$labels.namespace}}/{{$labels.pod}}
-      - alert: PrometheusNotificationQueueRunningFull
-        expr: predict_linear(prometheus_notifications_queue_length[5m], 60 * 30) > prometheus_notifications_queue_capacity
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Prometheus' alert notification queue is running full for {{$labels.namespace}}/{{
-            $labels.pod}}
-      - alert: PrometheusErrorSendingAlerts
-        expr: rate(prometheus_notifications_errors_total[5m]) / rate(prometheus_notifications_sent_total[5m])
-          > 0.01
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Errors while sending alerts from Prometheus {{$labels.namespace}}/{{
-            $labels.pod}} to Alertmanager {{$labels.Alertmanager}}
-      - alert: PrometheusErrorSendingAlerts
-        expr: rate(prometheus_notifications_errors_total[5m]) / rate(prometheus_notifications_sent_total[5m])
-          > 0.03
-        for: 10m
-        labels:
-          severity: critical
-        annotations:
-          description: Errors while sending alerts from Prometheus {{$labels.namespace}}/{{
-            $labels.pod}} to Alertmanager {{$labels.Alertmanager}}
-      - alert: PrometheusNotConnectedToAlertmanagers
-        expr: prometheus_notifications_alertmanagers_discovered < 1
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: Prometheus {{ $labels.namespace }}/{{ $labels.pod}} is not connected
-            to any Alertmanagers
-      - alert: PrometheusTSDBReloadsFailing
-        expr: increase(prometheus_tsdb_reloads_failures_total[2h]) > 0
-        for: 12h
-        labels:
-          severity: warning
-        annotations:
-          description: '{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}}
-            reload failures over the last four hours.'
-          summary: Prometheus has issues reloading data blocks from disk
-      - alert: PrometheusTSDBCompactionsFailing
-        expr: increase(prometheus_tsdb_compactions_failed_total[2h]) > 0
-        for: 12h
-        labels:
-          severity: warning
-        annotations:
-          description: '{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}}
-            compaction failures over the last four hours.'
-          summary: Prometheus has issues compacting sample blocks
-      - alert: PrometheusTSDBWALCorruptions
-        expr: tsdb_wal_corruptions_total > 0
-        for: 4h
-        labels:
-          severity: warning
-        annotations:
-          description: '{{$labels.job}} at {{$labels.instance}} has a corrupted write-ahead
-            log (WAL).'
-          summary: Prometheus write-ahead log is corrupted
-      - alert: PrometheusNotIngestingSamples
-        expr: rate(prometheus_tsdb_head_samples_appended_total[5m]) <= 0
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: "Prometheus {{ $labels.namespace }}/{{ $labels.pod}} isn't ingesting samples."
-          summary: "Prometheus isn't ingesting samples"
+  etcd.yaml: |-
+    {
+      "groups": [
+        {
+          "name": "etcd",
+          "rules": [
+            {
+              "alert": "etcdInsufficientMembers",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": insufficient members ({{ $value }})."
+              },
+              "expr": "sum(up{job=~\".*etcd.*\"} == bool 1) by (job) < ((count(up{job=~\".*etcd.*\"}) by (job) + 1) / 2)\n",
+              "for": "3m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "etcdNoLeader",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader."
+              },
+              "expr": "etcd_server_has_leader{job=~\".*etcd.*\"} == 0\n",
+              "for": "1m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "etcdHighNumberOfLeaderChanges",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": instance {{ $labels.instance }} has seen {{ $value }} leader changes within the last 30 minutes."
+              },
+              "expr": "rate(etcd_server_leader_changes_seen_total{job=~\".*etcd.*\"}[15m]) > 3\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdGRPCRequestsSlow",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": gRPC requests to {{ $labels.grpc_method }} are taking {{ $value }}s on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~\".*etcd.*\", grpc_type=\"unary\"}[5m])) by (job, instance, grpc_service, grpc_method, le))\n> 0.15\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "etcdMemberCommunicationSlow",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 0.15\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighNumberOfFailedProposals",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) > 5\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighFsyncDurations",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": 99th percentile fync durations are {{ $value }}s on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 0.5\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighCommitDurations",
+              "annotations": {
+                "message": "etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 0.25\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighNumberOfFailedHTTPRequests",
+              "annotations": {
+                "message": "{{ $value }}% of requests for {{ $labels.method }} failed on etcd instance {{ $labels.instance }}"
+              },
+              "expr": "sum(rate(etcd_http_failed_total{job=~\".*etcd.*\", code!=\"404\"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job=~\".*etcd.*\"}[5m]))\nBY (method) > 0.01\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "etcdHighNumberOfFailedHTTPRequests",
+              "annotations": {
+                "message": "{{ $value }}% of requests for {{ $labels.method }} failed on etcd instance {{ $labels.instance }}."
+              },
+              "expr": "sum(rate(etcd_http_failed_total{job=~\".*etcd.*\", code!=\"404\"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job=~\".*etcd.*\"}[5m]))\nBY (method) > 0.05\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "etcdHTTPRequestsSlow",
+              "annotations": {
+                "message": "etcd instance {{ $labels.instance }} HTTP requests to {{ $labels.method }} are slow."
+              },
+              "expr": "histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[5m]))\n> 0.15\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        }
+      ]
+    }
+  extra.yaml: |-
+    {
+      "groups": [
+        {
+          "name": "extra.rules",
+          "rules": [
+            {
+              "alert": "InactiveRAIDDisk",
+              "annotations": {
+                "message": "{{ $value }} RAID disk(s) on node {{ $labels.instance }} are inactive."
+              },
+              "expr": "node_md_disks - node_md_disks_active > 0",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        }
+      ]
+    }
+  kube.yaml: |-
+    {
+      "groups": [
+        {
+          "name": "k8s.rules",
+          "rules": [
+            {
+              "expr": "sum(rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\", container_name!=\"\"}[5m])) by (namespace)\n",
+              "record": "namespace:container_cpu_usage_seconds_total:sum_rate"
+            },
+            {
+              "expr": "sum by (namespace, pod_name, container_name) (\n  rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\", container_name!=\"\"}[5m])\n)\n",
+              "record": "namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate"
+            },
+            {
+              "expr": "sum(container_memory_usage_bytes{job=\"kubernetes-cadvisor\", image!=\"\", container_name!=\"\"}) by (namespace)\n",
+              "record": "namespace:container_memory_usage_bytes:sum"
+            },
+            {
+              "expr": "sum by (namespace, label_name) (\n   sum(rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\", container_name!=\"\"}[5m])) by (namespace, pod_name)\n * on (namespace, pod_name) group_left(label_name)\n   label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "record": "namespace_name:container_cpu_usage_seconds_total:sum_rate"
+            },
+            {
+              "expr": "sum by (namespace, label_name) (\n  sum(container_memory_usage_bytes{job=\"kubernetes-cadvisor\",image!=\"\", container_name!=\"\"}) by (pod_name, namespace)\n* on (namespace, pod_name) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "record": "namespace_name:container_memory_usage_bytes:sum"
+            },
+            {
+              "expr": "sum by (namespace, label_name) (\n  sum(kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"}) by (namespace, pod)\n* on (namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "record": "namespace_name:kube_pod_container_resource_requests_memory_bytes:sum"
+            },
+            {
+              "expr": "sum by (namespace, label_name) (\n  sum(kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"} and on(pod) kube_pod_status_scheduled{condition=\"true\"}) by (namespace, pod)\n* on (namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "record": "namespace_name:kube_pod_container_resource_requests_cpu_cores:sum"
+            }
+          ]
+        },
+        {
+          "name": "kube-scheduler.rules",
+          "rules": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(scheduler_e2e_scheduling_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.99"
+              },
+              "record": "cluster_quantile:scheduler_e2e_scheduling_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(scheduler_scheduling_algorithm_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.99"
+              },
+              "record": "cluster_quantile:scheduler_scheduling_algorithm_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(scheduler_binding_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.99"
+              },
+              "record": "cluster_quantile:scheduler_binding_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.9, sum(rate(scheduler_e2e_scheduling_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.9"
+              },
+              "record": "cluster_quantile:scheduler_e2e_scheduling_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.9, sum(rate(scheduler_scheduling_algorithm_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.9"
+              },
+              "record": "cluster_quantile:scheduler_scheduling_algorithm_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.9, sum(rate(scheduler_binding_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.9"
+              },
+              "record": "cluster_quantile:scheduler_binding_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.5, sum(rate(scheduler_e2e_scheduling_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.5"
+              },
+              "record": "cluster_quantile:scheduler_e2e_scheduling_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.5, sum(rate(scheduler_scheduling_algorithm_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.5"
+              },
+              "record": "cluster_quantile:scheduler_scheduling_algorithm_latency:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.5, sum(rate(scheduler_binding_latency_microseconds_bucket{job=\"kube-scheduler\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.5"
+              },
+              "record": "cluster_quantile:scheduler_binding_latency:histogram_quantile"
+            }
+          ]
+        },
+        {
+          "name": "kube-apiserver.rules",
+          "rules": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(apiserver_request_latencies_bucket{job=\"apiserver\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.99"
+              },
+              "record": "cluster_quantile:apiserver_request_latencies:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.9, sum(rate(apiserver_request_latencies_bucket{job=\"apiserver\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.9"
+              },
+              "record": "cluster_quantile:apiserver_request_latencies:histogram_quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.5, sum(rate(apiserver_request_latencies_bucket{job=\"apiserver\"}[5m])) without(instance, pod)) / 1e+06\n",
+              "labels": {
+                "quantile": "0.5"
+              },
+              "record": "cluster_quantile:apiserver_request_latencies:histogram_quantile"
+            }
+          ]
+        },
+        {
+          "name": "node.rules",
+          "rules": [
+            {
+              "expr": "sum(min(kube_pod_info) by (node))",
+              "record": ":kube_pod_info_node_count:"
+            },
+            {
+              "expr": "max(label_replace(kube_pod_info{job=\"kube-state-metrics\"}, \"pod\", \"$1\", \"pod\", \"(.*)\")) by (node, namespace, pod)\n",
+              "record": "node_namespace_pod:kube_pod_info:"
+            },
+            {
+              "expr": "count by (node) (sum by (node, cpu) (\n  node_cpu_seconds_total{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n))\n",
+              "record": "node:node_num_cpu:sum"
+            },
+            {
+              "expr": "1 - avg(rate(node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}[1m]))\n",
+              "record": ":node_cpu_utilisation:avg1m"
+            },
+            {
+              "expr": "1 - avg by (node) (\n  rate(node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}[1m])\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:)\n",
+              "record": "node:node_cpu_utilisation:avg1m"
+            },
+            {
+              "expr": "node:node_cpu_utilisation:avg1m\n  *\nnode:node_num_cpu:sum\n  /\nscalar(sum(node:node_num_cpu:sum))\n",
+              "record": "node:cluster_cpu_utilisation:ratio"
+            },
+            {
+              "expr": "sum(node_load1{job=\"node-exporter\"})\n/\nsum(node:node_num_cpu:sum)\n",
+              "record": ":node_cpu_saturation_load1:"
+            },
+            {
+              "expr": "sum by (node) (\n  node_load1{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n/\nnode:node_num_cpu:sum\n",
+              "record": "node:node_cpu_saturation_load1:"
+            },
+            {
+              "expr": "1 -\nsum(node_memory_MemFree_bytes{job=\"node-exporter\"} + node_memory_Cached_bytes{job=\"node-exporter\"} + node_memory_Buffers_bytes{job=\"node-exporter\"})\n/\nsum(node_memory_MemTotal_bytes{job=\"node-exporter\"})\n",
+              "record": ":node_memory_utilisation:"
+            },
+            {
+              "expr": "sum(node_memory_MemFree_bytes{job=\"node-exporter\"} + node_memory_Cached_bytes{job=\"node-exporter\"} + node_memory_Buffers_bytes{job=\"node-exporter\"})\n",
+              "record": ":node_memory_MemFreeCachedBuffers_bytes:sum"
+            },
+            {
+              "expr": "sum(node_memory_MemTotal_bytes{job=\"node-exporter\"})\n",
+              "record": ":node_memory_MemTotal_bytes:sum"
+            },
+            {
+              "expr": "sum by (node) (\n  (node_memory_MemFree_bytes{job=\"node-exporter\"} + node_memory_Cached_bytes{job=\"node-exporter\"} + node_memory_Buffers_bytes{job=\"node-exporter\"})\n  * on (namespace, pod) group_left(node)\n    node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_memory_bytes_available:sum"
+            },
+            {
+              "expr": "sum by (node) (\n  node_memory_MemTotal_bytes{job=\"node-exporter\"}\n  * on (namespace, pod) group_left(node)\n    node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_memory_bytes_total:sum"
+            },
+            {
+              "expr": "(node:node_memory_bytes_total:sum - node:node_memory_bytes_available:sum)\n/\nnode:node_memory_bytes_total:sum\n",
+              "record": "node:node_memory_utilisation:ratio"
+            },
+            {
+              "expr": "(node:node_memory_bytes_total:sum - node:node_memory_bytes_available:sum)\n/\nscalar(sum(node:node_memory_bytes_total:sum))\n",
+              "record": "node:cluster_memory_utilisation:ratio"
+            },
+            {
+              "expr": "1e3 * sum(\n  (rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m])\n + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n)\n",
+              "record": ":node_memory_swap_io_bytes:sum_rate"
+            },
+            {
+              "expr": "1 -\nsum by (node) (\n  (node_memory_MemFree_bytes{job=\"node-exporter\"} + node_memory_Cached_bytes{job=\"node-exporter\"} + node_memory_Buffers_bytes{job=\"node-exporter\"})\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n/\nsum by (node) (\n  node_memory_MemTotal_bytes{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_memory_utilisation:"
+            },
+            {
+              "expr": "1 - (node:node_memory_bytes_available:sum / node:node_memory_bytes_total:sum)\n",
+              "record": "node:node_memory_utilisation_2:"
+            },
+            {
+              "expr": "1e3 * sum by (node) (\n  (rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m])\n + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n * on (namespace, pod) group_left(node)\n   node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_memory_swap_io_bytes:sum_rate"
+            },
+            {
+              "expr": "avg(irate(node_disk_io_time_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m]))\n",
+              "record": ":node_disk_utilisation:avg_irate"
+            },
+            {
+              "expr": "avg by (node) (\n  irate(node_disk_io_time_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m])\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_disk_utilisation:avg_irate"
+            },
+            {
+              "expr": "avg(irate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m]) / 1e3)\n",
+              "record": ":node_disk_saturation:avg_irate"
+            },
+            {
+              "expr": "avg by (node) (\n  irate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m]) / 1e3\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_disk_saturation:avg_irate"
+            },
+            {
+              "expr": "max by (namespace, pod, device) ((node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"}\n- node_filesystem_avail_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"})\n/ node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"})\n",
+              "record": "node:node_filesystem_usage:"
+            },
+            {
+              "expr": "max by (namespace, pod, device) (node_filesystem_avail_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"} / node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"})\n",
+              "record": "node:node_filesystem_avail:"
+            },
+            {
+              "expr": "sum(irate(node_network_receive_bytes_total{job=\"node-exporter\",device!~\"veth.+\"}[1m])) +\nsum(irate(node_network_transmit_bytes_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]))\n",
+              "record": ":node_net_utilisation:sum_irate"
+            },
+            {
+              "expr": "sum by (node) (\n  (irate(node_network_receive_bytes_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]) +\n  irate(node_network_transmit_bytes_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]))\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_net_utilisation:sum_irate"
+            },
+            {
+              "expr": "sum(irate(node_network_receive_drop_total{job=\"node-exporter\",device!~\"veth.+\"}[1m])) +\nsum(irate(node_network_transmit_drop_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]))\n",
+              "record": ":node_net_saturation:sum_irate"
+            },
+            {
+              "expr": "sum by (node) (\n  (irate(node_network_receive_drop_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]) +\n  irate(node_network_transmit_drop_total{job=\"node-exporter\",device!~\"veth.+\"}[1m]))\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "record": "node:node_net_saturation:sum_irate"
+            },
+            {
+              "expr": "max(\n  max(\n    kube_pod_info{job=\"kube-state-metrics\", host_ip!=\"\"}\n  ) by (node, host_ip)\n  * on (host_ip) group_right (node)\n  label_replace(\n    (max(node_filesystem_files{job=\"node-exporter\", mountpoint=\"/\"}) by (instance)), \"host_ip\", \"$1\", \"instance\", \"(.*):.*\"\n  )\n) by (node)\n",
+              "record": "node:node_inodes_total:"
+            },
+            {
+              "expr": "max(\n  max(\n    kube_pod_info{job=\"kube-state-metrics\", host_ip!=\"\"}\n  ) by (node, host_ip)\n  * on (host_ip) group_right (node)\n  label_replace(\n    (max(node_filesystem_files_free{job=\"node-exporter\", mountpoint=\"/\"}) by (instance)), \"host_ip\", \"$1\", \"instance\", \"(.*):.*\"\n  )\n) by (node)\n",
+              "record": "node:node_inodes_free:"
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-absent",
+          "rules": [
+            {
+              "alert": "KubeAPIDown",
+              "annotations": {
+                "message": "KubeAPI has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapidown"
+              },
+              "expr": "absent(up{job=\"apiserver\"} == 1)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeControllerManagerDown",
+              "annotations": {
+                "message": "KubeControllerManager has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontrollermanagerdown"
+              },
+              "expr": "absent(up{job=\"kube-controller-manager\"} == 1)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeSchedulerDown",
+              "annotations": {
+                "message": "KubeScheduler has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeschedulerdown"
+              },
+              "expr": "absent(up{job=\"kube-scheduler\"} == 1)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeletDown",
+              "annotations": {
+                "message": "Kubelet has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletdown"
+              },
+              "expr": "absent(up{job=\"kubelet\"} == 1)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-apps",
+          "rules": [
+            {
+              "alert": "KubePodCrashLooping",
+              "annotations": {
+                "message": "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is restarting {{ printf \"%.2f\" $value }} times / 5 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping"
+              },
+              "expr": "rate(kube_pod_container_status_restarts_total{job=\"kube-state-metrics\"}[15m]) * 60 * 5 > 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubePodNotReady",
+              "annotations": {
+                "message": "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than an hour.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready"
+              },
+              "expr": "sum by (namespace, pod) (kube_pod_status_phase{job=\"kube-state-metrics\", phase=~\"Pending|Unknown\"}) > 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeDeploymentGenerationMismatch",
+              "annotations": {
+                "message": "Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentgenerationmismatch"
+              },
+              "expr": "kube_deployment_status_observed_generation{job=\"kube-state-metrics\"}\n  !=\nkube_deployment_metadata_generation{job=\"kube-state-metrics\"}\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeDeploymentReplicasMismatch",
+              "annotations": {
+                "message": "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than an hour.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch"
+              },
+              "expr": "kube_deployment_spec_replicas{job=\"kube-state-metrics\"}\n  !=\nkube_deployment_status_replicas_available{job=\"kube-state-metrics\"}\n",
+              "for": "1h",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeStatefulSetReplicasMismatch",
+              "annotations": {
+                "message": "StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetreplicasmismatch"
+              },
+              "expr": "kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\"}\n  !=\nkube_statefulset_status_replicas{job=\"kube-state-metrics\"}\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeStatefulSetGenerationMismatch",
+              "annotations": {
+                "message": "StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetgenerationmismatch"
+              },
+              "expr": "kube_statefulset_status_observed_generation{job=\"kube-state-metrics\"}\n  !=\nkube_statefulset_metadata_generation{job=\"kube-state-metrics\"}\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeStatefulSetUpdateNotRolledOut",
+              "annotations": {
+                "message": "StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetupdatenotrolledout"
+              },
+              "expr": "max without (revision) (\n  kube_statefulset_status_current_revision{job=\"kube-state-metrics\"}\n    unless\n  kube_statefulset_status_update_revision{job=\"kube-state-metrics\"}\n)\n  *\n(\n  kube_statefulset_replicas{job=\"kube-state-metrics\"}\n    !=\n  kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\"}\n)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeDaemonSetRolloutStuck",
+              "annotations": {
+                "message": "Only {{ $value }}% of the desired Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are scheduled and ready.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetrolloutstuck"
+              },
+              "expr": "kube_daemonset_status_number_ready{job=\"kube-state-metrics\"}\n  /\nkube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"} * 100 < 100\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeDaemonSetNotScheduled",
+              "annotations": {
+                "message": "{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetnotscheduled"
+              },
+              "expr": "kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  -\nkube_daemonset_status_current_number_scheduled{job=\"kube-state-metrics\"} > 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeDaemonSetMisScheduled",
+              "annotations": {
+                "message": "{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetmisscheduled"
+              },
+              "expr": "kube_daemonset_status_number_misscheduled{job=\"kube-state-metrics\"} > 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeCronJobRunning",
+              "annotations": {
+                "message": "CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} is taking more than 1h to complete.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecronjobrunning"
+              },
+              "expr": "time() - kube_cronjob_next_schedule_time{job=\"kube-state-metrics\"} > 3600\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeJobCompletion",
+              "annotations": {
+                "message": "Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than one hour to complete.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobcompletion"
+              },
+              "expr": "kube_job_spec_completions{job=\"kube-state-metrics\"} - kube_job_status_succeeded{job=\"kube-state-metrics\"}  > 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeJobFailed",
+              "annotations": {
+                "message": "Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobfailed"
+              },
+              "expr": "kube_job_status_failed{job=\"kube-state-metrics\"}  > 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-resources",
+          "rules": [
+            {
+              "alert": "KubeCPUOvercommit",
+              "annotations": {
+                "message": "Cluster has overcommitted CPU resource requests for Pods and cannot tolerate node failure.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit"
+              },
+              "expr": "sum(namespace_name:kube_pod_container_resource_requests_cpu_cores:sum)\n  /\nsum(node:node_num_cpu:sum)\n  >\n(count(node:node_num_cpu:sum)-1) / count(node:node_num_cpu:sum)\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeMemOvercommit",
+              "annotations": {
+                "message": "Cluster has overcommitted memory resource requests for Pods and cannot tolerate node failure.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememovercommit"
+              },
+              "expr": "sum(namespace_name:kube_pod_container_resource_requests_memory_bytes:sum)\n  /\nsum(node_memory_MemTotal_bytes)\n  >\n(count(node:node_num_cpu:sum)-1)\n  /\ncount(node:node_num_cpu:sum)\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeCPUOvercommit",
+              "annotations": {
+                "message": "Cluster has overcommitted CPU resource requests for Namespaces.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit"
+              },
+              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"requests.cpu\"})\n  /\nsum(node:node_num_cpu:sum)\n  > 1.5\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeMemOvercommit",
+              "annotations": {
+                "message": "Cluster has overcommitted memory resource requests for Namespaces.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememovercommit"
+              },
+              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"requests.memory\"})\n  /\nsum(node_memory_MemTotal_bytes{job=\"node-exporter\"})\n  > 1.5\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeQuotaExceeded",
+              "annotations": {
+                "message": "Namespace {{ $labels.namespace }} is using {{ printf \"%0.0f\" $value }}% of its {{ $labels.resource }} quota.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubequotaexceeded"
+              },
+              "expr": "100 * kube_resourcequota{job=\"kube-state-metrics\", type=\"used\"}\n  / ignoring(instance, job, type)\n(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\"} > 0)\n  > 90\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "CPUThrottlingHigh",
+              "annotations": {
+                "message": "{{ printf \"%0.0f\" $value }}% throttling of CPU in namespace {{ $labels.namespace }} for container {{ $labels.container_name }} in pod {{ $labels.pod_name }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-cputhrottlinghigh"
+              },
+              "expr": "100 * sum(increase(container_cpu_cfs_throttled_periods_total{container_name!=\"\", }[5m])) by (container_name, pod_name, namespace)\n  /\nsum(increase(container_cpu_cfs_periods_total{}[5m])) by (container_name, pod_name, namespace)\n  > 100 \n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-storage",
+          "rules": [
+            {
+              "alert": "KubePersistentVolumeUsageCritical",
+              "annotations": {
+                "message": "The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ printf \"%0.2f\" $value }}% free.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumeusagecritical"
+              },
+              "expr": "100 * kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n  /\nkubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n  < 3\n",
+              "for": "1m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubePersistentVolumeFullInFourDays",
+              "annotations": {
+                "message": "Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to fill up within four days. Currently {{ printf \"%0.2f\" $value }}% is available.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumefullinfourdays"
+              },
+              "expr": "100 * (\n  kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n    /\n  kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n) < 15\nand\npredict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\"}[6h], 4 * 24 * 3600) < 0\n",
+              "for": "5m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubePersistentVolumeErrors",
+              "annotations": {
+                "message": "The persistent volume {{ $labels.persistentvolume }} has status {{ $labels.phase }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumeerrors"
+              },
+              "expr": "kube_persistentvolume_status_phase{phase=~\"Failed|Pending\",job=\"kube-state-metrics\"} > 0\n",
+              "for": "5m",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
+        },
+        {
+          "name": "kubernetes-system",
+          "rules": [
+            {
+              "alert": "KubeNodeNotReady",
+              "annotations": {
+                "message": "{{ $labels.node }} has been unready for more than an hour.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubenodenotready"
+              },
+              "expr": "kube_node_status_condition{job=\"kube-state-metrics\",condition=\"Ready\",status=\"true\"} == 0\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeVersionMismatch",
+              "annotations": {
+                "message": "There are {{ $value }} different semantic versions of Kubernetes components running.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeversionmismatch"
+              },
+              "expr": "count(count by (gitVersion) (label_replace(kubernetes_build_info{job!=\"coredns\"},\"gitVersion\",\"$1\",\"gitVersion\",\"(v[0-9]*.[0-9]*.[0-9]*).*\"))) > 1\n",
+              "for": "1h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeClientErrors",
+              "annotations": {
+                "message": "Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ printf \"%0.0f\" $value }}% errors.'",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclienterrors"
+              },
+              "expr": "(sum(rate(rest_client_requests_total{code=~\"5..\"}[5m])) by (instance, job)\n  /\nsum(rate(rest_client_requests_total[5m])) by (instance, job))\n* 100 > 1\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeClientErrors",
+              "annotations": {
+                "message": "Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ printf \"%0.0f\" $value }} errors / second.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclienterrors"
+              },
+              "expr": "sum(rate(ksm_scrape_error_total{job=\"kube-state-metrics\"}[5m])) by (instance, job) > 0.1\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeletTooManyPods",
+              "annotations": {
+                "message": "Kubelet {{ $labels.instance }} is running {{ $value }} Pods, close to the limit of 110.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods"
+              },
+              "expr": "kubelet_running_pod_count{job=\"kubelet\"} > 110 * 0.9\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeAPILatencyHigh",
+              "annotations": {
+                "message": "The API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh"
+              },
+              "expr": "cluster_quantile:apiserver_request_latencies:histogram_quantile{job=\"apiserver\",quantile=\"0.99\",subresource!=\"log\",verb!~\"^(?:LIST|WATCH|WATCHLIST|PROXY|CONNECT)$\"} > 1\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeAPILatencyHigh",
+              "annotations": {
+                "message": "The API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh"
+              },
+              "expr": "cluster_quantile:apiserver_request_latencies:histogram_quantile{job=\"apiserver\",quantile=\"0.99\",subresource!=\"log\",verb!~\"^(?:LIST|WATCH|WATCHLIST|PROXY|CONNECT)$\"} > 4\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeAPIErrorsHigh",
+              "annotations": {
+                "message": "API server is returning errors for {{ $value }}% of requests.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
+              },
+              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m])) without(instance, pod)\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) without(instance, pod) * 100 > 10\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeAPIErrorsHigh",
+              "annotations": {
+                "message": "API server is returning errors for {{ $value }}% of requests.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
+              },
+              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m])) without(instance, pod)\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) without(instance, pod) * 100 > 5\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeClientCertificateExpiration",
+              "annotations": {
+                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 7 days.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
+              },
+              "expr": "histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 604800\n",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeClientCertificateExpiration",
+              "annotations": {
+                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 24 hours.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
+              },
+              "expr": "histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 86400\n",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
+        }
+      ]
+    }
+  kubeprom.yaml: |-
+    {
+      "groups": [
+        {
+          "name": "kube-prometheus-node-recording.rules",
+          "rules": [
+            {
+              "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\",mode!=\"iowait\"}[3m])) BY (instance)",
+              "record": "instance:node_cpu:rate:sum"
+            },
+            {
+              "expr": "sum((node_filesystem_size_bytes{mountpoint=\"/\"} - node_filesystem_free_bytes{mountpoint=\"/\"})) BY (instance)",
+              "record": "instance:node_filesystem_usage:sum"
+            },
+            {
+              "expr": "sum(rate(node_network_receive_bytes_total[3m])) BY (instance)",
+              "record": "instance:node_network_receive_bytes:rate:sum"
+            },
+            {
+              "expr": "sum(rate(node_network_transmit_bytes_total[3m])) BY (instance)",
+              "record": "instance:node_network_transmit_bytes:rate:sum"
+            },
+            {
+              "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\",mode!=\"iowait\"}[5m])) WITHOUT (cpu, mode) / ON(instance) GROUP_LEFT() count(sum(node_cpu_seconds_total) BY (instance, cpu)) BY (instance)",
+              "record": "instance:node_cpu:ratio"
+            },
+            {
+              "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\",mode!=\"iowait\"}[5m]))",
+              "record": "cluster:node_cpu:sum_rate5m"
+            },
+            {
+              "expr": "cluster:node_cpu_seconds_total:rate5m / count(sum(node_cpu_seconds_total) BY (instance, cpu))",
+              "record": "cluster:node_cpu:ratio"
+            }
+          ]
+        },
+        {
+          "name": "kube-prometheus-node-alerting.rules",
+          "rules": [
+            {
+              "alert": "NodeDiskRunningFull",
+              "annotations": {
+                "message": "Device {{ $labels.device }} of node-exporter {{ $labels.namespace }}/{{ $labels.pod }} will be full within the next 24 hours."
+              },
+              "expr": "(node:node_filesystem_usage: > 0.85) and (predict_linear(node:node_filesystem_avail:[6h], 3600 * 24) < 0)\n",
+              "for": "30m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "NodeDiskRunningFull",
+              "annotations": {
+                "message": "Device {{ $labels.device }} of node-exporter {{ $labels.namespace }}/{{ $labels.pod }} will be full within the next 2 hours."
+              },
+              "expr": "(node:node_filesystem_usage: > 0.85) and (predict_linear(node:node_filesystem_avail:[30m], 3600 * 2) < 0)\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
+        },
+        {
+          "name": "prometheus.rules",
+          "rules": [
+            {
+              "alert": "PrometheusConfigReloadFailed",
+              "annotations": {
+                "description": "Reloading Prometheus' configuration has failed for {{$labels.namespace}}/{{$labels.pod}}",
+                "summary": "Reloading Prometheus' configuration failed"
+              },
+              "expr": "prometheus_config_last_reload_successful{job=\"prometheus\"} == 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusNotificationQueueRunningFull",
+              "annotations": {
+                "description": "Prometheus' alert notification queue is running full for {{$labels.namespace}}/{{ $labels.pod}}",
+                "summary": "Prometheus' alert notification queue is running full"
+              },
+              "expr": "predict_linear(prometheus_notifications_queue_length{job=\"prometheus\"}[5m], 60 * 30) > prometheus_notifications_queue_capacity{job=\"prometheus\"}\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusErrorSendingAlerts",
+              "annotations": {
+                "description": "Errors while sending alerts from Prometheus {{$labels.namespace}}/{{ $labels.pod}} to Alertmanager {{$labels.Alertmanager}}",
+                "summary": "Errors while sending alert from Prometheus"
+              },
+              "expr": "rate(prometheus_notifications_errors_total{job=\"prometheus\"}[5m]) / rate(prometheus_notifications_sent_total{job=\"prometheus\"}[5m]) > 0.01\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusErrorSendingAlerts",
+              "annotations": {
+                "description": "Errors while sending alerts from Prometheus {{$labels.namespace}}/{{ $labels.pod}} to Alertmanager {{$labels.Alertmanager}}",
+                "summary": "Errors while sending alerts from Prometheus"
+              },
+              "expr": "rate(prometheus_notifications_errors_total{job=\"prometheus\"}[5m]) / rate(prometheus_notifications_sent_total{job=\"prometheus\"}[5m]) > 0.03\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "PrometheusNotConnectedToAlertmanagers",
+              "annotations": {
+                "description": "Prometheus {{ $labels.namespace }}/{{ $labels.pod}} is not connected to any Alertmanagers",
+                "summary": "Prometheus is not connected to any Alertmanagers"
+              },
+              "expr": "prometheus_notifications_alertmanagers_discovered{job=\"prometheus\"} < 1\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTSDBReloadsFailing",
+              "annotations": {
+                "description": "{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}} reload failures over the last four hours.",
+                "summary": "Prometheus has issues reloading data blocks from disk"
+              },
+              "expr": "increase(prometheus_tsdb_reloads_failures_total{job=\"prometheus\"}[2h]) > 0\n",
+              "for": "12h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTSDBCompactionsFailing",
+              "annotations": {
+                "description": "{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}} compaction failures over the last four hours.",
+                "summary": "Prometheus has issues compacting sample blocks"
+              },
+              "expr": "increase(prometheus_tsdb_compactions_failed_total{job=\"prometheus\"}[2h]) > 0\n",
+              "for": "12h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTSDBWALCorruptions",
+              "annotations": {
+                "description": "{{$labels.job}} at {{$labels.instance}} has a corrupted write-ahead log (WAL).",
+                "summary": "Prometheus write-ahead log is corrupted"
+              },
+              "expr": "tsdb_wal_corruptions_total{job=\"prometheus\"} > 0\n",
+              "for": "4h",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusNotIngestingSamples",
+              "annotations": {
+                "description": "Prometheus {{ $labels.namespace }}/{{ $labels.pod}} isn't ingesting samples.",
+                "summary": "Prometheus isn't ingesting samples"
+              },
+              "expr": "rate(prometheus_tsdb_head_samples_appended_total{job=\"prometheus\"}[5m]) <= 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTargetScrapesDuplicate",
+              "annotations": {
+                "description": "{{$labels.namespace}}/{{$labels.pod}} has many samples rejected due to duplicate timestamps but different values",
+                "summary": "Prometheus has many samples rejected"
+              },
+              "expr": "increase(prometheus_target_scrapes_sample_duplicate_timestamp_total{job=\"prometheus\"}[5m]) > 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "name": "general.rules",
+          "rules": [
+            {
+              "alert": "TargetDown",
+              "annotations": {
+                "message": "{{ $value }}% of the {{ $labels.job }} targets are down."
+              },
+              "expr": "100 * (count(up == 0) BY (job) / count(up) BY (job)) > 10",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        }
+      ]
+    }

aws/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

aws/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=4021467b7f280ceb54320333690e8574a3bd8d84"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

aws/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.12"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -78,7 +78,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -123,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

aws/container-linux/kubernetes/controllers.tf

@@ -68,10 +68,10 @@ data "template_file" "controller-configs" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    kubeconfig            = "${indent(10, module.bootkube.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

aws/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 # Outputs for Kubernetes Ingress
 
 output "ingress_dns_name" {
@@ -5,6 +9,11 @@ output "ingress_dns_name" {
   description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
 }
 
+output "ingress_zone_id" {
+  value       = "${aws_lb.nlb.zone_id}"
+  description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
+}
+
 # Outputs for worker pools
 
 output "vpc_id" {
@@ -23,7 +32,7 @@ output "worker_security_groups" {
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+  value = "${module.bootkube.kubeconfig-kubelet}"
 }
 
 # Outputs for custom load balancing

aws/container-linux/kubernetes/variables.tf

@@ -31,13 +31,13 @@ variable "worker_count" {
 
 variable "controller_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for controllers"
 }
 
 variable "worker_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for workers"
 }
 

aws/container-linux/kubernetes/workers.tf

@@ -13,7 +13,7 @@ module "workers" {
   spot_price      = "${var.worker_price}"
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig}"
+  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"

aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -51,7 +51,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -93,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -111,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.3 \
+            docker://k8s.gcr.io/hyperkube:v1.13.4 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

aws/container-linux/kubernetes/workers/variables.tf

@@ -30,7 +30,7 @@ variable "count" {
 
 variable "instance_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type"
 }
 

aws/container-linux/kubernetes/workers/workers.tf

@@ -77,9 +77,9 @@ data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {
-    kubeconfig            = "${indent(10, var.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, var.kubeconfig)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

aws/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [spot](https://typhoon.psdn.io/cl/aws/#spot) workers

aws/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=4021467b7f280ceb54320333690e8574a3bd8d84"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -40,7 +40,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -78,8 +78,8 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, start, --no-block, kubelet.service]

aws/fedora-atomic/kubernetes/controllers.tf

@@ -60,10 +60,10 @@ data "template_file" "controller-cloudinit" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    kubeconfig            = "${indent(6, module.bootkube.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(6, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

aws/fedora-atomic/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 # Outputs for Kubernetes Ingress
 
 output "ingress_dns_name" {
@@ -5,6 +9,11 @@ output "ingress_dns_name" {
   description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
 }
 
+output "ingress_zone_id" {
+  value       = "${aws_lb.nlb.zone_id}"
+  description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
+}
+
 # Outputs for worker pools
 
 output "vpc_id" {
@@ -23,7 +32,7 @@ output "worker_security_groups" {
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+  value = "${module.bootkube.kubeconfig-kubelet}"
 }
 
 # Outputs for custom load balancing

aws/fedora-atomic/kubernetes/variables.tf

@@ -31,13 +31,13 @@ variable "worker_count" {
 
 variable "controller_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for controllers"
 }
 
 variable "worker_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for workers"
 }
 

aws/fedora-atomic/kubernetes/workers.tf

@@ -12,7 +12,7 @@ module "workers" {
   spot_price      = "${var.worker_price}"
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig}"
+  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"

aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -19,7 +19,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -54,7 +54,7 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default

aws/fedora-atomic/kubernetes/workers/variables.tf

@@ -30,7 +30,7 @@ variable "count" {
 
 variable "instance_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type"
 }
 

aws/fedora-atomic/kubernetes/workers/workers.tf

@@ -70,9 +70,9 @@ data "template_file" "worker-cloudinit" {
   template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
 
   vars = {
-    kubeconfig            = "${indent(6, var.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(6, var.kubeconfig)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

azure/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

azure/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=4021467b7f280ceb54320333690e8574a3bd8d84"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

azure/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.12"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -78,7 +78,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -123,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

azure/container-linux/kubernetes/controllers.tf

@@ -121,10 +121,10 @@ resource "azurerm_public_ip" "controllers" {
   count               = "${var.controller_count}"
   resource_group_name = "${azurerm_resource_group.cluster.name}"
 
-  name                         = "${var.cluster_name}-controller-${count.index}"
-  location                     = "${azurerm_resource_group.cluster.location}"
-  sku                          = "Standard"
-  public_ip_address_allocation = "static"
+  name              = "${var.cluster_name}-controller-${count.index}"
+  location          = "${azurerm_resource_group.cluster.location}"
+  sku               = "Standard"
+  allocation_method = "Static"
 }
 
 # Controller Ignition configs
@@ -149,10 +149,10 @@ data "template_file" "controller-configs" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    kubeconfig            = "${indent(10, module.bootkube.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

azure/container-linux/kubernetes/lb.tf

@@ -17,20 +17,20 @@ resource "azurerm_dns_a_record" "apiserver" {
 resource "azurerm_public_ip" "apiserver-ipv4" {
   resource_group_name = "${azurerm_resource_group.cluster.name}"
 
-  name                         = "${var.cluster_name}-apiserver-ipv4"
-  location                     = "${var.region}"
-  sku                          = "Standard"
-  public_ip_address_allocation = "static"
+  name              = "${var.cluster_name}-apiserver-ipv4"
+  location          = "${var.region}"
+  sku               = "Standard"
+  allocation_method = "Static"
 }
 
 # Static IPv4 address for the ingress frontend
 resource "azurerm_public_ip" "ingress-ipv4" {
   resource_group_name = "${azurerm_resource_group.cluster.name}"
 
-  name                         = "${var.cluster_name}-ingress-ipv4"
-  location                     = "${var.region}"
-  sku                          = "Standard"
-  public_ip_address_allocation = "static"
+  name              = "${var.cluster_name}-ingress-ipv4"
+  location          = "${var.region}"
+  sku               = "Standard"
+  allocation_method = "Static"
 }
 
 # Network Load Balancer for apiservers and ingress

azure/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 # Outputs for Kubernetes Ingress
 
 output "ingress_static_ipv4" {
@@ -28,5 +32,5 @@ output "backend_address_pool_id" {
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+  value = "${module.bootkube.kubeconfig-kubelet}"
 }

azure/container-linux/kubernetes/require.tf

@@ -5,7 +5,7 @@ terraform {
 }
 
 provider "azurerm" {
-  version = "~> 1.19"
+  version = "~> 1.21"
 }
 
 provider "local" {

azure/container-linux/kubernetes/workers.tf

@@ -15,7 +15,7 @@ module "workers" {
   priority = "${var.worker_priority}"
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig}"
+  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"

azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -51,7 +51,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -93,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -111,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.3 \
+            docker://k8s.gcr.io/hyperkube:v1.13.4 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')

azure/container-linux/kubernetes/workers/workers.tf

@@ -106,9 +106,9 @@ data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {
-    kubeconfig            = "${indent(10, var.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, var.kubeconfig)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

bare-metal/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

bare-metal/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=4021467b7f280ceb54320333690e8574a3bd8d84"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name                    = "${var.cluster_name}"
   api_servers                     = ["${var.k8s_domain_name}"]

bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.12"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
@@ -90,7 +90,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -128,7 +128,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -63,7 +63,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -89,7 +89,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,3 @@
-output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
 }

bare-metal/container-linux/kubernetes/profiles.tf

@@ -160,12 +160,12 @@ data "template_file" "controller-configs" {
   template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.controller_domains, count.index)}"
-    etcd_name             = "${element(var.controller_names, count.index)}"
-    etcd_initial_cluster  = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.controller_domains, count.index)}"
+    etcd_name              = "${element(var.controller_names, count.index)}"
+    etcd_initial_cluster   = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }
 
@@ -191,10 +191,10 @@ data "template_file" "worker-configs" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.worker_domains, count.index)}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.worker_domains, count.index)}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }
 

bare-metal/container-linux/kubernetes/ssh.tf

@@ -18,7 +18,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -94,7 +94,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

bare-metal/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

bare-metal/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=4021467b7f280ceb54320333690e8574a3bd8d84"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${var.k8s_domain_name}"]

bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -40,7 +40,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -84,8 +84,8 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, kubelet.path]

bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -19,7 +19,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -60,7 +60,7 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

bare-metal/fedora-atomic/kubernetes/outputs.tf

@@ -1,3 +1,3 @@
-output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
 }

bare-metal/fedora-atomic/kubernetes/profiles.tf

@@ -55,12 +55,12 @@ data "template_file" "controller-configs" {
   template = "${file("${path.module}/cloudinit/controller.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.controller_domains, count.index)}"
-    etcd_name             = "${element(var.controller_names, count.index)}"
-    etcd_initial_cluster  = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.controller_domains, count.index)}"
+    etcd_name              = "${element(var.controller_names, count.index)}"
+    etcd_initial_cluster   = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }
 
@@ -79,9 +79,9 @@ data "template_file" "worker-configs" {
   template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.worker_domains, count.index)}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.worker_domains, count.index)}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }

bare-metal/fedora-atomic/kubernetes/ssh.tf

@@ -18,7 +18,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -92,7 +92,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

digital-ocean/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

digital-ocean/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=4021467b7f280ceb54320333690e8574a3bd8d84"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.12"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -86,7 +86,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -125,7 +125,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -59,7 +59,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -95,7 +95,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.3
+          KUBELET_IMAGE_TAG=v1.13.4
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -113,7 +113,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.3 \
+            docker://k8s.gcr.io/hyperkube:v1.13.4 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

digital-ocean/container-linux/kubernetes/controllers.tf

@@ -83,9 +83,9 @@ data "template_file" "controller-configs" {
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
 
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster  = "${join(",", data.template_file.etcds.*.rendered)}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    etcd_initial_cluster   = "${join(",", data.template_file.etcds.*.rendered)}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

digital-ocean/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 output "controllers_dns" {
   value = "${digitalocean_record.controllers.0.fqdn}"
 }

digital-ocean/container-linux/kubernetes/ssh.tf

@@ -10,7 +10,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -78,7 +78,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

digital-ocean/container-linux/kubernetes/workers.tf

@@ -66,7 +66,7 @@ data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

digital-ocean/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

digital-ocean/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=4021467b7f280ceb54320333690e8574a3bd8d84"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -40,7 +40,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -75,8 +75,8 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, kubelet.path]

digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -19,7 +19,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -51,7 +51,7 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

digital-ocean/fedora-atomic/kubernetes/controllers.tf

@@ -77,9 +77,9 @@ data "template_file" "controller-cloudinit" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

digital-ocean/fedora-atomic/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 output "controllers_dns" {
   value = "${digitalocean_record.controllers.0.fqdn}"
 }

digital-ocean/fedora-atomic/kubernetes/ssh.tf

@@ -10,7 +10,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -76,7 +76,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

digital-ocean/fedora-atomic/kubernetes/workers.tf

@@ -59,8 +59,8 @@ data "template_file" "worker-cloudinit" {
   template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
 
   vars = {
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

docs/addons/grafana.md

@@ -14,7 +14,8 @@ kubectl port-forward grafana-POD-ID 8080 -n monitoring
 
 Visit [127.0.0.1:8080](http://127.0.0.1:8080) to view the bundled dashboards.
 
-![Grafana Capacity Planning](../img/grafana-capacity.png)
-![Grafana Control Plane](../img/grafana-control-plane.png)
-![Grafana Node View](../img/grafana-node.png)
+![Grafana etcd](../img/grafana-etcd.png)
+![Grafana resources cluster](../img/grafana-resources-cluster.png)
+![Grafana usage cluster](../img/grafana-usage-cluster.png)
+![Grafana usage node](../img/grafana-usage-node.png)
 

docs/addons/heapster.md

@@ -1,6 +1,6 @@
 # Heapster
 
-[Heapster](https://kubernetes.io/docs/user-guide/monitoring/) collects data from apiservers and kubelets and exposes it through a REST API. This API powers the `kubectl top` command and Kubernetes dashboard graphs.
+[Heapster](https://kubernetes.io/docs/user-guide/monitoring/) collects data from apiservers and kubelets and exposes it through a REST API. This API powers the `kubectl top` command.
 
 ## Create
 

docs/advanced/worker-pools.md

@@ -16,7 +16,7 @@ Create a cluster following the AWS [tutorial](../cl/aws.md#cluster). Define a wo
 
 ```tf
 module "tempest-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.12.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.13.4"
   
   providers = {
     aws = "aws.default"
@@ -67,7 +67,7 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
 | count | Number of instances | 1 | 3 |
-| instance_type | EC2 instance type | "t2.small" | "t2.medium" |
+| instance_type | EC2 instance type | "t3.small" | "t3.medium" |
 | os_image | AMI channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | spot_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
@@ -82,7 +82,7 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define
 
 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.12.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.13.4"
   
   providers = {
     azurerm = "azurerm.default"
@@ -152,7 +152,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu
 
 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.12.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.13.4"
 
   providers = {
     google = "google.default"
@@ -187,11 +187,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.12.3
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.12.3
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.12.3
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.12.3
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.12.3
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.13.4
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.13.4
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.13.4
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.13.4
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.13.4
 ```
 
 ### Variables

docs/announce.md

@@ -18,7 +18,7 @@ Fedora Atomic is a container-optimized operating system designed for large-scale
 
 For newcomers, Typhoon is a free (cost and freedom) Kubernetes distribution providing upstream Kubernetes, declarative configuration via [Terraform](https://www.terraform.io/intro/index.html), and support for AWS, Google Cloud, DigitalOcean, and bare-metal. Typhoon clusters use a [self-hosted](https://github.com/kubernetes-incubator/bootkube) control plane, support [Calico](https://www.projectcalico.org/blog/) and [flannel](https://coreos.com/flannel/docs/latest/) CNI networking, and enable etcd TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/), and network policy.
 
-Typhoon for Fedora Atomic reflects many of the same principles that created Typhoon for Container Linux. Clusters are declared using plain Terraform configs that can be versioned. In lieu of Ignition, instances are declaratively provisioned with Cloud-Init and kickstart (bare-metal only). TLS assets are generated. Hosts run only a kubelet service, other components are scheduled (i.e. self-hosted). The upstream hyperkube is used directly[^1]. And clusters are kept minimal by offering optional addons for [Ingress](https://typhoon.psdn.io/addons/ingress/), [Prometheus](https://typhoon.psdn.io/addons/prometheus/), and [Grafana](https://typhoon.psdn.io/addons/grafana/). Typhoon compliments and enhances Fedora Atomic as a choice of operating system for Kubernetes.
+Typhoon for Fedora Atomic reflects many of the same principles that created Typhoon for Container Linux. Clusters are declared using plain Terraform configs that can be versioned. In lieu of Ignition, instances are declaratively provisioned with Cloud-Init and kickstart (bare-metal only). TLS assets are generated. Hosts run only a kubelet service, other components are scheduled (i.e. self-hosted). The upstream hyperkube is used directly[^1]. And clusters are kept minimal by offering optional addons for [Ingress](/addons/ingress/), [Prometheus](/addons/prometheus/), and [Grafana](/addons/grafana/). Typhoon compliments and enhances Fedora Atomic as a choice of operating system for Kubernetes.
 
 Meanwhile, Fedora Atomic adds some promising new low-level technologies:
 

docs/architecture/digitalocean.md

@@ -1,4 +1,4 @@
-# AWS
+# DigitalOcean
 
 ## IPv6
 

docs/atomic/aws.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.12.3 cluster on AWS with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on AWS with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -44,7 +44,7 @@ Configure the AWS provider to use your access key credentials in a `providers.tf
 
 ```tf
 provider "aws" {
-  version = "~> 1.13.0"
+  version = "~> 1.60.0"
   alias   = "default"
 
   region                  = "eu-central-1"
@@ -83,7 +83,7 @@ Define a Kubernetes cluster using the module `aws/fedora-atomic/kubernetes`.
 
 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.12.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.13.4"
 
   providers = {
     aws = "aws.default"
@@ -156,9 +156,9 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
 NAME           STATUS  ROLES              AGE  VERSION
-ip-10-0-3-155  Ready   controller,master  10m  v1.12.3
-ip-10-0-26-65  Ready   node               10m  v1.12.3
-ip-10-0-41-21  Ready   node               10m  v1.12.3
+ip-10-0-3-155  Ready   controller,master  10m  v1.13.4
+ip-10-0-26-65  Ready   node               10m  v1.13.4
+ip-10-0-41-21  Ready   node               10m  v1.13.4
 ```
 
 List the pods.
@@ -224,8 +224,8 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 |:-----|:------------|:--------|:--------|
 | controller_count | Number of controllers (i.e. masters) | 1 | 1 |
 | worker_count | Number of workers | 1 | 3 |
-| controller_type | EC2 instance type for controllers | "t2.small" | See below |
-| worker_type | EC2 instance type for workers | "t2.small" | See below |
+| controller_type | EC2 instance type for controllers | "t3.small" | See below |
+| worker_type | EC2 instance type for workers | "t3.small" | See below |
 | disk_size | Size of the EBS volume in GB | "40" | "100" |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
 | disk_iops | IOPS of the EBS volume | "0" (i.e. auto) | "400" |

docs/atomic/bare-metal.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.12.3 cluster on bare-metal with Fedora Atomic.
+In this tutorial, we'll network boot and provision a Kubernetes v1.13.4 cluster on bare-metal with Fedora Atomic.
 
 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora Atomic via kickstart, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via cloud-init.
 
@@ -174,20 +174,12 @@ $ terraform version
 Terraform v0.11.7
 ```
 
-Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system.
+Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
 
 ```sh
-wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.2/terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-tar xzf terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-sudo mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  matchbox = "/usr/local/bin/terraform-provider-matchbox"
-}
+wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.3/terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
+tar xzf terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
+mv terraform-provider-matchbox-v0.2.3-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.3
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -202,6 +194,7 @@ Configure the Matchbox provider to use your Matchbox API endpoint and client cer
 
 ```tf
 provider "matchbox" {
+  version     = "0.2.3"
   endpoint    = "matchbox.example.com:8081"
   client_cert = "${file("~/.config/matchbox/client.crt")}"
   client_key  = "${file("~/.config/matchbox/client.key")}"
@@ -235,7 +228,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-atomic/kubernete
 
 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.12.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.13.4"
   
   providers = {
     local = "local.default"
@@ -361,9 +354,9 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
 NAME                STATUS  ROLES              AGE  VERSION
-node1.example.com   Ready   controller,master  10m  v1.12.3
-node2.example.com   Ready   node               10m  v1.12.3
-node3.example.com   Ready   node               10m  v1.12.3
+node1.example.com   Ready   controller,master  10m  v1.13.4
+node2.example.com   Ready   node               10m  v1.13.4
+node3.example.com   Ready   node               10m  v1.13.4
 ```
 
 List the pods.

docs/atomic/digital-ocean.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.12.3 cluster on DigitalOcean with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on DigitalOcean with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -45,7 +45,7 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.
 
 ```tf
 provider "digitalocean" {
-  version = "1.0.0"
+  version = "~> 1.1.0"
   token = "${chomp(file("~/.config/digital-ocean/token"))}"
   alias = "default"
 }
@@ -77,7 +77,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-atomic/kubern
 
 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.12.3"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.13.4"
   
   providers = {
     digitalocean = "digitalocean.default"
@@ -152,9 +152,9 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
 NAME               STATUS  ROLES              AGE  VERSION
-nemo-controller-0  Ready   controller,master  10m  v1.12.3
-nemo-worker-0      Ready   node               10m  v1.12.3
-nemo-worker-1      Ready   node               10m  v1.12.3
+nemo-controller-0  Ready   controller,master  10m  v1.13.4
+nemo-worker-0      Ready   node               10m  v1.13.4
+nemo-worker-1      Ready   node               10m  v1.13.4
 ```
 
 List the pods.
@@ -204,9 +204,9 @@ Clusters create DNS A records `${cluster_name}.${dns_zone}` to resolve to contro
 You'll need a registered domain name or delegated subdomain in Digital Ocean Domains (i.e. DNS zones). You can set this up once and create many clusters with unique names.
 
 ```tf
+# Declare a DigitalOcean record to also create a zone file
 resource "digitalocean_domain" "zone-for-clusters" {
   name       = "do.example.com"
-  # Digital Ocean oddly requires an IP here. You may have to delete the A record it makes. :(
   ip_address = "8.8.8.8"
 }
 ```

docs/atomic/google-cloud.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Fedora does not publish official images for Google Cloud so you must prepare them yourself. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.12.3 cluster on Google Compute Engine with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.4 cluster on Google Compute Engine with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -35,7 +35,7 @@ cd infra/clusters
 
 Login to your Google Console [API Manager](https://console.cloud.google.com/apis/dashboard) and select a project, or [signup](https://cloud.google.com/free/) if you don't have an account.
 
-Select "Credentials" and create a service account key. Choose the "Compute Engine Admin" role and save the JSON private key to a file that can be referenced in configs.
+Select "Credentials" and create a service account key. Choose the "Compute Engine Admin" and "DNS Administrator" roles and save the JSON private key to a file that can be referenced in configs.
 
 ```sh
 mv ~/Downloads/project-id-43048204.json ~/.config/google-cloud/terraform.json
@@ -45,7 +45,7 @@ Configure the Google Cloud provider to use your service account key, project-id,
 
 ```tf
 provider "google" {
-  version = "1.6"
+  version = "~> 2.1.0"
   alias   = "default"
 
   credentials = "${file("~/.config/google-cloud/terraform.json")}"
@@ -121,7 +121,7 @@ Define a Kubernetes cluster using the module `google-cloud/fedora-atomic/kuberne
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.12.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.13.4"
   
   providers = {
     google   = "google.default"
@@ -197,9 +197,9 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                       ROLES              STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.12.3
-yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.12.3
-yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.12.3
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.4
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.4
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.4
 ```
 
 List the pods.