Add kubeconfig's for kube-scheduler and kube-controller-manager

* Generate TLS client certificates for `kube-scheduler` and
`kube-controller-manager` with `system:kube-scheduler` and
`system:kube-controller-manager` CNs
* Template separate kubeconfigs for kube-scheduler and
kube-controller manager (`scheduler.conf` and
`controller-manager.conf`). Rename admin for clarity
* Before v1.16.0, Typhoon scheduled a self-hosted control
plane, which allowed the steady-state kube-scheduler and
kube-controller-manager to use a scoped ServiceAccount.
With a static pod control plane, separate CN TLS client
certificates are the nearest equiv.
* https://kubernetes.io/docs/setup/best-practices/certificates/
* Remove unused Kubelet certificate, TLS bootstrap is used
instead

CHANGES.md

@@ -7,8 +7,9 @@ Notable changes between versions.
 * Add input variable validations ([#880](https://github.com/poseidon/typhoon/pull/880))
   * Require Terraform v0.13+ ([migration guide](https://typhoon.psdn.io/topics/maintenance/#terraform-versions))
 * Set output sensitive to suppress console display for some cases ([#885](https://github.com/poseidon/typhoon/pull/885))
+* Add service account token [volume projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection) ([#897](https://github.com/poseidon/typhoon/pull/897))
+* Scope kube-scheduler and kube-controller-manager permissions ([#898](https://github.com/poseidon/typhoon/pull/898))
 * Update etcd from v3.4.12 to [v3.4.14](https://github.com/etcd-io/etcd/releases/tag/v3.4.14)
-* Enable service account token [volume projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection) ([#897](https://github.com/poseidon/typhoon/pull/897))
 * Update Calico from v3.16.5 to v3.17.0 ([#890](https://github.com/poseidon/typhoon/pull/890))
   * Enable Calico MTU auto-detection
   * Remove [workaround](https://github.com/poseidon/typhoon/pull/724) to Calico cni-plugin [issue](https://github.com/projectcalico/cni-plugin/issues/874)
@@ -64,7 +65,6 @@ Notable changes between versions.
 ### Flatcar Linux
 
 * Rename `container-linux` modules to `flatcar-linux` ([#858](https://github.com/poseidon/typhoon/issues/858)) (**action required**)
-
 * Change on-host system containers from rkt to docker
   * Change `etcd-member.service` container runnner from rkt to docker ([#867](https://github.com/poseidon/typhoon/pull/867))
   * Change `kubelet.service` container runner from rkt-fly to docker ([#855](https://github.com/poseidon/typhoon/pull/855))

aws/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

aws/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -147,7 +147,7 @@ storage:
           mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -161,7 +161,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5

aws/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

aws/flatcar-linux/kubernetes/cl/controller.yaml

@@ -155,7 +155,7 @@ storage:
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
           chmod -R 700 /var/lib/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -169,7 +169,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5

azure/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

azure/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -146,7 +146,7 @@ storage:
           mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -160,7 +160,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5

azure/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

azure/flatcar-linux/kubernetes/cl/controller.yaml

@@ -155,7 +155,7 @@ storage:
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
           chmod -R 700 /var/lib/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -169,7 +169,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5

bare-metal/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]

bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -157,7 +157,7 @@ storage:
           mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -171,7 +171,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5

bare-metal/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]

bare-metal/flatcar-linux/kubernetes/cl/controller.yaml

@@ -169,7 +169,7 @@ storage:
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
           chmod -R 700 /var/lib/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -183,7 +183,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5

digital-ocean/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -153,7 +153,7 @@ storage:
           mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -167,7 +167,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5

digital-ocean/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml

@@ -162,7 +162,7 @@ storage:
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
           chmod -R 700 /var/lib/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -176,7 +176,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5

google-cloud/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -146,7 +146,7 @@ storage:
           mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -160,7 +160,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5

google-cloud/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=19c3ce61bd32801bef791e4848b2a3eac1b758c8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=ac5cb9577408cba65f66b0ce35a8881c3ca5d63b"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

google-cloud/flatcar-linux/kubernetes/cl/controller.yaml

@@ -153,7 +153,7 @@ storage:
           chown -R etcd:etcd /etc/ssl/etcd
           chmod -R 500 /etc/ssl/etcd
           chmod -R 700 /var/lib/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/bootstrap-secrets/
           mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
           mkdir -p /etc/kubernetes/manifests
           mv static-manifests/* /etc/kubernetes/manifests/
@@ -167,7 +167,7 @@ storage:
       contents:
         inline: |
           #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/secrets/admin.conf
           until kubectl version; do
             echo "Waiting for static pod control plane"
             sleep 5