Fix bootstrap mount to use shared volume SELinux label

* Race: During initial bootstrap, static control plane pods
could hang with Permission denied to bootstrap secrets. A
manual fix involved restarting Kubelet, which relabeled mounts
The race had no effect on subsequent reboots.
* bootstrap.service runs podman with a private unshared mount
of /etc/kubernetes/bootstrap-secrets which uses an SELinux MCS
label with a category pair. However, bootstrap-secrets should
be shared as its mounted by Docker pods kube-apiserver,
kube-scheduler, and kube-controller-manager. Restarting Kubelet
was a manual fix because Kubelet relabels all /etc/kubernetes
* Fix bootstrap Pod to use the shared volume label, which leaves
bootstrap-secrets files with SELinux level s0 without MCS
* Also allow failed bootstrap.service to be re-applied. This was
missing on bare-metal and AWS

CHANGES.md

@@ -13,6 +13,8 @@ Notable changes between versions.
 
 ### Fedora CoreOS
 
+* Fix race condition during bootstrap related to SELinux shared content label ([#708](https://github.com/poseidon/typhoon/pull/708))
+
 #### Azure
 
 * Add support for Fedora CoreOS ([#704](https://github.com/poseidon/typhoon/pull/704))

aws/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -116,9 +116,10 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
+        ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \

azure/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -119,7 +119,7 @@ systemd:
         ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \

bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -127,9 +127,10 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
+        ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \

digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -131,7 +131,7 @@ systemd:
         ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \

google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -119,7 +119,7 @@ systemd:
         ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \