Introduce cluster creation without local writes to asset_dir
* Allow generated assets (TLS materials, manifests) to be
securely distributed to controller node(s) via file provisioner
(i.e. ssh-agent) as an assets bundle file, rather than relying
on assets being locally rendered to disk in an asset_dir and
then securely distributed
* Change `asset_dir` from required to optional. Left unset,
asset_dir defaults to "" and no assets will be written to
files on the machine that runs terraform apply
* Enhancement: Managed cluster assets are kept only in Terraform
state, which supports different backends (GCS, S3, etcd, etc) and
optional encryption. terraform apply accesses state, runs in-memory,
and distributes sensitive materials to controllers without making
use of local disk (simplifies use in CI systems)
* Enhancement: Improve asset unpack and layout process to position
etcd certificates and control plane certificates more cleanly,
without unneeded secret materials
Details:
* Terraform file provisioner support for distributing directories of
contents (with unknown structure) has been limited to reading from a
local directory, meaning local writes to asset_dir were required.
https://github.com/poseidon/typhoon/issues/585 discusses the problem
and newer or upcoming Terraform features that might help.
* Observation: Terraform provisioner support for single files works
well, but iteration isn't viable. We're also constrained to Terraform
language features on the apply side (no extra plugins, no shelling out)
and CoreOS / Fedora tools on the receive side.
* Take a map representation of the contents that would have been splayed
out in asset_dir and pack/encode them into a single file format devised
for easy unpacking. Use an awk one-liner on the receive side to unpack.
In pratice, this has worked well and its rather nice that a single
assets file is transferred by file provisioner (all or none)
Rel: https://github.com/poseidon/terraform-render-bootstrap/pull/162
2019-12-05 07:10:55 +01:00
|
|
|
locals {
|
|
|
|
# format assets for distribution
|
|
|
|
assets_bundle = [
|
|
|
|
# header with the unpack location
|
2019-12-28 21:07:10 +01:00
|
|
|
for key, value in module.bootstrap.assets_dist :
|
Introduce cluster creation without local writes to asset_dir
* Allow generated assets (TLS materials, manifests) to be
securely distributed to controller node(s) via file provisioner
(i.e. ssh-agent) as an assets bundle file, rather than relying
on assets being locally rendered to disk in an asset_dir and
then securely distributed
* Change `asset_dir` from required to optional. Left unset,
asset_dir defaults to "" and no assets will be written to
files on the machine that runs terraform apply
* Enhancement: Managed cluster assets are kept only in Terraform
state, which supports different backends (GCS, S3, etcd, etc) and
optional encryption. terraform apply accesses state, runs in-memory,
and distributes sensitive materials to controllers without making
use of local disk (simplifies use in CI systems)
* Enhancement: Improve asset unpack and layout process to position
etcd certificates and control plane certificates more cleanly,
without unneeded secret materials
Details:
* Terraform file provisioner support for distributing directories of
contents (with unknown structure) has been limited to reading from a
local directory, meaning local writes to asset_dir were required.
https://github.com/poseidon/typhoon/issues/585 discusses the problem
and newer or upcoming Terraform features that might help.
* Observation: Terraform provisioner support for single files works
well, but iteration isn't viable. We're also constrained to Terraform
language features on the apply side (no extra plugins, no shelling out)
and CoreOS / Fedora tools on the receive side.
* Take a map representation of the contents that would have been splayed
out in asset_dir and pack/encode them into a single file format devised
for easy unpacking. Use an awk one-liner on the receive side to unpack.
In pratice, this has worked well and its rather nice that a single
assets file is transferred by file provisioner (all or none)
Rel: https://github.com/poseidon/terraform-render-bootstrap/pull/162
2019-12-05 07:10:55 +01:00
|
|
|
format("##### %s\n%s", key, value)
|
|
|
|
]
|
|
|
|
}
|
|
|
|
|
2019-09-05 07:05:29 +02:00
|
|
|
# Secure copy assets to controllers. Activates kubelet.service
|
2018-03-26 09:29:57 +02:00
|
|
|
resource "null_resource" "copy-controller-secrets" {
|
2019-10-06 21:57:15 +02:00
|
|
|
count = length(var.controllers)
|
2017-07-25 08:16:34 +02:00
|
|
|
|
2018-06-15 07:43:01 +02:00
|
|
|
# Without depends_on, remote-exec could start and wait for machines before
|
|
|
|
# matchbox groups are written, causing a deadlock.
|
|
|
|
depends_on = [
|
2019-05-29 04:19:23 +02:00
|
|
|
matchbox_group.install,
|
|
|
|
matchbox_group.controller,
|
|
|
|
matchbox_group.worker,
|
2019-09-15 01:24:32 +02:00
|
|
|
module.bootstrap,
|
2018-06-15 07:43:01 +02:00
|
|
|
]
|
|
|
|
|
2017-07-25 08:16:34 +02:00
|
|
|
connection {
|
|
|
|
type = "ssh"
|
2019-10-06 21:57:15 +02:00
|
|
|
host = var.controllers.*.domain[count.index]
|
2017-07-25 08:16:34 +02:00
|
|
|
user = "core"
|
|
|
|
timeout = "60m"
|
|
|
|
}
|
|
|
|
|
|
|
|
provisioner "file" {
|
2019-09-15 01:24:32 +02:00
|
|
|
content = module.bootstrap.kubeconfig-kubelet
|
2017-07-25 08:16:34 +02:00
|
|
|
destination = "$HOME/kubeconfig"
|
|
|
|
}
|
|
|
|
|
|
|
|
provisioner "file" {
|
Introduce cluster creation without local writes to asset_dir
* Allow generated assets (TLS materials, manifests) to be
securely distributed to controller node(s) via file provisioner
(i.e. ssh-agent) as an assets bundle file, rather than relying
on assets being locally rendered to disk in an asset_dir and
then securely distributed
* Change `asset_dir` from required to optional. Left unset,
asset_dir defaults to "" and no assets will be written to
files on the machine that runs terraform apply
* Enhancement: Managed cluster assets are kept only in Terraform
state, which supports different backends (GCS, S3, etcd, etc) and
optional encryption. terraform apply accesses state, runs in-memory,
and distributes sensitive materials to controllers without making
use of local disk (simplifies use in CI systems)
* Enhancement: Improve asset unpack and layout process to position
etcd certificates and control plane certificates more cleanly,
without unneeded secret materials
Details:
* Terraform file provisioner support for distributing directories of
contents (with unknown structure) has been limited to reading from a
local directory, meaning local writes to asset_dir were required.
https://github.com/poseidon/typhoon/issues/585 discusses the problem
and newer or upcoming Terraform features that might help.
* Observation: Terraform provisioner support for single files works
well, but iteration isn't viable. We're also constrained to Terraform
language features on the apply side (no extra plugins, no shelling out)
and CoreOS / Fedora tools on the receive side.
* Take a map representation of the contents that would have been splayed
out in asset_dir and pack/encode them into a single file format devised
for easy unpacking. Use an awk one-liner on the receive side to unpack.
In pratice, this has worked well and its rather nice that a single
assets file is transferred by file provisioner (all or none)
Rel: https://github.com/poseidon/terraform-render-bootstrap/pull/162
2019-12-05 07:10:55 +01:00
|
|
|
content = join("\n", local.assets_bundle)
|
2019-09-05 07:05:29 +02:00
|
|
|
destination = "$HOME/assets"
|
|
|
|
}
|
2017-07-25 08:16:34 +02:00
|
|
|
|
|
|
|
provisioner "remote-exec" {
|
|
|
|
inline = [
|
2019-09-19 08:56:17 +02:00
|
|
|
"sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
|
Introduce cluster creation without local writes to asset_dir
* Allow generated assets (TLS materials, manifests) to be
securely distributed to controller node(s) via file provisioner
(i.e. ssh-agent) as an assets bundle file, rather than relying
on assets being locally rendered to disk in an asset_dir and
then securely distributed
* Change `asset_dir` from required to optional. Left unset,
asset_dir defaults to "" and no assets will be written to
files on the machine that runs terraform apply
* Enhancement: Managed cluster assets are kept only in Terraform
state, which supports different backends (GCS, S3, etcd, etc) and
optional encryption. terraform apply accesses state, runs in-memory,
and distributes sensitive materials to controllers without making
use of local disk (simplifies use in CI systems)
* Enhancement: Improve asset unpack and layout process to position
etcd certificates and control plane certificates more cleanly,
without unneeded secret materials
Details:
* Terraform file provisioner support for distributing directories of
contents (with unknown structure) has been limited to reading from a
local directory, meaning local writes to asset_dir were required.
https://github.com/poseidon/typhoon/issues/585 discusses the problem
and newer or upcoming Terraform features that might help.
* Observation: Terraform provisioner support for single files works
well, but iteration isn't viable. We're also constrained to Terraform
language features on the apply side (no extra plugins, no shelling out)
and CoreOS / Fedora tools on the receive side.
* Take a map representation of the contents that would have been splayed
out in asset_dir and pack/encode them into a single file format devised
for easy unpacking. Use an awk one-liner on the receive side to unpack.
In pratice, this has worked well and its rather nice that a single
assets file is transferred by file provisioner (all or none)
Rel: https://github.com/poseidon/terraform-render-bootstrap/pull/162
2019-12-05 07:10:55 +01:00
|
|
|
"sudo /opt/bootstrap/layout",
|
2017-07-25 08:16:34 +02:00
|
|
|
]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-10-15 05:41:18 +02:00
|
|
|
# Secure copy kubeconfig to all workers. Activates kubelet.service
|
2018-03-26 09:29:57 +02:00
|
|
|
resource "null_resource" "copy-worker-secrets" {
|
2019-10-06 21:57:15 +02:00
|
|
|
count = length(var.workers)
|
2017-10-15 05:41:18 +02:00
|
|
|
|
2018-06-15 07:43:01 +02:00
|
|
|
# Without depends_on, remote-exec could start and wait for machines before
|
|
|
|
# matchbox groups are written, causing a deadlock.
|
|
|
|
depends_on = [
|
2019-05-29 04:19:23 +02:00
|
|
|
matchbox_group.install,
|
|
|
|
matchbox_group.controller,
|
|
|
|
matchbox_group.worker,
|
2018-06-15 07:43:01 +02:00
|
|
|
]
|
|
|
|
|
2017-10-15 05:41:18 +02:00
|
|
|
connection {
|
|
|
|
type = "ssh"
|
2019-10-06 21:57:15 +02:00
|
|
|
host = var.workers.*.domain[count.index]
|
2017-10-15 05:41:18 +02:00
|
|
|
user = "core"
|
|
|
|
timeout = "60m"
|
|
|
|
}
|
|
|
|
|
|
|
|
provisioner "file" {
|
2019-09-15 01:24:32 +02:00
|
|
|
content = module.bootstrap.kubeconfig-kubelet
|
2017-10-15 05:41:18 +02:00
|
|
|
destination = "$HOME/kubeconfig"
|
|
|
|
}
|
|
|
|
|
|
|
|
provisioner "remote-exec" {
|
|
|
|
inline = [
|
2018-03-26 09:29:57 +02:00
|
|
|
"sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
|
2017-10-15 05:41:18 +02:00
|
|
|
]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-09-05 07:05:29 +02:00
|
|
|
# Connect to a controller to perform one-time cluster bootstrap.
|
|
|
|
resource "null_resource" "bootstrap" {
|
2017-07-25 08:16:34 +02:00
|
|
|
# Without depends_on, this remote-exec may start before the kubeconfig copy.
|
|
|
|
# Terraform only does one task at a time, so it would try to bootstrap
|
2017-09-18 06:43:00 +02:00
|
|
|
# while no Kubelets are running.
|
2018-03-26 09:29:57 +02:00
|
|
|
depends_on = [
|
2019-05-29 04:19:23 +02:00
|
|
|
null_resource.copy-controller-secrets,
|
|
|
|
null_resource.copy-worker-secrets,
|
2018-03-26 09:29:57 +02:00
|
|
|
]
|
2017-07-25 08:16:34 +02:00
|
|
|
|
|
|
|
connection {
|
|
|
|
type = "ssh"
|
2019-10-06 21:57:15 +02:00
|
|
|
host = var.controllers[0].domain
|
2017-07-25 08:16:34 +02:00
|
|
|
user = "core"
|
2018-03-26 09:29:57 +02:00
|
|
|
timeout = "15m"
|
2017-07-25 08:16:34 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
provisioner "remote-exec" {
|
|
|
|
inline = [
|
2019-09-05 07:05:29 +02:00
|
|
|
"sudo systemctl start bootstrap",
|
2017-07-25 08:16:34 +02:00
|
|
|
]
|
|
|
|
}
|
|
|
|
}
|
2019-05-29 04:19:23 +02:00
|
|
|
|