Remove Fedora Atomic modules

* Typhoon for Fedora Atomic was deprecated in March 2019
* https://typhoon.psdn.io/announce/#march-27-2019

.github/ISSUE_TEMPLATE.md

@@ -5,7 +5,7 @@
 ### Environment
 
 * Platform: aws, azure, bare-metal, google-cloud, digital-ocean
-* OS: container-linux, flatcar-linux, or fedora-atomic
+* OS: container-linux, flatcar-linux
 * Release: Typhoon version or Git SHA (reporting latest is **not** helpful)
 * Terraform: `terraform version` (reporting latest is **not** helpful)
 * Plugins: Provider plugin versions (reporting latest is **not** helpful)

CHANGES.md

@@ -4,6 +4,46 @@ Notable changes between versions.
 
 ## Latest
 
+## v1.15.0
+
+* Kubernetes [v1.15.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md#v1150)
+* Migrate from Terraform v0.11 to v0.12.x (**action required!**)
+  * [Migration](https://typhoon.psdn.io/topics/maintenance/#terraform-v012x) instructions for Terraform v0.12
+* Require `terraform-provider-ct` v0.3.2+ to support Terraform v0.12 (action required)
+* Update Calico from v3.7.2 to [v3.7.3](https://docs.projectcalico.org/v3.7/release-notes/)
+* Remove Fedora Atomic modules (deprecated in March) ([#501](https://github.com/poseidon/typhoon/pull/501))
+
+#### AWS
+
+* Require `terraform-provider-aws` v2.7+ to support Terraform v0.12 (action required)
+* Allow using Flatcar Linux Edge by setting `os_image` to "flatcar-edge"
+
+#### Azure
+
+* Require `terraform-provider-azurerm` v1.27+ to support Terraform v0.12 (action required)
+* Avoid unneeded rotations of Regular priority virtual machine scale sets
+  * Azure only allows `eviction_policy` to be set for Low priority VMs. Supporting Low priority VMs meant when Regular VMs were used, each `terraform apply` rolled workers, to set eviction_policy to null.
+  * Terraform v0.12 nullable variables fix the issue so plan does not produce a diff. 
+
+#### Bare-Metal
+
+* Require `terraform-provider-matchbox` v0.3.0+ to support Terraform v0.12 (action required)
+* Allow using Flatcar Linux Edge by setting `os_channel` to "flatcar-edge"
+
+#### DigitalOcean
+
+* Require `terraform-provider-digitalocean` v1.3+ to support Terraform v0.12 (action required)
+* Change the default `worker_type` from `s-1vcpu1-1gb` to `s-1vcpu-2gb`
+
+#### Google Cloud
+
+* Require `terraform-provider-google` v2.5+ to support Terraform v0.12 (action required)
+
+#### Addons
+
+* Update Grafana from v6.2.1 to v6.2.4
+* Update node-exporter from v0.18.0 to v0.18.1
+
 ## v1.14.3
 
 * Kubernetes [v1.14.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#v1143)
@@ -12,7 +52,7 @@ Notable changes between versions.
 * Fix trailing slash in terraform-render-bootkube version ([#479](https://github.com/poseidon/typhoon/pull/479))
 * Recommend updating `terraform-provider-ct` plugin from v0.3.1 to [v0.3.2](https://github.com/poseidon/terraform-provider-ct/releases/tag/v0.3.2) ([#487](https://github.com/poseidon/typhoon/pull/487))
 
-### AWS
+#### AWS
 
 * Rename `worker` pool module `count` variable to `worker_count` ([#485](https://github.com/poseidon/typhoon/pull/485)) (action required)
   * `count` will become a reserved variable name in Terraform v0.12
@@ -20,7 +60,6 @@ Notable changes between versions.
 #### Azure
 
 * Replace `azurerm_autoscale_setting` with `azurerm_monitor_autoscale_setting` ([#482](https://github.com/poseidon/typhoon/pull/482))
-  * Require `terraform-provider-azurerm` v1.22+ (action required)
 * Rename `worker` pool module `count` variable to `worker_count` ([#485](https://github.com/poseidon/typhoon/pull/485)) (action required)
   * `count` will become a reserved variable name in Terraform v0.12
 
@@ -31,7 +70,7 @@ Notable changes between versions.
 #### Google Cloud
 
 * Rename `worker` pool module `count` variable to `worker_count` ([#485](https://github.com/poseidon/typhoon/pull/485)) (action required)
-  * `count` will become a reserved variable name in Terraform v0.12
+  * `count` is a reserved variable in Terraform v0.12
 
 #### Addons
 

README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.14.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.15.0 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
@@ -19,25 +19,16 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Modules
 
-Typhoon provides a Terraform Module for each supported operating system and platform. Container Linux is a mature and reliable choice. Also, Kinvolk's Flatcar Linux fork is selectable on AWS and bare-metal.
+Typhoon provides a Terraform Module for each supported operating system and platform.
 
 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
+| AWS           | Container Linux / Flatcar Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
 | Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
-| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
+| Bare-Metal    | Container Linux / Flatcar Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |
 
-Fedora Atomic support is alpha and will evolve as Fedora Atomic is replaced by Fedora CoreOS.
-
-| Platform      | Operating System | Terraform Module | Status |
-|---------------|------------------|------------------|--------|
-| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | deprecated |
-| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | deprecated |
-| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | deprecated |
-| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | deprecated |
-
 ## Documentation
 
 * [Docs](https://typhoon.psdn.io)
@@ -50,15 +41,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.14.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.15.0"
-  
-  providers = {
-    google   = "google.default"
-    local    = "local.default"
-    null     = "null.default"
-    template = "template.default"
-    tls      = "tls.default"
-  }
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -72,6 +55,7 @@ module "google-cloud-yavin" {
   
   # optional
   worker_count = 2
+  worker_preemptible = true
 }
 ```
 
@@ -91,9 +75,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                       ROLES              STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.14.3
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.15.0
-yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.14.3
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.15.0
-yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.14.3
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.15.0
 ```
 
 List the pods.
@@ -145,3 +129,5 @@ Typhoon clusters will contain only [free](https://www.debian.org/intro/free) com
 ## Donations
 
 Typhoon does not accept money donations. Instead, we encourage you to donate to one of [these organizations](https://github.com/poseidon/typhoon/wiki/Donations) to show your appreciation.
+
+* [DigitalOcean](https://www.digitalocean.com/) kindly provides credits to support Typhoon test clusters.

addons/grafana/deployment.yaml

@@ -23,7 +23,7 @@ spec:
     spec:
       containers:
         - name: grafana
-          image: grafana/grafana:6.2.1
+          image: grafana/grafana:6.2.4
           env:
             - name: GF_PATHS_CONFIG
               value: "/etc/grafana/custom.ini"

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -35,7 +35,7 @@ spec:
           initialDelaySeconds: 5
           timeoutSeconds: 5
       - name: addon-resizer
-        image: k8s.gcr.io/addon-resizer:1.8.4
+        image: k8s.gcr.io/addon-resizer:1.8.5
         resources:
           limits:
             cpu: 100m

addons/prometheus/exporters/node-exporter/daemonset.yaml

@@ -28,7 +28,7 @@ spec:
       hostPID: true
       containers:
       - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v0.18.0
+        image: quay.io/prometheus/node-exporter:v0.18.1
         args:
           - --path.procfs=/host/proc
           - --path.sysfs=/host/sys

aws/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.14.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.15.0 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

aws/container-linux/kubernetes/ami.tf

@@ -2,10 +2,10 @@ locals {
   # Pick a CoreOS Container Linux derivative
   # coreos-stable -> Container Linux AMI
   # flatcar-stable -> Flatcar Linux AMI
-  ami_id = "${local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id}"
+  ami_id = local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id
 
-  flavor  = "${element(split("-", var.os_image), 0)}"
+  flavor  = element(split("-", var.os_image), 0)
-  channel = "${element(split("-", var.os_image), 1)}"
+  channel = element(split("-", var.os_image), 1)
 }
 
 data "aws_ami" "coreos" {
@@ -24,7 +24,7 @@ data "aws_ami" "coreos" {
 
   filter {
     name   = "name"
-    values = ["CoreOS-${local.channel}-*"]
+    values = ["CoreOS-${local.flavor == "coreos" ? local.channel : "stable"}-*"]
   }
 }
 
@@ -44,6 +44,7 @@ data "aws_ami" "flatcar" {
 
   filter {
     name   = "name"
-    values = ["Flatcar-${local.channel}-*"]
+    values = ["Flatcar-${local.flavor == "flatcar" ? local.channel : "stable"}-*"]
   }
 }
+

aws/container-linux/kubernetes/bootkube.tf

@@ -1,16 +1,17 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=082921d67905417755609eebda7d39a7e26f7fdb"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=62df9ad69cc0da35f47d40fa981370c4503ad581"
 
-  cluster_name          = "${var.cluster_name}"
+  cluster_name          = var.cluster_name
-  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
+  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
-  etcd_servers          = ["${aws_route53_record.etcds.*.fqdn}"]
+  etcd_servers          = aws_route53_record.etcds.*.fqdn
-  asset_dir             = "${var.asset_dir}"
+  asset_dir             = var.asset_dir
-  networking            = "${var.networking}"
+  networking            = var.networking
-  network_mtu           = "${var.network_mtu}"
+  network_mtu           = var.network_mtu
-  pod_cidr              = "${var.pod_cidr}"
+  pod_cidr              = var.pod_cidr
-  service_cidr          = "${var.service_cidr}"
+  service_cidr          = var.service_cidr
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = "${var.enable_reporting}"
+  enable_reporting      = var.enable_reporting
-  enable_aggregation    = "${var.enable_aggregation}"
+  enable_aggregation    = var.enable_aggregation
 }
+

aws/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -63,6 +63,7 @@ systemd:
           --volume var-log,kind=host,source=/var/log \
           --mount volume=var-log,target=/var/log \
           --insecure-options=image"
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
@@ -77,6 +78,7 @@ systemd:
           --anonymous-auth=false \
           --authentication-token-webhook \
           --authorization-mode=Webhook \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -123,7 +125,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.14.3
+          KUBELET_IMAGE_TAG=v1.15.0
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

aws/container-linux/kubernetes/controllers.tf

@@ -1,87 +1,90 @@
 # Discrete DNS records for each controller's private IPv4 for etcd usage
 resource "aws_route53_record" "etcds" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
   # DNS Zone where record should be created
-  zone_id = "${var.dns_zone_id}"
+  zone_id = var.dns_zone_id
 
-  name = "${format("%s-etcd%d.%s.", var.cluster_name, count.index, var.dns_zone)}"
+  name = format("%s-etcd%d.%s.", var.cluster_name, count.index, var.dns_zone)
   type = "A"
   ttl  = 300
 
   # private IPv4 address for etcd
-  records = ["${element(aws_instance.controllers.*.private_ip, count.index)}"]
+  records = [element(aws_instance.controllers.*.private_ip, count.index)]
 }
 
 # Controller instances
 resource "aws_instance" "controllers" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
   tags = {
     Name = "${var.cluster_name}-controller-${count.index}"
   }
 
-  instance_type = "${var.controller_type}"
+  instance_type = var.controller_type
 
-  ami       = "${local.ami_id}"
+  ami       = local.ami_id
-  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
+  user_data = element(data.ct_config.controller-ignitions.*.rendered, count.index)
 
   # storage
   root_block_device {
-    volume_type = "${var.disk_type}"
+    volume_type = var.disk_type
-    volume_size = "${var.disk_size}"
+    volume_size = var.disk_size
-    iops        = "${var.disk_iops}"
+    iops        = var.disk_iops
   }
 
   # network
   associate_public_ip_address = true
-  subnet_id                   = "${element(aws_subnet.public.*.id, count.index)}"
+  subnet_id                   = element(aws_subnet.public.*.id, count.index)
-  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
+  vpc_security_group_ids      = [aws_security_group.controller.id]
 
   lifecycle {
     ignore_changes = [
-      "ami",
+      ami,
-      "user_data",
+      user_data,
     ]
   }
 }
 
 # Controller Ignition configs
 data "ct_config" "controller-ignitions" {
-  count        = "${var.controller_count}"
+  count = var.controller_count
-  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  content = element(
+    data.template_file.controller-configs.*.rendered,
+    count.index,
+  )
   pretty_print = false
-  snippets     = ["${var.controller_clc_snippets}"]
+  snippets     = var.controller_clc_snippets
 }
 
 # Controller Container Linux configs
 data "template_file" "controller-configs" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
-  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+  template = file("${path.module}/cl/controller.yaml.tmpl")
 
   vars = {
     # Cannot use cyclic dependencies on controllers or their DNS records
     etcd_name   = "etcd${count.index}"
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
-
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
+    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
-
+    cgroup_driver          = local.flavor == "flatcar" && local.channel == "edge" ? "systemd" : "cgroupfs"
-    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
+    kubeconfig             = indent(10, module.bootkube.kubeconfig-kubelet)
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    ssh_authorized_key     = var.ssh_authorized_key
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    cluster_domain_suffix  = var.cluster_domain_suffix
   }
 }
 
 data "template_file" "etcds" {
-  count    = "${var.controller_count}"
+  count    = var.controller_count
   template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
 
   vars = {
-    index        = "${count.index}"
+    index        = count.index
-    cluster_name = "${var.cluster_name}"
+    cluster_name = var.cluster_name
-    dns_zone     = "${var.dns_zone}"
+    dns_zone     = var.dns_zone
   }
 }
+

aws/container-linux/kubernetes/network.tf

@@ -1,57 +1,67 @@
-data "aws_availability_zones" "all" {}
+data "aws_availability_zones" "all" {
+}
 
 # Network VPC, gateway, and routes
 
 resource "aws_vpc" "network" {
-  cidr_block                       = "${var.host_cidr}"
+  cidr_block                       = var.host_cidr
   assign_generated_ipv6_cidr_block = true
   enable_dns_support               = true
   enable_dns_hostnames             = true
 
-  tags = "${map("Name", "${var.cluster_name}")}"
+  tags = {
+    "Name" = var.cluster_name
+  }
 }
 
 resource "aws_internet_gateway" "gateway" {
-  vpc_id = "${aws_vpc.network.id}"
+  vpc_id = aws_vpc.network.id
 
-  tags = "${map("Name", "${var.cluster_name}")}"
+  tags = {
+    "Name" = var.cluster_name
+  }
 }
 
 resource "aws_route_table" "default" {
-  vpc_id = "${aws_vpc.network.id}"
+  vpc_id = aws_vpc.network.id
 
   route {
     cidr_block = "0.0.0.0/0"
-    gateway_id = "${aws_internet_gateway.gateway.id}"
+    gateway_id = aws_internet_gateway.gateway.id
   }
 
   route {
     ipv6_cidr_block = "::/0"
-    gateway_id      = "${aws_internet_gateway.gateway.id}"
+    gateway_id      = aws_internet_gateway.gateway.id
   }
 
-  tags = "${map("Name", "${var.cluster_name}")}"
+  tags = {
+    "Name" = var.cluster_name
+  }
 }
 
 # Subnets (one per availability zone)
 
 resource "aws_subnet" "public" {
-  count = "${length(data.aws_availability_zones.all.names)}"
+  count = length(data.aws_availability_zones.all.names)
 
-  vpc_id            = "${aws_vpc.network.id}"
+  vpc_id            = aws_vpc.network.id
-  availability_zone = "${data.aws_availability_zones.all.names[count.index]}"
+  availability_zone = data.aws_availability_zones.all.names[count.index]
 
-  cidr_block                      = "${cidrsubnet(var.host_cidr, 4, count.index)}"
+  cidr_block                      = cidrsubnet(var.host_cidr, 4, count.index)
-  ipv6_cidr_block                 = "${cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)}"
+  ipv6_cidr_block                 = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
   map_public_ip_on_launch         = true
   assign_ipv6_address_on_creation = true
 
-  tags = "${map("Name", "${var.cluster_name}-public-${count.index}")}"
+  tags = {
+    "Name" = "${var.cluster_name}-public-${count.index}"
+  }
 }
 
 resource "aws_route_table_association" "public" {
-  count = "${length(data.aws_availability_zones.all.names)}"
+  count = length(data.aws_availability_zones.all.names)
 
-  route_table_id = "${aws_route_table.default.id}"
+  route_table_id = aws_route_table.default.id
-  subnet_id      = "${element(aws_subnet.public.*.id, count.index)}"
+  subnet_id      = element(aws_subnet.public.*.id, count.index)
 }
+

aws/container-linux/kubernetes/nlb.tf

@@ -1,14 +1,14 @@
 # Network Load Balancer DNS Record
 resource "aws_route53_record" "apiserver" {
-  zone_id = "${var.dns_zone_id}"
+  zone_id = var.dns_zone_id
 
-  name = "${format("%s.%s.", var.cluster_name, var.dns_zone)}"
+  name = format("%s.%s.", var.cluster_name, var.dns_zone)
   type = "A"
 
   # AWS recommends their special "alias" records for NLBs
   alias {
-    name                   = "${aws_lb.nlb.dns_name}"
+    name                   = aws_lb.nlb.dns_name
-    zone_id                = "${aws_lb.nlb.zone_id}"
+    zone_id                = aws_lb.nlb.zone_id
     evaluate_target_health = true
   }
 }
@@ -19,51 +19,51 @@ resource "aws_lb" "nlb" {
   load_balancer_type = "network"
   internal           = false
 
-  subnets = ["${aws_subnet.public.*.id}"]
+  subnets = aws_subnet.public.*.id
 
   enable_cross_zone_load_balancing = true
 }
 
 # Forward TCP apiserver traffic to controllers
 resource "aws_lb_listener" "apiserver-https" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
+  load_balancer_arn = aws_lb.nlb.arn
   protocol          = "TCP"
   port              = "6443"
 
   default_action {
     type             = "forward"
-    target_group_arn = "${aws_lb_target_group.controllers.arn}"
+    target_group_arn = aws_lb_target_group.controllers.arn
   }
 }
 
 # Forward HTTP ingress traffic to workers
 resource "aws_lb_listener" "ingress-http" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
+  load_balancer_arn = aws_lb.nlb.arn
   protocol          = "TCP"
   port              = 80
 
   default_action {
     type             = "forward"
-    target_group_arn = "${module.workers.target_group_http}"
+    target_group_arn = module.workers.target_group_http
   }
 }
 
 # Forward HTTPS ingress traffic to workers
 resource "aws_lb_listener" "ingress-https" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
+  load_balancer_arn = aws_lb.nlb.arn
   protocol          = "TCP"
   port              = 443
 
   default_action {
     type             = "forward"
-    target_group_arn = "${module.workers.target_group_https}"
+    target_group_arn = module.workers.target_group_https
   }
 }
 
 # Target group of controllers
 resource "aws_lb_target_group" "controllers" {
   name        = "${var.cluster_name}-controllers"
-  vpc_id      = "${aws_vpc.network.id}"
+  vpc_id      = aws_vpc.network.id
   target_type = "instance"
 
   protocol = "TCP"
@@ -85,9 +85,10 @@ resource "aws_lb_target_group" "controllers" {
 
 # Attach controller instances to apiserver NLB
 resource "aws_lb_target_group_attachment" "controllers" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
-  target_group_arn = "${aws_lb_target_group.controllers.arn}"
+  target_group_arn = aws_lb_target_group.controllers.arn
-  target_id        = "${element(aws_instance.controllers.*.id, count.index)}"
+  target_id        = element(aws_instance.controllers.*.id, count.index)
   port             = 6443
 }
+

aws/container-linux/kubernetes/outputs.tf

@@ -1,53 +1,54 @@
 output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
+  value = module.bootkube.kubeconfig-admin
 }
 
 # Outputs for Kubernetes Ingress
 
 output "ingress_dns_name" {
-  value       = "${aws_lb.nlb.dns_name}"
+  value       = aws_lb.nlb.dns_name
   description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
 }
 
 output "ingress_zone_id" {
-  value       = "${aws_lb.nlb.zone_id}"
+  value       = aws_lb.nlb.zone_id
   description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
 }
 
 # Outputs for worker pools
 
 output "vpc_id" {
-  value       = "${aws_vpc.network.id}"
+  value       = aws_vpc.network.id
   description = "ID of the VPC for creating worker instances"
 }
 
 output "subnet_ids" {
-  value       = ["${aws_subnet.public.*.id}"]
+  value       = aws_subnet.public.*.id
   description = "List of subnet IDs for creating worker instances"
 }
 
 output "worker_security_groups" {
-  value       = ["${aws_security_group.worker.id}"]
+  value       = [aws_security_group.worker.id]
   description = "List of worker security group IDs"
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig-kubelet}"
+  value = module.bootkube.kubeconfig-kubelet
 }
 
 # Outputs for custom load balancing
 
 output "nlb_id" {
   description = "ARN of the Network Load Balancer"
-  value       = "${aws_lb.nlb.id}"
+  value       = aws_lb.nlb.id
 }
 
 output "worker_target_group_http" {
   description = "ARN of a target group of workers for HTTP traffic"
-  value       = "${module.workers.target_group_http}"
+  value       = module.workers.target_group_http
 }
 
 output "worker_target_group_https" {
   description = "ARN of a target group of workers for HTTPS traffic"
-  value       = "${module.workers.target_group_https}"
+  value       = module.workers.target_group_https
 }
+

aws/container-linux/kubernetes/require.tf

@@ -1,25 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "aws" {
-  version = ">= 1.13, < 3.0"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}

aws/container-linux/kubernetes/security.tf

@@ -6,13 +6,15 @@ resource "aws_security_group" "controller" {
   name        = "${var.cluster_name}-controller"
   description = "${var.cluster_name} controller security group"
 
-  vpc_id = "${aws_vpc.network.id}"
+  vpc_id = aws_vpc.network.id
 
-  tags = "${map("Name", "${var.cluster_name}-controller")}"
+  tags = {
+    "Name" = "${var.cluster_name}-controller"
+  }
 }
 
 resource "aws_security_group_rule" "controller-ssh" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type        = "ingress"
   protocol    = "tcp"
@@ -22,7 +24,7 @@ resource "aws_security_group_rule" "controller-ssh" {
 }
 
 resource "aws_security_group_rule" "controller-etcd" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type      = "ingress"
   protocol  = "tcp"
@@ -33,31 +35,31 @@ resource "aws_security_group_rule" "controller-etcd" {
 
 # Allow Prometheus to scrape etcd metrics
 resource "aws_security_group_rule" "controller-etcd-metrics" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type                     = "ingress"
   protocol                 = "tcp"
   from_port                = 2381
   to_port                  = 2381
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }
 
 resource "aws_security_group_rule" "controller-vxlan" {
-  count = "${var.networking == "flannel" ? 1 : 0}"
+  count = var.networking == "flannel" ? 1 : 0
 
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type                     = "ingress"
   protocol                 = "udp"
   from_port                = 4789
   to_port                  = 4789
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }
 
 resource "aws_security_group_rule" "controller-vxlan-self" {
-  count = "${var.networking == "flannel" ? 1 : 0}"
+  count = var.networking == "flannel" ? 1 : 0
 
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type      = "ingress"
   protocol  = "udp"
@@ -67,7 +69,7 @@ resource "aws_security_group_rule" "controller-vxlan-self" {
 }
 
 resource "aws_security_group_rule" "controller-apiserver" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type        = "ingress"
   protocol    = "tcp"
@@ -78,28 +80,28 @@ resource "aws_security_group_rule" "controller-apiserver" {
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "controller-node-exporter" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type                     = "ingress"
   protocol                 = "tcp"
   from_port                = 9100
   to_port                  = 9100
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }
 
 # Allow apiserver to access kubelets for exec, log, port-forward
 resource "aws_security_group_rule" "controller-kubelet" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type                     = "ingress"
   protocol                 = "tcp"
   from_port                = 10250
   to_port                  = 10250
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }
 
 resource "aws_security_group_rule" "controller-kubelet-self" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type      = "ingress"
   protocol  = "tcp"
@@ -109,17 +111,17 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
 }
 
 resource "aws_security_group_rule" "controller-bgp" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type                     = "ingress"
   protocol                 = "tcp"
   from_port                = 179
   to_port                  = 179
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }
 
 resource "aws_security_group_rule" "controller-bgp-self" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type      = "ingress"
   protocol  = "tcp"
@@ -129,17 +131,17 @@ resource "aws_security_group_rule" "controller-bgp-self" {
 }
 
 resource "aws_security_group_rule" "controller-ipip" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type                     = "ingress"
   protocol                 = 4
   from_port                = 0
   to_port                  = 0
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }
 
 resource "aws_security_group_rule" "controller-ipip-self" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type      = "ingress"
   protocol  = 4
@@ -149,17 +151,17 @@ resource "aws_security_group_rule" "controller-ipip-self" {
 }
 
 resource "aws_security_group_rule" "controller-ipip-legacy" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type                     = "ingress"
   protocol                 = 94
   from_port                = 0
   to_port                  = 0
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }
 
 resource "aws_security_group_rule" "controller-ipip-legacy-self" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type      = "ingress"
   protocol  = 94
@@ -169,7 +171,7 @@ resource "aws_security_group_rule" "controller-ipip-legacy-self" {
 }
 
 resource "aws_security_group_rule" "controller-egress" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id
 
   type             = "egress"
   protocol         = "-1"
@@ -185,13 +187,15 @@ resource "aws_security_group" "worker" {
   name        = "${var.cluster_name}-worker"
   description = "${var.cluster_name} worker security group"
 
-  vpc_id = "${aws_vpc.network.id}"
+  vpc_id = aws_vpc.network.id
 
-  tags = "${map("Name", "${var.cluster_name}-worker")}"
+  tags = {
+    "Name" = "${var.cluster_name}-worker"
+  }
 }
 
 resource "aws_security_group_rule" "worker-ssh" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type        = "ingress"
   protocol    = "tcp"
@@ -201,7 +205,7 @@ resource "aws_security_group_rule" "worker-ssh" {
 }
 
 resource "aws_security_group_rule" "worker-http" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type        = "ingress"
   protocol    = "tcp"
@@ -211,7 +215,7 @@ resource "aws_security_group_rule" "worker-http" {
 }
 
 resource "aws_security_group_rule" "worker-https" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type        = "ingress"
   protocol    = "tcp"
@@ -221,21 +225,21 @@ resource "aws_security_group_rule" "worker-https" {
 }
 
 resource "aws_security_group_rule" "worker-vxlan" {
-  count = "${var.networking == "flannel" ? 1 : 0}"
+  count = var.networking == "flannel" ? 1 : 0
 
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type                     = "ingress"
   protocol                 = "udp"
   from_port                = 4789
   to_port                  = 4789
-  source_security_group_id = "${aws_security_group.controller.id}"
+  source_security_group_id = aws_security_group.controller.id
 }
 
 resource "aws_security_group_rule" "worker-vxlan-self" {
-  count = "${var.networking == "flannel" ? 1 : 0}"
+  count = var.networking == "flannel" ? 1 : 0
 
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type      = "ingress"
   protocol  = "udp"
@@ -246,7 +250,7 @@ resource "aws_security_group_rule" "worker-vxlan-self" {
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "worker-node-exporter" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type      = "ingress"
   protocol  = "tcp"
@@ -256,7 +260,7 @@ resource "aws_security_group_rule" "worker-node-exporter" {
 }
 
 resource "aws_security_group_rule" "ingress-health" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type        = "ingress"
   protocol    = "tcp"
@@ -267,18 +271,18 @@ resource "aws_security_group_rule" "ingress-health" {
 
 # Allow apiserver to access kubelets for exec, log, port-forward
 resource "aws_security_group_rule" "worker-kubelet" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type                     = "ingress"
   protocol                 = "tcp"
   from_port                = 10250
   to_port                  = 10250
-  source_security_group_id = "${aws_security_group.controller.id}"
+  source_security_group_id = aws_security_group.controller.id
 }
 
 # Allow Prometheus to scrape kubelet metrics
 resource "aws_security_group_rule" "worker-kubelet-self" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type      = "ingress"
   protocol  = "tcp"
@@ -288,17 +292,17 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
 }
 
 resource "aws_security_group_rule" "worker-bgp" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type                     = "ingress"
   protocol                 = "tcp"
   from_port                = 179
   to_port                  = 179
-  source_security_group_id = "${aws_security_group.controller.id}"
+  source_security_group_id = aws_security_group.controller.id
 }
 
 resource "aws_security_group_rule" "worker-bgp-self" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type      = "ingress"
   protocol  = "tcp"
@@ -308,17 +312,17 @@ resource "aws_security_group_rule" "worker-bgp-self" {
 }
 
 resource "aws_security_group_rule" "worker-ipip" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type                     = "ingress"
   protocol                 = 4
   from_port                = 0
   to_port                  = 0
-  source_security_group_id = "${aws_security_group.controller.id}"
+  source_security_group_id = aws_security_group.controller.id
 }
 
 resource "aws_security_group_rule" "worker-ipip-self" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type      = "ingress"
   protocol  = 4
@@ -328,17 +332,17 @@ resource "aws_security_group_rule" "worker-ipip-self" {
 }
 
 resource "aws_security_group_rule" "worker-ipip-legacy" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type                     = "ingress"
   protocol                 = 94
   from_port                = 0
   to_port                  = 0
-  source_security_group_id = "${aws_security_group.controller.id}"
+  source_security_group_id = aws_security_group.controller.id
 }
 
 resource "aws_security_group_rule" "worker-ipip-legacy-self" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type      = "ingress"
   protocol  = 94
@@ -348,7 +352,7 @@ resource "aws_security_group_rule" "worker-ipip-legacy-self" {
 }
 
 resource "aws_security_group_rule" "worker-egress" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id
 
   type             = "egress"
   protocol         = "-1"
@@ -357,3 +361,4 @@ resource "aws_security_group_rule" "worker-egress" {
   cidr_blocks      = ["0.0.0.0/0"]
   ipv6_cidr_blocks = ["::/0"]
 }
+

aws/container-linux/kubernetes/ssh.tf

@@ -1,46 +1,46 @@
 # Secure copy etcd TLS assets to controllers.
 resource "null_resource" "copy-controller-secrets" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
   connection {
     type    = "ssh"
-    host    = "${element(aws_instance.controllers.*.public_ip, count.index)}"
+    host    = element(aws_instance.controllers.*.public_ip, count.index)
     user    = "core"
     timeout = "15m"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
+    content     = module.bootkube.etcd_ca_cert
     destination = "$HOME/etcd-client-ca.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
+    content     = module.bootkube.etcd_client_cert
     destination = "$HOME/etcd-client.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
+    content     = module.bootkube.etcd_client_key
     destination = "$HOME/etcd-client.key"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
+    content     = module.bootkube.etcd_server_cert
     destination = "$HOME/etcd-server.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
+    content     = module.bootkube.etcd_server_key
     destination = "$HOME/etcd-server.key"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
+    content     = module.bootkube.etcd_peer_cert
     destination = "$HOME/etcd-peer.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
+    content     = module.bootkube.etcd_peer_key
     destination = "$HOME/etcd-peer.key"
   }
 
@@ -64,21 +64,21 @@ resource "null_resource" "copy-controller-secrets" {
 # one-time self-hosted cluster bootstrapping.
 resource "null_resource" "bootkube-start" {
   depends_on = [
-    "module.bootkube",
+    module.bootkube,
-    "module.workers",
+    module.workers,
-    "aws_route53_record.apiserver",
+    aws_route53_record.apiserver,
-    "null_resource.copy-controller-secrets",
+    null_resource.copy-controller-secrets,
   ]
 
   connection {
     type    = "ssh"
-    host    = "${aws_instance.controllers.0.public_ip}"
+    host    = aws_instance.controllers[0].public_ip
     user    = "core"
     timeout = "15m"
   }
 
   provisioner "file" {
-    source      = "${var.asset_dir}"
+    source      = var.asset_dir
     destination = "$HOME/assets"
   }
 
@@ -89,3 +89,4 @@ resource "null_resource" "bootkube-start" {
     ]
   }
 }
+

aws/container-linux/kubernetes/variables.tf

@@ -1,90 +1,90 @@
 variable "cluster_name" {
-  type        = "string"
+  type        = string
   description = "Unique cluster name (prepended to dns_zone)"
 }
 
 # AWS
 
 variable "dns_zone" {
-  type        = "string"
+  type        = string
   description = "AWS Route53 DNS Zone (e.g. aws.example.com)"
 }
 
 variable "dns_zone_id" {
-  type        = "string"
+  type        = string
   description = "AWS Route53 DNS Zone ID (e.g. Z3PAABBCFAKEC0)"
 }
 
 # instances
 
 variable "controller_count" {
-  type        = "string"
+  type        = string
   default     = "1"
   description = "Number of controllers (i.e. masters)"
 }
 
 variable "worker_count" {
-  type        = "string"
+  type        = string
   default     = "1"
   description = "Number of workers"
 }
 
 variable "controller_type" {
-  type        = "string"
+  type        = string
   default     = "t3.small"
   description = "EC2 instance type for controllers"
 }
 
 variable "worker_type" {
-  type        = "string"
+  type        = string
   default     = "t3.small"
   description = "EC2 instance type for workers"
 }
 
 variable "os_image" {
-  type        = "string"
+  type        = string
   default     = "coreos-stable"
-  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
+  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
 }
 
 variable "disk_size" {
-  type        = "string"
+  type        = string
   default     = "40"
   description = "Size of the EBS volume in GB"
 }
 
 variable "disk_type" {
-  type        = "string"
+  type        = string
   default     = "gp2"
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
 variable "disk_iops" {
-  type        = "string"
+  type        = string
   default     = "0"
   description = "IOPS of the EBS volume (e.g. 100)"
 }
 
 variable "worker_price" {
-  type        = "string"
+  type        = string
   default     = ""
   description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
 }
 
 variable "worker_target_groups" {
-  type        = "list"
+  type        = list(string)
   description = "Additional target group ARNs to which worker instances should be added"
   default     = []
 }
 
 variable "controller_clc_snippets" {
-  type        = "list"
+  type        = list(string)
   description = "Controller Container Linux Config snippets"
   default     = []
 }
 
 variable "worker_clc_snippets" {
-  type        = "list"
+  type        = list(string)
   description = "Worker Container Linux Config snippets"
   default     = []
 }
@@ -92,36 +92,36 @@ variable "worker_clc_snippets" {
 # configuration
 
 variable "ssh_authorized_key" {
-  type        = "string"
+  type        = string
   description = "SSH public key for user 'core'"
 }
 
 variable "asset_dir" {
   description = "Path to a directory where generated assets should be placed (contains secrets)"
-  type        = "string"
+  type        = string
 }
 
 variable "networking" {
   description = "Choice of networking provider (calico or flannel)"
-  type        = "string"
+  type        = string
   default     = "calico"
 }
 
 variable "network_mtu" {
   description = "CNI interface MTU (applies to calico only). Use 8981 if using instances types with Jumbo frames."
-  type        = "string"
+  type        = string
   default     = "1480"
 }
 
 variable "host_cidr" {
   description = "CIDR IPv4 range to assign to EC2 nodes"
-  type        = "string"
+  type        = string
   default     = "10.0.0.0/16"
 }
 
 variable "pod_cidr" {
   description = "CIDR IPv4 range to assign Kubernetes pods"
-  type        = "string"
+  type        = string
   default     = "10.2.0.0/16"
 }
 
@@ -131,24 +131,26 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD
 
-  type    = "string"
+
+  type = string
   default = "10.3.0.0/16"
 }
 
 variable "cluster_domain_suffix" {
   description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
+  type = string
-  default     = "cluster.local"
+  default = "cluster.local"
 }
 
 variable "enable_reporting" {
-  type        = "string"
+  type = string
   description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = "false"
+  default = "false"
 }
 
 variable "enable_aggregation" {
   description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  type        = "string"
+  type = string
-  default     = "false"
+  default = "false"
 }
+

aws/container-linux/kubernetes/versions.tf

@@ -0,0 +1,11 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = "~> 0.12.0"
+  required_providers {
+    aws      = "~> 2.7"
+    ct       = "~> 0.3.2"
+    template = "~> 2.1"
+    null     = "~> 2.1"
+  }
+}

aws/container-linux/kubernetes/workers.tf

@@ -1,22 +1,23 @@
 module "workers" {
   source = "./workers"
-  name   = "${var.cluster_name}"
+  name   = var.cluster_name
 
   # AWS
-  vpc_id          = "${aws_vpc.network.id}"
+  vpc_id          = aws_vpc.network.id
-  subnet_ids      = ["${aws_subnet.public.*.id}"]
+  subnet_ids      = aws_subnet.public.*.id
-  security_groups = ["${aws_security_group.worker.id}"]
+  security_groups = [aws_security_group.worker.id]
-  worker_count    = "${var.worker_count}"
+  worker_count    = var.worker_count
-  instance_type   = "${var.worker_type}"
+  instance_type   = var.worker_type
-  os_image        = "${var.os_image}"
+  os_image        = var.os_image
-  disk_size       = "${var.disk_size}"
+  disk_size       = var.disk_size
-  spot_price      = "${var.worker_price}"
+  spot_price      = var.worker_price
-  target_groups   = ["${var.worker_target_groups}"]
+  target_groups   = var.worker_target_groups
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
+  kubeconfig            = module.bootkube.kubeconfig-kubelet
-  ssh_authorized_key    = "${var.ssh_authorized_key}"
+  ssh_authorized_key    = var.ssh_authorized_key
-  service_cidr          = "${var.service_cidr}"
+  service_cidr          = var.service_cidr
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  cluster_domain_suffix = var.cluster_domain_suffix
-  clc_snippets          = "${var.worker_clc_snippets}"
+  clc_snippets          = var.worker_clc_snippets
 }
+

aws/container-linux/kubernetes/workers/ami.tf

@@ -2,10 +2,10 @@ locals {
   # Pick a CoreOS Container Linux derivative
   # coreos-stable -> Container Linux AMI
   # flatcar-stable -> Flatcar Linux AMI
-  ami_id = "${local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id}"
+  ami_id = local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id
 
-  flavor  = "${element(split("-", var.os_image), 0)}"
+  flavor  = element(split("-", var.os_image), 0)
-  channel = "${element(split("-", var.os_image), 1)}"
+  channel = element(split("-", var.os_image), 1)
 }
 
 data "aws_ami" "coreos" {
@@ -24,7 +24,7 @@ data "aws_ami" "coreos" {
 
   filter {
     name   = "name"
-    values = ["CoreOS-${local.channel}-*"]
+    values = ["CoreOS-${local.flavor == "coreos" ? local.channel : "stable"}-*"]
   }
 }
 
@@ -44,6 +44,7 @@ data "aws_ami" "flatcar" {
 
   filter {
     name   = "name"
-    values = ["Flatcar-${local.channel}-*"]
+    values = ["Flatcar-${local.flavor == "flatcar" ? local.channel : "stable"}-*"]
   }
 }
+

aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -38,6 +38,7 @@ systemd:
           --volume var-log,kind=host,source=/var/log \
           --mount volume=var-log,target=/var/log \
           --insecure-options=image"
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
@@ -50,6 +51,7 @@ systemd:
           --anonymous-auth=false \
           --authentication-token-webhook \
           --authorization-mode=Webhook \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -93,7 +95,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.14.3
+          KUBELET_IMAGE_TAG=v1.15.0
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -111,7 +113,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.14.3 \
+            docker://k8s.gcr.io/hyperkube:v1.15.0 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

aws/container-linux/kubernetes/workers/ingress.tf

@@ -2,7 +2,7 @@
 
 resource "aws_lb_target_group" "workers-http" {
   name        = "${var.name}-workers-http"
-  vpc_id      = "${var.vpc_id}"
+  vpc_id      = var.vpc_id
   target_type = "instance"
 
   protocol = "TCP"
@@ -25,7 +25,7 @@ resource "aws_lb_target_group" "workers-http" {
 
 resource "aws_lb_target_group" "workers-https" {
   name        = "${var.name}-workers-https"
-  vpc_id      = "${var.vpc_id}"
+  vpc_id      = var.vpc_id
   target_type = "instance"
 
   protocol = "TCP"
@@ -45,3 +45,4 @@ resource "aws_lb_target_group" "workers-https" {
     interval = 10
   }
 }
+

aws/container-linux/kubernetes/workers/outputs.tf

@@ -1,9 +1,10 @@
 output "target_group_http" {
   description = "ARN of a target group of workers for HTTP traffic"
-  value       = "${aws_lb_target_group.workers-http.arn}"
+  value       = aws_lb_target_group.workers-http.arn
 }
 
 output "target_group_https" {
   description = "ARN of a target group of workers for HTTPS traffic"
-  value       = "${aws_lb_target_group.workers-https.arn}"
+  value       = aws_lb_target_group.workers-https.arn
 }
+

aws/container-linux/kubernetes/workers/variables.tf

@@ -1,77 +1,77 @@
 variable "name" {
-  type        = "string"
+  type        = string
   description = "Unique name for the worker pool"
 }
 
 # AWS
 
 variable "vpc_id" {
-  type        = "string"
+  type        = string
   description = "Must be set to `vpc_id` output by cluster"
 }
 
 variable "subnet_ids" {
-  type        = "list"
+  type        = list(string)
   description = "Must be set to `subnet_ids` output by cluster"
 }
 
 variable "security_groups" {
-  type        = "list"
+  type        = list(string)
   description = "Must be set to `worker_security_groups` output by cluster"
 }
 
 # instances
 
 variable "worker_count" {
-  type        = "string"
+  type        = string
   default     = "1"
   description = "Number of instances"
 }
 
 variable "instance_type" {
-  type        = "string"
+  type        = string
   default     = "t3.small"
   description = "EC2 instance type"
 }
 
 variable "os_image" {
-  type        = "string"
+  type        = string
   default     = "coreos-stable"
-  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
+  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
 }
 
 variable "disk_size" {
-  type        = "string"
+  type        = string
   default     = "40"
   description = "Size of the EBS volume in GB"
 }
 
 variable "disk_type" {
-  type        = "string"
+  type        = string
   default     = "gp2"
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
 variable "disk_iops" {
-  type        = "string"
+  type        = string
   default     = "0"
   description = "IOPS of the EBS volume (required for io1)"
 }
 
 variable "spot_price" {
-  type        = "string"
+  type        = string
   default     = ""
   description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
 }
 
 variable "target_groups" {
-  type        = "list"
+  type        = list(string)
   description = "Additional target group ARNs to which instances should be added"
   default     = []
 }
 
 variable "clc_snippets" {
-  type        = "list"
+  type        = list(string)
   description = "Container Linux Config snippets"
   default     = []
 }
@@ -79,12 +79,12 @@ variable "clc_snippets" {
 # configuration
 
 variable "kubeconfig" {
-  type        = "string"
+  type        = string
   description = "Must be set to `kubeconfig` output by cluster"
 }
 
 variable "ssh_authorized_key" {
-  type        = "string"
+  type        = string
   description = "SSH public key for user 'core'"
 }
 
@@ -94,12 +94,14 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD
 
-  type    = "string"
+
+  type = string
   default = "10.3.0.0/16"
 }
 
 variable "cluster_domain_suffix" {
   description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
+  type = string
-  default     = "cluster.local"
+  default = "cluster.local"
 }
+

aws/container-linux/kubernetes/workers/versions.tf

@@ -0,0 +1,4 @@
+
+terraform {
+  required_version = ">= 0.12"
+}

aws/container-linux/kubernetes/workers/workers.tf

@@ -3,24 +3,24 @@ resource "aws_autoscaling_group" "workers" {
   name = "${var.name}-worker ${aws_launch_configuration.worker.name}"
 
   # count
-  desired_capacity          = "${var.worker_count}"
+  desired_capacity          = var.worker_count
-  min_size                  = "${var.worker_count}"
+  min_size                  = var.worker_count
-  max_size                  = "${var.worker_count + 2}"
+  max_size                  = var.worker_count + 2
   default_cooldown          = 30
   health_check_grace_period = 30
 
   # network
-  vpc_zone_identifier = ["${var.subnet_ids}"]
+  vpc_zone_identifier = var.subnet_ids
 
   # template
-  launch_configuration = "${aws_launch_configuration.worker.name}"
+  launch_configuration = aws_launch_configuration.worker.name
 
   # target groups to which instances should be added
-  target_group_arns = [
+  target_group_arns = flatten([
-    "${aws_lb_target_group.workers-http.id}",
+    aws_lb_target_group.workers-http.id,
-    "${aws_lb_target_group.workers-https.id}",
+    aws_lb_target_group.workers-https.id,
-    "${var.target_groups}",
+    var.target_groups,
-  ]
+  ])
 
   lifecycle {
     # override the default destroy and replace update behavior
@@ -33,54 +33,58 @@ resource "aws_autoscaling_group" "workers" {
   # used. Disable wait to avoid issues and align with other clouds.
   wait_for_capacity_timeout = "0"
 
-  tags = [{
+  tags = [
-    key                 = "Name"
+    {
-    value               = "${var.name}-worker"
+      key                 = "Name"
-    propagate_at_launch = true
+      value               = "${var.name}-worker"
-  }]
+      propagate_at_launch = true
+    },
+  ]
 }
 
 # Worker template
 resource "aws_launch_configuration" "worker" {
-  image_id          = "${local.ami_id}"
+  image_id          = local.ami_id
-  instance_type     = "${var.instance_type}"
+  instance_type     = var.instance_type
-  spot_price        = "${var.spot_price}"
+  spot_price        = var.spot_price
   enable_monitoring = false
 
-  user_data = "${data.ct_config.worker-ignition.rendered}"
+  user_data = data.ct_config.worker-ignition.rendered
 
   # storage
   root_block_device {
-    volume_type = "${var.disk_type}"
+    volume_type = var.disk_type
-    volume_size = "${var.disk_size}"
+    volume_size = var.disk_size
-    iops        = "${var.disk_iops}"
+    iops        = var.disk_iops
   }
 
   # network
-  security_groups = ["${var.security_groups}"]
+  security_groups = var.security_groups
 
   lifecycle {
     // Override the default destroy and replace update behavior
     create_before_destroy = true
-    ignore_changes        = ["image_id"]
+    ignore_changes        = [image_id]
   }
 }
 
 # Worker Ignition config
 data "ct_config" "worker-ignition" {
-  content      = "${data.template_file.worker-config.rendered}"
+  content      = data.template_file.worker-config.rendered
   pretty_print = false
-  snippets     = ["${var.clc_snippets}"]
+  snippets     = var.clc_snippets
 }
 
 # Worker Container Linux config
 data "template_file" "worker-config" {
-  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
+  template = file("${path.module}/cl/worker.yaml.tmpl")
 
   vars = {
-    kubeconfig             = "${indent(10, var.kubeconfig)}"
+    kubeconfig             = indent(10, var.kubeconfig)
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    ssh_authorized_key     = var.ssh_authorized_key
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    cluster_domain_suffix  = var.cluster_domain_suffix
+    cgroup_driver          = local.flavor == "flatcar" && local.channel == "edge" ? "systemd" : "cgroupfs"
   }
 }
+

aws/fedora-atomic/kubernetes/LICENSE

@@ -1,23 +0,0 @@
-The MIT License (MIT)
-
-Copyright (c) 2017 Typhoon Authors
-Copyright (c) 2017 Dalton Hubble
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
-

aws/fedora-atomic/kubernetes/README.md

@@ -1,23 +0,0 @@
-# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
-
-Typhoon is a minimal and free Kubernetes distribution.
-
-* Minimal, stable base Kubernetes distribution
-* Declarative infrastructure and configuration
-* Free (freedom and cost) and privacy-respecting
-* Practical for labs, datacenters, and clouds
-
-Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
-
-## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
-
-* Kubernetes v1.14.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [spot](https://typhoon.psdn.io/cl/aws/#spot) workers
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
-
-## Docs
-
-Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/cl/aws/).
-

aws/fedora-atomic/kubernetes/ami.tf

@@ -1,19 +0,0 @@
-data "aws_ami" "fedora" {
-  most_recent = true
-  owners      = ["125523088429"]
-
-  filter {
-    name   = "architecture"
-    values = ["x86_64"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "name"
-    values = ["Fedora-AtomicHost-28-20180625.1.x86_64-*-gp2-*"]
-  }
-}

aws/fedora-atomic/kubernetes/bootkube.tf

@@ -1,18 +0,0 @@
-# Self-hosted Kubernetes assets (kubeconfig, manifests)
-module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=082921d67905417755609eebda7d39a7e26f7fdb"
-
-  cluster_name          = "${var.cluster_name}"
-  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
-  etcd_servers          = ["${aws_route53_record.etcds.*.fqdn}"]
-  asset_dir             = "${var.asset_dir}"
-  networking            = "${var.networking}"
-  network_mtu           = "${var.network_mtu}"
-  pod_cidr              = "${var.pod_cidr}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
-  enable_reporting      = "${var.enable_reporting}"
-
-  # Fedora
-  trusted_certs_dir = "/etc/pki/tls/certs"
-}

aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -1,93 +0,0 @@
-#cloud-config
-write_files:
-  - path: /etc/etcd/etcd.conf
-    content: |
-      ETCD_NAME=${etcd_name}
-      ETCD_DATA_DIR=/var/lib/etcd
-      ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
-      ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
-      ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
-      ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
-      ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
-      ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
-      ETCD_STRICT_RECONFIG_CHECK=true
-      ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
-      ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
-      ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
-      ETCD_CLIENT_CERT_AUTH=true
-      ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
-      ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
-      ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
-      ETCD_PEER_CLIENT_CERT_AUTH=true
-  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
-    content: |
-      [Unit]
-      Wants=rpc-statd.service
-      [Service]
-      ExecStartPre=/bin/mkdir -p /opt/cni/bin
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
-      ExecStartPre=/bin/mkdir -p /var/lib/cni
-      ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-      ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-      Restart=always
-      RestartSec=10
-  - path: /etc/kubernetes/kubelet.conf
-    content: |
-      ARGS="--anonymous-auth=false \
-        --authentication-token-webhook \
-        --authorization-mode=Webhook \
-        --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${cluster_dns_service_ip} \
-        --cluster_domain=${cluster_domain_suffix} \
-        --cni-conf-dir=/etc/kubernetes/cni/net.d \
-        --exit-on-lock-contention \
-        --kubeconfig=/etc/kubernetes/kubeconfig \
-        --lock-file=/var/run/lock/kubelet.lock \
-        --network-plugin=cni \
-        --node-labels=node-role.kubernetes.io/master \
-        --node-labels=node-role.kubernetes.io/controller="true" \
-        --pod-manifest-path=/etc/kubernetes/manifests \
-        --read-only-port=0 \
-        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
-        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
-  - path: /etc/kubernetes/kubeconfig
-    permissions: '0644'
-    content: |
-      ${kubeconfig}
-  - path: /var/lib/bootkube/.keep
-  - path: /etc/NetworkManager/conf.d/typhoon.conf
-    content: |
-      [main]
-      plugins=keyfile
-      [keyfile]
-      unmanaged-devices=interface-name:cali*;interface-name:tunl*
-  - path: /etc/selinux/config
-    owner: root:root
-    permissions: '0644'
-    content: |
-      SELINUX=permissive
-      SELINUXTYPE=targeted
-bootcmd:
-  - [setenforce, Permissive]
-  - [systemctl, disable, firewalld, --now]
-  # https://github.com/kubernetes/kubernetes/issues/60869
-  - [modprobe, ip_vs]
-runcmd:
-  - [systemctl, daemon-reload]
-  - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.14.1"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
-  - [systemctl, start, --no-block, etcd.service]
-  - [systemctl, start, --no-block, kubelet.service]
-users:
-  - default
-  - name: fedora
-    gecos: Fedora Admin
-    sudo: ALL=(ALL) NOPASSWD:ALL
-    groups: wheel,adm,systemd-journal,docker
-    ssh-authorized-keys:
-      - "${ssh_authorized_key}"

aws/fedora-atomic/kubernetes/controllers.tf

@@ -1,79 +0,0 @@
-# Discrete DNS records for each controller's private IPv4 for etcd usage
-resource "aws_route53_record" "etcds" {
-  count = "${var.controller_count}"
-
-  # DNS Zone where record should be created
-  zone_id = "${var.dns_zone_id}"
-
-  name = "${format("%s-etcd%d.%s.", var.cluster_name, count.index, var.dns_zone)}"
-  type = "A"
-  ttl  = 300
-
-  # private IPv4 address for etcd
-  records = ["${element(aws_instance.controllers.*.private_ip, count.index)}"]
-}
-
-# Controller instances
-resource "aws_instance" "controllers" {
-  count = "${var.controller_count}"
-
-  tags = {
-    Name = "${var.cluster_name}-controller-${count.index}"
-  }
-
-  instance_type = "${var.controller_type}"
-
-  ami       = "${data.aws_ami.fedora.image_id}"
-  user_data = "${element(data.template_file.controller-cloudinit.*.rendered, count.index)}"
-
-  # storage
-  root_block_device {
-    volume_type = "${var.disk_type}"
-    volume_size = "${var.disk_size}"
-    iops        = "${var.disk_iops}"
-  }
-
-  # network
-  associate_public_ip_address = true
-  subnet_id                   = "${element(aws_subnet.public.*.id, count.index)}"
-  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
-
-  lifecycle {
-    ignore_changes = [
-      "ami",
-      "user_data",
-    ]
-  }
-}
-
-# Controller Cloud-Init
-data "template_file" "controller-cloudinit" {
-  count = "${var.controller_count}"
-
-  template = "${file("${path.module}/cloudinit/controller.yaml.tmpl")}"
-
-  vars = {
-    # Cannot use cyclic dependencies on controllers or their DNS records
-    etcd_name   = "etcd${count.index}"
-    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
-
-    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
-
-    kubeconfig             = "${indent(6, module.bootkube.kubeconfig-kubelet)}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
-  }
-}
-
-data "template_file" "etcds" {
-  count    = "${var.controller_count}"
-  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
-
-  vars = {
-    index        = "${count.index}"
-    cluster_name = "${var.cluster_name}"
-    dns_zone     = "${var.dns_zone}"
-  }
-}

aws/fedora-atomic/kubernetes/network.tf

@@ -1,57 +0,0 @@
-data "aws_availability_zones" "all" {}
-
-# Network VPC, gateway, and routes
-
-resource "aws_vpc" "network" {
-  cidr_block                       = "${var.host_cidr}"
-  assign_generated_ipv6_cidr_block = true
-  enable_dns_support               = true
-  enable_dns_hostnames             = true
-
-  tags = "${map("Name", "${var.cluster_name}")}"
-}
-
-resource "aws_internet_gateway" "gateway" {
-  vpc_id = "${aws_vpc.network.id}"
-
-  tags = "${map("Name", "${var.cluster_name}")}"
-}
-
-resource "aws_route_table" "default" {
-  vpc_id = "${aws_vpc.network.id}"
-
-  route {
-    cidr_block = "0.0.0.0/0"
-    gateway_id = "${aws_internet_gateway.gateway.id}"
-  }
-
-  route {
-    ipv6_cidr_block = "::/0"
-    gateway_id      = "${aws_internet_gateway.gateway.id}"
-  }
-
-  tags = "${map("Name", "${var.cluster_name}")}"
-}
-
-# Subnets (one per availability zone)
-
-resource "aws_subnet" "public" {
-  count = "${length(data.aws_availability_zones.all.names)}"
-
-  vpc_id            = "${aws_vpc.network.id}"
-  availability_zone = "${data.aws_availability_zones.all.names[count.index]}"
-
-  cidr_block                      = "${cidrsubnet(var.host_cidr, 4, count.index)}"
-  ipv6_cidr_block                 = "${cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)}"
-  map_public_ip_on_launch         = true
-  assign_ipv6_address_on_creation = true
-
-  tags = "${map("Name", "${var.cluster_name}-public-${count.index}")}"
-}
-
-resource "aws_route_table_association" "public" {
-  count = "${length(data.aws_availability_zones.all.names)}"
-
-  route_table_id = "${aws_route_table.default.id}"
-  subnet_id      = "${element(aws_subnet.public.*.id, count.index)}"
-}

aws/fedora-atomic/kubernetes/nlb.tf

@@ -1,93 +0,0 @@
-# Network Load Balancer DNS Record
-resource "aws_route53_record" "apiserver" {
-  zone_id = "${var.dns_zone_id}"
-
-  name = "${format("%s.%s.", var.cluster_name, var.dns_zone)}"
-  type = "A"
-
-  # AWS recommends their special "alias" records for NLBs
-  alias {
-    name                   = "${aws_lb.nlb.dns_name}"
-    zone_id                = "${aws_lb.nlb.zone_id}"
-    evaluate_target_health = true
-  }
-}
-
-# Network Load Balancer for apiservers and ingress
-resource "aws_lb" "nlb" {
-  name               = "${var.cluster_name}-nlb"
-  load_balancer_type = "network"
-  internal           = false
-
-  subnets = ["${aws_subnet.public.*.id}"]
-
-  enable_cross_zone_load_balancing = true
-}
-
-# Forward TCP apiserver traffic to controllers
-resource "aws_lb_listener" "apiserver-https" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
-  protocol          = "TCP"
-  port              = "6443"
-
-  default_action {
-    type             = "forward"
-    target_group_arn = "${aws_lb_target_group.controllers.arn}"
-  }
-}
-
-# Forward HTTP ingress traffic to workers
-resource "aws_lb_listener" "ingress-http" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
-  protocol          = "TCP"
-  port              = 80
-
-  default_action {
-    type             = "forward"
-    target_group_arn = "${module.workers.target_group_http}"
-  }
-}
-
-# Forward HTTPS ingress traffic to workers
-resource "aws_lb_listener" "ingress-https" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
-  protocol          = "TCP"
-  port              = 443
-
-  default_action {
-    type             = "forward"
-    target_group_arn = "${module.workers.target_group_https}"
-  }
-}
-
-# Target group of controllers
-resource "aws_lb_target_group" "controllers" {
-  name        = "${var.cluster_name}-controllers"
-  vpc_id      = "${aws_vpc.network.id}"
-  target_type = "instance"
-
-  protocol = "TCP"
-  port     = 6443
-
-  # TCP health check for apiserver
-  health_check {
-    protocol = "TCP"
-    port     = 6443
-
-    # NLBs required to use same healthy and unhealthy thresholds
-    healthy_threshold   = 3
-    unhealthy_threshold = 3
-
-    # Interval between health checks required to be 10 or 30
-    interval = 10
-  }
-}
-
-# Attach controller instances to apiserver NLB
-resource "aws_lb_target_group_attachment" "controllers" {
-  count = "${var.controller_count}"
-
-  target_group_arn = "${aws_lb_target_group.controllers.arn}"
-  target_id        = "${element(aws_instance.controllers.*.id, count.index)}"
-  port             = 6443
-}

aws/fedora-atomic/kubernetes/outputs.tf

@@ -1,48 +0,0 @@
-output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
-}
-
-# Outputs for Kubernetes Ingress
-
-output "ingress_dns_name" {
-  value       = "${aws_lb.nlb.dns_name}"
-  description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
-}
-
-output "ingress_zone_id" {
-  value       = "${aws_lb.nlb.zone_id}"
-  description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
-}
-
-# Outputs for worker pools
-
-output "vpc_id" {
-  value       = "${aws_vpc.network.id}"
-  description = "ID of the VPC for creating worker instances"
-}
-
-output "subnet_ids" {
-  value       = ["${aws_subnet.public.*.id}"]
-  description = "List of subnet IDs for creating worker instances"
-}
-
-output "worker_security_groups" {
-  value       = ["${aws_security_group.worker.id}"]
-  description = "List of worker security group IDs"
-}
-
-output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig-kubelet}"
-}
-
-# Outputs for custom load balancing
-
-output "worker_target_group_http" {
-  description = "ARN of a target group of workers for HTTP traffic"
-  value       = "${module.workers.target_group_http}"
-}
-
-output "worker_target_group_https" {
-  description = "ARN of a target group of workers for HTTPS traffic"
-  value       = "${module.workers.target_group_https}"
-}

aws/fedora-atomic/kubernetes/require.tf

@@ -1,25 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "aws" {
-  version = ">= 1.13, < 3.0"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}

aws/fedora-atomic/kubernetes/security.tf

@@ -1,359 +0,0 @@
-# Security Groups (instance firewalls)
-
-# Controller security group
-
-resource "aws_security_group" "controller" {
-  name        = "${var.cluster_name}-controller"
-  description = "${var.cluster_name} controller security group"
-
-  vpc_id = "${aws_vpc.network.id}"
-
-  tags = "${map("Name", "${var.cluster_name}-controller")}"
-}
-
-resource "aws_security_group_rule" "controller-ssh" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 22
-  to_port     = 22
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-resource "aws_security_group_rule" "controller-etcd" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 2379
-  to_port   = 2380
-  self      = true
-}
-
-# Allow Prometheus to scrape etcd metrics
-resource "aws_security_group_rule" "controller-etcd-metrics" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 2381
-  to_port                  = 2381
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-vxlan" {
-  count = "${var.networking == "flannel" ? 1 : 0}"
-
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "udp"
-  from_port                = 4789
-  to_port                  = 4789
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-vxlan-self" {
-  count = "${var.networking == "flannel" ? 1 : 0}"
-
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "udp"
-  from_port = 4789
-  to_port   = 4789
-  self      = true
-}
-
-resource "aws_security_group_rule" "controller-apiserver" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 6443
-  to_port     = 6443
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-# Allow Prometheus to scrape node-exporter daemonset
-resource "aws_security_group_rule" "controller-node-exporter" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 9100
-  to_port                  = 9100
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-# Allow apiserver to access kubelets for exec, log, port-forward
-resource "aws_security_group_rule" "controller-kubelet" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10250
-  to_port                  = 10250
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10250
-  to_port   = 10250
-  self      = true
-}
-
-resource "aws_security_group_rule" "controller-bgp" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 179
-  to_port                  = 179
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-bgp-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 179
-  to_port   = 179
-  self      = true
-}
-
-resource "aws_security_group_rule" "controller-ipip" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = 4
-  from_port                = 0
-  to_port                  = 0
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-ipip-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = 4
-  from_port = 0
-  to_port   = 0
-  self      = true
-}
-
-resource "aws_security_group_rule" "controller-ipip-legacy" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = 94
-  from_port                = 0
-  to_port                  = 0
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-ipip-legacy-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = 94
-  from_port = 0
-  to_port   = 0
-  self      = true
-}
-
-resource "aws_security_group_rule" "controller-egress" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type             = "egress"
-  protocol         = "-1"
-  from_port        = 0
-  to_port          = 0
-  cidr_blocks      = ["0.0.0.0/0"]
-  ipv6_cidr_blocks = ["::/0"]
-}
-
-# Worker security group
-
-resource "aws_security_group" "worker" {
-  name        = "${var.cluster_name}-worker"
-  description = "${var.cluster_name} worker security group"
-
-  vpc_id = "${aws_vpc.network.id}"
-
-  tags = "${map("Name", "${var.cluster_name}-worker")}"
-}
-
-resource "aws_security_group_rule" "worker-ssh" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 22
-  to_port     = 22
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-resource "aws_security_group_rule" "worker-http" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 80
-  to_port     = 80
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-resource "aws_security_group_rule" "worker-https" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 443
-  to_port     = 443
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-resource "aws_security_group_rule" "worker-vxlan" {
-  count = "${var.networking == "flannel" ? 1 : 0}"
-
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "udp"
-  from_port                = 4789
-  to_port                  = 4789
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-vxlan-self" {
-  count = "${var.networking == "flannel" ? 1 : 0}"
-
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "udp"
-  from_port = 4789
-  to_port   = 4789
-  self      = true
-}
-
-# Allow Prometheus to scrape node-exporter daemonset
-resource "aws_security_group_rule" "worker-node-exporter" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 9100
-  to_port   = 9100
-  self      = true
-}
-
-resource "aws_security_group_rule" "ingress-health" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 10254
-  to_port     = 10254
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-# Allow apiserver to access kubelets for exec, log, port-forward
-resource "aws_security_group_rule" "worker-kubelet" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10250
-  to_port                  = 10250
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-# Allow Prometheus to scrape kubelet metrics
-resource "aws_security_group_rule" "worker-kubelet-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10250
-  to_port   = 10250
-  self      = true
-}
-
-resource "aws_security_group_rule" "worker-bgp" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 179
-  to_port                  = 179
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-bgp-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 179
-  to_port   = 179
-  self      = true
-}
-
-resource "aws_security_group_rule" "worker-ipip" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = 4
-  from_port                = 0
-  to_port                  = 0
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-ipip-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = 4
-  from_port = 0
-  to_port   = 0
-  self      = true
-}
-
-resource "aws_security_group_rule" "worker-ipip-legacy" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = 94
-  from_port                = 0
-  to_port                  = 0
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-ipip-legacy-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = 94
-  from_port = 0
-  to_port   = 0
-  self      = true
-}
-
-resource "aws_security_group_rule" "worker-egress" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type             = "egress"
-  protocol         = "-1"
-  from_port        = 0
-  to_port          = 0
-  cidr_blocks      = ["0.0.0.0/0"]
-  ipv6_cidr_blocks = ["::/0"]
-}

aws/fedora-atomic/kubernetes/ssh.tf

@@ -1,89 +0,0 @@
-# Secure copy etcd TLS assets to controllers.
-resource "null_resource" "copy-controller-secrets" {
-  count = "${var.controller_count}"
-
-  connection {
-    type    = "ssh"
-    host    = "${element(aws_instance.controllers.*.public_ip, count.index)}"
-    user    = "fedora"
-    timeout = "15m"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
-    destination = "$HOME/etcd-client-ca.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
-    destination = "$HOME/etcd-client.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
-    destination = "$HOME/etcd-client.key"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
-    destination = "$HOME/etcd-server.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
-    destination = "$HOME/etcd-server.key"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
-    destination = "$HOME/etcd-peer.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
-    destination = "$HOME/etcd-peer.key"
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      "sudo mkdir -p /etc/ssl/etcd/etcd",
-      "sudo mv etcd-client* /etc/ssl/etcd/",
-      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/server-ca.crt",
-      "sudo mv etcd-server.crt /etc/ssl/etcd/etcd/server.crt",
-      "sudo mv etcd-server.key /etc/ssl/etcd/etcd/server.key",
-      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/peer-ca.crt",
-      "sudo mv etcd-peer.crt /etc/ssl/etcd/etcd/peer.crt",
-      "sudo mv etcd-peer.key /etc/ssl/etcd/etcd/peer.key",
-    ]
-  }
-}
-
-# Secure copy bootkube assets to ONE controller and start bootkube to perform
-# one-time self-hosted cluster bootstrapping.
-resource "null_resource" "bootkube-start" {
-  depends_on = [
-    "null_resource.copy-controller-secrets",
-    "module.workers",
-    "aws_route53_record.apiserver",
-  ]
-
-  connection {
-    type    = "ssh"
-    host    = "${aws_instance.controllers.0.public_ip}"
-    user    = "fedora"
-    timeout = "15m"
-  }
-
-  provisioner "file" {
-    source      = "${var.asset_dir}"
-    destination = "$HOME/assets"
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do sleep 4; done",
-      "sudo mv $HOME/assets /var/lib/bootkube",
-      "sudo systemctl start bootkube",
-    ]
-  }
-}

aws/fedora-atomic/kubernetes/variables.tf

@@ -1,124 +0,0 @@
-variable "cluster_name" {
-  type        = "string"
-  description = "Unique cluster name (prepended to dns_zone)"
-}
-
-# AWS
-
-variable "dns_zone" {
-  type        = "string"
-  description = "AWS DNS Zone (e.g. aws.example.com)"
-}
-
-variable "dns_zone_id" {
-  type        = "string"
-  description = "AWS DNS Zone ID (e.g. Z3PAABBCFAKEC0)"
-}
-
-# instances
-
-variable "controller_count" {
-  type        = "string"
-  default     = "1"
-  description = "Number of controllers (i.e. masters)"
-}
-
-variable "worker_count" {
-  type        = "string"
-  default     = "1"
-  description = "Number of workers"
-}
-
-variable "controller_type" {
-  type        = "string"
-  default     = "t3.small"
-  description = "EC2 instance type for controllers"
-}
-
-variable "worker_type" {
-  type        = "string"
-  default     = "t3.small"
-  description = "EC2 instance type for workers"
-}
-
-variable "disk_size" {
-  type        = "string"
-  default     = "40"
-  description = "Size of the EBS volume in GB"
-}
-
-variable "disk_type" {
-  type        = "string"
-  default     = "gp2"
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-}
-
-variable "disk_iops" {
-  type        = "string"
-  default     = "0"
-  description = "IOPS of the EBS volume (e.g. 100)"
-}
-
-variable "worker_price" {
-  type        = "string"
-  default     = ""
-  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
-}
-
-# configuration
-
-variable "ssh_authorized_key" {
-  type        = "string"
-  description = "SSH public key for user 'fedora'"
-}
-
-variable "asset_dir" {
-  description = "Path to a directory where generated assets should be placed (contains secrets)"
-  type        = "string"
-}
-
-variable "networking" {
-  description = "Choice of networking provider (calico or flannel)"
-  type        = "string"
-  default     = "calico"
-}
-
-variable "network_mtu" {
-  description = "CNI interface MTU (applies to calico only). Use 8981 if using instances types with Jumbo frames."
-  type        = "string"
-  default     = "1480"
-}
-
-variable "host_cidr" {
-  description = "CIDR IPv4 range to assign to EC2 nodes"
-  type        = "string"
-  default     = "10.0.0.0/16"
-}
-
-variable "pod_cidr" {
-  description = "CIDR IPv4 range to assign Kubernetes pods"
-  type        = "string"
-  default     = "10.2.0.0/16"
-}
-
-variable "service_cidr" {
-  description = <<EOD
-CIDR IPv4 range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
-EOD
-
-  type    = "string"
-  default = "10.3.0.0/16"
-}
-
-variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
-  default     = "cluster.local"
-}
-
-variable "enable_reporting" {
-  type        = "string"
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = "false"
-}

aws/fedora-atomic/kubernetes/workers.tf

@@ -1,19 +0,0 @@
-module "workers" {
-  source = "./workers"
-  name   = "${var.cluster_name}"
-
-  # AWS
-  vpc_id          = "${aws_vpc.network.id}"
-  subnet_ids      = ["${aws_subnet.public.*.id}"]
-  security_groups = ["${aws_security_group.worker.id}"]
-  count           = "${var.worker_count}"
-  instance_type   = "${var.worker_type}"
-  disk_size       = "${var.disk_size}"
-  spot_price      = "${var.worker_price}"
-
-  # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
-  ssh_authorized_key    = "${var.ssh_authorized_key}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
-}

aws/fedora-atomic/kubernetes/workers/ami.tf

@@ -1,19 +0,0 @@
-data "aws_ami" "fedora" {
-  most_recent = true
-  owners      = ["125523088429"]
-
-  filter {
-    name   = "architecture"
-    values = ["x86_64"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "name"
-    values = ["Fedora-AtomicHost-28-20180625.1.x86_64-*-gp2-*"]
-  }
-}

aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -1,66 +0,0 @@
-#cloud-config
-write_files:
-  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
-    content: |
-      [Unit]
-      Wants=rpc-statd.service
-      [Service]
-      ExecStartPre=/bin/mkdir -p /opt/cni/bin
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-      ExecStartPre=/bin/mkdir -p /var/lib/cni
-      ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-      ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-      Restart=always
-      RestartSec=10
-  - path: /etc/kubernetes/kubelet.conf
-    content: |
-      ARGS="--anonymous-auth=false \
-        --authentication-token-webhook \
-        --authorization-mode=Webhook \
-        --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${cluster_dns_service_ip} \
-        --cluster_domain=${cluster_domain_suffix} \
-        --cni-conf-dir=/etc/kubernetes/cni/net.d \
-        --exit-on-lock-contention \
-        --kubeconfig=/etc/kubernetes/kubeconfig \
-        --lock-file=/var/run/lock/kubelet.lock \
-        --network-plugin=cni \
-        --node-labels=node-role.kubernetes.io/node \
-        --pod-manifest-path=/etc/kubernetes/manifests \
-        --read-only-port=0 \
-        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
-  - path: /etc/kubernetes/kubeconfig
-    permissions: '0644'
-    content: |
-      ${kubeconfig}
-  - path: /etc/NetworkManager/conf.d/typhoon.conf
-    content: |
-      [main]
-      plugins=keyfile
-      [keyfile]
-      unmanaged-devices=interface-name:cali*;interface-name:tunl*
-  - path: /etc/selinux/config
-    owner: root:root
-    permissions: '0644'
-    content: |
-      SELINUX=permissive
-      SELINUXTYPE=targeted
-bootcmd:
-  - [setenforce, Permissive]
-  - [systemctl, disable, firewalld, --now]
-  # https://github.com/kubernetes/kubernetes/issues/60869
-  - [modprobe, ip_vs]
-runcmd:
-  - [systemctl, daemon-reload]
-  - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.14.1"
-  - [systemctl, start, --no-block, kubelet.service]
-users:
-  - default
-  - name: fedora
-    gecos: Fedora Admin
-    sudo: ALL=(ALL) NOPASSWD:ALL
-    groups: wheel,adm,systemd-journal,docker
-    ssh-authorized-keys:
-      - "${ssh_authorized_key}"

aws/fedora-atomic/kubernetes/workers/ingress.tf

@@ -1,47 +0,0 @@
-# Target groups of instances for use with load balancers
-
-resource "aws_lb_target_group" "workers-http" {
-  name        = "${var.name}-workers-http"
-  vpc_id      = "${var.vpc_id}"
-  target_type = "instance"
-
-  protocol = "TCP"
-  port     = 80
-
-  # HTTP health check for ingress
-  health_check {
-    protocol = "HTTP"
-    port     = 10254
-    path     = "/healthz"
-
-    # NLBs required to use same healthy and unhealthy thresholds
-    healthy_threshold   = 3
-    unhealthy_threshold = 3
-
-    # Interval between health checks required to be 10 or 30
-    interval = 10
-  }
-}
-
-resource "aws_lb_target_group" "workers-https" {
-  name        = "${var.name}-workers-https"
-  vpc_id      = "${var.vpc_id}"
-  target_type = "instance"
-
-  protocol = "TCP"
-  port     = 443
-
-  # HTTP health check for ingress
-  health_check {
-    protocol = "HTTP"
-    port     = 10254
-    path     = "/healthz"
-
-    # NLBs required to use same healthy and unhealthy thresholds
-    healthy_threshold   = 3
-    unhealthy_threshold = 3
-
-    # Interval between health checks required to be 10 or 30
-    interval = 10
-  }
-}

aws/fedora-atomic/kubernetes/workers/outputs.tf

@@ -1,9 +0,0 @@
-output "target_group_http" {
-  description = "ARN of a target group of workers for HTTP traffic"
-  value       = "${aws_lb_target_group.workers-http.arn}"
-}
-
-output "target_group_https" {
-  description = "ARN of a target group of workers for HTTPS traffic"
-  value       = "${aws_lb_target_group.workers-https.arn}"
-}

aws/fedora-atomic/kubernetes/workers/variables.tf

@@ -1,87 +0,0 @@
-variable "name" {
-  type        = "string"
-  description = "Unique name for the worker pool"
-}
-
-# AWS
-
-variable "vpc_id" {
-  type        = "string"
-  description = "Must be set to `vpc_id` output by cluster"
-}
-
-variable "subnet_ids" {
-  type        = "list"
-  description = "Must be set to `subnet_ids` output by cluster"
-}
-
-variable "security_groups" {
-  type        = "list"
-  description = "Must be set to `worker_security_groups` output by cluster"
-}
-
-# instances
-
-variable "count" {
-  type        = "string"
-  default     = "1"
-  description = "Number of instances"
-}
-
-variable "instance_type" {
-  type        = "string"
-  default     = "t3.small"
-  description = "EC2 instance type"
-}
-
-variable "disk_size" {
-  type        = "string"
-  default     = "40"
-  description = "Size of the EBS volume in GB"
-}
-
-variable "disk_type" {
-  type        = "string"
-  default     = "gp2"
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-}
-
-variable "disk_iops" {
-  type        = "string"
-  default     = "0"
-  description = "IOPS of the EBS volume (required for io1)"
-}
-
-variable "spot_price" {
-  type        = "string"
-  default     = ""
-  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
-}
-
-# configuration
-
-variable "kubeconfig" {
-  type        = "string"
-  description = "Must be set to `kubeconfig` output by cluster"
-}
-
-variable "ssh_authorized_key" {
-  type        = "string"
-  description = "SSH public key for user 'fedora'"
-}
-
-variable "service_cidr" {
-  description = <<EOD
-CIDR IPv4 range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
-EOD
-
-  type    = "string"
-  default = "10.3.0.0/16"
-}
-
-variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
-  default     = "cluster.local"
-}

aws/fedora-atomic/kubernetes/workers/workers.tf

@@ -1,78 +0,0 @@
-# Workers AutoScaling Group
-resource "aws_autoscaling_group" "workers" {
-  name = "${var.name}-worker ${aws_launch_configuration.worker.name}"
-
-  # count
-  desired_capacity          = "${var.count}"
-  min_size                  = "${var.count}"
-  max_size                  = "${var.count + 2}"
-  default_cooldown          = 30
-  health_check_grace_period = 30
-
-  # network
-  vpc_zone_identifier = ["${var.subnet_ids}"]
-
-  # template
-  launch_configuration = "${aws_launch_configuration.worker.name}"
-
-  # target groups to which instances should be added
-  target_group_arns = [
-    "${aws_lb_target_group.workers-http.id}",
-    "${aws_lb_target_group.workers-https.id}",
-  ]
-
-  lifecycle {
-    # override the default destroy and replace update behavior
-    create_before_destroy = true
-  }
-
-  # Waiting for instance creation delays adding the ASG to state. If instances
-  # can't be created (e.g. spot price too low), the ASG will be orphaned.
-  # Orphaned ASGs escape cleanup, can't be updated, and keep bidding if spot is
-  # used. Disable wait to avoid issues and align with other clouds.
-  wait_for_capacity_timeout = "0"
-
-  tags = [{
-    key                 = "Name"
-    value               = "${var.name}-worker"
-    propagate_at_launch = true
-  }]
-}
-
-# Worker template
-resource "aws_launch_configuration" "worker" {
-  image_id          = "${data.aws_ami.fedora.image_id}"
-  instance_type     = "${var.instance_type}"
-  spot_price        = "${var.spot_price}"
-  enable_monitoring = false
-
-  user_data = "${data.template_file.worker-cloudinit.rendered}"
-
-  # storage
-  root_block_device {
-    volume_type = "${var.disk_type}"
-    volume_size = "${var.disk_size}"
-    iops        = "${var.disk_iops}"
-  }
-
-  # network
-  security_groups = ["${var.security_groups}"]
-
-  lifecycle {
-    // Override the default destroy and replace update behavior
-    create_before_destroy = true
-    ignore_changes        = ["image_id"]
-  }
-}
-
-# Worker Cloud-Init
-data "template_file" "worker-cloudinit" {
-  template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
-
-  vars = {
-    kubeconfig             = "${indent(6, var.kubeconfig)}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
-  }
-}

azure/container-linux/kubernetes/README.md

@@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.14.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.15.0 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 

azure/container-linux/kubernetes/bootkube.tf

@@ -1,22 +1,23 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=082921d67905417755609eebda7d39a7e26f7fdb"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=62df9ad69cc0da35f47d40fa981370c4503ad581"
 
-  cluster_name = "${var.cluster_name}"
+  cluster_name = var.cluster_name
-  api_servers  = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
+  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
-  etcd_servers = ["${formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)}"]
+  etcd_servers = formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)
-  asset_dir    = "${var.asset_dir}"
+  asset_dir    = var.asset_dir
 
-  networking = "${var.networking}"
+  networking = var.networking
 
   # only effective with Calico networking
   # we should be able to use 1450 MTU, but in practice, 1410 was needed
   network_encapsulation = "vxlan"
   network_mtu           = "1410"
 
-  pod_cidr              = "${var.pod_cidr}"
+  pod_cidr              = var.pod_cidr
-  service_cidr          = "${var.service_cidr}"
+  service_cidr          = var.service_cidr
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = "${var.enable_reporting}"
+  enable_reporting      = var.enable_reporting
-  enable_aggregation    = "${var.enable_aggregation}"
+  enable_aggregation    = var.enable_aggregation
 }
+

azure/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -123,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.14.3
+          KUBELET_IMAGE_TAG=v1.15.0
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

azure/container-linux/kubernetes/controllers.tf

@@ -1,31 +1,34 @@
 # Discrete DNS records for each controller's private IPv4 for etcd usage
 resource "azurerm_dns_a_record" "etcds" {
-  count               = "${var.controller_count}"
+  count               = var.controller_count
-  resource_group_name = "${var.dns_zone_group}"
+  resource_group_name = var.dns_zone_group
 
   # DNS Zone name where record should be created
-  zone_name = "${var.dns_zone}"
+  zone_name = var.dns_zone
 
   # DNS record
-  name = "${format("%s-etcd%d", var.cluster_name, count.index)}"
+  name = format("%s-etcd%d", var.cluster_name, count.index)
   ttl  = 300
 
   # private IPv4 address for etcd
-  records = ["${element(azurerm_network_interface.controllers.*.private_ip_address, count.index)}"]
+  records = [element(
+    azurerm_network_interface.controllers.*.private_ip_address,
+    count.index,
+  )]
 }
 
 locals {
   # Channel for a Container Linux derivative
   # coreos-stable -> Container Linux Stable
-  channel = "${element(split("-", var.os_image), 1)}"
+  channel = element(split("-", var.os_image), 1)
 }
 
 # Controller availability set to spread controllers
 resource "azurerm_availability_set" "controllers" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                         = "${var.cluster_name}-controllers"
-  location                     = "${var.region}"
+  location                     = var.region
   platform_fault_domain_count  = 2
   platform_update_domain_count = 4
   managed                      = true
@@ -33,19 +36,19 @@ resource "azurerm_availability_set" "controllers" {
 
 # Controller instances
 resource "azurerm_virtual_machine" "controllers" {
-  count               = "${var.controller_count}"
+  count               = var.controller_count
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                = "${var.cluster_name}-controller-${count.index}"
-  location            = "${var.region}"
+  location            = var.region
-  availability_set_id = "${azurerm_availability_set.controllers.id}"
+  availability_set_id = azurerm_availability_set.controllers.id
-  vm_size             = "${var.controller_type}"
+  vm_size             = var.controller_type
 
   # boot
   storage_image_reference {
     publisher = "CoreOS"
     offer     = "CoreOS"
-    sku       = "${local.channel}"
+    sku       = local.channel
     version   = "latest"
   }
 
@@ -54,18 +57,18 @@ resource "azurerm_virtual_machine" "controllers" {
     name              = "${var.cluster_name}-controller-${count.index}"
     create_option     = "FromImage"
     caching           = "ReadWrite"
-    disk_size_gb      = "${var.disk_size}"
+    disk_size_gb      = var.disk_size
     os_type           = "Linux"
     managed_disk_type = "Premium_LRS"
   }
 
   # network
-  network_interface_ids = ["${element(azurerm_network_interface.controllers.*.id, count.index)}"]
+  network_interface_ids = [element(azurerm_network_interface.controllers.*.id, count.index)]
 
   os_profile {
     computer_name  = "${var.cluster_name}-controller-${count.index}"
     admin_username = "core"
-    custom_data    = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
+    custom_data    = element(data.ct_config.controller-ignitions.*.rendered, count.index)
   }
 
   # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
@@ -74,7 +77,7 @@ resource "azurerm_virtual_machine" "controllers" {
 
     ssh_keys {
       path     = "/home/core/.ssh/authorized_keys"
-      key_data = "${var.ssh_authorized_key}"
+      key_data = var.ssh_authorized_key
     }
   }
 
@@ -84,85 +87,87 @@ resource "azurerm_virtual_machine" "controllers" {
 
   lifecycle {
     ignore_changes = [
-      "storage_os_disk",
+      storage_os_disk,
-      "os_profile",
+      os_profile,
     ]
   }
 }
 
 # Controller NICs with public and private IPv4
 resource "azurerm_network_interface" "controllers" {
-  count               = "${var.controller_count}"
+  count               = var.controller_count
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                      = "${var.cluster_name}-controller-${count.index}"
-  location                  = "${azurerm_resource_group.cluster.location}"
+  location                  = azurerm_resource_group.cluster.location
-  network_security_group_id = "${azurerm_network_security_group.controller.id}"
+  network_security_group_id = azurerm_network_security_group.controller.id
 
   ip_configuration {
     name                          = "ip0"
-    subnet_id                     = "${azurerm_subnet.controller.id}"
+    subnet_id                     = azurerm_subnet.controller.id
     private_ip_address_allocation = "dynamic"
 
     # public IPv4
-    public_ip_address_id = "${element(azurerm_public_ip.controllers.*.id, count.index)}"
+    public_ip_address_id = element(azurerm_public_ip.controllers.*.id, count.index)
   }
 }
 
 # Add controller NICs to the controller backend address pool
 resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
-  network_interface_id    = "${azurerm_network_interface.controllers.id}"
+  network_interface_id    = azurerm_network_interface.controllers[0].id
   ip_configuration_name   = "ip0"
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
 }
 
 # Controller public IPv4 addresses
 resource "azurerm_public_ip" "controllers" {
-  count               = "${var.controller_count}"
+  count               = var.controller_count
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name              = "${var.cluster_name}-controller-${count.index}"
-  location          = "${azurerm_resource_group.cluster.location}"
+  location          = azurerm_resource_group.cluster.location
   sku               = "Standard"
   allocation_method = "Static"
 }
 
 # Controller Ignition configs
 data "ct_config" "controller-ignitions" {
-  count        = "${var.controller_count}"
+  count = var.controller_count
-  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  content = element(
+    data.template_file.controller-configs.*.rendered,
+    count.index,
+  )
   pretty_print = false
-  snippets     = ["${var.controller_clc_snippets}"]
+  snippets     = var.controller_clc_snippets
 }
 
 # Controller Container Linux configs
 data "template_file" "controller-configs" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
-  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+  template = file("${path.module}/cl/controller.yaml.tmpl")
 
   vars = {
     # Cannot use cyclic dependencies on controllers or their DNS records
     etcd_name   = "etcd${count.index}"
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
-
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
+    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
-
+    kubeconfig             = indent(10, module.bootkube.kubeconfig-kubelet)
-    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = var.ssh_authorized_key
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = var.cluster_domain_suffix
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 
 data "template_file" "etcds" {
-  count    = "${var.controller_count}"
+  count    = var.controller_count
   template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
 
   vars = {
-    index        = "${count.index}"
+    index        = count.index
-    cluster_name = "${var.cluster_name}"
+    cluster_name = var.cluster_name
-    dns_zone     = "${var.dns_zone}"
+    dns_zone     = var.dns_zone
   }
 }
+

azure/container-linux/kubernetes/lb.tf

@@ -1,123 +1,123 @@
 # DNS record for the apiserver load balancer
 resource "azurerm_dns_a_record" "apiserver" {
-  resource_group_name = "${var.dns_zone_group}"
+  resource_group_name = var.dns_zone_group
 
   # DNS Zone name where record should be created
-  zone_name = "${var.dns_zone}"
+  zone_name = var.dns_zone
 
   # DNS record
-  name = "${var.cluster_name}"
+  name = var.cluster_name
   ttl  = 300
 
   # IPv4 address of apiserver load balancer
-  records = ["${azurerm_public_ip.apiserver-ipv4.ip_address}"]
+  records = [azurerm_public_ip.apiserver-ipv4.ip_address]
 }
 
 # Static IPv4 address for the apiserver frontend
 resource "azurerm_public_ip" "apiserver-ipv4" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name              = "${var.cluster_name}-apiserver-ipv4"
-  location          = "${var.region}"
+  location          = var.region
   sku               = "Standard"
   allocation_method = "Static"
 }
 
 # Static IPv4 address for the ingress frontend
 resource "azurerm_public_ip" "ingress-ipv4" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name              = "${var.cluster_name}-ingress-ipv4"
-  location          = "${var.region}"
+  location          = var.region
   sku               = "Standard"
   allocation_method = "Static"
 }
 
 # Network Load Balancer for apiservers and ingress
 resource "azurerm_lb" "cluster" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
-  name     = "${var.cluster_name}"
+  name     = var.cluster_name
-  location = "${var.region}"
+  location = var.region
   sku      = "Standard"
 
   frontend_ip_configuration {
     name                 = "apiserver"
-    public_ip_address_id = "${azurerm_public_ip.apiserver-ipv4.id}"
+    public_ip_address_id = azurerm_public_ip.apiserver-ipv4.id
   }
 
   frontend_ip_configuration {
     name                 = "ingress"
-    public_ip_address_id = "${azurerm_public_ip.ingress-ipv4.id}"
+    public_ip_address_id = azurerm_public_ip.ingress-ipv4.id
   }
 }
 
 resource "azurerm_lb_rule" "apiserver" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                           = "apiserver"
-  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  loadbalancer_id                = azurerm_lb.cluster.id
   frontend_ip_configuration_name = "apiserver"
 
   protocol                = "Tcp"
   frontend_port           = 6443
   backend_port            = 6443
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
-  probe_id                = "${azurerm_lb_probe.apiserver.id}"
+  probe_id                = azurerm_lb_probe.apiserver.id
 }
 
 resource "azurerm_lb_rule" "ingress-http" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                           = "ingress-http"
-  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  loadbalancer_id                = azurerm_lb.cluster.id
   frontend_ip_configuration_name = "ingress"
 
   protocol                = "Tcp"
   frontend_port           = 80
   backend_port            = 80
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = "${azurerm_lb_probe.ingress.id}"
+  probe_id                = azurerm_lb_probe.ingress.id
 }
 
 resource "azurerm_lb_rule" "ingress-https" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                           = "ingress-https"
-  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  loadbalancer_id                = azurerm_lb.cluster.id
   frontend_ip_configuration_name = "ingress"
 
   protocol                = "Tcp"
   frontend_port           = 443
   backend_port            = 443
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = "${azurerm_lb_probe.ingress.id}"
+  probe_id                = azurerm_lb_probe.ingress.id
 }
 
 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name            = "controller"
-  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  loadbalancer_id = azurerm_lb.cluster.id
 }
 
 # Address pool of workers
 resource "azurerm_lb_backend_address_pool" "worker" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name            = "worker"
-  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  loadbalancer_id = azurerm_lb.cluster.id
 }
 
 # Health checks / probes
 
 # TCP health check for apiserver
 resource "azurerm_lb_probe" "apiserver" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name            = "apiserver"
-  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  loadbalancer_id = azurerm_lb.cluster.id
   protocol        = "Tcp"
   port            = 6443
 
@@ -129,10 +129,10 @@ resource "azurerm_lb_probe" "apiserver" {
 
 # HTTP health check for ingress
 resource "azurerm_lb_probe" "ingress" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name            = "ingress"
-  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  loadbalancer_id = azurerm_lb.cluster.id
   protocol        = "Http"
   port            = 10254
   request_path    = "/healthz"
@@ -142,3 +142,4 @@ resource "azurerm_lb_probe" "ingress" {
 
   interval_in_seconds = 5
 }
+

azure/container-linux/kubernetes/network.tf

@@ -1,15 +1,15 @@
 # Organize cluster into a resource group
 resource "azurerm_resource_group" "cluster" {
-  name     = "${var.cluster_name}"
+  name     = var.cluster_name
-  location = "${var.region}"
+  location = var.region
 }
 
 resource "azurerm_virtual_network" "network" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
-  name          = "${var.cluster_name}"
+  name          = var.cluster_name
-  location      = "${azurerm_resource_group.cluster.location}"
+  location      = azurerm_resource_group.cluster.location
-  address_space = ["${var.host_cidr}"]
+  address_space = [var.host_cidr]
 }
 
 # Subnets - separate subnets for controller and workers because Azure
@@ -17,17 +17,18 @@ resource "azurerm_virtual_network" "network" {
 # tags like GCP or security group membership like AWS
 
 resource "azurerm_subnet" "controller" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                 = "controller"
-  virtual_network_name = "${azurerm_virtual_network.network.name}"
+  virtual_network_name = azurerm_virtual_network.network.name
-  address_prefix       = "${cidrsubnet(var.host_cidr, 1, 0)}"
+  address_prefix       = cidrsubnet(var.host_cidr, 1, 0)
 }
 
 resource "azurerm_subnet" "worker" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                 = "worker"
-  virtual_network_name = "${azurerm_virtual_network.network.name}"
+  virtual_network_name = azurerm_virtual_network.network.name
-  address_prefix       = "${cidrsubnet(var.host_cidr, 1, 1)}"
+  address_prefix       = cidrsubnet(var.host_cidr, 1, 1)
 }
+

azure/container-linux/kubernetes/outputs.tf

@@ -1,55 +1,56 @@
 output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
+  value = module.bootkube.kubeconfig-admin
 }
 
 # Outputs for Kubernetes Ingress
 
 output "ingress_static_ipv4" {
-  value       = "${azurerm_public_ip.ingress-ipv4.ip_address}"
+  value       = azurerm_public_ip.ingress-ipv4.ip_address
   description = "IPv4 address of the load balancer for distributing traffic to Ingress controllers"
 }
 
 # Outputs for worker pools
 
 output "region" {
-  value = "${azurerm_resource_group.cluster.location}"
+  value = azurerm_resource_group.cluster.location
 }
 
 output "resource_group_name" {
-  value = "${azurerm_resource_group.cluster.name}"
+  value = azurerm_resource_group.cluster.name
 }
 
 output "subnet_id" {
-  value = "${azurerm_subnet.worker.id}"
+  value = azurerm_subnet.worker.id
 }
 
 output "security_group_id" {
-  value = "${azurerm_network_security_group.worker.id}"
+  value = azurerm_network_security_group.worker.id
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig-kubelet}"
+  value = module.bootkube.kubeconfig-kubelet
 }
 
 # Outputs for custom firewalling
 
 output "worker_security_group_name" {
-  value = "${azurerm_network_security_group.worker.name}"
+  value = azurerm_network_security_group.worker.name
 }
 
 output "worker_address_prefix" {
   description = "Worker network subnet CIDR address (for source/destination)"
-  value       = "${azurerm_subnet.worker.address_prefix}"
+  value       = azurerm_subnet.worker.address_prefix
 }
 
 # Outputs for custom load balancing
 
 output "loadbalancer_id" {
   description = "ID of the cluster load balancer"
-  value       = "${azurerm_lb.cluster.id}"
+  value       = azurerm_lb.cluster.id
 }
 
 output "backend_address_pool_id" {
   description = "ID of the worker backend address pool"
-  value       = "${azurerm_lb_backend_address_pool.worker.id}"
+  value       = azurerm_lb_backend_address_pool.worker.id
 }
+

azure/container-linux/kubernetes/require.tf

@@ -1,25 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "azurerm" {
-  version = "~> 1.22"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}

azure/container-linux/kubernetes/security.tf

@@ -1,17 +1,17 @@
 # Controller security group
 
 resource "azurerm_network_security_group" "controller" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name     = "${var.cluster_name}-controller"
-  location = "${azurerm_resource_group.cluster.location}"
+  location = azurerm_resource_group.cluster.location
 }
 
 resource "azurerm_network_security_rule" "controller-ssh" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-ssh"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
   priority                    = "2000"
   access                      = "Allow"
   direction                   = "Inbound"
@@ -19,45 +19,45 @@ resource "azurerm_network_security_rule" "controller-ssh" {
   source_port_range           = "*"
   destination_port_range      = "22"
   source_address_prefix       = "*"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
 
 resource "azurerm_network_security_rule" "controller-etcd" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-etcd"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
   priority                    = "2005"
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "2379-2380"
-  source_address_prefix       = "${azurerm_subnet.controller.address_prefix}"
+  source_address_prefix       = azurerm_subnet.controller.address_prefix
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
 
 # Allow Prometheus to scrape etcd metrics
 resource "azurerm_network_security_rule" "controller-etcd-metrics" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-etcd-metrics"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
   priority                    = "2010"
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "2381"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
 
 resource "azurerm_network_security_rule" "controller-apiserver" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-apiserver"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
   priority                    = "2015"
   access                      = "Allow"
   direction                   = "Inbound"
@@ -65,46 +65,46 @@ resource "azurerm_network_security_rule" "controller-apiserver" {
   source_port_range           = "*"
   destination_port_range      = "6443"
   source_address_prefix       = "*"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
 
 resource "azurerm_network_security_rule" "controller-vxlan" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-vxlan"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
   priority                    = "2020"
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Udp"
   source_port_range           = "*"
   destination_port_range      = "4789"
-  source_address_prefixes     = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "controller-node-exporter" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-node-exporter"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
   priority                    = "2025"
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "9100"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
 
 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "controller-kubelet" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-kubelet"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
   priority                    = "2030"
   access                      = "Allow"
   direction                   = "Inbound"
@@ -113,18 +113,18 @@ resource "azurerm_network_security_rule" "controller-kubelet" {
   destination_port_range      = "10250"
 
   # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix = azurerm_subnet.controller.address_prefix
 }
 
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
 
 resource "azurerm_network_security_rule" "controller-allow-loadblancer" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-loadbalancer"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
   priority                    = "3000"
   access                      = "Allow"
   direction                   = "Inbound"
@@ -136,10 +136,10 @@ resource "azurerm_network_security_rule" "controller-allow-loadblancer" {
 }
 
 resource "azurerm_network_security_rule" "controller-deny-all" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "deny-all"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
   priority                    = "3005"
   access                      = "Deny"
   direction                   = "Inbound"
@@ -153,32 +153,32 @@ resource "azurerm_network_security_rule" "controller-deny-all" {
 # Worker security group
 
 resource "azurerm_network_security_group" "worker" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name     = "${var.cluster_name}-worker"
-  location = "${azurerm_resource_group.cluster.location}"
+  location = azurerm_resource_group.cluster.location
 }
 
 resource "azurerm_network_security_rule" "worker-ssh" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-ssh"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
   priority                    = "2000"
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "22"
-  source_address_prefix       = "${azurerm_subnet.controller.address_prefix}"
+  source_address_prefix       = azurerm_subnet.controller.address_prefix
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }
 
 resource "azurerm_network_security_rule" "worker-http" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-http"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
   priority                    = "2005"
   access                      = "Allow"
   direction                   = "Inbound"
@@ -186,14 +186,14 @@ resource "azurerm_network_security_rule" "worker-http" {
   source_port_range           = "*"
   destination_port_range      = "80"
   source_address_prefix       = "*"
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }
 
 resource "azurerm_network_security_rule" "worker-https" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-https"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
   priority                    = "2010"
   access                      = "Allow"
   direction                   = "Inbound"
@@ -201,46 +201,46 @@ resource "azurerm_network_security_rule" "worker-https" {
   source_port_range           = "*"
   destination_port_range      = "443"
   source_address_prefix       = "*"
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }
 
 resource "azurerm_network_security_rule" "worker-vxlan" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-vxlan"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
   priority                    = "2015"
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Udp"
   source_port_range           = "*"
   destination_port_range      = "4789"
-  source_address_prefixes     = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "worker-node-exporter" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-node-exporter"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
   priority                    = "2020"
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "9100"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }
 
 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "worker-kubelet" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-kubelet"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
   priority                    = "2025"
   access                      = "Allow"
   direction                   = "Inbound"
@@ -249,18 +249,18 @@ resource "azurerm_network_security_rule" "worker-kubelet" {
   destination_port_range      = "10250"
 
   # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix = azurerm_subnet.worker.address_prefix
 }
 
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
 
 resource "azurerm_network_security_rule" "worker-allow-loadblancer" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "allow-loadbalancer"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
   priority                    = "3000"
   access                      = "Allow"
   direction                   = "Inbound"
@@ -272,10 +272,10 @@ resource "azurerm_network_security_rule" "worker-allow-loadblancer" {
 }
 
 resource "azurerm_network_security_rule" "worker-deny-all" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name
 
   name                        = "deny-all"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
   priority                    = "3005"
   access                      = "Deny"
   direction                   = "Inbound"
@@ -285,3 +285,4 @@ resource "azurerm_network_security_rule" "worker-deny-all" {
   source_address_prefix       = "*"
   destination_address_prefix  = "*"
 }
+

azure/container-linux/kubernetes/ssh.tf

@@ -1,50 +1,48 @@
 # Secure copy etcd TLS assets to controllers.
 resource "null_resource" "copy-controller-secrets" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
-  depends_on = [
+  depends_on = [azurerm_virtual_machine.controllers]
-    "azurerm_virtual_machine.controllers",
-  ]
 
   connection {
     type    = "ssh"
-    host    = "${element(azurerm_public_ip.controllers.*.ip_address, count.index)}"
+    host    = element(azurerm_public_ip.controllers.*.ip_address, count.index)
     user    = "core"
     timeout = "15m"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
+    content     = module.bootkube.etcd_ca_cert
     destination = "$HOME/etcd-client-ca.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
+    content     = module.bootkube.etcd_client_cert
     destination = "$HOME/etcd-client.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
+    content     = module.bootkube.etcd_client_key
     destination = "$HOME/etcd-client.key"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
+    content     = module.bootkube.etcd_server_cert
     destination = "$HOME/etcd-server.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
+    content     = module.bootkube.etcd_server_key
     destination = "$HOME/etcd-server.key"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
+    content     = module.bootkube.etcd_peer_cert
     destination = "$HOME/etcd-peer.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
+    content     = module.bootkube.etcd_peer_key
     destination = "$HOME/etcd-peer.key"
   }
 
@@ -68,21 +66,21 @@ resource "null_resource" "copy-controller-secrets" {
 # one-time self-hosted cluster bootstrapping.
 resource "null_resource" "bootkube-start" {
   depends_on = [
-    "module.bootkube",
+    module.bootkube,
-    "module.workers",
+    module.workers,
-    "azurerm_dns_a_record.apiserver",
+    azurerm_dns_a_record.apiserver,
-    "null_resource.copy-controller-secrets",
+    null_resource.copy-controller-secrets,
   ]
 
   connection {
     type    = "ssh"
-    host    = "${element(azurerm_public_ip.controllers.*.ip_address, 0)}"
+    host    = element(azurerm_public_ip.controllers.*.ip_address, 0)
     user    = "core"
     timeout = "15m"
   }
 
   provisioner "file" {
-    source      = "${var.asset_dir}"
+    source      = var.asset_dir
     destination = "$HOME/assets"
   }
 
@@ -93,3 +91,4 @@ resource "null_resource" "bootkube-start" {
     ]
   }
 }
+

azure/container-linux/kubernetes/variables.tf

@@ -1,77 +1,77 @@
 variable "cluster_name" {
-  type        = "string"
+  type        = string
   description = "Unique cluster name (prepended to dns_zone)"
 }
 
 # Azure
 
 variable "region" {
-  type        = "string"
+  type        = string
   description = "Azure Region (e.g. centralus , see `az account list-locations --output table`)"
 }
 
 variable "dns_zone" {
-  type        = "string"
+  type        = string
   description = "Azure DNS Zone (e.g. azure.example.com)"
 }
 
 variable "dns_zone_group" {
-  type        = "string"
+  type        = string
   description = "Resource group where the Azure DNS Zone resides (e.g. global)"
 }
 
 # instances
 
 variable "controller_count" {
-  type        = "string"
+  type        = string
   default     = "1"
   description = "Number of controllers (i.e. masters)"
 }
 
 variable "worker_count" {
-  type        = "string"
+  type        = string
   default     = "1"
   description = "Number of workers"
 }
 
 variable "controller_type" {
-  type        = "string"
+  type        = string
   default     = "Standard_DS1_v2"
   description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
 }
 
 variable "worker_type" {
-  type        = "string"
+  type        = string
   default     = "Standard_F1"
   description = "Machine type for workers (see `az vm list-skus --location centralus`)"
 }
 
 variable "os_image" {
-  type        = "string"
+  type        = string
   default     = "coreos-stable"
   description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha)"
 }
 
 variable "disk_size" {
-  type        = "string"
+  type        = string
   default     = "40"
   description = "Size of the disk in GB"
 }
 
 variable "worker_priority" {
-  type        = "string"
+  type        = string
   default     = "Regular"
   description = "Set worker priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
 }
 
 variable "controller_clc_snippets" {
-  type        = "list"
+  type        = list(string)
   description = "Controller Container Linux Config snippets"
   default     = []
 }
 
 variable "worker_clc_snippets" {
-  type        = "list"
+  type        = list(string)
   description = "Worker Container Linux Config snippets"
   default     = []
 }
@@ -79,30 +79,30 @@ variable "worker_clc_snippets" {
 # configuration
 
 variable "ssh_authorized_key" {
-  type        = "string"
+  type        = string
   description = "SSH public key for user 'core'"
 }
 
 variable "asset_dir" {
   description = "Path to a directory where generated assets should be placed (contains secrets)"
-  type        = "string"
+  type        = string
 }
 
 variable "networking" {
   description = "Choice of networking provider (flannel or calico)"
-  type        = "string"
+  type        = string
   default     = "flannel"
 }
 
 variable "host_cidr" {
   description = "CIDR IPv4 range to assign to instances"
-  type        = "string"
+  type        = string
   default     = "10.0.0.0/16"
 }
 
 variable "pod_cidr" {
   description = "CIDR IPv4 range to assign Kubernetes pods"
-  type        = "string"
+  type        = string
   default     = "10.2.0.0/16"
 }
 
@@ -112,24 +112,26 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD
 
-  type    = "string"
+
+  type = string
   default = "10.3.0.0/16"
 }
 
 variable "cluster_domain_suffix" {
   description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
+  type = string
-  default     = "cluster.local"
+  default = "cluster.local"
 }
 
 variable "enable_reporting" {
-  type        = "string"
+  type = string
   description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = "false"
+  default = "false"
 }
 
 variable "enable_aggregation" {
   description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  type        = "string"
+  type = string
-  default     = "false"
+  default = "false"
 }
+

azure/container-linux/kubernetes/versions.tf

@@ -0,0 +1,12 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = "~> 0.12.0"
+  required_providers {
+    azurerm  = "~> 1.27"
+    ct       = "~> 0.3.2"
+    template = "~> 2.1"
+    null     = "~> 2.1"
+  }
+}
+

azure/container-linux/kubernetes/workers.tf

@@ -1,23 +1,24 @@
 module "workers" {
   source = "./workers"
-  name   = "${var.cluster_name}"
+  name   = var.cluster_name
 
   # Azure
-  resource_group_name     = "${azurerm_resource_group.cluster.name}"
+  resource_group_name     = azurerm_resource_group.cluster.name
-  region                  = "${azurerm_resource_group.cluster.location}"
+  region                  = azurerm_resource_group.cluster.location
-  subnet_id               = "${azurerm_subnet.worker.id}"
+  subnet_id               = azurerm_subnet.worker.id
-  security_group_id       = "${azurerm_network_security_group.worker.id}"
+  security_group_id       = azurerm_network_security_group.worker.id
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
 
-  worker_count    = "${var.worker_count}"
+  worker_count = var.worker_count
-  vm_type  = "${var.worker_type}"
+  vm_type      = var.worker_type
-  os_image = "${var.os_image}"
+  os_image     = var.os_image
-  priority = "${var.worker_priority}"
+  priority     = var.worker_priority
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
+  kubeconfig            = module.bootkube.kubeconfig-kubelet
-  ssh_authorized_key    = "${var.ssh_authorized_key}"
+  ssh_authorized_key    = var.ssh_authorized_key
-  service_cidr          = "${var.service_cidr}"
+  service_cidr          = var.service_cidr
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  cluster_domain_suffix = var.cluster_domain_suffix
-  clc_snippets          = "${var.worker_clc_snippets}"
+  clc_snippets          = var.worker_clc_snippets
 }
+

azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -93,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.14.3
+          KUBELET_IMAGE_TAG=v1.15.0
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -111,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.14.3 \
+            docker://k8s.gcr.io/hyperkube:v1.15.0 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')

azure/container-linux/kubernetes/workers/variables.tf

@@ -1,63 +1,63 @@
 variable "name" {
-  type        = "string"
+  type        = string
   description = "Unique name for the worker pool"
 }
 
 # Azure
 
 variable "region" {
-  type        = "string"
+  type        = string
   description = "Must be set to the Azure Region of cluster"
 }
 
 variable "resource_group_name" {
-  type        = "string"
+  type        = string
   description = "Must be set to the resource group name of cluster"
 }
 
 variable "subnet_id" {
-  type        = "string"
+  type        = string
   description = "Must be set to the `worker_subnet_id` output by cluster"
 }
 
 variable "security_group_id" {
-  type        = "string"
+  type        = string
   description = "Must be set to the `worker_security_group_id` output by cluster"
 }
 
 variable "backend_address_pool_id" {
-  type        = "string"
+  type        = string
   description = "Must be set to the `worker_backend_address_pool_id` output by cluster"
 }
 
 # instances
 
 variable "worker_count" {
-  type        = "string"
+  type        = string
   default     = "1"
   description = "Number of instances"
 }
 
 variable "vm_type" {
-  type        = "string"
+  type        = string
   default     = "Standard_F1"
   description = "Machine type for instances (see `az vm list-skus --location centralus`)"
 }
 
 variable "os_image" {
-  type        = "string"
+  type        = string
   default     = "coreos-stable"
   description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha)"
 }
 
 variable "priority" {
-  type        = "string"
+  type        = string
   default     = "Regular"
   description = "Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."
 }
 
 variable "clc_snippets" {
-  type        = "list"
+  type        = list(string)
   description = "Container Linux Config snippets"
   default     = []
 }
@@ -65,12 +65,12 @@ variable "clc_snippets" {
 # configuration
 
 variable "kubeconfig" {
-  type        = "string"
+  type        = string
   description = "Must be set to `kubeconfig` output by cluster"
 }
 
 variable "ssh_authorized_key" {
-  type        = "string"
+  type        = string
   description = "SSH public key for user 'core'"
 }
 
@@ -80,12 +80,14 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD
 
-  type    = "string"
+
+  type = string
   default = "10.3.0.0/16"
 }
 
 variable "cluster_domain_suffix" {
   description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
+  type = string
-  default     = "cluster.local"
+  default = "cluster.local"
 }
+

azure/container-linux/kubernetes/workers/versions.tf

@@ -0,0 +1,4 @@
+
+terraform {
+  required_version = ">= 0.12"
+}

azure/container-linux/kubernetes/workers/workers.tf

@@ -1,28 +1,28 @@
 locals {
   # Channel for a Container Linux derivative
   # coreos-stable -> Container Linux Stable
-  channel = "${element(split("-", var.os_image), 1)}"
+  channel = element(split("-", var.os_image), 1)
 }
 
 # Workers scale set
 resource "azurerm_virtual_machine_scale_set" "workers" {
-  resource_group_name = "${var.resource_group_name}"
+  resource_group_name = var.resource_group_name
 
   name                   = "${var.name}-workers"
-  location               = "${var.region}"
+  location               = var.region
   single_placement_group = false
 
   sku {
-    name     = "${var.vm_type}"
+    name     = var.vm_type
     tier     = "standard"
-    capacity = "${var.worker_count}"
+    capacity = var.worker_count
   }
 
   # boot
   storage_profile_image_reference {
     publisher = "CoreOS"
     offer     = "CoreOS"
-    sku       = "${local.channel}"
+    sku       = local.channel
     version   = "latest"
   }
 
@@ -37,7 +37,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
   os_profile {
     computer_name_prefix = "${var.name}-worker-"
     admin_username       = "core"
-    custom_data          = "${data.ct_config.worker-ignition.rendered}"
+    custom_data          = data.ct_config.worker-ignition.rendered
   }
 
   # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
@@ -46,7 +46,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
 
     ssh_keys {
       path     = "/home/core/.ssh/authorized_keys"
-      key_data = "${var.ssh_authorized_key}"
+      key_data = var.ssh_authorized_key
     }
   }
 
@@ -54,61 +54,63 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
   network_profile {
     name                      = "nic0"
     primary                   = true
-    network_security_group_id = "${var.security_group_id}"
+    network_security_group_id = var.security_group_id
 
     ip_configuration {
       name      = "ip0"
       primary   = true
-      subnet_id = "${var.subnet_id}"
+      subnet_id = var.subnet_id
 
       # backend address pool to which the NIC should be added
-      load_balancer_backend_address_pool_ids = ["${var.backend_address_pool_id}"]
+      load_balancer_backend_address_pool_ids = [var.backend_address_pool_id]
     }
   }
 
   # lifecycle
   upgrade_policy_mode = "Manual"
-  priority            = "${var.priority}"
+  # eviction policy may only be set when priority is Low
-  eviction_policy     = "Delete"
+  priority            = var.priority
+  eviction_policy     = var.priority == "Low" ? "Delete" : null
 }
 
 # Scale up or down to maintain desired number, tolerating deallocations.
 resource "azurerm_monitor_autoscale_setting" "workers" {
-  resource_group_name = "${var.resource_group_name}"
+  resource_group_name = var.resource_group_name
 
   name     = "${var.name}-maintain-desired"
-  location = "${var.region}"
+  location = var.region
 
   # autoscale
   enabled            = true
-  target_resource_id = "${azurerm_virtual_machine_scale_set.workers.id}"
+  target_resource_id = azurerm_virtual_machine_scale_set.workers.id
 
   profile {
     name = "default"
 
     capacity {
-      minimum = "${var.worker_count}"
+      minimum = var.worker_count
-      default = "${var.worker_count}"
+      default = var.worker_count
-      maximum = "${var.worker_count}"
+      maximum = var.worker_count
     }
   }
 }
 
 # Worker Ignition configs
 data "ct_config" "worker-ignition" {
-  content      = "${data.template_file.worker-config.rendered}"
+  content      = data.template_file.worker-config.rendered
   pretty_print = false
-  snippets     = ["${var.clc_snippets}"]
+  snippets     = var.clc_snippets
 }
 
 # Worker Container Linux configs
 data "template_file" "worker-config" {
-  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
+  template = file("${path.module}/cl/worker.yaml.tmpl")
 
   vars = {
-    kubeconfig             = "${indent(10, var.kubeconfig)}"
+    kubeconfig             = indent(10, var.kubeconfig)
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    ssh_authorized_key     = var.ssh_authorized_key
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    cluster_domain_suffix  = var.cluster_domain_suffix
   }
 }
+

bare-metal/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.14.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.15.0 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization

bare-metal/container-linux/kubernetes/bootkube.tf

@@ -1,17 +1,18 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=082921d67905417755609eebda7d39a7e26f7fdb"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=62df9ad69cc0da35f47d40fa981370c4503ad581"
 
-  cluster_name                    = "${var.cluster_name}"
+  cluster_name                    = var.cluster_name
-  api_servers                     = ["${var.k8s_domain_name}"]
+  api_servers                     = [var.k8s_domain_name]
-  etcd_servers                    = ["${var.controller_domains}"]
+  etcd_servers                    = var.controller_domains
-  asset_dir                       = "${var.asset_dir}"
+  asset_dir                       = var.asset_dir
-  networking                      = "${var.networking}"
+  networking                      = var.networking
-  network_mtu                     = "${var.network_mtu}"
+  network_mtu                     = var.network_mtu
-  network_ip_autodetection_method = "${var.network_ip_autodetection_method}"
+  network_ip_autodetection_method = var.network_ip_autodetection_method
-  pod_cidr                        = "${var.pod_cidr}"
+  pod_cidr                        = var.pod_cidr
-  service_cidr                    = "${var.service_cidr}"
+  service_cidr                    = var.service_cidr
-  cluster_domain_suffix           = "${var.cluster_domain_suffix}"
+  cluster_domain_suffix           = var.cluster_domain_suffix
-  enable_reporting                = "${var.enable_reporting}"
+  enable_reporting                = var.enable_reporting
-  enable_aggregation              = "${var.enable_aggregation}"
+  enable_aggregation              = var.enable_aggregation
 }
+

bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -75,6 +75,7 @@ systemd:
           --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
           --mount volume=iscsiadm,target=/sbin/iscsiadm \
           --insecure-options=image"
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
@@ -89,6 +90,7 @@ systemd:
           --anonymous-auth=false \
           --authentication-token-webhook \
           --authorization-mode=Webhook \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -128,7 +130,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.14.3
+          KUBELET_IMAGE_TAG=v1.15.0
     - path: /etc/hostname
       filesystem: root
       mode: 0644
@@ -160,7 +162,7 @@ storage:
             --mount volume=assets,target=/assets \
             --volume bootstrap,kind=host,source=/etc/kubernetes \
             --mount volume=bootstrap,target=/etc/kubernetes \
-            $$RKT_OPTS \
+            $${RKT_OPTS} \
             quay.io/coreos/bootkube:v0.14.0 \
             --net=host \
             --dns=host \

bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -50,6 +50,7 @@ systemd:
           --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
           --mount volume=iscsiadm,target=/sbin/iscsiadm \
           --insecure-options=image"
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
@@ -62,6 +63,7 @@ systemd:
           --anonymous-auth=false \
           --authentication-token-webhook \
           --authorization-mode=Webhook \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -89,7 +91,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.14.3
+          KUBELET_IMAGE_TAG=v1.15.0
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/groups.tf

@@ -1,33 +1,35 @@
 resource "matchbox_group" "install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)
 
-  name = "${format("install-%s", element(concat(var.controller_names, var.worker_names), count.index))}"
+  name = format("install-%s", element(concat(var.controller_names, var.worker_names), count.index))
 
-  profile = "${local.flavor == "flatcar" ? var.cached_install == "true" ? element(matchbox_profile.cached-flatcar-linux-install.*.name, count.index) : element(matchbox_profile.flatcar-install.*.name, count.index) : var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)}"
+  # pick one of 4 Matchbox profiles (Container Linux or Flatcar, cached or non-cached)
+  profile = local.flavor == "flatcar" ? var.cached_install == "true" ? element(matchbox_profile.cached-flatcar-linux-install.*.name, count.index) : element(matchbox_profile.flatcar-install.*.name, count.index) : var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)
 
   selector = {
-    mac = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"
+    mac = element(concat(var.controller_macs, var.worker_macs), count.index)
   }
 }
 
 resource "matchbox_group" "controller" {
-  count   = "${length(var.controller_names)}"
+  count   = length(var.controller_names)
-  name    = "${format("%s-%s", var.cluster_name, element(var.controller_names, count.index))}"
+  name    = format("%s-%s", var.cluster_name, element(var.controller_names, count.index))
-  profile = "${element(matchbox_profile.controllers.*.name, count.index)}"
+  profile = element(matchbox_profile.controllers.*.name, count.index)
 
   selector = {
-    mac = "${element(var.controller_macs, count.index)}"
+    mac = element(var.controller_macs, count.index)
     os  = "installed"
   }
 }
 
 resource "matchbox_group" "worker" {
-  count   = "${length(var.worker_names)}"
+  count   = length(var.worker_names)
-  name    = "${format("%s-%s", var.cluster_name, element(var.worker_names, count.index))}"
+  name    = format("%s-%s", var.cluster_name, element(var.worker_names, count.index))
-  profile = "${element(matchbox_profile.workers.*.name, count.index)}"
+  profile = element(matchbox_profile.workers.*.name, count.index)
 
   selector = {
-    mac = "${element(var.worker_macs, count.index)}"
+    mac = element(var.worker_macs, count.index)
     os  = "installed"
   }
 }
+

bare-metal/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,4 @@
 output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
+  value = module.bootkube.kubeconfig-admin
 }
+

bare-metal/container-linux/kubernetes/profiles.tf

@@ -1,15 +1,15 @@
 locals {
   # coreos-stable -> coreos flavor, stable channel
   # flatcar-stable -> flatcar flavor, stable channel
-  flavor = "${element(split("-", var.os_channel), 0)}"
+  flavor = element(split("-", var.os_channel), 0)
 
-  channel = "${element(split("-", var.os_channel), 1)}"
+  channel = element(split("-", var.os_channel), 1)
 }
 
 // Container Linux Install profile (from release.core-os.net)
 resource "matchbox_profile" "container-linux-install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)
-  name  = "${format("%s-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+  name  = format("%s-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))
 
   kernel = "${var.download_protocol}://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe.vmlinuz"
 
@@ -17,32 +17,31 @@ resource "matchbox_profile" "container-linux-install" {
     "${var.download_protocol}://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe_image.cpio.gz",
   ]
 
-  args = [
+  args = flatten([
     "initrd=coreos_production_pxe_image.cpio.gz",
     "coreos.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
     "coreos.first_boot=yes",
     "console=tty0",
     "console=ttyS0",
-    "${var.kernel_args}",
+    var.kernel_args,
-  ]
+  ])
 
-  container_linux_config = "${element(data.template_file.container-linux-install-configs.*.rendered, count.index)}"
+  container_linux_config = element(data.template_file.container-linux-install-configs.*.rendered, count.index)
 }
 
 data "template_file" "container-linux-install-configs" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)
 
-  template = "${file("${path.module}/cl/install.yaml.tmpl")}"
+  template = file("${path.module}/cl/install.yaml.tmpl")
 
   vars = {
-    os_flavor           = "${local.flavor}"
+    os_flavor           = local.flavor
-    os_channel          = "${local.channel}"
+    os_channel          = local.channel
-    os_version          = "${var.os_version}"
+    os_version          = var.os_version
-    ignition_endpoint   = "${format("%s/ignition", var.matchbox_http_endpoint)}"
+    ignition_endpoint   = format("%s/ignition", var.matchbox_http_endpoint)
-    install_disk        = "${var.install_disk}"
+    install_disk        = var.install_disk
-    container_linux_oem = "${var.container_linux_oem}"
+    container_linux_oem = var.container_linux_oem
-    ssh_authorized_key  = "${var.ssh_authorized_key}"
+    ssh_authorized_key  = var.ssh_authorized_key
-
     # only cached-container-linux profile adds -b baseurl
     baseurl_flag = ""
   }
@@ -51,8 +50,8 @@ data "template_file" "container-linux-install-configs" {
 // Container Linux Install profile (from matchbox /assets cache)
 // Note: Admin must have downloaded os_version into matchbox assets/coreos.
 resource "matchbox_profile" "cached-container-linux-install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)
-  name  = "${format("%s-cached-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+  name  = format("%s-cached-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))
 
   kernel = "/assets/coreos/${var.os_version}/coreos_production_pxe.vmlinuz"
 
@@ -60,32 +59,31 @@ resource "matchbox_profile" "cached-container-linux-install" {
     "/assets/coreos/${var.os_version}/coreos_production_pxe_image.cpio.gz",
   ]
 
-  args = [
+  args = flatten([
     "initrd=coreos_production_pxe_image.cpio.gz",
     "coreos.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
     "coreos.first_boot=yes",
     "console=tty0",
     "console=ttyS0",
-    "${var.kernel_args}",
+    var.kernel_args,
-  ]
+  ])
 
-  container_linux_config = "${element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)}"
+  container_linux_config = element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)
 }
 
 data "template_file" "cached-container-linux-install-configs" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)
 
-  template = "${file("${path.module}/cl/install.yaml.tmpl")}"
+  template = file("${path.module}/cl/install.yaml.tmpl")
 
   vars = {
-    os_flavor           = "${local.flavor}"
+    os_flavor           = local.flavor
-    os_channel          = "${local.channel}"
+    os_channel          = local.channel
-    os_version          = "${var.os_version}"
+    os_version          = var.os_version
-    ignition_endpoint   = "${format("%s/ignition", var.matchbox_http_endpoint)}"
+    ignition_endpoint   = format("%s/ignition", var.matchbox_http_endpoint)
-    install_disk        = "${var.install_disk}"
+    install_disk        = var.install_disk
-    container_linux_oem = "${var.container_linux_oem}"
+    container_linux_oem = var.container_linux_oem
-    ssh_authorized_key  = "${var.ssh_authorized_key}"
+    ssh_authorized_key  = var.ssh_authorized_key
-
     # profile uses -b baseurl to install from matchbox cache
     baseurl_flag = "-b ${var.matchbox_http_endpoint}/assets/${local.flavor}"
   }
@@ -93,8 +91,8 @@ data "template_file" "cached-container-linux-install-configs" {
 
 // Flatcar Linux install profile (from release.flatcar-linux.net)
 resource "matchbox_profile" "flatcar-install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)
-  name  = "${format("%s-flatcar-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+  name  = format("%s-flatcar-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))
 
   kernel = "${var.download_protocol}://${local.channel}.release.flatcar-linux.net/amd64-usr/${var.os_version}/flatcar_production_pxe.vmlinuz"
 
@@ -102,23 +100,23 @@ resource "matchbox_profile" "flatcar-install" {
     "${var.download_protocol}://${local.channel}.release.flatcar-linux.net/amd64-usr/${var.os_version}/flatcar_production_pxe_image.cpio.gz",
   ]
 
-  args = [
+  args = flatten([
     "initrd=flatcar_production_pxe_image.cpio.gz",
     "flatcar.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
     "flatcar.first_boot=yes",
     "console=tty0",
     "console=ttyS0",
-    "${var.kernel_args}",
+    var.kernel_args,
-  ]
+  ])
 
-  container_linux_config = "${element(data.template_file.container-linux-install-configs.*.rendered, count.index)}"
+  container_linux_config = element(data.template_file.container-linux-install-configs.*.rendered, count.index)
 }
 
 // Flatcar Linux Install profile (from matchbox /assets cache)
 // Note: Admin must have downloaded os_version into matchbox assets/flatcar.
 resource "matchbox_profile" "cached-flatcar-linux-install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)
-  name  = "${format("%s-cached-flatcar-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+  name  = format("%s-cached-flatcar-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))
 
   kernel = "/assets/flatcar/${var.os_version}/flatcar_production_pxe.vmlinuz"
 
@@ -126,90 +124,93 @@ resource "matchbox_profile" "cached-flatcar-linux-install" {
     "/assets/flatcar/${var.os_version}/flatcar_production_pxe_image.cpio.gz",
   ]
 
-  args = [
+  args = flatten([
     "initrd=flatcar_production_pxe_image.cpio.gz",
     "flatcar.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
     "flatcar.first_boot=yes",
     "console=tty0",
     "console=ttyS0",
-    "${var.kernel_args}",
+    var.kernel_args,
-  ]
+  ])
 
-  container_linux_config = "${element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)}"
+  container_linux_config = element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)
 }
 
 // Kubernetes Controller profiles
 resource "matchbox_profile" "controllers" {
-  count        = "${length(var.controller_names)}"
+  count        = length(var.controller_names)
-  name         = "${format("%s-controller-%s", var.cluster_name, element(var.controller_names, count.index))}"
+  name         = format("%s-controller-%s", var.cluster_name, element(var.controller_names, count.index))
-  raw_ignition = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
+  raw_ignition = element(data.ct_config.controller-ignitions.*.rendered, count.index)
 }
 
 data "ct_config" "controller-ignitions" {
-  count        = "${length(var.controller_names)}"
+  count        = length(var.controller_names)
-  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  content      = element(data.template_file.controller-configs.*.rendered, count.index)
   pretty_print = false
-
+  snippets     = local.clc_map[element(var.controller_names, count.index)]
-  # Must use direct lookup. Cannot use lookup(map, key) since it only works for flat maps
-  snippets = ["${local.clc_map[element(var.controller_names, count.index)]}"]
 }
 
 data "template_file" "controller-configs" {
-  count = "${length(var.controller_names)}"
+  count = length(var.controller_names)
 
-  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+  template = file("${path.module}/cl/controller.yaml.tmpl")
 
   vars = {
-    domain_name            = "${element(var.controller_domains, count.index)}"
+    domain_name            = element(var.controller_domains, count.index)
-    etcd_name              = "${element(var.controller_names, count.index)}"
+    etcd_name              = element(var.controller_names, count.index)
-    etcd_initial_cluster   = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
+    etcd_initial_cluster   = join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))
-    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cgroup_driver = var.os_channel == "flatcar-edge" ? "systemd" : "cgroupfs"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    cluster_dns_service_ip = module.bootkube.cluster_dns_service_ip
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_domain_suffix  = var.cluster_domain_suffix
+    ssh_authorized_key     = var.ssh_authorized_key
   }
 }
 
 // Kubernetes Worker profiles
 resource "matchbox_profile" "workers" {
-  count        = "${length(var.worker_names)}"
+  count        = length(var.worker_names)
-  name         = "${format("%s-worker-%s", var.cluster_name, element(var.worker_names, count.index))}"
+  name         = format("%s-worker-%s", var.cluster_name, element(var.worker_names, count.index))
-  raw_ignition = "${element(data.ct_config.worker-ignitions.*.rendered, count.index)}"
+  raw_ignition = element(data.ct_config.worker-ignitions.*.rendered, count.index)
 }
 
 data "ct_config" "worker-ignitions" {
-  count        = "${length(var.worker_names)}"
+  count        = length(var.worker_names)
-  content      = "${element(data.template_file.worker-configs.*.rendered, count.index)}"
+  content      = element(data.template_file.worker-configs.*.rendered, count.index)
   pretty_print = false
-
+  snippets     = local.clc_map[element(var.worker_names, count.index)]
-  # Must use direct lookup. Cannot use lookup(map, key) since it only works for flat maps
-  snippets = ["${local.clc_map[element(var.worker_names, count.index)]}"]
 }
 
 data "template_file" "worker-configs" {
-  count = "${length(var.worker_names)}"
+  count = length(var.worker_names)
 
-  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
+  template = file("${path.module}/cl/worker.yaml.tmpl")
 
   vars = {
-    domain_name            = "${element(var.worker_domains, count.index)}"
+    domain_name            = element(var.worker_domains, count.index)
-    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cgroup_driver = var.os_channel == "flatcar-edge" ? "systemd" : "cgroupfs"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    cluster_dns_service_ip = module.bootkube.cluster_dns_service_ip
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_domain_suffix  = var.cluster_domain_suffix
+    ssh_authorized_key     = var.ssh_authorized_key
   }
 }
 
 locals {
   # Hack to workaround https://github.com/hashicorp/terraform/issues/17251
+  # Still an issue in Terraform v0.12 https://github.com/hashicorp/terraform/issues/20572
   # Default Container Linux config snippets map every node names to list("\n") so
   # all lookups succeed
-  clc_defaults = "${zipmap(concat(var.controller_names, var.worker_names), chunklist(data.template_file.clc-default-snippets.*.rendered, 1))}"
+  clc_defaults = zipmap(
+    concat(var.controller_names, var.worker_names),
+    chunklist(data.template_file.clc-default-snippets.*.rendered, 1),
+  )
 
   # Union of the default and user specific snippets, later overrides prior.
-  clc_map = "${merge(local.clc_defaults, var.clc_snippets)}"
+  clc_map = merge(local.clc_defaults, var.clc_snippets)
 }
 
 // Horrible hack to generate a Terraform list of node count length
 data "template_file" "clc-default-snippets" {
-  count    = "${length(var.controller_names) + length(var.worker_names)}"
+  count    = length(var.controller_names) + length(var.worker_names)
   template = "\n"
 }
+

bare-metal/container-linux/kubernetes/require.tf

@@ -1,21 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}

bare-metal/container-linux/kubernetes/ssh.tf

@@ -1,59 +1,59 @@
 # Secure copy etcd TLS assets and kubeconfig to controllers. Activates kubelet.service
 resource "null_resource" "copy-controller-secrets" {
-  count = "${length(var.controller_names)}"
+  count = length(var.controller_names)
 
   # Without depends_on, remote-exec could start and wait for machines before
   # matchbox groups are written, causing a deadlock.
   depends_on = [
-    "matchbox_group.install",
+    matchbox_group.install,
-    "matchbox_group.controller",
+    matchbox_group.controller,
-    "matchbox_group.worker",
+    matchbox_group.worker,
   ]
 
   connection {
     type    = "ssh"
-    host    = "${element(var.controller_domains, count.index)}"
+    host    = element(var.controller_domains, count.index)
     user    = "core"
     timeout = "60m"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig-kubelet}"
+    content     = module.bootkube.kubeconfig-kubelet
     destination = "$HOME/kubeconfig"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
+    content     = module.bootkube.etcd_ca_cert
     destination = "$HOME/etcd-client-ca.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
+    content     = module.bootkube.etcd_client_cert
     destination = "$HOME/etcd-client.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
+    content     = module.bootkube.etcd_client_key
     destination = "$HOME/etcd-client.key"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
+    content     = module.bootkube.etcd_server_cert
     destination = "$HOME/etcd-server.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
+    content     = module.bootkube.etcd_server_key
     destination = "$HOME/etcd-server.key"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
+    content     = module.bootkube.etcd_peer_cert
     destination = "$HOME/etcd-peer.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
+    content     = module.bootkube.etcd_peer_key
     destination = "$HOME/etcd-peer.key"
   }
 
@@ -76,25 +76,25 @@ resource "null_resource" "copy-controller-secrets" {
 
 # Secure copy kubeconfig to all workers. Activates kubelet.service
 resource "null_resource" "copy-worker-secrets" {
-  count = "${length(var.worker_names)}"
+  count = length(var.worker_names)
 
   # Without depends_on, remote-exec could start and wait for machines before
   # matchbox groups are written, causing a deadlock.
   depends_on = [
-    "matchbox_group.install",
+    matchbox_group.install,
-    "matchbox_group.controller",
+    matchbox_group.controller,
-    "matchbox_group.worker",
+    matchbox_group.worker,
   ]
 
   connection {
     type    = "ssh"
-    host    = "${element(var.worker_domains, count.index)}"
+    host    = element(var.worker_domains, count.index)
     user    = "core"
     timeout = "60m"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig-kubelet}"
+    content     = module.bootkube.kubeconfig-kubelet
     destination = "$HOME/kubeconfig"
   }
 
@@ -112,19 +112,19 @@ resource "null_resource" "bootkube-start" {
   # Terraform only does one task at a time, so it would try to bootstrap
   # while no Kubelets are running.
   depends_on = [
-    "null_resource.copy-controller-secrets",
+    null_resource.copy-controller-secrets,
-    "null_resource.copy-worker-secrets",
+    null_resource.copy-worker-secrets,
   ]
 
   connection {
     type    = "ssh"
-    host    = "${element(var.controller_domains, 0)}"
+    host    = element(var.controller_domains, 0)
     user    = "core"
     timeout = "15m"
   }
 
   provisioner "file" {
-    source      = "${var.asset_dir}"
+    source      = var.asset_dir
     destination = "$HOME/assets"
   }
 
@@ -135,3 +135,4 @@ resource "null_resource" "bootkube-start" {
     ]
   }
 }
+

bare-metal/container-linux/kubernetes/variables.tf

@@ -1,60 +1,60 @@
 variable "cluster_name" {
-  type        = "string"
+  type        = string
   description = "Unique cluster name"
 }
 
 # bare-metal
 
 variable "matchbox_http_endpoint" {
-  type        = "string"
+  type        = string
   description = "Matchbox HTTP read-only endpoint (e.g. http://matchbox.example.com:8080)"
 }
 
 variable "os_channel" {
-  type        = "string"
+  type        = string
-  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
+  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
 }
 
 variable "os_version" {
-  type        = "string"
+  type        = string
-  description = "Version for a Container Linux derivative to PXE and install (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
+  description = "Version for a Container Linux derivative to PXE and install (e.g. 2079.5.1)"
 }
 
 # machines
 # Terraform's crude "type system" does not properly support lists of maps so we do this.
 
 variable "controller_names" {
-  type        = "list"
+  type        = list(string)
   description = "Ordered list of controller names (e.g. [node1])"
 }
 
 variable "controller_macs" {
-  type        = "list"
+  type        = list(string)
   description = "Ordered list of controller identifying MAC addresses (e.g. [52:54:00:a1:9c:ae])"
 }
 
 variable "controller_domains" {
-  type        = "list"
+  type        = list(string)
   description = "Ordered list of controller FQDNs (e.g. [node1.example.com])"
 }
 
 variable "worker_names" {
-  type        = "list"
+  type        = list(string)
   description = "Ordered list of worker names (e.g. [node2, node3])"
 }
 
 variable "worker_macs" {
-  type        = "list"
+  type        = list(string)
   description = "Ordered list of worker identifying MAC addresses (e.g. [52:54:00:b2:2f:86, 52:54:00:c3:61:77])"
 }
 
 variable "worker_domains" {
-  type        = "list"
+  type        = list(string)
   description = "Ordered list of worker FQDNs (e.g. [node2.example.com, node3.example.com])"
 }
 
 variable "clc_snippets" {
-  type        = "map"
+  type        = map(list(string))
   description = "Map from machine names to lists of Container Linux Config snippets"
   default     = {}
 }
@@ -63,40 +63,40 @@ variable "clc_snippets" {
 
 variable "k8s_domain_name" {
   description = "Controller DNS name which resolves to a controller instance. Workers and kubeconfig's will communicate with this endpoint (e.g. cluster.example.com)"
-  type        = "string"
+  type        = string
 }
 
 variable "ssh_authorized_key" {
-  type        = "string"
+  type        = string
   description = "SSH public key for user 'core'"
 }
 
 variable "asset_dir" {
   description = "Path to a directory where generated assets should be placed (contains secrets)"
-  type        = "string"
+  type        = string
 }
 
 variable "networking" {
   description = "Choice of networking provider (flannel or calico)"
-  type        = "string"
+  type        = string
   default     = "calico"
 }
 
 variable "network_mtu" {
   description = "CNI interface MTU (applies to calico only)"
-  type        = "string"
+  type        = string
   default     = "1480"
 }
 
 variable "network_ip_autodetection_method" {
   description = "Method to autodetect the host IPv4 address (applies to calico only)"
-  type        = "string"
+  type        = string
   default     = "first-found"
 }
 
 variable "pod_cidr" {
   description = "CIDR IPv4 range to assign Kubernetes pods"
-  type        = "string"
+  type        = string
   default     = "10.2.0.0/16"
 }
 
@@ -106,7 +106,8 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD
 
-  type    = "string"
+
+  type = string
   default = "10.3.0.0/16"
 }
 
@@ -114,48 +115,49 @@ EOD
 
 variable "cluster_domain_suffix" {
   description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
+  type = string
-  default     = "cluster.local"
+  default = "cluster.local"
 }
 
 variable "download_protocol" {
-  type        = "string"
+  type = string
-  default     = "https"
+  default = "https"
   description = "Protocol iPXE should use to download the kernel and initrd. Defaults to https, which requires iPXE compiled with crypto support. Unused if cached_install is true."
 }
 
 variable "cached_install" {
-  type        = "string"
+  type = string
-  default     = "false"
+  default = "false"
   description = "Whether Container Linux should PXE boot and install from matchbox /assets cache. Note that the admin must have downloaded the os_version into matchbox assets."
 }
 
 variable "install_disk" {
-  type        = "string"
+  type = string
-  default     = "/dev/sda"
+  default = "/dev/sda"
   description = "Disk device to which the install profiles should install Container Linux (e.g. /dev/sda)"
 }
 
 variable "container_linux_oem" {
-  type        = "string"
+  type = string
-  default     = ""
+  default = ""
   description = "DEPRECATED: Specify an OEM image id to use as base for the installation (e.g. ami, vmware_raw, xen) or leave blank for the default image"
 }
 
 variable "kernel_args" {
   description = "Additional kernel arguments to provide at PXE boot."
-  type        = "list"
+  type = list(string)
-  default     = []
+  default = []
 }
 
 variable "enable_reporting" {
-  type        = "string"
+  type = string
   description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = "false"
+  default = "false"
 }
 
 variable "enable_aggregation" {
   description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  type        = "string"
+  type = string
-  default     = "false"
+  default = "false"
 }
+

bare-metal/container-linux/kubernetes/versions.tf

@@ -0,0 +1,12 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = "~> 0.12.0"
+  required_providers {
+    matchbox = "~> 0.3.0"
+    ct       = "~> 0.3.2"
+    template = "~> 2.1"
+    null     = "~> 2.1"
+  }
+}
+

bare-metal/fedora-atomic/kubernetes/LICENSE

@@ -1,23 +0,0 @@
-The MIT License (MIT)
-
-Copyright (c) 2017 Typhoon Authors
-Copyright (c) 2017 Dalton Hubble
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
-

bare-metal/fedora-atomic/kubernetes/README.md

@@ -1,22 +0,0 @@
-# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
-
-Typhoon is a minimal and free Kubernetes distribution.
-
-* Minimal, stable base Kubernetes distribution
-* Declarative infrastructure and configuration
-* Free (freedom and cost) and privacy-respecting
-* Practical for labs, datacenters, and clouds
-
-Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
-
-## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
-
-* Kubernetes v1.14.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
-
-## Docs
-
-Please see the [official docs](https://typhoon.psdn.io) and the bare-metal [tutorial](https://typhoon.psdn.io/cl/bare-metal/).
-

bare-metal/fedora-atomic/kubernetes/bootkube.tf

@@ -1,18 +0,0 @@
-# Self-hosted Kubernetes assets (kubeconfig, manifests)
-module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=082921d67905417755609eebda7d39a7e26f7fdb"
-
-  cluster_name          = "${var.cluster_name}"
-  api_servers           = ["${var.k8s_domain_name}"]
-  etcd_servers          = ["${var.controller_domains}"]
-  asset_dir             = "${var.asset_dir}"
-  networking            = "${var.networking}"
-  network_mtu           = "${var.network_mtu}"
-  pod_cidr              = "${var.pod_cidr}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
-  enable_reporting      = "${var.enable_reporting}"
-
-  # Fedora
-  trusted_certs_dir = "/etc/pki/tls/certs"
-}

bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -1,100 +0,0 @@
-#cloud-config
-write_files:
-  - path: /etc/etcd/etcd.conf
-    content: |
-      ETCD_NAME=${etcd_name}
-      ETCD_DATA_DIR=/var/lib/etcd
-      ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379
-      ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380
-      ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
-      ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
-      ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
-      ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
-      ETCD_STRICT_RECONFIG_CHECK=true
-      ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
-      ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
-      ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
-      ETCD_CLIENT_CERT_AUTH=true
-      ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
-      ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
-      ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
-      ETCD_PEER_CLIENT_CERT_AUTH=true
-  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
-    content: |
-      [Unit]
-      Wants=rpc-statd.service
-      [Service]
-      ExecStartPre=/bin/mkdir -p /opt/cni/bin
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
-      ExecStartPre=/bin/mkdir -p /var/lib/cni
-      ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-      ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-      Restart=always
-      RestartSec=10
-  - path: /etc/kubernetes/kubelet.conf
-    content: |
-      ARGS="--anonymous-auth=false \
-        --authentication-token-webhook \
-        --authorization-mode=Webhook \
-        --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${cluster_dns_service_ip} \
-        --cluster_domain=${cluster_domain_suffix} \
-        --cni-conf-dir=/etc/kubernetes/cni/net.d \
-        --exit-on-lock-contention \
-        --hostname-override=${domain_name} \
-        --kubeconfig=/etc/kubernetes/kubeconfig \
-        --lock-file=/var/run/lock/kubelet.lock \
-        --network-plugin=cni \
-        --node-labels=node-role.kubernetes.io/master \
-        --node-labels=node-role.kubernetes.io/controller="true" \
-        --pod-manifest-path=/etc/kubernetes/manifests \
-        --read-only-port=0 \
-        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
-        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
-  - path: /etc/systemd/system/kubelet.path
-    content: |
-      [Unit]
-      Description=Watch for kubeconfig
-      [Path]
-      PathExists=/etc/kubernetes/kubeconfig
-      [Install]
-      WantedBy=multi-user.target
-  - path: /var/lib/bootkube/.keep
-  - path: /etc/NetworkManager/conf.d/typhoon.conf
-    content: |
-      [main]
-      plugins=keyfile
-      [keyfile]
-      unmanaged-devices=interface-name:cali*;interface-name:tunl*
-  - path: /etc/selinux/config
-    owner: root:root
-    permissions: '0644'
-    content: |
-      SELINUX=permissive
-      SELINUXTYPE=targeted
-bootcmd:
-  - [setenforce, Permissive]
-  - [systemctl, disable, firewalld, --now]
-  # https://github.com/kubernetes/kubernetes/issues/60869
-  - [modprobe, ip_vs]
-runcmd:
-  - [systemctl, daemon-reload]
-  - [systemctl, restart, NetworkManager]
-  - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.14.1"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
-  - [systemctl, start, --no-block, etcd.service]
-  - [systemctl, enable, kubelet.path]
-  - [systemctl, start, --no-block, kubelet.path]
-users:
-  - default
-  - name: fedora
-    gecos: Fedora Admin
-    sudo: ALL=(ALL) NOPASSWD:ALL
-    groups: wheel,adm,systemd-journal,docker
-    ssh-authorized-keys:
-      - "${ssh_authorized_key}"

bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -1,73 +0,0 @@
-#cloud-config
-write_files:
-  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
-    content: |
-      [Unit]
-      Wants=rpc-statd.service
-      [Service]
-      ExecStartPre=/bin/mkdir -p /opt/cni/bin
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-      ExecStartPre=/bin/mkdir -p /var/lib/cni
-      ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-      ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-      Restart=always
-      RestartSec=10
-  - path: /etc/kubernetes/kubelet.conf
-    content: |
-      ARGS="--anonymous-auth=false \
-        --authentication-token-webhook \
-        --authorization-mode=Webhook \
-        --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${cluster_dns_service_ip} \
-        --cluster_domain=${cluster_domain_suffix} \
-        --cni-conf-dir=/etc/kubernetes/cni/net.d \
-        --exit-on-lock-contention \
-        --hostname-override=${domain_name} \
-        --kubeconfig=/etc/kubernetes/kubeconfig \
-        --lock-file=/var/run/lock/kubelet.lock \
-        --network-plugin=cni \
-        --node-labels=node-role.kubernetes.io/node \
-        --pod-manifest-path=/etc/kubernetes/manifests \
-        --read-only-port=0 \
-        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
-  - path: /etc/systemd/system/kubelet.path
-    content: |
-      [Unit]
-      Description=Watch for kubeconfig
-      [Path]
-      PathExists=/etc/kubernetes/kubeconfig
-      [Install]
-      WantedBy=multi-user.target
-  - path: /etc/NetworkManager/conf.d/typhoon.conf
-    content: |
-      [main]
-      plugins=keyfile
-      [keyfile]
-      unmanaged-devices=interface-name:cali*;interface-name:tunl*
-  - path: /etc/selinux/config
-    owner: root:root
-    permissions: '0644'
-    content: |
-      SELINUX=permissive
-      SELINUXTYPE=targeted
-bootcmd:
-  - [setenforce, Permissive]
-  - [systemctl, disable, firewalld, --now]
-  # https://github.com/kubernetes/kubernetes/issues/60869
-  - [modprobe, ip_vs]
-runcmd:
-  - [systemctl, daemon-reload]
-  - [systemctl, restart, NetworkManager]
-  - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.14.1"
-  - [systemctl, enable, kubelet.path]
-  - [systemctl, start, --no-block, kubelet.path]
-users:
-  - default
-  - name: fedora
-    gecos: Fedora Admin
-    sudo: ALL=(ALL) NOPASSWD:ALL
-    groups: wheel,adm,systemd-journal,docker
-    ssh-authorized-keys:
-      - "${ssh_authorized_key}"

bare-metal/fedora-atomic/kubernetes/groups.tf

@@ -1,37 +0,0 @@
-// Install Fedora to disk
-resource "matchbox_group" "install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
-
-  name    = "${format("fedora-install-%s", element(concat(var.controller_names, var.worker_names), count.index))}"
-  profile = "${element(matchbox_profile.cached-fedora-install.*.name, count.index)}"
-
-  selector = {
-    mac = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"
-  }
-
-  metadata = {
-    ssh_authorized_key = "${var.ssh_authorized_key}"
-  }
-}
-
-resource "matchbox_group" "controller" {
-  count   = "${length(var.controller_names)}"
-  name    = "${format("%s-%s", var.cluster_name, element(var.controller_names, count.index))}"
-  profile = "${element(matchbox_profile.controllers.*.name, count.index)}"
-
-  selector = {
-    mac = "${element(var.controller_macs, count.index)}"
-    os  = "installed"
-  }
-}
-
-resource "matchbox_group" "worker" {
-  count   = "${length(var.worker_names)}"
-  name    = "${format("%s-%s", var.cluster_name, element(var.worker_names, count.index))}"
-  profile = "${element(matchbox_profile.workers.*.name, count.index)}"
-
-  selector = {
-    mac = "${element(var.worker_macs, count.index)}"
-    os  = "installed"
-  }
-}

bare-metal/fedora-atomic/kubernetes/kickstart/fedora-atomic.ks.tmpl

@@ -1,36 +0,0 @@
-# required
-lang en_US.UTF-8
-keyboard us
-timezone --utc Etc/UTC
-
-# wipe disks
-zerombr
-clearpart --all --initlabel
-
-# locked root and temporary user
-rootpw --lock --iscrypted locked
-user --name=none
-
-# config
-autopart --type=lvm --noswap
-network --bootproto=dhcp --device=link --activate --onboot=on
-bootloader --timeout=1 --append="ds=nocloud\;seedfrom=/var/cloud-init/"
-services --enabled=cloud-init,cloud-init-local,cloud-config,cloud-final
-
-ostreesetup --osname="fedora-atomic" --remote="fedora-atomic" --url="${atomic_assets_endpoint}/repo" --ref=fedora/28/x86_64/atomic-host --nogpg
-
-reboot
-
-%post --erroronfail
-mkdir /var/cloud-init
-curl --retry 10 "${matchbox_http_endpoint}/generic?mac=${mac}&os=installed" -o /var/cloud-init/user-data
-echo "instance-id: iid-local01" > /var/cloud-init/meta-data
-
-rm -f /etc/ostree/remotes.d/fedora-atomic.conf
-ostree remote add fedora-atomic https://dl.fedoraproject.org/atomic/repo/ --set=gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-28-primary
-
-# lock root user
-passwd -l root
-# remove temporary user
-userdel -r none
-%end

bare-metal/fedora-atomic/kubernetes/outputs.tf

@@ -1,3 +0,0 @@
-output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
-}

bare-metal/fedora-atomic/kubernetes/profiles.tf

@@ -1,87 +0,0 @@
-locals {
-  default_assets_endpoint = "${var.matchbox_http_endpoint}/assets/fedora/28"
-  atomic_assets_endpoint  = "${var.atomic_assets_endpoint != "" ? var.atomic_assets_endpoint : local.default_assets_endpoint}"
-}
-
-// Cached Fedora Install profile (from matchbox /assets cache)
-// Note: Admin must have downloaded Fedora kernel, initrd, and repo into
-// matchbox assets.
-resource "matchbox_profile" "cached-fedora-install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
-  name  = "${format("%s-cached-fedora-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
-
-  kernel = "${local.atomic_assets_endpoint}/images/pxeboot/vmlinuz"
-
-  initrd = [
-    "${local.atomic_assets_endpoint}/images/pxeboot/initrd.img",
-  ]
-
-  args = [
-    "initrd=initrd.img",
-    "inst.repo=${local.atomic_assets_endpoint}",
-    "inst.ks=${var.matchbox_http_endpoint}/generic?mac=${element(concat(var.controller_macs, var.worker_macs), count.index)}",
-    "inst.text",
-    "${var.kernel_args}",
-  ]
-
-  # kickstart
-  generic_config = "${element(data.template_file.install-kickstarts.*.rendered, count.index)}"
-}
-
-data "template_file" "install-kickstarts" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
-
-  template = "${file("${path.module}/kickstart/fedora-atomic.ks.tmpl")}"
-
-  vars = {
-    matchbox_http_endpoint = "${var.matchbox_http_endpoint}"
-    atomic_assets_endpoint = "${local.atomic_assets_endpoint}"
-    mac                    = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"
-  }
-}
-
-// Kubernetes Controller profiles
-resource "matchbox_profile" "controllers" {
-  count = "${length(var.controller_names)}"
-  name  = "${format("%s-controller-%s", var.cluster_name, element(var.controller_names, count.index))}"
-
-  # cloud-init
-  generic_config = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
-}
-
-data "template_file" "controller-configs" {
-  count = "${length(var.controller_names)}"
-
-  template = "${file("${path.module}/cloudinit/controller.yaml.tmpl")}"
-
-  vars = {
-    domain_name            = "${element(var.controller_domains, count.index)}"
-    etcd_name              = "${element(var.controller_names, count.index)}"
-    etcd_initial_cluster   = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
-    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-  }
-}
-
-// Kubernetes Worker profiles
-resource "matchbox_profile" "workers" {
-  count = "${length(var.worker_names)}"
-  name  = "${format("%s-worker-%s", var.cluster_name, element(var.worker_names, count.index))}"
-
-  # cloud-init
-  generic_config = "${element(data.template_file.worker-configs.*.rendered, count.index)}"
-}
-
-data "template_file" "worker-configs" {
-  count = "${length(var.worker_names)}"
-
-  template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
-
-  vars = {
-    domain_name            = "${element(var.worker_domains, count.index)}"
-    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-  }
-}

bare-metal/fedora-atomic/kubernetes/require.tf

@@ -1,21 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}

bare-metal/fedora-atomic/kubernetes/ssh.tf

@@ -1,136 +0,0 @@
-# Secure copy etcd TLS assets and kubeconfig to controllers. Activates kubelet.service
-resource "null_resource" "copy-controller-secrets" {
-  count = "${length(var.controller_names)}"
-
-  # Without depends_on, remote-exec could start and wait for machines before
-  # matchbox groups are written, causing a deadlock.
-  depends_on = [
-    "matchbox_group.install",
-    "matchbox_group.controller",
-    "matchbox_group.worker",
-  ]
-
-  connection {
-    type    = "ssh"
-    host    = "${element(var.controller_domains, count.index)}"
-    user    = "fedora"
-    timeout = "60m"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.kubeconfig-kubelet}"
-    destination = "$HOME/kubeconfig"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
-    destination = "$HOME/etcd-client-ca.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
-    destination = "$HOME/etcd-client.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
-    destination = "$HOME/etcd-client.key"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
-    destination = "$HOME/etcd-server.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
-    destination = "$HOME/etcd-server.key"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
-    destination = "$HOME/etcd-peer.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
-    destination = "$HOME/etcd-peer.key"
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      "sudo mkdir -p /etc/ssl/etcd/etcd",
-      "sudo mv etcd-client* /etc/ssl/etcd/",
-      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/server-ca.crt",
-      "sudo mv etcd-server.crt /etc/ssl/etcd/etcd/server.crt",
-      "sudo mv etcd-server.key /etc/ssl/etcd/etcd/server.key",
-      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/peer-ca.crt",
-      "sudo mv etcd-peer.crt /etc/ssl/etcd/etcd/peer.crt",
-      "sudo mv etcd-peer.key /etc/ssl/etcd/etcd/peer.key",
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
-    ]
-  }
-}
-
-# Secure copy kubeconfig to all workers. Activates kubelet.service
-resource "null_resource" "copy-worker-secrets" {
-  count = "${length(var.worker_names)}"
-
-  # Without depends_on, remote-exec could start and wait for machines before
-  # matchbox groups are written, causing a deadlock.
-  depends_on = [
-    "matchbox_group.install",
-    "matchbox_group.controller",
-    "matchbox_group.worker",
-  ]
-
-  connection {
-    type    = "ssh"
-    host    = "${element(var.worker_domains, count.index)}"
-    user    = "fedora"
-    timeout = "60m"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.kubeconfig-kubelet}"
-    destination = "$HOME/kubeconfig"
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
-    ]
-  }
-}
-
-# Secure copy bootkube assets to ONE controller and start bootkube to perform
-# one-time self-hosted cluster bootstrapping.
-resource "null_resource" "bootkube-start" {
-  # Without depends_on, this remote-exec may start before the kubeconfig copy.
-  # Terraform only does one task at a time, so it would try to bootstrap
-  # while no Kubelets are running.
-  depends_on = [
-    "null_resource.copy-controller-secrets",
-    "null_resource.copy-worker-secrets",
-  ]
-
-  connection {
-    type    = "ssh"
-    host    = "${element(var.controller_domains, 0)}"
-    user    = "fedora"
-    timeout = "15m"
-  }
-
-  provisioner "file" {
-    source      = "${var.asset_dir}"
-    destination = "$HOME/assets"
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do sleep 4; done",
-      "sudo mv $HOME/assets /var/lib/bootkube",
-      "sudo systemctl start bootkube",
-    ]
-  }
-}

bare-metal/fedora-atomic/kubernetes/variables.tf

@@ -1,118 +0,0 @@
-variable "cluster_name" {
-  type        = "string"
-  description = "Unique cluster name"
-}
-
-# bare-metal
-
-variable "matchbox_http_endpoint" {
-  type        = "string"
-  description = "Matchbox HTTP read-only endpoint (e.g. http://matchbox.example.com:8080)"
-}
-
-variable "atomic_assets_endpoint" {
-  type    = "string"
-  default = ""
-
-  description = <<EOD
-HTTP endpoint serving the Fedora Atomic Host vmlinuz, initrd, os repo, and ostree repo (.e.g `http://example.com/some/path`).
-
-Ensure the HTTP server directory contains `vmlinuz` and `initrd` files and `os` and `repo` directories. Leave unset to assume ${matchbox_http_endpoint}/assets/fedora/28
-EOD
-}
-
-# machines
-# Terraform's crude "type system" does not properly support lists of maps so we do this.
-
-variable "controller_names" {
-  type        = "list"
-  description = "Ordered list of controller names (e.g. [node1])"
-}
-
-variable "controller_macs" {
-  type        = "list"
-  description = "Ordered list of controller identifying MAC addresses (e.g. [52:54:00:a1:9c:ae])"
-}
-
-variable "controller_domains" {
-  type        = "list"
-  description = "Ordered list of controller FQDNs (e.g. [node1.example.com])"
-}
-
-variable "worker_names" {
-  type        = "list"
-  description = "Ordered list of worker names (e.g. [node2, node3])"
-}
-
-variable "worker_macs" {
-  type        = "list"
-  description = "Ordered list of worker identifying MAC addresses (e.g. [52:54:00:b2:2f:86, 52:54:00:c3:61:77])"
-}
-
-variable "worker_domains" {
-  type        = "list"
-  description = "Ordered list of worker FQDNs (e.g. [node2.example.com, node3.example.com])"
-}
-
-# configuration
-
-variable "k8s_domain_name" {
-  description = "Controller DNS name which resolves to a controller instance. Workers and kubeconfig's will communicate with this endpoint (e.g. cluster.example.com)"
-  type        = "string"
-}
-
-variable "ssh_authorized_key" {
-  type        = "string"
-  description = "SSH public key for user 'fedora'"
-}
-
-variable "asset_dir" {
-  description = "Path to a directory where generated assets should be placed (contains secrets)"
-  type        = "string"
-}
-
-variable "networking" {
-  description = "Choice of networking provider (flannel or calico)"
-  type        = "string"
-  default     = "calico"
-}
-
-variable "network_mtu" {
-  description = "CNI interface MTU (applies to calico only)"
-  type        = "string"
-  default     = "1480"
-}
-
-variable "pod_cidr" {
-  description = "CIDR IPv4 range to assign Kubernetes pods"
-  type        = "string"
-  default     = "10.2.0.0/16"
-}
-
-variable "service_cidr" {
-  description = <<EOD
-CIDR IPv4 range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
-EOD
-
-  type    = "string"
-  default = "10.3.0.0/16"
-}
-
-variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
-  default     = "cluster.local"
-}
-
-variable "kernel_args" {
-  description = "Additional kernel arguments to provide at PXE boot."
-  type        = "list"
-  default     = []
-}
-
-variable "enable_reporting" {
-  type        = "string"
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = "false"
-}

digital-ocean/container-linux/kubernetes/README.md

@@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.14.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.15.0 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, and other [addons](https://typhoon.psdn.io/addons/overview/)
 

digital-ocean/container-linux/kubernetes/bootkube.tf

@@ -1,21 +1,22 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=082921d67905417755609eebda7d39a7e26f7fdb"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=62df9ad69cc0da35f47d40fa981370c4503ad581"
 
-  cluster_name = "${var.cluster_name}"
+  cluster_name = var.cluster_name
-  api_servers  = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
+  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
-  etcd_servers = "${digitalocean_record.etcds.*.fqdn}"
+  etcd_servers = digitalocean_record.etcds.*.fqdn
-  asset_dir    = "${var.asset_dir}"
+  asset_dir    = var.asset_dir
 
-  networking = "${var.networking}"
+  networking = var.networking
 
   # only effective with Calico networking
   network_encapsulation = "vxlan"
   network_mtu           = "1450"
 
-  pod_cidr              = "${var.pod_cidr}"
+  pod_cidr              = var.pod_cidr
-  service_cidr          = "${var.service_cidr}"
+  service_cidr          = var.service_cidr
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = "${var.enable_reporting}"
+  enable_reporting      = var.enable_reporting
-  enable_aggregation    = "${var.enable_aggregation}"
+  enable_aggregation    = var.enable_aggregation
 }
+

digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -129,7 +129,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.14.3
+          KUBELET_IMAGE_TAG=v1.15.0
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -99,7 +99,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.14.3
+          KUBELET_IMAGE_TAG=v1.15.0
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -117,7 +117,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.14.3 \
+            docker://k8s.gcr.io/hyperkube:v1.15.0 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

digital-ocean/container-linux/kubernetes/controllers.tf

@@ -1,25 +1,25 @@
 # Controller Instance DNS records
 resource "digitalocean_record" "controllers" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
   # DNS zone where record should be created
-  domain = "${var.dns_zone}"
+  domain = var.dns_zone
 
   # DNS record (will be prepended to domain)
-  name = "${var.cluster_name}"
+  name = var.cluster_name
   type = "A"
   ttl  = 300
 
   # IPv4 addresses of controllers
-  value = "${element(digitalocean_droplet.controllers.*.ipv4_address, count.index)}"
+  value = element(digitalocean_droplet.controllers.*.ipv4_address, count.index)
 }
 
 # Discrete DNS records for each controller's private IPv4 for etcd usage
 resource "digitalocean_record" "etcds" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
   # DNS zone where record should be created
-  domain = "${var.dns_zone}"
+  domain = var.dns_zone
 
   # DNS record (will be prepended to domain)
   name = "${var.cluster_name}-etcd${count.index}"
@@ -27,34 +27,32 @@ resource "digitalocean_record" "etcds" {
   ttl  = 300
 
   # private IPv4 address for etcd
-  value = "${element(digitalocean_droplet.controllers.*.ipv4_address_private, count.index)}"
+  value = element(digitalocean_droplet.controllers.*.ipv4_address_private, count.index)
 }
 
 # Controller droplet instances
 resource "digitalocean_droplet" "controllers" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
   name   = "${var.cluster_name}-controller-${count.index}"
-  region = "${var.region}"
+  region = var.region
 
-  image = "${var.image}"
+  image = var.image
-  size  = "${var.controller_type}"
+  size  = var.controller_type
 
   # network
   ipv6               = true
   private_networking = true
 
-  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
+  user_data = element(data.ct_config.controller-ignitions.*.rendered, count.index)
-  ssh_keys  = ["${var.ssh_fingerprints}"]
+  ssh_keys  = var.ssh_fingerprints
 
   tags = [
-    "${digitalocean_tag.controllers.id}",
+    digitalocean_tag.controllers.id,
   ]
 
   lifecycle {
-    ignore_changes = [
+    ignore_changes = [user_data]
-      "user_data",
-    ]
   }
 }
 
@@ -65,37 +63,37 @@ resource "digitalocean_tag" "controllers" {
 
 # Controller Ignition configs
 data "ct_config" "controller-ignitions" {
-  count        = "${var.controller_count}"
+  count        = var.controller_count
-  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  content      = element(data.template_file.controller-configs.*.rendered, count.index)
   pretty_print = false
-  snippets     = ["${var.controller_clc_snippets}"]
+  snippets     = var.controller_clc_snippets
 }
 
 # Controller Container Linux configs
 data "template_file" "controller-configs" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
-  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+  template = file("${path.module}/cl/controller.yaml.tmpl")
 
   vars = {
     # Cannot use cyclic dependencies on controllers or their DNS records
     etcd_name   = "etcd${count.index}"
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
-
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster   = "${join(",", data.template_file.etcds.*.rendered)}"
+    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    cluster_domain_suffix  = var.cluster_domain_suffix
   }
 }
 
 data "template_file" "etcds" {
-  count    = "${var.controller_count}"
+  count    = var.controller_count
   template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
 
   vars = {
-    index        = "${count.index}"
+    index        = count.index
-    cluster_name = "${var.cluster_name}"
+    cluster_name = var.cluster_name
-    dns_zone     = "${var.dns_zone}"
+    dns_zone     = var.dns_zone
   }
 }
+

digital-ocean/container-linux/kubernetes/network.tf

@@ -1,50 +1,51 @@
 resource "digitalocean_firewall" "rules" {
-  name = "${var.cluster_name}"
+  name = var.cluster_name
 
   tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 
   # allow ssh, internal flannel, internal node-exporter, internal kubelet
-  inbound_rule = [
+  inbound_rule {
-    {
+    protocol         = "tcp"
-      protocol         = "tcp"
+    port_range       = "22"
-      port_range       = "22"
+    source_addresses = ["0.0.0.0/0", "::/0"]
-      source_addresses = ["0.0.0.0/0", "::/0"]
+  }
-    },
+
-    {
+  inbound_rule {
-      protocol    = "udp"
+    protocol    = "udp"
-      port_range  = "4789"
+    port_range  = "4789"
-      source_tags = ["${digitalocean_tag.controllers.name}", "${digitalocean_tag.workers.name}"]
+    source_tags = [digitalocean_tag.controllers.name, digitalocean_tag.workers.name]
-    },
+  }
-    {
+
-      protocol    = "tcp"
+  inbound_rule {
-      port_range  = "9100"
+    protocol    = "tcp"
-      source_tags = ["${digitalocean_tag.workers.name}"]
+    port_range  = "9100"
-    },
+    source_tags = [digitalocean_tag.workers.name]
-    {
+  }
-      protocol    = "tcp"
+
-      port_range  = "10250"
+  inbound_rule {
-      source_tags = ["${digitalocean_tag.controllers.name}", "${digitalocean_tag.workers.name}"]
+    protocol    = "tcp"
-    },
+    port_range  = "10250"
-  ]
+    source_tags = [digitalocean_tag.controllers.name, digitalocean_tag.workers.name]
+  }
 
   # allow all outbound traffic
-  outbound_rule = [
+  outbound_rule {
-    {
+    protocol              = "tcp"
-      protocol              = "tcp"
+    port_range            = "1-65535"
-      port_range            = "1-65535"
+    destination_addresses = ["0.0.0.0/0", "::/0"]
-      destination_addresses = ["0.0.0.0/0", "::/0"]
+  }
-    },
+
-    {
+  outbound_rule {
-      protocol              = "udp"
+    protocol              = "udp"
-      port_range            = "1-65535"
+    port_range            = "1-65535"
-      destination_addresses = ["0.0.0.0/0", "::/0"]
+    destination_addresses = ["0.0.0.0/0", "::/0"]
-    },
+  }
-    {
+
-      protocol              = "icmp"
+  outbound_rule {
-      port_range            = "1-65535"
+    protocol              = "icmp"
-      destination_addresses = ["0.0.0.0/0", "::/0"]
+    port_range            = "1-65535"
-    },
+    destination_addresses = ["0.0.0.0/0", "::/0"]
-  ]
+  }
 }
 
 resource "digitalocean_firewall" "controllers" {
@@ -53,23 +54,23 @@ resource "digitalocean_firewall" "controllers" {
   tags = ["${var.cluster_name}-controller"]
 
   # etcd, kube-apiserver, kubelet
-  inbound_rule = [
+  inbound_rule {
-    {
+    protocol    = "tcp"
-      protocol    = "tcp"
+    port_range  = "2379-2380"
-      port_range  = "2379-2380"
+    source_tags = [digitalocean_tag.controllers.name]
-      source_tags = ["${digitalocean_tag.controllers.name}"]
+  }
-    },
+
-    {
+  inbound_rule {
-      protocol    = "tcp"
+    protocol    = "tcp"
-      port_range  = "2381"
+    port_range  = "2381"
-      source_tags = ["${digitalocean_tag.workers.name}"]
+    source_tags = [digitalocean_tag.workers.name]
-    },
+  }
-    {
+
-      protocol         = "tcp"
+  inbound_rule {
-      port_range       = "6443"
+    protocol         = "tcp"
-      source_addresses = ["0.0.0.0/0", "::/0"]
+    port_range       = "6443"
-    },
+    source_addresses = ["0.0.0.0/0", "::/0"]
-  ]
+  }
 }
 
 resource "digitalocean_firewall" "workers" {
@@ -78,21 +79,22 @@ resource "digitalocean_firewall" "workers" {
   tags = ["${var.cluster_name}-worker"]
 
   # allow HTTP/HTTPS ingress
-  inbound_rule = [
+  inbound_rule {
-    {
+    protocol         = "tcp"
-      protocol         = "tcp"
+    port_range       = "80"
-      port_range       = "80"
+    source_addresses = ["0.0.0.0/0", "::/0"]
-      source_addresses = ["0.0.0.0/0", "::/0"]
+  }
-    },
+
-    {
+  inbound_rule {
-      protocol         = "tcp"
+    protocol         = "tcp"
-      port_range       = "443"
+    port_range       = "443"
-      source_addresses = ["0.0.0.0/0", "::/0"]
+    source_addresses = ["0.0.0.0/0", "::/0"]
-    },
+  }
-    {
+
-      protocol         = "tcp"
+  inbound_rule {
-      port_range       = "10254"
+    protocol         = "tcp"
-      source_addresses = ["0.0.0.0/0"]
+    port_range       = "10254"
-    },
+    source_addresses = ["0.0.0.0/0"]
-  ]
+  }
 }
+

digital-ocean/container-linux/kubernetes/outputs.tf

@@ -1,40 +1,41 @@
 output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
+  value = module.bootkube.kubeconfig-admin
 }
 
 output "controllers_dns" {
-  value = "${digitalocean_record.controllers.0.fqdn}"
+  value = digitalocean_record.controllers[0].fqdn
 }
 
 output "workers_dns" {
   # Multiple A and AAAA records with the same FQDN
-  value = "${digitalocean_record.workers-record-a.0.fqdn}"
+  value = digitalocean_record.workers-record-a[0].fqdn
 }
 
 output "controllers_ipv4" {
-  value = ["${digitalocean_droplet.controllers.*.ipv4_address}"]
+  value = [digitalocean_droplet.controllers.*.ipv4_address]
 }
 
 output "controllers_ipv6" {
-  value = ["${digitalocean_droplet.controllers.*.ipv6_address}"]
+  value = [digitalocean_droplet.controllers.*.ipv6_address]
 }
 
 output "workers_ipv4" {
-  value = ["${digitalocean_droplet.workers.*.ipv4_address}"]
+  value = [digitalocean_droplet.workers.*.ipv4_address]
 }
 
 output "workers_ipv6" {
-  value = ["${digitalocean_droplet.workers.*.ipv6_address}"]
+  value = [digitalocean_droplet.workers.*.ipv6_address]
 }
 
 # Outputs for custom firewalls
 
 output "controller_tag" {
   description = "Tag applied to controller droplets"
-  value       = "${digitalocean_tag.controllers.name}"
+  value       = digitalocean_tag.controllers.name
 }
 
 output "worker_tag" {
   description = "Tag applied to worker droplets"
-  value       = "${digitalocean_tag.workers.name}"
+  value       = digitalocean_tag.workers.name
 }
+

digital-ocean/container-linux/kubernetes/require.tf

@@ -1,25 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "digitalocean" {
-  version = "~> 1.0"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}

digital-ocean/container-linux/kubernetes/ssh.tf

@@ -1,55 +1,55 @@
 # Secure copy etcd TLS assets and kubeconfig to controllers. Activates kubelet.service
 resource "null_resource" "copy-controller-secrets" {
-  count = "${var.controller_count}"
+  count = var.controller_count
 
   depends_on = [
-    "digitalocean_firewall.rules",
+    digitalocean_firewall.rules
   ]
 
   connection {
     type    = "ssh"
-    host    = "${element(concat(digitalocean_droplet.controllers.*.ipv4_address), count.index)}"
+    host    = element(digitalocean_droplet.controllers.*.ipv4_address, count.index)
     user    = "core"
     timeout = "15m"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig-kubelet}"
+    content     = module.bootkube.kubeconfig-kubelet
     destination = "$HOME/kubeconfig"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
+    content     = module.bootkube.etcd_ca_cert
     destination = "$HOME/etcd-client-ca.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
+    content     = module.bootkube.etcd_client_cert
     destination = "$HOME/etcd-client.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
+    content     = module.bootkube.etcd_client_key
     destination = "$HOME/etcd-client.key"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
+    content     = module.bootkube.etcd_server_cert
     destination = "$HOME/etcd-server.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
+    content     = module.bootkube.etcd_server_key
     destination = "$HOME/etcd-server.key"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
+    content     = module.bootkube.etcd_peer_cert
     destination = "$HOME/etcd-peer.crt"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
+    content     = module.bootkube.etcd_peer_key
     destination = "$HOME/etcd-peer.key"
   }
 
@@ -72,17 +72,17 @@ resource "null_resource" "copy-controller-secrets" {
 
 # Secure copy kubeconfig to all workers. Activates kubelet.service.
 resource "null_resource" "copy-worker-secrets" {
-  count = "${var.worker_count}"
+  count = var.worker_count
 
   connection {
     type    = "ssh"
-    host    = "${element(concat(digitalocean_droplet.workers.*.ipv4_address), count.index)}"
+    host    = element(digitalocean_droplet.workers.*.ipv4_address, count.index)
     user    = "core"
     timeout = "15m"
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig-kubelet}"
+    content     = module.bootkube.kubeconfig-kubelet
     destination = "$HOME/kubeconfig"
   }
 
@@ -97,20 +97,20 @@ resource "null_resource" "copy-worker-secrets" {
 # one-time self-hosted cluster bootstrapping.
 resource "null_resource" "bootkube-start" {
   depends_on = [
-    "module.bootkube",
+    module.bootkube,
-    "null_resource.copy-controller-secrets",
+    null_resource.copy-controller-secrets,
-    "null_resource.copy-worker-secrets",
+    null_resource.copy-worker-secrets,
   ]
 
   connection {
     type    = "ssh"
-    host    = "${digitalocean_droplet.controllers.0.ipv4_address}"
+    host    = digitalocean_droplet.controllers[0].ipv4_address
     user    = "core"
     timeout = "15m"
   }
 
   provisioner "file" {
-    source      = "${var.asset_dir}"
+    source      = var.asset_dir
     destination = "$HOME/assets"
   }
 
@@ -121,3 +121,4 @@ resource "null_resource" "bootkube-start" {
     ]
   }
 }
+

digital-ocean/container-linux/kubernetes/variables.tf

@@ -1,60 +1,60 @@
 variable "cluster_name" {
-  type        = "string"
+  type        = string
   description = "Unique cluster name (prepended to dns_zone)"
 }
 
 # Digital Ocean
 
 variable "region" {
-  type        = "string"
+  type        = string
   description = "Digital Ocean region (e.g. nyc1, sfo2, fra1, tor1)"
 }
 
 variable "dns_zone" {
-  type        = "string"
+  type        = string
   description = "Digital Ocean domain (i.e. DNS zone) (e.g. do.example.com)"
 }
 
 # instances
 
 variable "controller_count" {
-  type        = "string"
+  type        = string
   default     = "1"
   description = "Number of controllers (i.e. masters)"
 }
 
 variable "worker_count" {
-  type        = "string"
+  type        = string
   default     = "1"
   description = "Number of workers"
 }
 
 variable "controller_type" {
-  type        = "string"
+  type        = string
   default     = "s-2vcpu-2gb"
   description = "Droplet type for controllers (e.g. s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb)."
 }
 
 variable "worker_type" {
-  type        = "string"
+  type        = string
-  default     = "s-1vcpu-1gb"
+  default     = "s-1vcpu-2gb"
-  description = "Droplet type for workers (e.g. s-1vcpu-1gb, s-1vcpu-2gb, s-2vcpu-2gb)"
+  description = "Droplet type for workers (e.g. s-1vcpu-2gb, s-2vcpu-2gb)"
 }
 
 variable "image" {
-  type        = "string"
+  type        = string
   default     = "coreos-stable"
   description = "Container Linux image for instances (e.g. coreos-stable)"
 }
 
 variable "controller_clc_snippets" {
-  type        = "list"
+  type        = list(string)
   description = "Controller Container Linux Config snippets"
   default     = []
 }
 
 variable "worker_clc_snippets" {
-  type        = "list"
+  type        = list(string)
   description = "Worker Container Linux Config snippets"
   default     = []
 }
@@ -62,24 +62,24 @@ variable "worker_clc_snippets" {
 # configuration
 
 variable "ssh_fingerprints" {
-  type        = "list"
+  type        = list(string)
   description = "SSH public key fingerprints. (e.g. see `ssh-add -l -E md5`)"
 }
 
 variable "asset_dir" {
   description = "Path to a directory where generated assets should be placed (contains secrets)"
-  type        = "string"
+  type        = string
 }
 
 variable "networking" {
   description = "Choice of networking provider (flannel or calico)"
-  type        = "string"
+  type        = string
   default     = "flannel"
 }
 
 variable "pod_cidr" {
   description = "CIDR IPv4 range to assign Kubernetes pods"
-  type        = "string"
+  type        = string
   default     = "10.2.0.0/16"
 }
 
@@ -89,24 +89,26 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD
 
-  type    = "string"
+
+  type = string
   default = "10.3.0.0/16"
 }
 
 variable "cluster_domain_suffix" {
   description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
+  type = string
-  default     = "cluster.local"
+  default = "cluster.local"
 }
 
 variable "enable_reporting" {
-  type        = "string"
+  type = string
   description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = "false"
+  default = "false"
 }
 
 variable "enable_aggregation" {
   description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  type        = "string"
+  type = string
-  default     = "false"
+  default = "false"
 }
+

digital-ocean/container-linux/kubernetes/versions.tf

@@ -0,0 +1,12 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = "~> 0.12.0"
+  required_providers {
+    digitalocean = "~> 1.3"
+    ct           = "~> 0.3.2"
+    template     = "~> 2.1"
+    null         = "~> 2.1"
+  }
+}
+