Add support for worker pools on AWS
* Allow groups of workers to be defined and joined to a cluster (i.e. worker pools) * Move worker resources into a Terraform submodule * Output variables needed for passing to worker pools * Add usage docs for AWS worker pools (advanced)
This commit is contained in:
parent
160ae34e71
commit
73126eb7f8
|
@ -4,10 +4,11 @@ Notable changes between versions.
|
||||||
|
|
||||||
## Latest
|
## Latest
|
||||||
|
|
||||||
|
* Add [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) for AWS and Google Cloud (advanced)
|
||||||
|
* [Recommend](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action recommended)
|
||||||
* Upgrade etcd from v3.2.15 to v3.3.1
|
* Upgrade etcd from v3.2.15 to v3.3.1
|
||||||
* Update Calico from v3.0.2 to v3.0.3
|
* Update Calico from v3.0.2 to v3.0.3
|
||||||
* Use kubernetes-incubator/bootkube v0.10.0
|
* Use kubernetes-incubator/bootkube v0.10.0
|
||||||
* [Recommend](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action recommended)
|
|
||||||
|
|
||||||
#### AWS
|
#### AWS
|
||||||
|
|
||||||
|
@ -15,6 +16,7 @@ Notable changes between versions.
|
||||||
* Switch Ingress elastic load balancer to a network load balancer ([#141](https://github.com/poseidon/typhoon/pull/141))
|
* Switch Ingress elastic load balancer to a network load balancer ([#141](https://github.com/poseidon/typhoon/pull/141))
|
||||||
* AWS [NLBs](https://aws.amazon.com/blogs/aws/new-network-load-balancer-effortless-scaling-to-millions-of-requests-per-second/) can handle millions of RPS with high throughput and low latency.
|
* AWS [NLBs](https://aws.amazon.com/blogs/aws/new-network-load-balancer-effortless-scaling-to-millions-of-requests-per-second/) can handle millions of RPS with high throughput and low latency.
|
||||||
* Require terraform-provider-aws 1.7.0 or higher
|
* Require terraform-provider-aws 1.7.0 or higher
|
||||||
|
* Allow groups of workers to be defined and joined to a cluster (i.e. worker pools) ([#150](https://github.com/poseidon/typhoon/pull/150))
|
||||||
* Add kubelet `--volume-plugin-dir` flag to allow flexvolume plugins ([#142](https://github.com/poseidon/typhoon/pull/142))
|
* Add kubelet `--volume-plugin-dir` flag to allow flexvolume plugins ([#142](https://github.com/poseidon/typhoon/pull/142))
|
||||||
|
|
||||||
#### Digital Ocean
|
#### Digital Ocean
|
||||||
|
@ -24,7 +26,7 @@ Notable changes between versions.
|
||||||
|
|
||||||
#### Google Cloud
|
#### Google Cloud
|
||||||
|
|
||||||
* Add support for "worker pools" - groups of homogeneous workers joined to an existing cluster ([#148](https://github.com/poseidon/typhoon/pull/148))
|
* Allow groups of workers to be defined and joined to a cluster (i.e. worker pools) ([#148](https://github.com/poseidon/typhoon/pull/148))
|
||||||
* Add kubelet `--volume-plugin-dir` flag to allow flexvolume plugins ([#142](https://github.com/poseidon/typhoon/pull/142))
|
* Add kubelet `--volume-plugin-dir` flag to allow flexvolume plugins ([#142](https://github.com/poseidon/typhoon/pull/142))
|
||||||
* Add `kubeconfig` variable to `controllers` and `workers` submodules ([#147](https://github.com/poseidon/typhoon/pull/147))
|
* Add `kubeconfig` variable to `controllers` and `workers` submodules ([#147](https://github.com/poseidon/typhoon/pull/147))
|
||||||
* Remove `kubeconfig_*` variables from `controllers` and `workers` submodules ([#147](https://github.com/poseidon/typhoon/pull/147))
|
* Remove `kubeconfig_*` variables from `controllers` and `workers` submodules ([#147](https://github.com/poseidon/typhoon/pull/147))
|
||||||
|
|
|
@ -14,6 +14,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
|
||||||
* Kubernetes v1.9.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
|
* Kubernetes v1.9.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
|
||||||
* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
|
* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
|
||||||
* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
|
* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
|
||||||
|
* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/google-cloud/#preemption) (varies by platform)
|
||||||
* Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
|
* Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
|
||||||
|
|
||||||
## Modules
|
## Modules
|
||||||
|
|
|
@ -14,6 +14,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
|
||||||
* Kubernetes v1.9.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
|
* Kubernetes v1.9.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
|
||||||
* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
|
* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
|
||||||
* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
|
* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
|
||||||
|
* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
|
||||||
* Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
|
* Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
|
||||||
|
|
||||||
## Docs
|
## Docs
|
||||||
|
|
|
@ -79,185 +79,3 @@ data "ct_config" "controller_ign" {
|
||||||
content = "${element(data.template_file.controller_config.*.rendered, count.index)}"
|
content = "${element(data.template_file.controller_config.*.rendered, count.index)}"
|
||||||
pretty_print = false
|
pretty_print = false
|
||||||
}
|
}
|
||||||
|
|
||||||
# Security Group (instance firewall)
|
|
||||||
|
|
||||||
resource "aws_security_group" "controller" {
|
|
||||||
name = "${var.cluster_name}-controller"
|
|
||||||
description = "${var.cluster_name} controller security group"
|
|
||||||
|
|
||||||
vpc_id = "${aws_vpc.network.id}"
|
|
||||||
|
|
||||||
tags = "${map("Name", "${var.cluster_name}-controller")}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-icmp" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "icmp"
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-ssh" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 22
|
|
||||||
to_port = 22
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-apiserver" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 443
|
|
||||||
to_port = 443
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-etcd" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 2379
|
|
||||||
to_port = 2380
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-flannel" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "udp"
|
|
||||||
from_port = 8472
|
|
||||||
to_port = 8472
|
|
||||||
source_security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-flannel-self" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "udp"
|
|
||||||
from_port = 8472
|
|
||||||
to_port = 8472
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-node-exporter" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 9100
|
|
||||||
to_port = 9100
|
|
||||||
source_security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-kubelet-self" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 10250
|
|
||||||
to_port = 10250
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-kubelet-read" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 10255
|
|
||||||
to_port = 10255
|
|
||||||
source_security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-kubelet-read-self" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 10255
|
|
||||||
to_port = 10255
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-bgp" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 179
|
|
||||||
to_port = 179
|
|
||||||
source_security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-bgp-self" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 179
|
|
||||||
to_port = 179
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-ipip" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = 4
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
source_security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-ipip-self" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = 4
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-ipip-legacy" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = 94
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
source_security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-ipip-legacy-self" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = 94
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "controller-egress" {
|
|
||||||
security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
|
|
||||||
type = "egress"
|
|
||||||
protocol = "-1"
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
ipv6_cidr_blocks = ["::/0"]
|
|
||||||
}
|
|
||||||
|
|
|
@ -1,4 +1,25 @@
|
||||||
output "ingress_dns_name" {
|
output "ingress_dns_name" {
|
||||||
value = "${aws_lb.ingress.dns_name}"
|
value = "${module.workers.ingress_dns_name}"
|
||||||
description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
|
description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Outputs for worker pools
|
||||||
|
|
||||||
|
output "vpc_id" {
|
||||||
|
value = "${aws_vpc.network.id}"
|
||||||
|
description = "ID of the VPC for creating worker instances"
|
||||||
|
}
|
||||||
|
|
||||||
|
output "subnet_ids" {
|
||||||
|
value = ["${aws_subnet.public.*.id}"]
|
||||||
|
description = "List of subnet IDs for creating worker instances"
|
||||||
|
}
|
||||||
|
|
||||||
|
output "worker_security_groups" {
|
||||||
|
value = ["${aws_security_group.worker.id}"]
|
||||||
|
description = "List of worker security group IDs"
|
||||||
|
}
|
||||||
|
|
||||||
|
output "kubeconfig" {
|
||||||
|
value = "${module.bootkube.kubeconfig}"
|
||||||
|
}
|
||||||
|
|
|
@ -0,0 +1,385 @@
|
||||||
|
# Security Groups (instance firewalls)
|
||||||
|
|
||||||
|
# Controller security group
|
||||||
|
|
||||||
|
resource "aws_security_group" "controller" {
|
||||||
|
name = "${var.cluster_name}-controller"
|
||||||
|
description = "${var.cluster_name} controller security group"
|
||||||
|
|
||||||
|
vpc_id = "${aws_vpc.network.id}"
|
||||||
|
|
||||||
|
tags = "${map("Name", "${var.cluster_name}-controller")}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-icmp" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "icmp"
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-ssh" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 22
|
||||||
|
to_port = 22
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-apiserver" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 443
|
||||||
|
to_port = 443
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-etcd" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 2379
|
||||||
|
to_port = 2380
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-flannel" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "udp"
|
||||||
|
from_port = 8472
|
||||||
|
to_port = 8472
|
||||||
|
source_security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-flannel-self" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "udp"
|
||||||
|
from_port = 8472
|
||||||
|
to_port = 8472
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-node-exporter" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 9100
|
||||||
|
to_port = 9100
|
||||||
|
source_security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-kubelet-self" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 10250
|
||||||
|
to_port = 10250
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-kubelet-read" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 10255
|
||||||
|
to_port = 10255
|
||||||
|
source_security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-kubelet-read-self" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 10255
|
||||||
|
to_port = 10255
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-bgp" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 179
|
||||||
|
to_port = 179
|
||||||
|
source_security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-bgp-self" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 179
|
||||||
|
to_port = 179
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-ipip" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = 4
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
source_security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-ipip-self" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = 4
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-ipip-legacy" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = 94
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
source_security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-ipip-legacy-self" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = 94
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "controller-egress" {
|
||||||
|
security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
|
||||||
|
type = "egress"
|
||||||
|
protocol = "-1"
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
ipv6_cidr_blocks = ["::/0"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Worker security group
|
||||||
|
|
||||||
|
resource "aws_security_group" "worker" {
|
||||||
|
name = "${var.cluster_name}-worker"
|
||||||
|
description = "${var.cluster_name} worker security group"
|
||||||
|
|
||||||
|
vpc_id = "${aws_vpc.network.id}"
|
||||||
|
|
||||||
|
tags = "${map("Name", "${var.cluster_name}-worker")}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-icmp" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "icmp"
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-ssh" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 22
|
||||||
|
to_port = 22
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-http" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 80
|
||||||
|
to_port = 80
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-https" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 443
|
||||||
|
to_port = 443
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-flannel" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "udp"
|
||||||
|
from_port = 8472
|
||||||
|
to_port = 8472
|
||||||
|
source_security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-flannel-self" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "udp"
|
||||||
|
from_port = 8472
|
||||||
|
to_port = 8472
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-node-exporter" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 9100
|
||||||
|
to_port = 9100
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "ingress-health" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 10254
|
||||||
|
to_port = 10254
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-kubelet" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 10250
|
||||||
|
to_port = 10250
|
||||||
|
source_security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-kubelet-self" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 10250
|
||||||
|
to_port = 10250
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-kubelet-read" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 10255
|
||||||
|
to_port = 10255
|
||||||
|
source_security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-kubelet-read-self" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 10255
|
||||||
|
to_port = 10255
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-bgp" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 179
|
||||||
|
to_port = 179
|
||||||
|
source_security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-bgp-self" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = "tcp"
|
||||||
|
from_port = 179
|
||||||
|
to_port = 179
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-ipip" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = 4
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
source_security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-ipip-self" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = 4
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-ipip-legacy" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = 94
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
source_security_group_id = "${aws_security_group.controller.id}"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-ipip-legacy-self" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "ingress"
|
||||||
|
protocol = 94
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
self = true
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "aws_security_group_rule" "worker-egress" {
|
||||||
|
security_group_id = "${aws_security_group.worker.id}"
|
||||||
|
|
||||||
|
type = "egress"
|
||||||
|
protocol = "-1"
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
ipv6_cidr_blocks = ["::/0"]
|
||||||
|
}
|
|
@ -1,276 +1,19 @@
|
||||||
# Workers AutoScaling Group
|
module "workers" {
|
||||||
resource "aws_autoscaling_group" "workers" {
|
source = "workers"
|
||||||
name = "${var.cluster_name}-worker ${aws_launch_configuration.worker.name}"
|
cluster_name = "${var.cluster_name}"
|
||||||
|
|
||||||
# count
|
# AWS
|
||||||
desired_capacity = "${var.worker_count}"
|
vpc_id = "${aws_vpc.network.id}"
|
||||||
min_size = "${var.worker_count}"
|
subnet_ids = ["${aws_subnet.public.*.id}"]
|
||||||
max_size = "${var.worker_count + 2}"
|
|
||||||
default_cooldown = 30
|
|
||||||
health_check_grace_period = 30
|
|
||||||
|
|
||||||
# network
|
|
||||||
vpc_zone_identifier = ["${aws_subnet.public.*.id}"]
|
|
||||||
|
|
||||||
# template
|
|
||||||
launch_configuration = "${aws_launch_configuration.worker.name}"
|
|
||||||
|
|
||||||
# target groups to which instances should be added
|
|
||||||
target_group_arns = [
|
|
||||||
"${aws_lb_target_group.workers-http.id}",
|
|
||||||
"${aws_lb_target_group.workers-https.id}",
|
|
||||||
]
|
|
||||||
|
|
||||||
lifecycle {
|
|
||||||
# override the default destroy and replace update behavior
|
|
||||||
create_before_destroy = true
|
|
||||||
ignore_changes = ["image_id"]
|
|
||||||
}
|
|
||||||
|
|
||||||
tags = [{
|
|
||||||
key = "Name"
|
|
||||||
value = "${var.cluster_name}-worker"
|
|
||||||
propagate_at_launch = true
|
|
||||||
}]
|
|
||||||
}
|
|
||||||
|
|
||||||
# Worker template
|
|
||||||
resource "aws_launch_configuration" "worker" {
|
|
||||||
image_id = "${data.aws_ami.coreos.image_id}"
|
|
||||||
instance_type = "${var.worker_type}"
|
|
||||||
|
|
||||||
user_data = "${data.ct_config.worker_ign.rendered}"
|
|
||||||
|
|
||||||
# storage
|
|
||||||
root_block_device {
|
|
||||||
volume_type = "standard"
|
|
||||||
volume_size = "${var.disk_size}"
|
|
||||||
}
|
|
||||||
|
|
||||||
# network
|
|
||||||
security_groups = ["${aws_security_group.worker.id}"]
|
security_groups = ["${aws_security_group.worker.id}"]
|
||||||
|
count = "${var.worker_count}"
|
||||||
|
instance_type = "${var.worker_type}"
|
||||||
|
os_channel = "${var.os_channel}"
|
||||||
|
disk_size = "${var.disk_size}"
|
||||||
|
|
||||||
lifecycle {
|
# configuration
|
||||||
// Override the default destroy and replace update behavior
|
kubeconfig = "${module.bootkube.kubeconfig}"
|
||||||
create_before_destroy = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# Worker Container Linux Config
|
|
||||||
data "template_file" "worker_config" {
|
|
||||||
template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
|
|
||||||
|
|
||||||
vars = {
|
|
||||||
kubeconfig = "${indent(10, module.bootkube.kubeconfig)}"
|
|
||||||
ssh_authorized_key = "${var.ssh_authorized_key}"
|
ssh_authorized_key = "${var.ssh_authorized_key}"
|
||||||
k8s_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
|
service_cidr = "${var.service_cidr}"
|
||||||
cluster_domain_suffix = "${var.cluster_domain_suffix}"
|
cluster_domain_suffix = "${var.cluster_domain_suffix}"
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
data "ct_config" "worker_ign" {
|
|
||||||
content = "${data.template_file.worker_config.rendered}"
|
|
||||||
pretty_print = false
|
|
||||||
}
|
|
||||||
|
|
||||||
# Security Group (instance firewall)
|
|
||||||
|
|
||||||
resource "aws_security_group" "worker" {
|
|
||||||
name = "${var.cluster_name}-worker"
|
|
||||||
description = "${var.cluster_name} worker security group"
|
|
||||||
|
|
||||||
vpc_id = "${aws_vpc.network.id}"
|
|
||||||
|
|
||||||
tags = "${map("Name", "${var.cluster_name}-worker")}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-icmp" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "icmp"
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-ssh" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 22
|
|
||||||
to_port = 22
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-http" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 80
|
|
||||||
to_port = 80
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-https" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 443
|
|
||||||
to_port = 443
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-flannel" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "udp"
|
|
||||||
from_port = 8472
|
|
||||||
to_port = 8472
|
|
||||||
source_security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-flannel-self" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "udp"
|
|
||||||
from_port = 8472
|
|
||||||
to_port = 8472
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-node-exporter" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 9100
|
|
||||||
to_port = 9100
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "ingress-health" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 10254
|
|
||||||
to_port = 10254
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-kubelet" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 10250
|
|
||||||
to_port = 10250
|
|
||||||
source_security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-kubelet-self" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 10250
|
|
||||||
to_port = 10250
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-kubelet-read" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 10255
|
|
||||||
to_port = 10255
|
|
||||||
source_security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-kubelet-read-self" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 10255
|
|
||||||
to_port = 10255
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-bgp" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 179
|
|
||||||
to_port = 179
|
|
||||||
source_security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-bgp-self" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = "tcp"
|
|
||||||
from_port = 179
|
|
||||||
to_port = 179
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-ipip" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = 4
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
source_security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-ipip-self" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = 4
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-ipip-legacy" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = 94
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
source_security_group_id = "${aws_security_group.controller.id}"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-ipip-legacy-self" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "ingress"
|
|
||||||
protocol = 94
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
self = true
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "aws_security_group_rule" "worker-egress" {
|
|
||||||
security_group_id = "${aws_security_group.worker.id}"
|
|
||||||
|
|
||||||
type = "egress"
|
|
||||||
protocol = "-1"
|
|
||||||
from_port = 0
|
|
||||||
to_port = 0
|
|
||||||
cidr_blocks = ["0.0.0.0/0"]
|
|
||||||
ipv6_cidr_blocks = ["::/0"]
|
|
||||||
}
|
|
||||||
|
|
|
@ -0,0 +1,19 @@
|
||||||
|
data "aws_ami" "coreos" {
|
||||||
|
most_recent = true
|
||||||
|
owners = ["595879546273"]
|
||||||
|
|
||||||
|
filter {
|
||||||
|
name = "architecture"
|
||||||
|
values = ["x86_64"]
|
||||||
|
}
|
||||||
|
|
||||||
|
filter {
|
||||||
|
name = "virtualization-type"
|
||||||
|
values = ["hvm"]
|
||||||
|
}
|
||||||
|
|
||||||
|
filter {
|
||||||
|
name = "name"
|
||||||
|
values = ["CoreOS-${var.os_channel}-*"]
|
||||||
|
}
|
||||||
|
}
|
|
@ -4,7 +4,7 @@ resource "aws_lb" "ingress" {
|
||||||
load_balancer_type = "network"
|
load_balancer_type = "network"
|
||||||
internal = false
|
internal = false
|
||||||
|
|
||||||
subnets = ["${aws_subnet.public.*.id}"]
|
subnets = ["${var.subnet_ids}"]
|
||||||
}
|
}
|
||||||
|
|
||||||
# Forward HTTP traffic to workers
|
# Forward HTTP traffic to workers
|
||||||
|
@ -35,7 +35,7 @@ resource "aws_lb_listener" "ingress-https" {
|
||||||
|
|
||||||
resource "aws_lb_target_group" "workers-http" {
|
resource "aws_lb_target_group" "workers-http" {
|
||||||
name = "${var.cluster_name}-workers-http"
|
name = "${var.cluster_name}-workers-http"
|
||||||
vpc_id = "${aws_vpc.network.id}"
|
vpc_id = "${var.vpc_id}"
|
||||||
target_type = "instance"
|
target_type = "instance"
|
||||||
|
|
||||||
protocol = "TCP"
|
protocol = "TCP"
|
||||||
|
@ -58,7 +58,7 @@ resource "aws_lb_target_group" "workers-http" {
|
||||||
|
|
||||||
resource "aws_lb_target_group" "workers-https" {
|
resource "aws_lb_target_group" "workers-https" {
|
||||||
name = "${var.cluster_name}-workers-https"
|
name = "${var.cluster_name}-workers-https"
|
||||||
vpc_id = "${aws_vpc.network.id}"
|
vpc_id = "${var.vpc_id}"
|
||||||
target_type = "instance"
|
target_type = "instance"
|
||||||
|
|
||||||
protocol = "TCP"
|
protocol = "TCP"
|
|
@ -0,0 +1,4 @@
|
||||||
|
output "ingress_dns_name" {
|
||||||
|
value = "${aws_lb.ingress.dns_name}"
|
||||||
|
description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
|
||||||
|
}
|
|
@ -0,0 +1,73 @@
|
||||||
|
variable "cluster_name" {
|
||||||
|
type = "string"
|
||||||
|
description = "Unique name"
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "vpc_id" {
|
||||||
|
type = "string"
|
||||||
|
description = "ID of the VPC for creating instances"
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "subnet_ids" {
|
||||||
|
type = "list"
|
||||||
|
description = "List of subnet IDs for creating instances"
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "security_groups" {
|
||||||
|
type = "list"
|
||||||
|
description = "List of security group IDs"
|
||||||
|
}
|
||||||
|
|
||||||
|
# instances
|
||||||
|
|
||||||
|
variable "count" {
|
||||||
|
type = "string"
|
||||||
|
default = "1"
|
||||||
|
description = "Number of instances"
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "instance_type" {
|
||||||
|
type = "string"
|
||||||
|
default = "t2.small"
|
||||||
|
description = "EC2 instance type"
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "os_channel" {
|
||||||
|
type = "string"
|
||||||
|
default = "stable"
|
||||||
|
description = "Container Linux AMI channel (stable, beta, alpha)"
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "disk_size" {
|
||||||
|
type = "string"
|
||||||
|
default = "40"
|
||||||
|
description = "Size of the disk in GB"
|
||||||
|
}
|
||||||
|
|
||||||
|
# configuration
|
||||||
|
|
||||||
|
variable "kubeconfig" {
|
||||||
|
type = "string"
|
||||||
|
description = "Generated Kubelet kubeconfig"
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "ssh_authorized_key" {
|
||||||
|
type = "string"
|
||||||
|
description = "SSH public key for user 'core'"
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "service_cidr" {
|
||||||
|
description = <<EOD
|
||||||
|
CIDR IPv4 range to assign Kubernetes services.
|
||||||
|
The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns.
|
||||||
|
EOD
|
||||||
|
|
||||||
|
type = "string"
|
||||||
|
default = "10.3.0.0/16"
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "cluster_domain_suffix" {
|
||||||
|
description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
|
||||||
|
type = "string"
|
||||||
|
default = "cluster.local"
|
||||||
|
}
|
|
@ -0,0 +1,74 @@
|
||||||
|
# Workers AutoScaling Group
|
||||||
|
resource "aws_autoscaling_group" "workers" {
|
||||||
|
name = "${var.cluster_name}-worker ${aws_launch_configuration.worker.name}"
|
||||||
|
|
||||||
|
# count
|
||||||
|
desired_capacity = "${var.count}"
|
||||||
|
min_size = "${var.count}"
|
||||||
|
max_size = "${var.count + 2}"
|
||||||
|
default_cooldown = 30
|
||||||
|
health_check_grace_period = 30
|
||||||
|
|
||||||
|
# network
|
||||||
|
vpc_zone_identifier = ["${var.subnet_ids}"]
|
||||||
|
|
||||||
|
# template
|
||||||
|
launch_configuration = "${aws_launch_configuration.worker.name}"
|
||||||
|
|
||||||
|
# target groups to which instances should be added
|
||||||
|
target_group_arns = [
|
||||||
|
"${aws_lb_target_group.workers-http.id}",
|
||||||
|
"${aws_lb_target_group.workers-https.id}",
|
||||||
|
]
|
||||||
|
|
||||||
|
lifecycle {
|
||||||
|
# override the default destroy and replace update behavior
|
||||||
|
create_before_destroy = true
|
||||||
|
ignore_changes = ["image_id"]
|
||||||
|
}
|
||||||
|
|
||||||
|
tags = [{
|
||||||
|
key = "Name"
|
||||||
|
value = "${var.cluster_name}-worker"
|
||||||
|
propagate_at_launch = true
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Worker template
|
||||||
|
resource "aws_launch_configuration" "worker" {
|
||||||
|
image_id = "${data.aws_ami.coreos.image_id}"
|
||||||
|
instance_type = "${var.instance_type}"
|
||||||
|
|
||||||
|
user_data = "${data.ct_config.worker_ign.rendered}"
|
||||||
|
|
||||||
|
# storage
|
||||||
|
root_block_device {
|
||||||
|
volume_type = "standard"
|
||||||
|
volume_size = "${var.disk_size}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# network
|
||||||
|
security_groups = ["${var.security_groups}"]
|
||||||
|
|
||||||
|
lifecycle {
|
||||||
|
// Override the default destroy and replace update behavior
|
||||||
|
create_before_destroy = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Worker Container Linux Config
|
||||||
|
data "template_file" "worker_config" {
|
||||||
|
template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
|
||||||
|
|
||||||
|
vars = {
|
||||||
|
kubeconfig = "${indent(10, var.kubeconfig)}"
|
||||||
|
ssh_authorized_key = "${var.ssh_authorized_key}"
|
||||||
|
k8s_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
|
||||||
|
cluster_domain_suffix = "${var.cluster_domain_suffix}"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
data "ct_config" "worker_ign" {
|
||||||
|
content = "${data.template_file.worker_config.rendered}"
|
||||||
|
pretty_print = false
|
||||||
|
}
|
|
@ -1,26 +1,97 @@
|
||||||
# Worker Pools
|
# Worker Pools
|
||||||
|
|
||||||
Typhoon can create "worker pools", groups of homogeneous workers that are part of an existing cluster. For example, you may wish to augment a Kubernetes cluster with groups of workers with a different machine type, larger disks, or preemptibility.
|
Typhoon AWS and Google Cloud allow additional groups of workers to be defined and joined to a cluster. For example, add worker pools of instances with different types, disk sizes, Container Linux channels, or preemptibility modes.
|
||||||
|
|
||||||
|
Internal Terraform Modules:
|
||||||
|
|
||||||
|
* `aws/container-linux/kubernetes/workers`
|
||||||
|
* `google-cloud/container-linux/kubernetes/workers`
|
||||||
|
|
||||||
|
## AWS
|
||||||
|
|
||||||
|
Create a cluster following the AWS [tutorial](../aws.md#cluster). Define a worker pool using the AWS internal `workers` module.
|
||||||
|
|
||||||
|
```tf
|
||||||
|
module "tempest-worker-pool" {
|
||||||
|
source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.9.4"
|
||||||
|
|
||||||
|
providers = {
|
||||||
|
aws = "aws.default"
|
||||||
|
}
|
||||||
|
|
||||||
|
# AWS
|
||||||
|
vpc_id = "${module.aws-tempest.vpc_id}"
|
||||||
|
subnet_ids = "${module.aws-tempest.subnet_ids}"
|
||||||
|
security_groups = "${module.aws-tempest.worker_security_groups}"
|
||||||
|
|
||||||
|
# configuration
|
||||||
|
cluster_name = "tempest-worker-pool"
|
||||||
|
kubeconfig = "${module.aws-tempest.kubeconfig}"
|
||||||
|
ssh_authorized_key = "${var.ssh_authorized_key}"
|
||||||
|
|
||||||
|
count = 2
|
||||||
|
instance_type = "m5.large"
|
||||||
|
os_channel = "beta"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Apply the change.
|
||||||
|
|
||||||
|
```
|
||||||
|
terraform apply
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify an auto-scaling group of workers join the cluster within a few minutes.
|
||||||
|
|
||||||
|
### Variables
|
||||||
|
|
||||||
|
The AWS internal `workers` module supports a number of [variables](https://github.com/poseidon/typhoon/blob/master/aws/container-linux/kubernetes/workers/variables.tf).
|
||||||
|
|
||||||
|
#### Required
|
||||||
|
|
||||||
|
| Name | Description | Example |
|
||||||
|
|:-----|:------------|:--------|
|
||||||
|
| vpc_id | Must be set to `vpc_id` output by cluster | "${module.cluster.vpc_id}" |
|
||||||
|
| subnet_ids | Must be set to `subnet_ids` output by cluster | "${module.cluster.subnet_ids}" |
|
||||||
|
| security_groups | Must be set to `worker_security_groups` output by cluster | "${module.cluster.worker_security_groups}" |
|
||||||
|
| cluster_name | Unique name | "tempest-worker-pool" |
|
||||||
|
| kubeconfig | Must be set to `kubeconfig` output by cluster | "${module.cluster.kubeconfig}" |
|
||||||
|
| ssh_authorized_key | SSH public key for ~/.ssh_authorized_keys | "ssh-rsa AAAAB3NZ..." |
|
||||||
|
|
||||||
|
#### Optional
|
||||||
|
|
||||||
|
| Name | Description | Default | Example |
|
||||||
|
|:-----|:------------|:--------|:--------|
|
||||||
|
| count | Number of instances | 1 | 3 |
|
||||||
|
| instance_type | EC2 instance type | "t2.small" | "t2.medium" |
|
||||||
|
| os_channel | Container Linux AMI channel | stable| "beta", "alpha" |
|
||||||
|
| disk_size | Size of the disk in GB | 40 | 100 |
|
||||||
|
| service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
|
||||||
|
| cluster_domain_suffix | Must match `cluster_domain_suffix` of cluster | "cluster.local" | "k8s.example.com" |
|
||||||
|
|
||||||
|
Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-types/).
|
||||||
|
|
||||||
## Google Cloud
|
## Google Cloud
|
||||||
|
|
||||||
Create a cluster following the Google Cloud [tutorial](../google-cloud.md#cluster). Then define a worker pool using the internal `workers` Terraform module.
|
Create a cluster following the Google Cloud [tutorial](../google-cloud.md#cluster). Define a worker pool using the Google Cloud internal `workers` module.
|
||||||
|
|
||||||
```tf
|
```tf
|
||||||
module "yavin-worker-pool" {
|
module "yavin-worker-pool" {
|
||||||
source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.9.4"
|
source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.9.4"
|
||||||
|
|
||||||
# Google Cloud
|
# Google Cloud
|
||||||
network = "${module.google-cloud-yavin.network_name}"
|
|
||||||
region = "us-central1"
|
region = "us-central1"
|
||||||
count = 2
|
network = "${module.google-cloud-yavin.network_name}"
|
||||||
machine_type = "n1-standard-16"
|
|
||||||
preemptible = true
|
|
||||||
|
|
||||||
|
# configuration
|
||||||
cluster_name = "yavin-16x"
|
cluster_name = "yavin-16x"
|
||||||
|
kubeconfig = "${module.google-cloud-yavin.kubeconfig}"
|
||||||
ssh_authorized_key = "${var.ssh_authorized_key}"
|
ssh_authorized_key = "${var.ssh_authorized_key}"
|
||||||
|
|
||||||
kubeconfig = "${module.google-cloud-yavin.kubeconfig}"
|
count = 2
|
||||||
|
machine_type = "n1-standard-16"
|
||||||
|
os_image = "coreos-beta"
|
||||||
|
preemptible = true
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -50,22 +121,23 @@ The Google Cloud internal `workers` module supports a number of [variables](http
|
||||||
|
|
||||||
| Name | Description | Example |
|
| Name | Description | Example |
|
||||||
|:-----|:------------|:--------|
|
|:-----|:------------|:--------|
|
||||||
|
| region | Must be set to `region` of cluster | "us-central1" |
|
||||||
|
| network | Must be set to `network_name` output by cluster | "${module.cluster.network_name}" |
|
||||||
| cluster_name | Unique name | "yavin-worker-pool" |
|
| cluster_name | Unique name | "yavin-worker-pool" |
|
||||||
| region | Must match region of cluster | "us-central1" |
|
| kubeconfig | Must be set to `kubeconfig` output by cluster | "${module.cluster.kubeconfig}" |
|
||||||
| network | Must match network name output by cluster | "${module.cluster.network_name}" |
|
|
||||||
| ssh_authorized_key | SSH public key for ~/.ssh_authorized_keys | "ssh-rsa AAAAB3NZ..." |
|
| ssh_authorized_key | SSH public key for ~/.ssh_authorized_keys | "ssh-rsa AAAAB3NZ..." |
|
||||||
|
|
||||||
#### Optional
|
#### Optional
|
||||||
|
|
||||||
| Name | Description | Default | Example |
|
| Name | Description | Default | Example |
|
||||||
|:-----|:------------|:--------|:--------|
|
|:-----|:------------|:--------|:--------|
|
||||||
| count | Number of workers | 1 | 3 |
|
| count | Number of instances | 1 | 3 |
|
||||||
| machine_type | Machine type for compute instances | "n1-standard-1" | See below |
|
| machine_type | Compute instance machine type | "n1-standard-1" | See below |
|
||||||
| os_image | OS image for compute instances | "coreos-stable" | "coreos-alpha" |
|
| os_image | OS image for compute instances | "coreos-stable" | "coreos-alpha", "coreos-beta" |
|
||||||
| disk_size | Size of the disk in GB | 40 | 100 |
|
| disk_size | Size of the disk in GB | 40 | 100 |
|
||||||
| preemptible | If enabled, Compute Engine will terminate instances randomly within 24 hours | false | true |
|
| preemptible | If true, Compute Engine will terminate instances randomly within 24 hours | false | true |
|
||||||
| service_cidr | Must match service_cidr of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
|
| service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
|
||||||
| cluster_domain_suffix | Must match domain suffix of cluster | "cluster.local" | "k8s.example.com" |
|
| cluster_domain_suffix | Must match `cluster_domain_suffix` of cluster | "cluster.local" | "k8s.example.com" |
|
||||||
|
|
||||||
Check the list of valid [machine types](https://cloud.google.com/compute/docs/machine-types).
|
Check the list of valid [machine types](https://cloud.google.com/compute/docs/machine-types).
|
||||||
|
|
||||||
|
|
|
@ -14,6 +14,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
|
||||||
* Kubernetes v1.9.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
|
* Kubernetes v1.9.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
|
||||||
* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
|
* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
|
||||||
* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
|
* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
|
||||||
|
* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/google-cloud/#preemption) (varies by platform)
|
||||||
* Ready for Ingress, Dashboards, Metrics and other optional [addons](addons/overview.md)
|
* Ready for Ingress, Dashboards, Metrics and other optional [addons](addons/overview.md)
|
||||||
* Provided via Terraform Modules
|
* Provided via Terraform Modules
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue