Add new load balancing, TCP/UDP, and firewall docs/diagrams

* Describe kube-apiserver load balancing on each platform
* Describe HTTP/S Ingress load balancing on each platform
* Describe TCP/UDP load balancing apps on each platform
(some clouds don't support UDP)
* Describe firewall customization (e.g. for TCP/UDP apps)
* Update IPv6 status for each platform
This commit is contained in:
Dalton Hubble 2019-07-29 22:41:10 -07:00
parent dcd6733649
commit a12833531e
10 changed files with 331 additions and 9 deletions

View File

@ -101,7 +101,7 @@ resource "google_dns_record_set" "some-application" {
## Digital Ocean
On Digital Ocean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
On DigitalOcean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
Create the Ingress controller daemonset, service, RBAC roles, RBAC bindings, and namespace.
@ -127,7 +127,7 @@ resource "google_dns_record_set" "some-application" {
!!! note
Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon to use `hostNetwork: true`.
[^1]: Digital Ocean does offer load balancers. We've opted not to use them to keep the Digital Ocean setup simple and cheap for developers.
[^1]: DigitalOcean does offer load balancers. We've opted not to use them to keep the DigitalOcean cluster cheap for developers.
## Google Cloud

View File

@ -1,8 +1,87 @@
# AWS
## Load Balancing
![Load Balancing](/img/typhoon-aws-load-balancing.png)
### kube-apiserver
A network load balancer (NLB) distributes IPv4 TCP/6443 traffic across a target group of controller nodes with a healthy `kube-apiserver`. Clusters with multiple controllers span zones in a region to tolerate zone outages.
### HTTP/HTTPS Ingress
A network load balancer (NLB) distributes IPv4 TCP/80 and TCP/443 traffic across two target groups of worker nodes with a healthy Ingress controller. Workers span the zones in a region to tolerate zone outages.
The AWS NLB has a DNS alias record (regional) resolving to 3 zonal IPv4 addresses. The alias record is output as `ingress_dns_name` for use in application DNS CNAME records. See [Ingress on AWS](/addons/ingress/#aws).
### TCP Services
Load balance TCP applications by adding a listener and target group. A listener and target group may map different ports (e.g 3333 external, 30333 internal).
```tf
# Forward TCP traffic to a target group
resource "aws_lb_listener" "some-app" {
load_balancer_arn = module.tempest.nlb_id
protocol = "TCP"
port = "3333"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.some-app.arn
}
}
# Target group of workers for some-app
resource "aws_lb_target_group" "some-app" {
name = "some-app"
vpc_id = module.tempest.vpc_id
target_type = "instance"
protocol = "TCP"
port = 3333
health_check {
protocol = "TCP"
port = 30333
}
}
```
Pass `worker_target_groups` to the cluster to register worker instances into custom target groups.
```tf
module "tempest" {
...
worker_target_groups = [
aws_lb_target_group.some-app.id,
]
}
```
Notes:
* AWS NLBs and target groups do not support UDP
* Global Accelerator does support UDP, but its expensive
## Firewalls
Add firewall rules to the worker security group.
```tf
resource "aws_security_group_rule" "some-app" {
security_group_id = module.tempest.worker_security_groups[0]
type = "ingress"
protocol = "tcp"
from_port = 3333
to_port = 30333
cidr_blocks = ["0.0.0.0/0"]
}
```
## IPv6
Status of IPv6 on Typhoon AWS clusters.
AWS Network Load Balancers do not support `dualstack`.
| IPv6 Feature | Supported |
|-------------------------|-----------|
@ -10,4 +89,3 @@ Status of IPv6 on Typhoon AWS clusters.
| Node Outbound IPv6 | Yes |
| Kubernetes Ingress IPv6 | No |
* AWS Network Load Balancers do not support `dualstack`.

View File

@ -1,13 +1,77 @@
# Azure
## Load Balancing
![Load Balancing](/img/typhoon-azure-load-balancing.png)
### kube-apiserver
A load balancer distributes IPv4 TCP/6443 traffic across a backend address pool of controllers with a healthy `kube-apiserver`. Clusters with multiple controllers use an availability set with 2 fault domains to tolerate hardware failures within Azure.
### HTTP/HTTPS Ingress
A load balancer distributes IPv4 TCP/80 and TCP/443 traffic across a backend address pool of workers with a healthy Ingress controller.
The Azure LB IPv4 address is output as `ingress_static_ipv4` for use in DNS A records. See [Ingress on Azure](/addons/ingress/#azure).
### TCP/UDP Services
Load balance TCP/UDP applications by adding rules to the Azure LB (output). A rule may map different ports (e.g. 3333 external, 30333 internal).
```tf
# Forward traffic to the worker backend address pool
resource "azurerm_lb_rule" "some-app-tcp" {
resource_group_name = module.ramius.resource_group_name
name = "some-app-tcp"
loadbalancer_id = module.ramius.loadbalancer_id
frontend_ip_configuration_name = "ingress"
protocol = "Tcp"
frontend_port = 3333
backend_port = 30333
backend_address_pool_id = module.ramius.backend_address_pool_id
probe_id = azurerm_lb_probe.some-app.id
}
# Health check some-app
resource "azurerm_lb_probe" "some-app" {
resource_group_name = module.ramius.resource_group_name
name = "some-app"
loadbalancer_id = module.ramius.loadbalancer_id
protocol = "Tcp"
port = 30333
}
```
## Firewalls
Add firewall rules to the worker security group.
```tf
resource "azurerm_network_security_rule" "some-app" {
resource_group_name = "${module.ramius.resource_group_name}"
name = "some-app"
network_security_group_name = module.ramius.worker_security_group_name
priority = "3001"
access = "Allow"
direction = "Inbound"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "30333"
source_address_prefix = "*"
destination_address_prefix = module.ramius.worker_address_prefix
}
```
## IPv6
Status of IPv6 on Typhoon Azure clusters.
Azure does not provide public IPv6 addresses at the standard SKU.
| IPv6 Feature | Supported |
|-------------------------|-----------|
| Node IPv6 address | No |
| Node Outbound IPv6 | No |
| Kubernetes Ingress IPv6 | No |
* Azure does not allow reserving a static IPv6 address

View File

@ -1,5 +1,21 @@
# Bare-Metal
## Load Balancing
### kube-apiserver
Load balancing across controller nodes with a healthy `kube-apiserver` is determined by your unique bare-metal environment and its capabilities.
### HTTP/HTTPS Ingress
Load balancing across worker nodes with a healthy Ingress Controller is determined by your unique bare-metal environment and its capabilities.
See the `nginx-ingress` addon to run [Nginx as the Ingress Controller](/addons/ingress/#bare-metal) for bare-metal.
### TCP/UDP Services
Load balancing across worker nodes with TCP/UDP services is determined by your unique bare-metal environment and its capabilities.
## IPv6
Status of IPv6 on Typhoon bare-metal clusters.

View File

@ -1,11 +1,92 @@
# DigitalOcean
## Load Balancing
![Load Balancing](/img/typhoon-digitalocean-load-balancing.png)
### kube-apiserver
DNS A records round-robin[^1] resolve IPv4 TCP/6443 traffic to controller droplets (regardless of whether their `kube-apiserver` is healthy). Clusters with multiple controllers are supported, but round-robin means 1/3 down causes ~1/3 of apiserver requests will fail).
[^1]: DigitalOcean does offer load balancers. We've opted not to use them to keep the DigitalOcean cluster cheap for developers.
### HTTP/HTTPS Ingress
DNS records (A and AAAA) round-robin[^1] resolve the `workers_dns` name (e.g. `nemo-workers.example.com`) to a worker droplet's IPv4 and IPv6 address. This allows running an Ingress controller Daemonset across workers (resolved regardless of whether its the controller is healthy).
The DNS record name is output as `workers_dns` for use in application DNS CNAME records. See [Ingess on DigitalOcean](/addons/ingress/#digital-ocean).
### TCP/UDP Services
DNS records (A and AAAA) round-robin[^1] resolve the `workers_dns` name (e.g. `nemo-workers.example.com`) to a worker droplet's IPv4 and IPv6 address. The DNS record name is output as `workers_dns` for use in application DNS CNAME records.
With round-robin as "load balancing", TCP/UDP services can be served via the same CNAME. Don't forget to add a firewall rule for the application.
### Custom Load Balancer
Add a DigitalOcean load balancer to distribute IPv4 TCP traffic (HTTP/HTTPS Ingress or TCP service) across worker droplets (tagged with `worker_tag`) with a healthy Ingress controller. A load balancer adds cost, but adds redundancy against worker failures (closer to Typhoon clusters on other platforms).
```tf
resource "digitalocean_loadbalancer" "ingress" {
name = "ingress"
region = "fra1"
droplet_tag = module.nemo.worker_tag
healthcheck {
protocol = "http"
port = "10254"
path = "/healthz"
healthy_threshold = 2
}
forwarding_rule {
entry_protocol = "tcp"
entry_port = 80
target_protocol = "tcp"
target_port = 80
}
forwarding_rule {
entry_protocol = "tcp"
entry_port = 443
target_protocol = "tcp"
target_port = 443
}
forwarding_rule {
entry_protocol = "tcp"
entry_port = 3333
target_protocol = "tcp"
target_port = 30300
}
}
```
Define DNS A records to `digitalocean_loadbalancer.ingress.ip` instead of CNAMEs.
## Firewalls
Add firewall rules matching worker droplets with `worker_tag`.
```tf
resource "digitalocean_firewall" "some-app" {
name = "some-app"
tags = [module.nemo.worker_tag]
inbound_rule {
protocol = "tcp"
port_range = "30300"
source_addresses = ["0.0.0.0/0"]
}
}
```
## IPv6
Status of IPv6 on Typhoon DigitalOcean clusters.
DigitalOcean load balancers do not have an IPv6 address. Resolving individual droplets' IPv6 addresses and using an Ingress controller with `hostNetwork: true` is a possible way to serve IPv6 traffic, if one must.
| IPv6 Feature | Supported |
|-------------------------|-----------|
| Node IPv6 address | Yes |
| Node Outbound IPv6 | Yes |
| Kubernetes Ingress IPv6 | Possible |

View File

@ -1,11 +1,94 @@
# Google Cloud
## Load Balancing
![Load Balancing](/img/typhoon-gcp-load-balancing.png)
### kube-apiserver
A global forwarding rule (IPv4 anycast) and TCP Proxy distribute IPv4 TCP/443 traffic across a backend service with zonal instance groups of controller(s) with a healthy `kube-apiserver` (TCP/6443). Clusters with multiple controllers span zones in a region to tolerate zone outages.
Notes:
* GCP TCP Proxy limits external port options (e.g. must use 443, not 6443)
* A regional NLB cannot be used for multi-controller (see [#190](https://github.com/poseidon/typhoon/pull/190))
### HTTP/HTTP Ingress
Global forwarding rules and a TCP Proxy distribute IPv4/IPv6 TCP/80 and TCP/443 traffic across a managed instance group of workers with a healthy Ingress Controller. Workers span zones in a region to tolerate zone outages.
The IPv4 and IPv6 anycast addresses are output as `ingress_static_ipv4` and `ingress_static_ipv6` for use in DNS A and AAAA records. See [Ingress on Google Cloud](/addons/ingress/#google-cloud).
### TCP/UDP Services
Load balance TCP/UDP applications by adding a forwarding rule to the worker target pool (output).
```tf
# Static IPv4 address for some-app Load Balancing
resource "google_compute_address" "some-app-ipv4" {
name = "some-app-ipv4"
}
# Forward IPv4 TCP traffic to the target pool
resource "google_compute_forwarding_rule" "some-app-tcp" {
name = "some-app-tcp"
ip_address = google_compute_address.some-app-ipv4.address
ip_protocol = "TCP"
port_range = "3333"
target = module.yavin.worker_target_pool
}
# Forward IPv4 UDP traffic to the target pool
resource "google_compute_forwarding_rule" "some-app-udp" {
name = "some-app-udp"
ip_address = google_compute_address.some-app-ipv4.address
ip_protocol = "UDP"
port_range = "3333"
target = module.yavin.worker_target_pool
}
```
Notes:
* GCP Global Load Balancers aren't appropriate for custom TCP/UDP.
* Backend Services require a named port corresponding to an instance group (output by Typhoon) port. Typhoon shouldn't accept a list of every TCP/UDP service that may later be hosted on the cluster.
* Backend Services don't support UDP (i.e. rules out global load balancers)
* IPv4 Only: Regional Load Balancers use a regional IPv4 address (e.g. `google_compute_address`), no IPv6.
* Forward rules don't support differing external and internal ports. Some Ingress controllers (e.g. nginx) can proxy TCP/UDP traffic to achieve this.
* Worker target pool health checks workers `HTTP:10254/healthz` (i.e. `nginx-ingress`)
## Firewalls
Add firewall rules to the cluster's network.
```tf
resource "google_compute_firewall" "some-app" {
name = "some-app"
network = module.yavin.network_self_link
allow {
protocol = "tcp"
ports = [3333]
}
allow {
protocol = "udp"
ports = [3333]
}
source_ranges = ["0.0.0.0/0"]
target_tags = ["yavin-worker"]
}
```
## IPv6
Status of IPv6 on Typhoon Google Cloud clusters.
Applications exposed via HTTP/HTTPS Ingress can be served over IPv6.
| IPv6 Feature | Supported |
|-------------------------|-----------|
| Node IPv6 address | No |
| Node Outbound IPv6 | No |
| Kubernetes Ingress IPv6 | Yes |

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB