Add new load balancing, TCP/UDP, and firewall docs/diagrams
* Describe kube-apiserver load balancing on each platform * Describe HTTP/S Ingress load balancing on each platform * Describe TCP/UDP load balancing apps on each platform (some clouds don't support UDP) * Describe firewall customization (e.g. for TCP/UDP apps) * Update IPv6 status for each platform
This commit is contained in:
parent
dcd6733649
commit
a12833531e
|
@ -101,7 +101,7 @@ resource "google_dns_record_set" "some-application" {
|
|||
|
||||
## Digital Ocean
|
||||
|
||||
On Digital Ocean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
|
||||
On DigitalOcean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
|
||||
|
||||
Create the Ingress controller daemonset, service, RBAC roles, RBAC bindings, and namespace.
|
||||
|
||||
|
@ -127,7 +127,7 @@ resource "google_dns_record_set" "some-application" {
|
|||
!!! note
|
||||
Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon to use `hostNetwork: true`.
|
||||
|
||||
[^1]: Digital Ocean does offer load balancers. We've opted not to use them to keep the Digital Ocean setup simple and cheap for developers.
|
||||
[^1]: DigitalOcean does offer load balancers. We've opted not to use them to keep the DigitalOcean cluster cheap for developers.
|
||||
|
||||
## Google Cloud
|
||||
|
||||
|
|
|
@ -1,8 +1,87 @@
|
|||
# AWS
|
||||
|
||||
## Load Balancing
|
||||
|
||||
![Load Balancing](/img/typhoon-aws-load-balancing.png)
|
||||
|
||||
### kube-apiserver
|
||||
|
||||
A network load balancer (NLB) distributes IPv4 TCP/6443 traffic across a target group of controller nodes with a healthy `kube-apiserver`. Clusters with multiple controllers span zones in a region to tolerate zone outages.
|
||||
|
||||
### HTTP/HTTPS Ingress
|
||||
|
||||
A network load balancer (NLB) distributes IPv4 TCP/80 and TCP/443 traffic across two target groups of worker nodes with a healthy Ingress controller. Workers span the zones in a region to tolerate zone outages.
|
||||
|
||||
The AWS NLB has a DNS alias record (regional) resolving to 3 zonal IPv4 addresses. The alias record is output as `ingress_dns_name` for use in application DNS CNAME records. See [Ingress on AWS](/addons/ingress/#aws).
|
||||
|
||||
### TCP Services
|
||||
|
||||
Load balance TCP applications by adding a listener and target group. A listener and target group may map different ports (e.g 3333 external, 30333 internal).
|
||||
|
||||
```tf
|
||||
# Forward TCP traffic to a target group
|
||||
resource "aws_lb_listener" "some-app" {
|
||||
load_balancer_arn = module.tempest.nlb_id
|
||||
protocol = "TCP"
|
||||
port = "3333"
|
||||
|
||||
default_action {
|
||||
type = "forward"
|
||||
target_group_arn = aws_lb_target_group.some-app.arn
|
||||
}
|
||||
}
|
||||
|
||||
# Target group of workers for some-app
|
||||
resource "aws_lb_target_group" "some-app" {
|
||||
name = "some-app"
|
||||
vpc_id = module.tempest.vpc_id
|
||||
target_type = "instance"
|
||||
|
||||
protocol = "TCP"
|
||||
port = 3333
|
||||
|
||||
health_check {
|
||||
protocol = "TCP"
|
||||
port = 30333
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Pass `worker_target_groups` to the cluster to register worker instances into custom target groups.
|
||||
|
||||
```tf
|
||||
module "tempest" {
|
||||
...
|
||||
worker_target_groups = [
|
||||
aws_lb_target_group.some-app.id,
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
* AWS NLBs and target groups do not support UDP
|
||||
* Global Accelerator does support UDP, but its expensive
|
||||
|
||||
## Firewalls
|
||||
|
||||
Add firewall rules to the worker security group.
|
||||
|
||||
```tf
|
||||
resource "aws_security_group_rule" "some-app" {
|
||||
security_group_id = module.tempest.worker_security_groups[0]
|
||||
|
||||
type = "ingress"
|
||||
protocol = "tcp"
|
||||
from_port = 3333
|
||||
to_port = 30333
|
||||
cidr_blocks = ["0.0.0.0/0"]
|
||||
}
|
||||
```
|
||||
|
||||
## IPv6
|
||||
|
||||
Status of IPv6 on Typhoon AWS clusters.
|
||||
AWS Network Load Balancers do not support `dualstack`.
|
||||
|
||||
| IPv6 Feature | Supported |
|
||||
|-------------------------|-----------|
|
||||
|
@ -10,4 +89,3 @@ Status of IPv6 on Typhoon AWS clusters.
|
|||
| Node Outbound IPv6 | Yes |
|
||||
| Kubernetes Ingress IPv6 | No |
|
||||
|
||||
* AWS Network Load Balancers do not support `dualstack`.
|
||||
|
|
|
@ -1,13 +1,77 @@
|
|||
# Azure
|
||||
|
||||
## Load Balancing
|
||||
|
||||
![Load Balancing](/img/typhoon-azure-load-balancing.png)
|
||||
|
||||
### kube-apiserver
|
||||
|
||||
A load balancer distributes IPv4 TCP/6443 traffic across a backend address pool of controllers with a healthy `kube-apiserver`. Clusters with multiple controllers use an availability set with 2 fault domains to tolerate hardware failures within Azure.
|
||||
|
||||
### HTTP/HTTPS Ingress
|
||||
|
||||
A load balancer distributes IPv4 TCP/80 and TCP/443 traffic across a backend address pool of workers with a healthy Ingress controller.
|
||||
|
||||
The Azure LB IPv4 address is output as `ingress_static_ipv4` for use in DNS A records. See [Ingress on Azure](/addons/ingress/#azure).
|
||||
|
||||
### TCP/UDP Services
|
||||
|
||||
Load balance TCP/UDP applications by adding rules to the Azure LB (output). A rule may map different ports (e.g. 3333 external, 30333 internal).
|
||||
|
||||
```tf
|
||||
# Forward traffic to the worker backend address pool
|
||||
resource "azurerm_lb_rule" "some-app-tcp" {
|
||||
resource_group_name = module.ramius.resource_group_name
|
||||
|
||||
name = "some-app-tcp"
|
||||
loadbalancer_id = module.ramius.loadbalancer_id
|
||||
frontend_ip_configuration_name = "ingress"
|
||||
|
||||
protocol = "Tcp"
|
||||
frontend_port = 3333
|
||||
backend_port = 30333
|
||||
backend_address_pool_id = module.ramius.backend_address_pool_id
|
||||
probe_id = azurerm_lb_probe.some-app.id
|
||||
}
|
||||
|
||||
# Health check some-app
|
||||
resource "azurerm_lb_probe" "some-app" {
|
||||
resource_group_name = module.ramius.resource_group_name
|
||||
|
||||
name = "some-app"
|
||||
loadbalancer_id = module.ramius.loadbalancer_id
|
||||
protocol = "Tcp"
|
||||
port = 30333
|
||||
}
|
||||
```
|
||||
|
||||
## Firewalls
|
||||
|
||||
Add firewall rules to the worker security group.
|
||||
|
||||
```tf
|
||||
resource "azurerm_network_security_rule" "some-app" {
|
||||
resource_group_name = "${module.ramius.resource_group_name}"
|
||||
|
||||
name = "some-app"
|
||||
network_security_group_name = module.ramius.worker_security_group_name
|
||||
priority = "3001"
|
||||
access = "Allow"
|
||||
direction = "Inbound"
|
||||
protocol = "Tcp"
|
||||
source_port_range = "*"
|
||||
destination_port_range = "30333"
|
||||
source_address_prefix = "*"
|
||||
destination_address_prefix = module.ramius.worker_address_prefix
|
||||
}
|
||||
```
|
||||
|
||||
## IPv6
|
||||
|
||||
Status of IPv6 on Typhoon Azure clusters.
|
||||
Azure does not provide public IPv6 addresses at the standard SKU.
|
||||
|
||||
| IPv6 Feature | Supported |
|
||||
|-------------------------|-----------|
|
||||
| Node IPv6 address | No |
|
||||
| Node Outbound IPv6 | No |
|
||||
| Kubernetes Ingress IPv6 | No |
|
||||
|
||||
* Azure does not allow reserving a static IPv6 address
|
||||
|
|
|
@ -1,5 +1,21 @@
|
|||
# Bare-Metal
|
||||
|
||||
## Load Balancing
|
||||
|
||||
### kube-apiserver
|
||||
|
||||
Load balancing across controller nodes with a healthy `kube-apiserver` is determined by your unique bare-metal environment and its capabilities.
|
||||
|
||||
### HTTP/HTTPS Ingress
|
||||
|
||||
Load balancing across worker nodes with a healthy Ingress Controller is determined by your unique bare-metal environment and its capabilities.
|
||||
|
||||
See the `nginx-ingress` addon to run [Nginx as the Ingress Controller](/addons/ingress/#bare-metal) for bare-metal.
|
||||
|
||||
### TCP/UDP Services
|
||||
|
||||
Load balancing across worker nodes with TCP/UDP services is determined by your unique bare-metal environment and its capabilities.
|
||||
|
||||
## IPv6
|
||||
|
||||
Status of IPv6 on Typhoon bare-metal clusters.
|
||||
|
|
|
@ -1,11 +1,92 @@
|
|||
# DigitalOcean
|
||||
|
||||
## Load Balancing
|
||||
|
||||
![Load Balancing](/img/typhoon-digitalocean-load-balancing.png)
|
||||
|
||||
### kube-apiserver
|
||||
|
||||
DNS A records round-robin[^1] resolve IPv4 TCP/6443 traffic to controller droplets (regardless of whether their `kube-apiserver` is healthy). Clusters with multiple controllers are supported, but round-robin means 1/3 down causes ~1/3 of apiserver requests will fail).
|
||||
|
||||
[^1]: DigitalOcean does offer load balancers. We've opted not to use them to keep the DigitalOcean cluster cheap for developers.
|
||||
|
||||
### HTTP/HTTPS Ingress
|
||||
|
||||
DNS records (A and AAAA) round-robin[^1] resolve the `workers_dns` name (e.g. `nemo-workers.example.com`) to a worker droplet's IPv4 and IPv6 address. This allows running an Ingress controller Daemonset across workers (resolved regardless of whether its the controller is healthy).
|
||||
|
||||
The DNS record name is output as `workers_dns` for use in application DNS CNAME records. See [Ingess on DigitalOcean](/addons/ingress/#digital-ocean).
|
||||
|
||||
### TCP/UDP Services
|
||||
|
||||
DNS records (A and AAAA) round-robin[^1] resolve the `workers_dns` name (e.g. `nemo-workers.example.com`) to a worker droplet's IPv4 and IPv6 address. The DNS record name is output as `workers_dns` for use in application DNS CNAME records.
|
||||
|
||||
With round-robin as "load balancing", TCP/UDP services can be served via the same CNAME. Don't forget to add a firewall rule for the application.
|
||||
|
||||
### Custom Load Balancer
|
||||
|
||||
Add a DigitalOcean load balancer to distribute IPv4 TCP traffic (HTTP/HTTPS Ingress or TCP service) across worker droplets (tagged with `worker_tag`) with a healthy Ingress controller. A load balancer adds cost, but adds redundancy against worker failures (closer to Typhoon clusters on other platforms).
|
||||
|
||||
```tf
|
||||
resource "digitalocean_loadbalancer" "ingress" {
|
||||
name = "ingress"
|
||||
region = "fra1"
|
||||
droplet_tag = module.nemo.worker_tag
|
||||
|
||||
healthcheck {
|
||||
protocol = "http"
|
||||
port = "10254"
|
||||
path = "/healthz"
|
||||
healthy_threshold = 2
|
||||
}
|
||||
|
||||
forwarding_rule {
|
||||
entry_protocol = "tcp"
|
||||
entry_port = 80
|
||||
target_protocol = "tcp"
|
||||
target_port = 80
|
||||
}
|
||||
|
||||
forwarding_rule {
|
||||
entry_protocol = "tcp"
|
||||
entry_port = 443
|
||||
target_protocol = "tcp"
|
||||
target_port = 443
|
||||
}
|
||||
|
||||
forwarding_rule {
|
||||
entry_protocol = "tcp"
|
||||
entry_port = 3333
|
||||
target_protocol = "tcp"
|
||||
target_port = 30300
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Define DNS A records to `digitalocean_loadbalancer.ingress.ip` instead of CNAMEs.
|
||||
|
||||
## Firewalls
|
||||
|
||||
Add firewall rules matching worker droplets with `worker_tag`.
|
||||
|
||||
```tf
|
||||
resource "digitalocean_firewall" "some-app" {
|
||||
name = "some-app"
|
||||
tags = [module.nemo.worker_tag]
|
||||
inbound_rule {
|
||||
protocol = "tcp"
|
||||
port_range = "30300"
|
||||
source_addresses = ["0.0.0.0/0"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## IPv6
|
||||
|
||||
Status of IPv6 on Typhoon DigitalOcean clusters.
|
||||
DigitalOcean load balancers do not have an IPv6 address. Resolving individual droplets' IPv6 addresses and using an Ingress controller with `hostNetwork: true` is a possible way to serve IPv6 traffic, if one must.
|
||||
|
||||
| IPv6 Feature | Supported |
|
||||
|-------------------------|-----------|
|
||||
| Node IPv6 address | Yes |
|
||||
| Node Outbound IPv6 | Yes |
|
||||
| Kubernetes Ingress IPv6 | Possible |
|
||||
|
||||
|
|
|
@ -1,11 +1,94 @@
|
|||
# Google Cloud
|
||||
|
||||
## Load Balancing
|
||||
|
||||
![Load Balancing](/img/typhoon-gcp-load-balancing.png)
|
||||
|
||||
### kube-apiserver
|
||||
|
||||
A global forwarding rule (IPv4 anycast) and TCP Proxy distribute IPv4 TCP/443 traffic across a backend service with zonal instance groups of controller(s) with a healthy `kube-apiserver` (TCP/6443). Clusters with multiple controllers span zones in a region to tolerate zone outages.
|
||||
|
||||
Notes:
|
||||
|
||||
* GCP TCP Proxy limits external port options (e.g. must use 443, not 6443)
|
||||
* A regional NLB cannot be used for multi-controller (see [#190](https://github.com/poseidon/typhoon/pull/190))
|
||||
|
||||
### HTTP/HTTP Ingress
|
||||
|
||||
Global forwarding rules and a TCP Proxy distribute IPv4/IPv6 TCP/80 and TCP/443 traffic across a managed instance group of workers with a healthy Ingress Controller. Workers span zones in a region to tolerate zone outages.
|
||||
|
||||
The IPv4 and IPv6 anycast addresses are output as `ingress_static_ipv4` and `ingress_static_ipv6` for use in DNS A and AAAA records. See [Ingress on Google Cloud](/addons/ingress/#google-cloud).
|
||||
|
||||
### TCP/UDP Services
|
||||
|
||||
Load balance TCP/UDP applications by adding a forwarding rule to the worker target pool (output).
|
||||
|
||||
```tf
|
||||
# Static IPv4 address for some-app Load Balancing
|
||||
resource "google_compute_address" "some-app-ipv4" {
|
||||
name = "some-app-ipv4"
|
||||
}
|
||||
|
||||
# Forward IPv4 TCP traffic to the target pool
|
||||
resource "google_compute_forwarding_rule" "some-app-tcp" {
|
||||
name = "some-app-tcp"
|
||||
ip_address = google_compute_address.some-app-ipv4.address
|
||||
ip_protocol = "TCP"
|
||||
port_range = "3333"
|
||||
target = module.yavin.worker_target_pool
|
||||
}
|
||||
|
||||
|
||||
# Forward IPv4 UDP traffic to the target pool
|
||||
resource "google_compute_forwarding_rule" "some-app-udp" {
|
||||
name = "some-app-udp"
|
||||
ip_address = google_compute_address.some-app-ipv4.address
|
||||
ip_protocol = "UDP"
|
||||
port_range = "3333"
|
||||
target = module.yavin.worker_target_pool
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
* GCP Global Load Balancers aren't appropriate for custom TCP/UDP.
|
||||
* Backend Services require a named port corresponding to an instance group (output by Typhoon) port. Typhoon shouldn't accept a list of every TCP/UDP service that may later be hosted on the cluster.
|
||||
* Backend Services don't support UDP (i.e. rules out global load balancers)
|
||||
* IPv4 Only: Regional Load Balancers use a regional IPv4 address (e.g. `google_compute_address`), no IPv6.
|
||||
* Forward rules don't support differing external and internal ports. Some Ingress controllers (e.g. nginx) can proxy TCP/UDP traffic to achieve this.
|
||||
* Worker target pool health checks workers `HTTP:10254/healthz` (i.e. `nginx-ingress`)
|
||||
|
||||
## Firewalls
|
||||
|
||||
Add firewall rules to the cluster's network.
|
||||
|
||||
```tf
|
||||
resource "google_compute_firewall" "some-app" {
|
||||
name = "some-app"
|
||||
network = module.yavin.network_self_link
|
||||
|
||||
allow {
|
||||
protocol = "tcp"
|
||||
ports = [3333]
|
||||
}
|
||||
|
||||
allow {
|
||||
protocol = "udp"
|
||||
ports = [3333]
|
||||
}
|
||||
|
||||
source_ranges = ["0.0.0.0/0"]
|
||||
target_tags = ["yavin-worker"]
|
||||
}
|
||||
```
|
||||
|
||||
## IPv6
|
||||
|
||||
Status of IPv6 on Typhoon Google Cloud clusters.
|
||||
Applications exposed via HTTP/HTTPS Ingress can be served over IPv6.
|
||||
|
||||
| IPv6 Feature | Supported |
|
||||
|-------------------------|-----------|
|
||||
| Node IPv6 address | No |
|
||||
| Node Outbound IPv6 | No |
|
||||
| Kubernetes Ingress IPv6 | Yes |
|
||||
|
||||
|
|
Binary file not shown.
After Width: | Height: | Size: 38 KiB |
Binary file not shown.
After Width: | Height: | Size: 39 KiB |
Binary file not shown.
After Width: | Height: | Size: 49 KiB |
Binary file not shown.
After Width: | Height: | Size: 69 KiB |
Loading…
Reference in New Issue