Add new load balancing, TCP/UDP, and firewall docs/diagrams

* Describe kube-apiserver load balancing on each platform
* Describe HTTP/S Ingress load balancing on each platform
* Describe TCP/UDP load balancing apps on each platform
(some clouds don't support UDP)
* Describe firewall customization (e.g. for TCP/UDP apps)
* Update IPv6 status for each platform

docs/addons/ingress.md

@@ -101,7 +101,7 @@ resource "google_dns_record_set" "some-application" {
 
 ## Digital Ocean
 
-On Digital Ocean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
+On DigitalOcean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
 
 Create the Ingress controller daemonset, service, RBAC roles, RBAC bindings, and namespace.
 
@@ -127,7 +127,7 @@ resource "google_dns_record_set" "some-application" {
 !!! note
     Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon to use `hostNetwork: true`.
 
-[^1]: Digital Ocean does offer load balancers. We've opted not to use them to keep the Digital Ocean setup simple and cheap for developers.
+[^1]: DigitalOcean does offer load balancers. We've opted not to use them to keep the DigitalOcean cluster cheap for developers.
 
 ## Google Cloud
 

docs/architecture/aws.md

@@ -1,8 +1,87 @@
 # AWS
 
+## Load Balancing
+
+![Load Balancing](/img/typhoon-aws-load-balancing.png)
+
+### kube-apiserver
+
+A network load balancer (NLB) distributes IPv4 TCP/6443 traffic across a target group of controller nodes with a healthy `kube-apiserver`. Clusters with multiple controllers span zones in a region to tolerate zone outages.
+
+### HTTP/HTTPS Ingress
+
+A network load balancer (NLB) distributes IPv4 TCP/80 and TCP/443 traffic across two target groups of worker nodes with a healthy Ingress controller. Workers span the zones in a region to tolerate zone outages.
+
+The AWS NLB has a DNS alias record (regional) resolving to 3 zonal IPv4 addresses. The alias record is output as `ingress_dns_name` for use in application DNS CNAME records. See [Ingress on AWS](/addons/ingress/#aws).
+
+### TCP Services
+
+Load balance TCP applications by adding a listener and target group. A listener and target group may map different ports (e.g 3333 external, 30333 internal).
+
+```tf
+# Forward TCP traffic to a target group
+resource "aws_lb_listener" "some-app" {
+  load_balancer_arn = module.tempest.nlb_id
+  protocol          = "TCP"
+  port              = "3333"
+
+  default_action {
+    type             = "forward"
+    target_group_arn = aws_lb_target_group.some-app.arn
+  }
+}
+
+# Target group of workers for some-app
+resource "aws_lb_target_group" "some-app" {
+  name        = "some-app"
+  vpc_id      = module.tempest.vpc_id
+  target_type = "instance"
+
+  protocol = "TCP"
+  port     = 3333
+
+  health_check {
+    protocol = "TCP"
+    port     = 30333
+  }
+}
+```
+
+Pass `worker_target_groups` to the cluster to register worker instances into custom target groups.
+
+```tf
+module "tempest" {
+...
+  worker_target_groups = [
+    aws_lb_target_group.some-app.id,
+  ]
+}
+```
+
+Notes:
+
+* AWS NLBs and target groups do not support UDP
+* Global Accelerator does support UDP, but its expensive
+
+## Firewalls
+
+Add firewall rules to the worker security group.
+
+```tf
+resource "aws_security_group_rule" "some-app" {
+  security_group_id = module.tempest.worker_security_groups[0]
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 3333
+  to_port     = 30333
+  cidr_blocks = ["0.0.0.0/0"]
+}
+```
+
 ## IPv6
 
-Status of IPv6 on Typhoon AWS clusters.
+AWS Network Load Balancers do not support `dualstack`.
 
 | IPv6 Feature            | Supported |
 |-------------------------|-----------|
@@ -10,4 +89,3 @@ Status of IPv6 on Typhoon AWS clusters.
 | Node Outbound IPv6      | Yes       |
 | Kubernetes Ingress IPv6 | No        |
 
-* AWS Network Load Balancers do not support `dualstack`.

docs/architecture/azure.md

@@ -1,13 +1,77 @@
 # Azure
 
+## Load Balancing
+
+![Load Balancing](/img/typhoon-azure-load-balancing.png)
+
+### kube-apiserver
+
+A load balancer distributes IPv4 TCP/6443 traffic across a backend address pool of controllers with a healthy `kube-apiserver`. Clusters with multiple controllers use an availability set with 2 fault domains to tolerate hardware failures within Azure.
+
+### HTTP/HTTPS Ingress
+
+A load balancer distributes IPv4 TCP/80 and TCP/443 traffic across a backend address pool of workers with a healthy Ingress controller.
+
+The Azure LB IPv4 address is output as `ingress_static_ipv4` for use in DNS A records. See [Ingress on Azure](/addons/ingress/#azure).
+
+### TCP/UDP Services
+
+Load balance TCP/UDP applications by adding rules to the Azure LB (output). A rule may map different ports (e.g. 3333 external, 30333 internal).
+
+```tf
+# Forward traffic to the worker backend address pool
+resource "azurerm_lb_rule" "some-app-tcp" {
+  resource_group_name = module.ramius.resource_group_name
+
+  name                           = "some-app-tcp"
+  loadbalancer_id                = module.ramius.loadbalancer_id
+  frontend_ip_configuration_name = "ingress"
+
+  protocol                = "Tcp"
+  frontend_port           = 3333
+  backend_port            = 30333
+  backend_address_pool_id = module.ramius.backend_address_pool_id
+  probe_id                = azurerm_lb_probe.some-app.id
+}
+
+# Health check some-app
+resource "azurerm_lb_probe" "some-app" {
+  resource_group_name = module.ramius.resource_group_name
+
+  name            = "some-app"
+  loadbalancer_id = module.ramius.loadbalancer_id
+  protocol        = "Tcp"
+  port            = 30333
+}
+```
+
+## Firewalls
+
+Add firewall rules to the worker security group.
+
+```tf
+resource "azurerm_network_security_rule" "some-app" {
+  resource_group_name = "${module.ramius.resource_group_name}"
+
+  name                        = "some-app"
+  network_security_group_name = module.ramius.worker_security_group_name
+  priority                    = "3001"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "30333"
+  source_address_prefix       = "*"
+  destination_address_prefix  = module.ramius.worker_address_prefix
+}
+```
+
 ## IPv6
 
-Status of IPv6 on Typhoon Azure clusters.
+Azure does not provide public IPv6 addresses at the standard SKU.
 
 | IPv6 Feature            | Supported |
 |-------------------------|-----------|
 | Node IPv6 address       | No        |
 | Node Outbound IPv6      | No        |
 | Kubernetes Ingress IPv6 | No        |
-
-* Azure does not allow reserving a static IPv6 address 

docs/architecture/bare-metal.md

@@ -1,5 +1,21 @@
 # Bare-Metal
 
+## Load Balancing
+
+### kube-apiserver
+
+Load balancing across controller nodes with a healthy `kube-apiserver` is determined by your unique bare-metal environment and its capabilities.
+
+### HTTP/HTTPS Ingress
+
+Load balancing across worker nodes with a healthy Ingress Controller is determined by your unique bare-metal environment and its capabilities.
+
+See the `nginx-ingress` addon to run [Nginx as the Ingress Controller](/addons/ingress/#bare-metal) for bare-metal.
+
+### TCP/UDP Services
+
+Load balancing across worker nodes with TCP/UDP services is determined by your unique bare-metal environment and its capabilities.
+
 ## IPv6
 
 Status of IPv6 on Typhoon bare-metal clusters.

docs/architecture/digitalocean.md

@@ -1,11 +1,92 @@
 # DigitalOcean
 
+## Load Balancing
+
+![Load Balancing](/img/typhoon-digitalocean-load-balancing.png)
+
+### kube-apiserver
+
+DNS A records round-robin[^1] resolve IPv4 TCP/6443 traffic to controller droplets (regardless of whether their `kube-apiserver` is healthy). Clusters with multiple controllers are supported, but round-robin means 1/3 down causes ~1/3 of apiserver requests will fail).
+
+[^1]: DigitalOcean does offer load balancers. We've opted not to use them to keep the DigitalOcean cluster cheap for developers.
+
+### HTTP/HTTPS Ingress
+
+DNS records (A and AAAA) round-robin[^1] resolve the `workers_dns` name (e.g. `nemo-workers.example.com`) to a worker droplet's IPv4 and IPv6 address. This allows running an Ingress controller Daemonset across workers (resolved regardless of whether its the controller is healthy).
+
+The DNS record name is output as `workers_dns` for use in application DNS CNAME records. See [Ingess on DigitalOcean](/addons/ingress/#digital-ocean).
+
+### TCP/UDP Services
+
+DNS records (A and AAAA) round-robin[^1] resolve the `workers_dns` name (e.g. `nemo-workers.example.com`) to a worker droplet's IPv4 and IPv6 address. The DNS record name is output as `workers_dns` for use in application DNS CNAME records.
+
+With round-robin as "load balancing", TCP/UDP services can be served via the same CNAME. Don't forget to add a firewall rule for the application.
+
+### Custom Load Balancer
+
+Add a DigitalOcean load balancer to distribute IPv4 TCP traffic (HTTP/HTTPS Ingress or TCP service) across worker droplets (tagged with `worker_tag`) with a healthy Ingress controller. A load balancer adds cost, but adds redundancy against worker failures (closer to Typhoon clusters on other platforms).
+
+```tf
+resource "digitalocean_loadbalancer" "ingress" {
+  name        = "ingress"
+  region      = "fra1"
+  droplet_tag = module.nemo.worker_tag
+
+  healthcheck {
+    protocol          = "http"
+    port              = "10254"
+    path              = "/healthz"
+    healthy_threshold = 2
+  }
+
+  forwarding_rule {
+    entry_protocol  = "tcp"
+    entry_port      = 80
+    target_protocol = "tcp"
+    target_port     = 80
+  }
+
+  forwarding_rule {
+    entry_protocol  = "tcp"
+    entry_port      = 443
+    target_protocol = "tcp"
+    target_port     = 443
+  }
+
+  forwarding_rule {
+    entry_protocol  = "tcp"
+    entry_port      = 3333
+    target_protocol = "tcp"
+    target_port     = 30300
+  }
+}
+```
+
+Define DNS A records to `digitalocean_loadbalancer.ingress.ip` instead of CNAMEs.
+
+## Firewalls
+
+Add firewall rules matching worker droplets with `worker_tag`.
+
+```tf
+resource "digitalocean_firewall" "some-app" {
+  name = "some-app"
+  tags = [module.nemo.worker_tag]
+  inbound_rule {
+    protocol         = "tcp"
+    port_range       = "30300"
+    source_addresses = ["0.0.0.0/0"]
+  }
+}
+```
+
 ## IPv6
 
-Status of IPv6 on Typhoon DigitalOcean clusters.
+DigitalOcean load balancers do not have an IPv6 address. Resolving individual droplets' IPv6 addresses and using an Ingress controller with `hostNetwork: true` is a possible way to serve IPv6 traffic, if one must.
 
 | IPv6 Feature            | Supported |
 |-------------------------|-----------|
 | Node IPv6 address       | Yes       |
 | Node Outbound IPv6      | Yes       |
 | Kubernetes Ingress IPv6 | Possible  |
+

docs/architecture/google-cloud.md

@@ -1,11 +1,94 @@
 # Google Cloud
 
+## Load Balancing
+
+![Load Balancing](/img/typhoon-gcp-load-balancing.png)
+
+### kube-apiserver
+
+A global forwarding rule (IPv4 anycast) and TCP Proxy distribute IPv4 TCP/443 traffic across a backend service with zonal instance groups of controller(s) with a healthy `kube-apiserver` (TCP/6443). Clusters with multiple controllers span zones in a region to tolerate zone outages.
+
+Notes:
+
+* GCP TCP Proxy limits external port options (e.g. must use 443, not 6443)
+* A regional NLB cannot be used for multi-controller (see [#190](https://github.com/poseidon/typhoon/pull/190))
+
+### HTTP/HTTP Ingress
+
+Global forwarding rules and a TCP Proxy distribute IPv4/IPv6 TCP/80 and TCP/443 traffic across a managed instance group of workers with a healthy Ingress Controller. Workers span zones in a region to tolerate zone outages.
+
+The IPv4 and IPv6 anycast addresses are output as `ingress_static_ipv4` and `ingress_static_ipv6` for use in DNS A and AAAA records. See [Ingress on Google Cloud](/addons/ingress/#google-cloud).
+
+### TCP/UDP Services
+
+Load balance TCP/UDP applications by adding a forwarding rule to the worker target pool (output).
+
+```tf
+# Static IPv4 address for some-app Load Balancing
+resource "google_compute_address" "some-app-ipv4" {
+  name = "some-app-ipv4"
+}
+
+# Forward IPv4 TCP traffic to the target pool
+resource "google_compute_forwarding_rule" "some-app-tcp" {
+  name        = "some-app-tcp"
+  ip_address  = google_compute_address.some-app-ipv4.address
+  ip_protocol = "TCP"
+  port_range  = "3333"
+  target      = module.yavin.worker_target_pool
+}
+
+
+# Forward IPv4 UDP traffic to the target pool
+resource "google_compute_forwarding_rule" "some-app-udp" {
+  name        = "some-app-udp"
+  ip_address  = google_compute_address.some-app-ipv4.address
+  ip_protocol = "UDP"
+  port_range  = "3333"
+  target      = module.yavin.worker_target_pool
+}
+```
+
+Notes:
+
+* GCP Global Load Balancers aren't appropriate for custom TCP/UDP.
+    * Backend Services require a named port corresponding to an instance group (output by Typhoon) port. Typhoon shouldn't accept a list of every TCP/UDP service that may later be hosted on the cluster.
+    * Backend Services don't support UDP (i.e. rules out global load balancers)
+* IPv4 Only: Regional Load Balancers use a regional IPv4 address (e.g. `google_compute_address`), no IPv6.
+* Forward rules don't support differing external and internal ports. Some Ingress controllers (e.g. nginx) can proxy TCP/UDP traffic to achieve this.
+* Worker target pool health checks workers `HTTP:10254/healthz` (i.e. `nginx-ingress`)
+
+## Firewalls
+
+Add firewall rules to the cluster's network.
+
+```tf
+resource "google_compute_firewall" "some-app" {
+  name    = "some-app"
+  network = module.yavin.network_self_link
+
+  allow {
+    protocol = "tcp"
+    ports    = [3333]
+  }
+  
+  allow {
+    protocol = "udp"
+    ports    = [3333]
+  }
+
+  source_ranges = ["0.0.0.0/0"]
+  target_tags   = ["yavin-worker"]
+}
+```
+
 ## IPv6
 
-Status of IPv6 on Typhoon Google Cloud clusters.
+Applications exposed via HTTP/HTTPS Ingress can be served over IPv6.
 
 | IPv6 Feature            | Supported |
 |-------------------------|-----------|
 | Node IPv6 address       | No        |
 | Node Outbound IPv6      | No        |
 | Kubernetes Ingress IPv6 | Yes       |
+