Add new load balancing, TCP/UDP, and firewall docs/diagrams

* Describe kube-apiserver load balancing on each platform
* Describe HTTP/S Ingress load balancing on each platform
* Describe TCP/UDP load balancing apps on each platform
(some clouds don't support UDP)
* Describe firewall customization (e.g. for TCP/UDP apps)
* Update IPv6 status for each platform

docs/addons/ingress.md

@@ -127,7 +127,7 @@ resource "google_dns_record_set" "some-application" {
 !!! note
     Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon to use `hostNetwork: true`.
 
-[^1]: Digital Ocean does offer load balancers. We've opted not to use them to keep the Digital Ocean setup simple and cheap for developers.
+[^1]: DigitalOcean does offer load balancers. We've opted not to use them to keep the DigitalOcean cluster cheap for developers.
 
 ## Google Cloud
 

docs/architecture/aws.md

@@ -1,8 +1,87 @@
 # AWS
 
+## Load Balancing
+
+![Load Balancing](/img/typhoon-aws-load-balancing.png)
+
+### kube-apiserver
+
+A network load balancer (NLB) distributes IPv4 TCP/6443 traffic across a target group of controller nodes with a healthy `kube-apiserver`. Clusters with multiple controllers span zones in a region to tolerate zone outages.
+
+### HTTP/HTTPS Ingress
+
+A network load balancer (NLB) distributes IPv4 TCP/80 and TCP/443 traffic across two target groups of worker nodes with a healthy Ingress controller. Workers span the zones in a region to tolerate zone outages.
+
+The AWS NLB has a DNS alias record (regional) resolving to 3 zonal IPv4 addresses. The alias record is output as `ingress_dns_name` for use in application DNS CNAME records. See [Ingress on AWS](/addons/ingress/#aws).
+
+### TCP Services
+
+Load balance TCP applications by adding a listener and target group. A listener and target group may map different ports (e.g 3333 external, 30333 internal).
+
+```tf
+# Forward TCP traffic to a target group
+resource "aws_lb_listener" "some-app" {
+  load_balancer_arn = module.tempest.nlb_id
+  protocol          = "TCP"
+  port              = "3333"
+
+  default_action {
+    type             = "forward"
+    target_group_arn = aws_lb_target_group.some-app.arn
+  }
+}
+
+# Target group of workers for some-app
+resource "aws_lb_target_group" "some-app" {
+  name        = "some-app"
+  vpc_id      = module.tempest.vpc_id
+  target_type = "instance"
+
+  protocol = "TCP"
+  port     = 3333
+
+  health_check {
+    protocol = "TCP"
+    port     = 30333
+  }
+}
+```
+
+Pass `worker_target_groups` to the cluster to register worker instances into custom target groups.
+
+```tf
+module "tempest" {
+...
+  worker_target_groups = [
+    aws_lb_target_group.some-app.id,
+  ]
+}
+```
+
+Notes:
+
+* AWS NLBs and target groups do not support UDP
+* Global Accelerator does support UDP, but its expensive
+
+## Firewalls
+
+Add firewall rules to the worker security group.
+
+```tf
+resource "aws_security_group_rule" "some-app" {
+  security_group_id = module.tempest.worker_security_groups[0]
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 3333
+  to_port     = 30333
+  cidr_blocks = ["0.0.0.0/0"]
+}
+```
+
 ## IPv6
 
-Status of IPv6 on Typhoon AWS clusters.
+AWS Network Load Balancers do not support `dualstack`.
 
 | IPv6 Feature            | Supported |
 |-------------------------|-----------|
@@ -10,4 +89,3 @@ Status of IPv6 on Typhoon AWS clusters.
 | Node Outbound IPv6      | Yes       |
 | Kubernetes Ingress IPv6 | No        |
 
-* AWS Network Load Balancers do not support `dualstack`.

docs/architecture/azure.md

@@ -1,13 +1,77 @@
 # Azure
 
+## Load Balancing
+
+![Load Balancing](/img/typhoon-azure-load-balancing.png)
+
+### kube-apiserver
+
+A load balancer distributes IPv4 TCP/6443 traffic across a backend address pool of controllers with a healthy `kube-apiserver`. Clusters with multiple controllers use an availability set with 2 fault domains to tolerate hardware failures within Azure.
+
+### HTTP/HTTPS Ingress
+
+A load balancer distributes IPv4 TCP/80 and TCP/443 traffic across a backend address pool of workers with a healthy Ingress controller.
+
+The Azure LB IPv4 address is output as `ingress_static_ipv4` for use in DNS A records. See [Ingress on Azure](/addons/ingress/#azure).
+
+### TCP/UDP Services
+
+Load balance TCP/UDP applications by adding rules to the Azure LB (output). A rule may map different ports (e.g. 3333 external, 30333 internal).
+
+```tf
+# Forward traffic to the worker backend address pool
+resource "azurerm_lb_rule" "some-app-tcp" {
+  resource_group_name = module.ramius.resource_group_name
+
+  name                           = "some-app-tcp"
+  loadbalancer_id                = module.ramius.loadbalancer_id
+  frontend_ip_configuration_name = "ingress"
+
+  protocol                = "Tcp"
+  frontend_port           = 3333
+  backend_port            = 30333
+  backend_address_pool_id = module.ramius.backend_address_pool_id
+  probe_id                = azurerm_lb_probe.some-app.id
+}
+
+# Health check some-app
+resource "azurerm_lb_probe" "some-app" {
+  resource_group_name = module.ramius.resource_group_name
+
+  name            = "some-app"
+  loadbalancer_id = module.ramius.loadbalancer_id
+  protocol        = "Tcp"
+  port            = 30333
+}
+```
+
+## Firewalls
+
+Add firewall rules to the worker security group.
+
+```tf
+resource "azurerm_network_security_rule" "some-app" {
+  resource_group_name = "${module.ramius.resource_group_name}"
+
+  name                        = "some-app"
+  network_security_group_name = module.ramius.worker_security_group_name
+  priority                    = "3001"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "30333"
+  source_address_prefix       = "*"
+  destination_address_prefix  = module.ramius.worker_address_prefix
+}
+```
+
 ## IPv6
 
-Status of IPv6 on Typhoon Azure clusters.
+Azure does not provide public IPv6 addresses at the standard SKU.
 
 | IPv6 Feature            | Supported |
 |-------------------------|-----------|
 | Node IPv6 address       | No        |
 | Node Outbound IPv6      | No        |
 | Kubernetes Ingress IPv6 | No        |
-
-* Azure does not allow reserving a static IPv6 address 

docs/architecture/bare-metal.md

@@ -1,5 +1,21 @@
 # Bare-Metal
 
+## Load Balancing
+
+### kube-apiserver
+
+Load balancing across controller nodes with a healthy `kube-apiserver` is determined by your unique bare-metal environment and its capabilities.
+
+### HTTP/HTTPS Ingress
+
+Load balancing across worker nodes with a healthy Ingress Controller is determined by your unique bare-metal environment and its capabilities.
+
+See the `nginx-ingress` addon to run [Nginx as the Ingress Controller](/addons/ingress/#bare-metal) for bare-metal.
+
+### TCP/UDP Services
+
+Load balancing across worker nodes with TCP/UDP services is determined by your unique bare-metal environment and its capabilities.
+
 ## IPv6
 
 Status of IPv6 on Typhoon bare-metal clusters.

docs/architecture/digitalocean.md

@@ -1,11 +1,92 @@
 # DigitalOcean
 
+## Load Balancing
+
+![Load Balancing](/img/typhoon-digitalocean-load-balancing.png)
+
+### kube-apiserver
+
+DNS A records round-robin[^1] resolve IPv4 TCP/6443 traffic to controller droplets (regardless of whether their `kube-apiserver` is healthy). Clusters with multiple controllers are supported, but round-robin means 1/3 down causes ~1/3 of apiserver requests will fail).
+
+[^1]: DigitalOcean does offer load balancers. We've opted not to use them to keep the DigitalOcean cluster cheap for developers.
+
+### HTTP/HTTPS Ingress
+
+DNS records (A and AAAA) round-robin[^1] resolve the `workers_dns` name (e.g. `nemo-workers.example.com`) to a worker droplet's IPv4 and IPv6 address. This allows running an Ingress controller Daemonset across workers (resolved regardless of whether its the controller is healthy).
+
+The DNS record name is output as `workers_dns` for use in application DNS CNAME records. See [Ingess on DigitalOcean](/addons/ingress/#digital-ocean).
+
+### TCP/UDP Services
+
+DNS records (A and AAAA) round-robin[^1] resolve the `workers_dns` name (e.g. `nemo-workers.example.com`) to a worker droplet's IPv4 and IPv6 address. The DNS record name is output as `workers_dns` for use in application DNS CNAME records.
+
+With round-robin as "load balancing", TCP/UDP services can be served via the same CNAME. Don't forget to add a firewall rule for the application.
+
+### Custom Load Balancer
+
+Add a DigitalOcean load balancer to distribute IPv4 TCP traffic (HTTP/HTTPS Ingress or TCP service) across worker droplets (tagged with `worker_tag`) with a healthy Ingress controller. A load balancer adds cost, but adds redundancy against worker failures (closer to Typhoon clusters on other platforms).
+
+```tf
+resource "digitalocean_loadbalancer" "ingress" {
+  name        = "ingress"
+  region      = "fra1"
+  droplet_tag = module.nemo.worker_tag
+
+  healthcheck {
+    protocol          = "http"
+    port              = "10254"
+    path              = "/healthz"
+    healthy_threshold = 2
+  }
+
+  forwarding_rule {
+    entry_protocol  = "tcp"
+    entry_port      = 80
+    target_protocol = "tcp"
+    target_port     = 80
+  }
+
+  forwarding_rule {
+    entry_protocol  = "tcp"
+    entry_port      = 443
+    target_protocol = "tcp"
+    target_port     = 443
+  }
+
+  forwarding_rule {
+    entry_protocol  = "tcp"
+    entry_port      = 3333
+    target_protocol = "tcp"
+    target_port     = 30300
+  }
+}
+```
+
+Define DNS A records to `digitalocean_loadbalancer.ingress.ip` instead of CNAMEs.
+
+## Firewalls
+
+Add firewall rules matching worker droplets with `worker_tag`.
+
+```tf
+resource "digitalocean_firewall" "some-app" {
+  name = "some-app"
+  tags = [module.nemo.worker_tag]
+  inbound_rule {
+    protocol         = "tcp"
+    port_range       = "30300"
+    source_addresses = ["0.0.0.0/0"]
+  }
+}
+```
+
 ## IPv6
 
-Status of IPv6 on Typhoon DigitalOcean clusters.
+DigitalOcean load balancers do not have an IPv6 address. Resolving individual droplets' IPv6 addresses and using an Ingress controller with `hostNetwork: true` is a possible way to serve IPv6 traffic, if one must.
 
 | IPv6 Feature            | Supported |
 |-------------------------|-----------|
 | Node IPv6 address       | Yes       |
 | Node Outbound IPv6      | Yes       |
 | Kubernetes Ingress IPv6 | Possible  |
+

docs/architecture/google-cloud.md

@@ -1,11 +1,94 @@
 # Google Cloud
 
+## Load Balancing
+
+![Load Balancing](/img/typhoon-gcp-load-balancing.png)
+
+### kube-apiserver
+
+A global forwarding rule (IPv4 anycast) and TCP Proxy distribute IPv4 TCP/443 traffic across a backend service with zonal instance groups of controller(s) with a healthy `kube-apiserver` (TCP/6443). Clusters with multiple controllers span zones in a region to tolerate zone outages.
+
+Notes:
+
+* GCP TCP Proxy limits external port options (e.g. must use 443, not 6443)
+* A regional NLB cannot be used for multi-controller (see [#190](https://github.com/poseidon/typhoon/pull/190))
+
+### HTTP/HTTP Ingress
+
+Global forwarding rules and a TCP Proxy distribute IPv4/IPv6 TCP/80 and TCP/443 traffic across a managed instance group of workers with a healthy Ingress Controller. Workers span zones in a region to tolerate zone outages.
+
+The IPv4 and IPv6 anycast addresses are output as `ingress_static_ipv4` and `ingress_static_ipv6` for use in DNS A and AAAA records. See [Ingress on Google Cloud](/addons/ingress/#google-cloud).
+
+### TCP/UDP Services
+
+Load balance TCP/UDP applications by adding a forwarding rule to the worker target pool (output).
+
+```tf
+# Static IPv4 address for some-app Load Balancing
+resource "google_compute_address" "some-app-ipv4" {
+  name = "some-app-ipv4"
+}
+
+# Forward IPv4 TCP traffic to the target pool
+resource "google_compute_forwarding_rule" "some-app-tcp" {
+  name        = "some-app-tcp"
+  ip_address  = google_compute_address.some-app-ipv4.address
+  ip_protocol = "TCP"
+  port_range  = "3333"
+  target      = module.yavin.worker_target_pool
+}
+
+
+# Forward IPv4 UDP traffic to the target pool
+resource "google_compute_forwarding_rule" "some-app-udp" {
+  name        = "some-app-udp"
+  ip_address  = google_compute_address.some-app-ipv4.address
+  ip_protocol = "UDP"
+  port_range  = "3333"
+  target      = module.yavin.worker_target_pool
+}
+```
+
+Notes:
+
+* GCP Global Load Balancers aren't appropriate for custom TCP/UDP.
+    * Backend Services require a named port corresponding to an instance group (output by Typhoon) port. Typhoon shouldn't accept a list of every TCP/UDP service that may later be hosted on the cluster.
+    * Backend Services don't support UDP (i.e. rules out global load balancers)
+* IPv4 Only: Regional Load Balancers use a regional IPv4 address (e.g. `google_compute_address`), no IPv6.
+* Forward rules don't support differing external and internal ports. Some Ingress controllers (e.g. nginx) can proxy TCP/UDP traffic to achieve this.
+* Worker target pool health checks workers `HTTP:10254/healthz` (i.e. `nginx-ingress`)
+
+## Firewalls
+
+Add firewall rules to the cluster's network.
+
+```tf
+resource "google_compute_firewall" "some-app" {
+  name    = "some-app"
+  network = module.yavin.network_self_link
+
+  allow {
+    protocol = "tcp"
+    ports    = [3333]
+  }
+  
+  allow {
+    protocol = "udp"
+    ports    = [3333]
+  }
+
+  source_ranges = ["0.0.0.0/0"]
+  target_tags   = ["yavin-worker"]
+}
+```
+
 ## IPv6
 
-Status of IPv6 on Typhoon Google Cloud clusters.
+Applications exposed via HTTP/HTTPS Ingress can be served over IPv6.
 
 | IPv6 Feature            | Supported |
 |-------------------------|-----------|
 | Node IPv6 address       | No        |
 | Node Outbound IPv6      | No        |
 | Kubernetes Ingress IPv6 | Yes       |
+