typhoon/docs/topics/hardware.md

Hardware

Typhoon ensures certain networking hardware integrates well with bare-metal Kubernetes.

Ubiquiti

Ubiquiti EdgeRouters and EdgeOS work well with bare-metal Kubernetes clusters. Familiarity with EdgeRouter setup and CLI usage is required.

PXE

Ubiquiti EdgeRouters can provide a PXE-enabled network boot environment for client machines.

ISC DHCP

With ISC DHCP, add a subnet parameter to the LAN DHCP server to include an ISC DHCP config file.

configure
show service dhcp-server shared-network-name NAME subnet SUBNET
set service dhcp-server shared-network-name NAME subnet SUBNET subnet-parameters "include "/config/scripts/ipxe.conf";"
commit-confirm

Switch to root (i.e. sudo -i) and write the ISC DHCP config /config/scripts/ipxe.conf. iPXE client machines will chainload to matchbox.example.com, while non-iPXE clients will chainload to undionly.kpxe (requires TFTP).

allow bootp;
allow booting;
next-server ADD_ROUTER_IP_HERE;

if exists user-class and option user-class = "iPXE" {
  filename "http://matchbox.example.com/boot.ipxe";
} else {
  filename "undionly.kpxe";
}

dnsmasq

With dnsmasq for DHCP, add options to chainload PXE clients to iPXE undionly.kpxe (requires TFTP), tag iPXE clients, and chainload iPXE clients to matchbox.example.com.

set service dns forwarding options 'dhcp-userclass=set:ipxe,iPXE'
set service dns forwarding options 'pxe-service=tag:#ipxe,x86PC,PXE chainload to iPXE,undionly.kpxe'
set service dns forwarding options 'pxe-service=tag:ipxe,x86PC,iPXE,http://matchbox.example.com/boot.ipxe'

TFTP

Use dnsmasq as a TFTP server to serve undionly.kpxe. Compiling from source with TLS support is strongly recommended. If you use a pre-compiled copy, you must set download_protocol = "http" in your cluster definition (discouraged).

sudo -i
mkdir /config/tftpboot && cd /config/tftpboot
curl http://boot.ipxe.org/undionly.kpxe -o undionly.kpxe

Add dnsmasq command line options to enable the TFTP file server.

configure
show service dns forwarding
set service dns forwarding options enable-tftp
set service dns forwarding options tftp-root=/config/tftpboot
commit-confirm

DHCP

Assign static IPs to clients with known MAC addresses. This is called a static mapping by EdgeOS. Configure the router with the commands based on region inventory.

configure
show service dhcp-server shared-network
set service dhcp-server shared-network-name LAN subnet SUBNET static-mapping NAME mac-address MACADDR
set service dhcp-server shared-network-name LAN subnet SUBNET static-mapping NAME ip-address 10.0.0.20

DNS

Assign DNS A records to nodes as options to dnsmasq.

configure
set service dns forwarding options host-record=node.example.com,10.0.0.20

Restart dnsmasq.

sudo /etc/init.d/dnsmasq restart

Configure queries for *.svc.cluster.local to be forwarded to the Kubernetes coredns service IP to allow hosts to resolve cluster-local Kubernetes names.

configure
show service dns forwarding
set service dns forwarding options server=/svc.cluster.local/10.3.0.10
commit-confirm

Kubernetes Services

Add static routes for the Kubernetes IPv4 service range to Kubernetes node(s) so hosts can route to Kubernetes services (default: 10.3.0.0/16).

configure
show protocols static route
set protocols static route 10.3.0.0/16 next-hop NODE_IP
...
commit-confirm

!!! note Adding multiple next-hop nodes provides equal-cost multi-path (ECMP) routing. EdgeOS v2.0+ is required. The kernel in prior versions used flow-hash to balanced packets, whereas with v2.0, round-robin sessions are used.

Port Forwarding

Expose the Ingress Controller by adding port-forward rules that DNAT a port on the router's WAN interface to an internal IP and port. By convention, a public Ingress controller is assigned a fixed service IP (e.g. 10.3.0.12).

configure
set port-forward wan-interface eth0
set port-forward lan-interface eth1
set port-forward auto-firewall enable
set port-forward hairpin-nat enable
set port-forward rule 1 description 'ingress http'
set port-forward rule 1 forward-to address 10.3.0.12
set port-forward rule 1 forward-to port 80
set port-forward rule 1 original-port 80
set port-forward rule 1 protocol tcp_udp
set port-forward rule 2 description 'ingress https'
set port-forward rule 2 forward-to address 10.3.0.12
set port-forward rule 2 forward-to port 443
set port-forward rule 2 original-port 443
set port-forward rule 2 protocol tcp_udp
commit-confirm

Web UI

The web UI is often accessible from the LAN on ports 80/443 by default. Edit the ports to 8080 and 4443 to avoid a conflict.

configure
show service gui
set service gui http-port 8080
set service gui https-port 4443
commit-confirm

BGP

Add the EdgeRouter as a global BGP peer for nodes in a Kubernetes cluster (requires Calico). Neighbors will exchange podCIDR routes and individual pods will become routable on the LAN.

Configure node(s) as BGP neighbors.

show protocols bgp 1
set protocols bgp 1 parameters router-id LAN_IP
set protocols bgp 1 neighbor NODE1_IP remote-as 64512
set protocols bgp 1 neighbor NODE2_IP remote-as 64512
set protocols bgp 1 neighbor NODE3_IP remote-as 64512

View the neighbors and exchanged routes.

show ip bgp neighbors
show ip route bgp

Be sure to register the peer by creating a Calico BGPPeer CRD with kubectl apply.

apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
  name: NAME
spec:
  peerIP: LAN_IP
  asNumber: 64512