typhoon/docs/advanced/arm64.md

# ARM64

Typhoon has experimental support for ARM64 on AWS, with Fedora CoreOS or Flatcar Linux. Clusters can be created with ARM64 controller and worker nodes. Or worker pools of ARM64 nodes can be attached to an AMD64 cluster to create a hybrid/mixed architecture cluster.

!!! note
    Currently, CNI networking must be set to `flannel` or `cilium`.

## Cluster

Create a cluster with ARM64 controller and worker nodes. Container workloads must be `arm64` compatible and use `arm64` container images.

=== "Fedora CoreOS Cluster (arm64)"

    ```tf
    module "gravitas" {
      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.4"

      # AWS
      cluster_name = "gravitas"
      dns_zone     = "aws.example.com"
      dns_zone_id  = "Z3PAABBCFAKEC0"

      # configuration
      ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

      # optional
      arch         = "arm64"
      networking   = "cilium"
      worker_count = 2
      worker_price = "0.0168"

      controller_type = "t4g.small"
      worker_type     = "t4g.small"
    }
    ```

=== "Flatcar Linux Cluster (arm64)"

    ```tf
    module "gravitas" {
      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.23.4"

      # AWS
      cluster_name = "gravitas"
      dns_zone     = "aws.example.com"
      dns_zone_id  = "Z3PAABBCFAKEC0"

      # configuration
      ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

      # optional
      arch         = "arm64"
      networking   = "cilium"
      worker_count = 2
      worker_price = "0.0168"

      controller_type = "t4g.small"
      worker_type     = "t4g.small"
    }
    ```

Verify the cluster has only arm64 (`aarch64`) nodes. For Flatcar Linux, describe nodes.

```
$ kubectl get nodes -o wide
NAME             STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION            CONTAINER-RUNTIME
ip-10-0-21-119   Ready    <none>   77s   v1.23.4   10.0.21.119   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
ip-10-0-32-166   Ready    <none>   80s   v1.23.4   10.0.32.166   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
ip-10-0-5-79     Ready    <none>   77s   v1.23.4   10.0.5.79     <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
```

## Hybrid

Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [worker pool](worker-pools.md#aws) with ARM64 workers. Optional taints are added to aid in scheduling.

=== "FCOS Cluster"

    ```tf
    module "gravitas" {
      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.4"

      # AWS
      cluster_name = "gravitas"
      dns_zone     = "aws.example.com"
      dns_zone_id  = "Z3PAABBCFAKEC0"

      # configuration
      ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

      # optional
      networking   = "cilium"
      worker_count = 2
      worker_price = "0.021"

      daemonset_tolerations = ["arch"]     # important
    }
    ```

=== "Flatcar Cluster"

    ```tf
    module "gravitas" {
      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.23.4"

      # AWS
      cluster_name = "gravitas"
      dns_zone     = "aws.example.com"
      dns_zone_id  = "Z3PAABBCFAKEC0"

      # configuration
      ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

      # optional
      networking   = "cilium"
      worker_count = 2
      worker_price = "0.021"

      daemonset_tolerations = ["arch"]     # important
    }
    ```

=== "FCOS ARM64 Workers"

    ```tf
    module "gravitas-arm64" {
      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.4"

      # AWS
      vpc_id          = module.gravitas.vpc_id
      subnet_ids      = module.gravitas.subnet_ids
      security_groups = module.gravitas.worker_security_groups

      # configuration
      name               = "gravitas-arm64"
      kubeconfig         = module.gravitas.kubeconfig
      ssh_authorized_key = var.ssh_authorized_key

      # optional
      arch          = "arm64"
      instance_type = "t4g.small"
      spot_price    = "0.0168"
      node_taints   = ["arch=arm64:NoSchedule"]
    }
    ```

=== "Flatcar ARM64 Workers"

    ```tf
    module "gravitas-arm64" {
      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.23.4"

      # AWS
      vpc_id          = module.gravitas.vpc_id
      subnet_ids      = module.gravitas.subnet_ids
      security_groups = module.gravitas.worker_security_groups

      # configuration
      name               = "gravitas-arm64"
      kubeconfig         = module.gravitas.kubeconfig
      ssh_authorized_key = var.ssh_authorized_key

      # optional
      arch          = "arm64"
      instance_type = "t4g.small"
      spot_price    = "0.0168"
      node_taints   = ["arch=arm64:NoSchedule"]
    }
    ```

Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.

```
$ kubectl get nodes -o wide
NAME                       STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                             KERNEL-VERSION            CONTAINER-RUNTIME
ip-10-0-1-73               Ready    <none>   111m   v1.23.4   10.0.1.73     <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
ip-10-0-22-79...           Ready    <none>   111m   v1.23.4   10.0.22.79    <none>        Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)   5.10.84-flatcar           containerd://1.5.8
ip-10-0-24-130             Ready    <none>   111m   v1.23.4   10.0.24.130   <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
ip-10-0-39-19              Ready    <none>   111m   v1.23.4   10.0.39.19    <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
```