typhoon/docs/topics/performance.md

# Performance

## Provision Time

Provisioning times vary based on the operating system and platform. Sampling the time to create (apply) and destroy clusters with 1 controller and 2 workers shows (roughly) what to expect.

| Platform      | Apply | Destroy |
|---------------|-------|---------|
| AWS           | 6 min | 4 min   |
| Azure         | 7 min | 7 min   |
| Bare-Metal    | 10-15 min | NA  |
| Digital Ocean | 3 min 30 sec | 20 sec |
| Google Cloud  | 7 min | 6 min   |

Notes:

* SOA TTL and NXDOMAIN caching can have a large impact on provision time
* Platforms with auto-scaling take more time to provision (AWS, Azure, Google)
* Bare-metal POST times and network bandwidth will affect provision times

## Network Performance

Network performance varies based on the platform and CNI plugin. `iperf` was used to measure the bandwidth between different hosts and different pods. Host-to-host shows typical bandwidth between host machines. Pod-to-pod shows the bandwidth between two `iperf` containers.

| Platform / Plugin          | Theory | Host to Host | Pod to Pod   |
|----------------------------|-------:|-------------:|-------------:|
| AWS (flannel)              | 5 Gb/s | 4.94 Gb/s    | 4.89 Gb/s    |
| AWS (calico, MTU 1480)     | 5 Gb/s | 4.94 Gb/s    | 4.42 Gb/s    |
| AWS (calico, MTU 8981)     | 5 Gb/s | 4.94 Gb/s    | 4.75 Gb/s    |
| Azure (flannel)            | Varies |  749 Mb/s    | 680 Mb/s     |
| Bare-Metal (flannel)       | 1 Gb/s |  940 Mb/s    | 903 Mb/s     |
| Bare-Metal (calico)        | 1 Gb/s |  940 Mb/s    | 931 Mb/s     |
| Bare-Metal (flannel, bond) | 3 Gb/s |  2.3 Gb/s    | 1.17 Gb/s    |
| Bare-Metal (calico, bond)  | 3 Gb/s |  2.3 Gb/s    | 1.17 Gb/s    |
| Digital Ocean              | 2 Gb/s | 1.97 Gb/s    | 1.64 Gb/s    |
| Google Cloud (flannel)     | 2 Gb/s | 1.94 Gb/s    | 1.76 Gb/s    |
| Google Cloud (calico)      | 2 Gb/s | 1.94 Gb/s    | 1.81 Gb/s    |

Notes:

* Calico and Flannel have comparable performance. Platform and configuration differences dominate.
* AWS and Azure node bandwidth (i.e. upper bound) depends greatly on machine type
* Only [certain AWS EC2 instance types](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) allow jumbo frames. This is why the default MTU on AWS must be 1480.
* Neither CNI provider seems to be able to leverage bonded NICs well (bare-metal)
Add docs on provision time and network performance 2017-10-04 08:44:26 +02:00			`# Performance`

			`## Provision Time`

Write documentation for Fedora Atomic 2018-04-24 06:36:20 +02:00			`Provisioning times vary based on the operating system and platform. Sampling the time to create (apply) and destroy clusters with 1 controller and 2 workers shows (roughly) what to expect.`
Add docs on provision time and network performance 2017-10-04 08:44:26 +02:00
			`\| Platform \| Apply \| Destroy \|`
			`\|---------------\|-------\|---------\|`
Change AWS default type from t2.small to t3.small * T3 is the next generation general purpose burstable instance type. Compared with t2.small, the t3.small is cheaper, has 2 vCPU (instead of 1) and provides 5 Gbps of pod-to-pod bandwidth (instead of 1 Gbps) 2018-12-18 21:29:27 +01:00			`\| AWS \| 6 min \| 4 min \|`
Add new tutorial docs and links 2018-08-27 08:39:41 +02:00			`\| Azure \| 7 min \| 7 min \|`
Write documentation for Fedora Atomic 2018-04-24 06:36:20 +02:00			`\| Bare-Metal \| 10-15 min \| NA \|`
digital-ocean: Run etcd cluster on-host, across controllers * Run etcd peers with TLS across controller nodes * Deprecate self-hosted-etcd on the Digital Ocean platform * Distribute etcd TLS certificates as part of initial provisioning * Check the status of etcd by running `systemctl status etcd-member` 2017-10-04 09:07:48 +02:00			`\| Digital Ocean \| 3 min 30 sec \| 20 sec \|`
Add new tutorial docs and links 2018-08-27 08:39:41 +02:00			`\| Google Cloud \| 7 min \| 6 min \|`
Add docs on provision time and network performance 2017-10-04 08:44:26 +02:00
			`Notes:`

Promote AWS platform from alpha to beta 2017-11-07 06:19:11 +01:00			`* SOA TTL and NXDOMAIN caching can have a large impact on provision time`
Add new tutorial docs and links 2018-08-27 08:39:41 +02:00			`* Platforms with auto-scaling take more time to provision (AWS, Azure, Google)`
Promote AWS platform from alpha to beta 2017-11-07 06:19:11 +01:00			`* Bare-metal POST times and network bandwidth will affect provision times`
Add docs on provision time and network performance 2017-10-04 08:44:26 +02:00
			`## Network Performance`

Fix typos in docs and CONTRIBUTING.md 2017-12-09 18:26:26 +01:00			Network performance varies based on the platform and CNI plugin. `iperf` was used to measure the bandwidth between different hosts and different pods. Host-to-host shows typical bandwidth between host machines. Pod-to-pod shows the bandwidth between two `iperf` containers.
Add docs on provision time and network performance 2017-10-04 08:44:26 +02:00
			`\| Platform / Plugin \| Theory \| Host to Host \| Pod to Pod \|`
			`\|----------------------------\|-------:\|-------------:\|-------------:\|`
Change AWS default type from t2.small to t3.small * T3 is the next generation general purpose burstable instance type. Compared with t2.small, the t3.small is cheaper, has 2 vCPU (instead of 1) and provides 5 Gbps of pod-to-pod bandwidth (instead of 1 Gbps) 2018-12-18 21:29:27 +01:00			`\| AWS (flannel) \| 5 Gb/s \| 4.94 Gb/s \| 4.89 Gb/s \|`
			`\| AWS (calico, MTU 1480) \| 5 Gb/s \| 4.94 Gb/s \| 4.42 Gb/s \|`
			`\| AWS (calico, MTU 8981) \| 5 Gb/s \| 4.94 Gb/s \| 4.75 Gb/s \|`
			`\| Azure (flannel) \| Varies \| 749 Mb/s \| 680 Mb/s \|`
			`\| Bare-Metal (flannel) \| 1 Gb/s \| 940 Mb/s \| 903 Mb/s \|`
			`\| Bare-Metal (calico) \| 1 Gb/s \| 940 Mb/s \| 931 Mb/s \|`
Measure DigitalOcean network performance * Measuring pod-to-pod bandwidth in a few regions (NYC3, FRA1, SFO1) shows DigitalOcean has made some improvements 2018-11-12 06:05:07 +01:00			`\| Bare-Metal (flannel, bond) \| 3 Gb/s \| 2.3 Gb/s \| 1.17 Gb/s \|`
			`\| Bare-Metal (calico, bond) \| 3 Gb/s \| 2.3 Gb/s \| 1.17 Gb/s \|`
			`\| Digital Ocean \| 2 Gb/s \| 1.97 Gb/s \| 1.64 Gb/s \|`
			`\| Google Cloud (flannel) \| 2 Gb/s \| 1.94 Gb/s \| 1.76 Gb/s \|`
			`\| Google Cloud (calico) \| 2 Gb/s \| 1.94 Gb/s \| 1.81 Gb/s \|`
Add docs on provision time and network performance 2017-10-04 08:44:26 +02:00
			`Notes:`

Fix typos in docs and CONTRIBUTING.md 2017-12-09 18:26:26 +01:00			`* Calico and Flannel have comparable performance. Platform and configuration differences dominate.`
Add new tutorial docs and links 2018-08-27 08:39:41 +02:00			`* AWS and Azure node bandwidth (i.e. upper bound) depends greatly on machine type`
Add docs on provision time and network performance 2017-10-04 08:44:26 +02:00			`* Only [certain AWS EC2 instance types](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) allow jumbo frames. This is why the default MTU on AWS must be 1480.`
Add new tutorial docs and links 2018-08-27 08:39:41 +02:00			`* Neither CNI provider seems to be able to leverage bonded NICs well (bare-metal)`