mirror of
https://github.com/puppetmaster/typhoon.git
synced 2024-12-25 07:09:34 +01:00
125 KiB
125 KiB
Typhoon
Notable changes between versions.
Latest
v1.24.4
- Kubernetes v1.24.4
- Update CoreDNS from v1.8.6 to v1.9.3
- Update Cilium from v1.11.7 to v1.12.1
- Update Calico from v3.23.1 to v3.23.3
- Switch Kubernetes registry from
k8s.gcr.io
toregistry.k8s.io
(#1206) - Remove use of deprecated Terraform template provider (#1194)
Fedora CoreOS
- Remove ineffective
/etc/fedora-coreos/iptables-legacy.stamp
(#1201)- Typhoon already uses iptables v1.8.7 (nf_tables) since FCOS 36
- Staying on legacy iptables required a file in
/etc/coreos
instead
Flatcar Linux
- Migrate Flatcar Linux from Ignition spec v2.3.0 to v3.3.0 (#1196) (action required)
- Please modify any Flatcar Linux snippets to use the Butane Config format (action required)
variant: flatcar
version: 1.0.0
...
AWS
- Refresh instances in autoscaling group when launch configuration changes (#1208) (docs, important)
- Worker launch configuration changes start an autoscaling group instance refresh to replace instances
- Instance refresh creates surge instances, waits for a warm-up period, then deletes old instances
- Changing
worker_type
,disk_*
,worker_price
,worker_target_groups
, or Butaneworker_snippets
on existing worker nodes will replace instances - New AMIs or changing
os_stream
will be ignored, to allow Fedora CoreOS or Flatcar Linux to keep themselves updated - Previously, new launch configurations were made in the same way, but not applied to instances unless manually replaced
- Rename worker autoscaling group
${cluster_name}-worker
(#1202)- Rename launch configuration
${cluster_name}-worker
instead of a random id
- Rename launch configuration
- Roll instance template changes to worker managed instance groups (#1207) (docs, important)
- Worker instance template changes roll out by gradually replacing instances
- Automatic rollouts create surge instances, wait for health checks, then delete old instances (0 unavailable instances)
- Changing
worker_type
,disk_size
,worker_preemptible
, or Butaneworker_snippets
on existing worker nodes will replace instances - New compute images or changing
os_stream
will be ignored, to allow Fedora CoreOS or Flatcar Linux to keep themselves updated - Previously, new instance templates were made in the same way, but not applied to instances unless manually replaced
- Add health checks to worker managed instance groups (i.e. "autohealing") (#1207)
- Use health checks to probe kube-proxy every 30s
- Replace worker nodes that fail the health check 6 times (3min)
- Name
kube-apiserver
andworker
health checks consistently (#1207)- Use name
${cluster_name}-apiserver-health
and${cluster_name}-worker-health
- Use name
- Rename managed instance group from
${cluster_name}-worker-group
to${cluster_name}-worker
(#1207) - Fix bug provisioning clusters with multiple controller nodes (#1195)
Addons
v1.24.3
Addons
- Update ingress-nginx from v1.2.1 to v1.3.0
- Update Prometheus from v2.36.1 to v2.37.0
- Update Grafana from v8.5.6 to v9.0.3
Notes
- Poseidon repos will soon change their default branch from
master
tomain
v1.24.2
Addons
- Update Prometheus from v2.36.0 to v2.36.1
- Update Grafana from v8.5.3 to v8.5.6
- Update kube-state-metrics from v2.4.2 to v2.5.0
Known Issues
- Skip AWS Terraform provider v4.17.0 to v4.19.0, which had a regression affecting workers joining (#1173)
v1.24.1
Addons
- Update Prometheus from v2.35.0 to v2.36.0
- Update Grafana from v8.5.1 to v8.5.3
- Update nginx-ingress from v1.2.0 to v1.2.1
v1.24.0
- Kubernetes v1.24.0
- Update etcd from v3.5.2 to v3.5.4
- Add Kubelet mounts to enable relabeling workload volumes (#1152)
- StorageClass no longer require explicit SELinux mount contexts
Addons
- Update nginx-ingress from v1.1.3 to v1.2.0
- Update Prometheus from v2.34.0 to v2.35.0
- Update Grafana from v8.4.5 to v8.5.1
v1.23.6
- Kubernetes v1.23.6
- Update Cilium from v1.11.2 to v1.11.4
- Rename Cilium DaemonSet from
cilium-agent
tocilium
to match Cilium CLI tools (#303) - Update Calico from v3.22.1 to v3.22.2
- Mount /etc/machine-id from host into Kubelet (#1143)
- Remove deprecated use of
key_algorithm
inhashicorp/tls
resources
Azure
- Allow upgrading Azure Terraform provider to v3.x (#1144)
- Rename
worker_address_prefix
output toworker_address_prefixes
Google Cloud
- Fix issue on Flatcar Linux with controller nodes not ignoring os image changes (#1149)
- Nodes will auto-update, Terraform should not attempt to delete/recreate them
Addons
- Update nginx-ingress from v1.1.2 to v1.1.3
- Update Prometheus from v2.33.5 to v2.34.0
- Update Grafana from v8.4.4 to v8.4.5
v1.23.5
- Kubernetes v1.23.5
- Update Cilium from v1.11.1 to v1.11.2
- Update Calico from v3.21.2 to v3.22.1
- Fix calico#5011, broken since v1.23.0
Addons
- Refresh Prometheus rules and Grafana dashboards (#1136)
- Update nginx-ingress from v1.1.1 to v1.1.2
- Update Prometheus from v2.33.3 to v2.33.5
- Update Grafana from v8.4.1 to v8.4.3
- Update kube-state-metrics from v2.3.0 to v2.4.2
v1.23.4
- Kubernetes v1.23.4
- Update etcd from v3.5.1 to v3.5.2
- Change default CNI
networking
provider fromcalico
tocilium
(#1114)
AWS
- Allow upgrading AWS Terraform Provider to v4.x
Addons
- Align nginx-ingress
--controller-class
withIngressClass
- Watch only
public
IngressClass objects, better example
- Watch only
- Update Prometheus from v2.32.1 to v2.33.3
- Update Grafana from v8.3.6 to v8.4.1
V1.23.3
- Kubernetes v1.23.3
Flatcar Linux
Google Cloud
- Switch to using official Kinvolk Flatcar Linux images
- Promote Typhoon on Flatcar Linux / Google Cloud to stable
- Change
os_image
toflatcar-stable
,flatcar-beta
, orflatcar-alpha
(action required)
v1.23.2
- Kubernetes v1.23.2
- Update Cilium from v1.11.0 to v1.11.1
- Remove Kubelet flag
--network-plugin
. Unused sincedocker-shim
isn't used (#1106)
Fedora CoreOS
- Switch Kubernetes Container Runtime from
docker
tocontainerd
(#1101) - Mask
docker.service
to prevent it from being socket activated (#1105)
Flatcar Linux
AWS
- Add experimental Flatcar Linux ARM64 support (docs, #1102)
- Add
arch
variable to AWSkubernetes
andworkers
modules - Allow arm64 full-cluster or mixed/hybrid cluster with arm64 workers
- Requires
flannel
orcilium
CNI provider
- Add
DigitalOcean
Addons
v1.23.1
Flatcar Linux
- Switch Kubernetes Container Runtime from
docker
tocontainerd
(#1087)
Addons
- Configure Prometheus to allow a custom scrape query parameter (#1095)
- Configure Prometheus to probe Kubernetes Ingress via
blackbox-exporter
(#1096) - Fix Prometheus Service probes to use
blackbox-exporter
, notblackbox
(#1096)
v1.23.0
- Kubernetes v1.23.0
- Normalize CA cert mounts in static Pods and kube-proxy (#1078)
- Set Kubelet resolver config to
/run/systemd/resolve/resolv.conf
(#1082) - Update Cilium from v1.10.5 to v1.11.0 (#1083)
- With Calico, add missing
caliconodestatuses
CRD (#289) - Change
enable_aggregation
default to true (#279) - Remove deprecated
--port
fromkube-scheduler
(#1078)
AWS
- Change controller node default
disk_iops
to 3000 (#1073)
Azure
- Fix warning about deprecated
backend_address_pool_id
(#1086)
Fedora CoreOS
- Fix Fedora ARM64 workers to official Fedora CoreOS AMIs (#1072)
- Should have been changed alongside controller AMIs in (#1038)
- Old Poseidon built ARM64 AMIs have been deleted
Addons
- Update nginx-ingress from v1.0.5 to v1.1.0
- Update Prometheus from v2.31.1 to v2.32.0
- Update kube-state-metrics from v2.2.4 to v2.3.0
- Update node-exporter from v1.3.0 to v1.3.1
- Update Grafana from v8.2.4 to v8.3.3
Known Issues
- Calico does not yet support Kubernetes v1.23.0, use
flannel
orcilium
(calico#5011)
v1.22.4
- Kubernetes v1.22.4
- Update CoreDNS from v1.8.4 to v1.8.6
- Update Calico from v3.20.2 to v3.21.0
- Update flannel from v0.14.0 to v0.15.1
- Allow use of Terraform provider
google
v4.0+
Flatcar Linux
- Change Kubelet mounts for cgroups v2 (#1064)
- Update cgroup driver from cgroupfs to systemd (Flatcar Linux changed default) (#1064)
Addons
- Update Prometheus from v2.30.3 to v2.31.1
- Update node-exporter from v1.2.2 to v1.3.0
- Update kube-state-metrics from v2.2.3 to v2.2.4
- Update Grafana from v8.2.1 to v8.2.4
- Update nginx-ingress from v1.0.4 to v1.0.5
v1.23.3
- Kubernetes v1.22.3
- Update etcd from v3.5.0 to v3.5.1
- Update Cilium from v1.10.4 to v1.10.5
- Update Calico from v3.20.1 to v3.20.2
- Use Calico's iptables legacy vs nft auto-detection
- Update flannel from v0.13.0 to v0.14.0
Bare-Metal
- Require Terraform provider
poseidon/matchbox
v0.5+ (#1048)
Addons
- Update nginx-ingress from v1.0.0 to v1.0.4
- Update Prometheus from v2.29.2 to v2.30.3
- Update kube-state-metrics from v2.2.0 to v2.2.3
- Update Grafana from v8.1.2 to v8.2.1
v1.22.2
- Kubernetes v1.22.2
- Update Cilium from v1.10.3 to v1.10.4
- Update Calico from v3.20.0 to v3.20.1
- Fix access to ClusterIP services with Cilium (#276)
Fedora CoreOS
- Use Fedora CoreOS ARM64 AMIs (#1038)
Addons
v1.22.1
Addons
- Update nginx-ingress from v1.0.0-beta.1 to v1.0.0
- Update Prometheus from v2.28.1 to v2.29.1
- Update Grafana from v8.1.1 to v8.1.2
v1.22.0
- Kubernetes v1.22.0
- Update etcd from v3.4.16 to v3.5.0
- Switch
kube-controller-manager
andkube-scheduler
to use secure port only- Update Prometheus config to discover endpoints and use a bearer token to scrape
Fedora CoreOS
- Add Cilium cgroups v2 support on Fedora CoreOS
- Update Butane Config version from v1.2.0 to v1.4.0
- Rename Fedora CoreOS Config to Butane Config
- Require any snippets customizations to update to v1.4.0
Addons
- Update nginx-ingress from v0.47.0 to v1.0.0-beta.1
- Update node-exporter from v1.2.0 to v1.2.2
- Update kube-state-metrics from v2.1.0 to v2.1.1
- Update Grafana from v8.0.6 to v8.1.1
v1.21.3
- Kubernetes v1.21.3
- Update Cilium from v1.10.1 to v1.10.3
- Require poseidon/ct Terraform provider v0.9+ (notes)
AWS
- Change default disk type from
gp2
togp3
(#1012)
Addons
- Update Prometheus from v2.28.0 to v2.28.1
- Update node-exporter from v1.1.2 to v1.2.0
- Update Grafana from v8.0.3 to v8.0.6
Known Issues
- Cilium with recent Fedora CoreOS will have networking issues (fedora-coreos#881) (fixed in v1.21.4)
v1.21.2
- Kubernetes v1.21.2
- Add Terraform v1.0.x support (#974)
- Continue to support Terraform v0.13.x, v0.14.4+, and v0.15.x
- Update CoreDNS from v1.8.0 to v1.8.4
- Update Cilium from v1.9.6 to v1.10.1
- Update Calico from v3.19.0 to v3.19.1
Addons
- Update kube-state-metrics from v2.0.0 to v2.1.0
- Update Prometheus from v2.27.0 to v2.28.0
- Update Grafana from v7.5.6 to v8.0.3
- Update nginx-ingress from v0.46.0 to v0.47.0
Fedora CoreOS
AWS
- Extend experimental Fedora CoreOS arm64 support with Cilium
- CNI provider may now be
flannel
orcilium
(new)
- CNI provider may now be
Bare-Metal
- Workaround systemd path unit issue fedora-coreos-tracker/#861
DigitalOcean
- Workaround systemd path unit issue fedora-coreos-tracker/#861
Known Issues
- Cilium with recent Fedora CoreOS will have networking issues (fedora-coreos#881) (fixed in v1.21.4)
v1.21.1
- Kubernetes v1.21.1
- Add Terraform v0.15.x support (#974)
- Continue to support Terraform v0.13.x and v0.14.4+
- Update etcd from v3.4.15 to v3.4.16
- Update Cilium from v1.9.5 to v1.9.6
- Update Calico from v3.18.1 to v3.19.0
AWS
- Reduce the default
disk_size
from 40GB to 30GB (#983)
Azure
- Reduce the default
disk_size
from 40GB to 30GB (#983)
Google Cloud
- Reduce the default
disk_size
from 40GB to 30GB (#983)
Fedora CoreOS
- Update Kubelet mounts for cgroups v2 (#978)
Addons
- Update kube-state-metrics from v2.0.0-rc.1 to v2.0.0
- Update Prometheus from v2.25.2 to v2.27.0
- Update Grafana from v7.5.3 to v7.5.6
- Update nginx-ingress from v0.45.0 to v0.46.0
v1.21.0
- Kubernetes v1.21.0
- Enable
tokencleaner
controller (#969) - Enable
kube-scheduler
andkube-controller-manager
separate authn/z kubeconfig - Change CNI config location from /etc/kubernetes/cni/net.d to /etc/cni/net.d (#965)
- Change
kube-controller-manager
to mount/var/lib/kubelet/volumeplugins
directly - Remove unused
cloud-provider
flags
- Enable
- Update Fedora CoreOS Config version from v1.1.0 to v1.2.0 (#970)
- Require poseidon/ct Terraform provider v0.8+ (notes)
- Require any snippets customizations to update to v1.2.0
AWS
- Allow setting custom initial node taints on worker pools (#968)
- Add
node_taints
variable to internalworkers
pool module to set initial node taints - Add
daemonset_tolerations
sokube-system
DaemonSets can tolerate custom taints
- Add
Azure
- Allow setting custom initial node taints on worker pools (#968)
- Add
node_taints
variable to internalworkers
pool module to set initial node taints - Add
daemonset_tolerations
sokube-system
DaemonSets can tolerate custom taints
- Add
- Remove deprecated
azurerm_lb_backend_address_pool
fieldresource_group_name
(#972)
Google Cloud
- Allow setting custom initial node taints on worker pools (#968)
- Add
node_taints
variable to internalworkers
pool module to set initial node taints - Add
daemonset_tolerations
sokube-system
DaemonSets can tolerate custom taints
- Add
Addons
- Update nginx-ingress from v0.44.0 to v0.45.0
- Update kube-state-metrics from v2.0.0-rc.0 to v2.0.0-rc.1
- Update Grafana from v7.4.5 to v7.5.3
v1.20.5
- Kubernetes v1.20.5
- Update etcd from v3.4.14 to v3.4.15
- Update Cilium from v1.9.4 to v1.9.5
- Update Calico from v3.17.3 to v3.18.1
- Update CoreDNS from v1.7.0 to v1.8.0
- Mark bootstrap token as sensitive in Terraform plans (#949)
Fedora CoreOS
- Set Kubelet
provider-id
(#951)
Flatcar Linux
AWS
Azure
- Remove
os_image
optionflatcar-edge
(#943)
Bare-Metal
- Remove
os_channel
optionflatcar-edge
(#943)
Addons
- Update Prometheus from v2.25.0 to v2.25.2
- Update kube-state-metrics from v2.0.0-alpha.3 to v2.0.0-rc.0
- Switch image from
quay.io
tok8s.gcr.io
(#946)
- Switch image from
- Update node-exporter from v1.1.1 to v1.1.2
- Update Grafana from v7.4.2 to v7.4.5
v1.20.4
- Kubernetes v1.20.4
- Update Cilium from v1.9.1 to v1.9.4
- Update Calico from v3.17.1 to v3.17.3
- Update flannel-cni from v0.4.1 to v0.4.2
Addons
- Update nginx-ingress from v0.43.0 to v0.44.0
- Update Prometheus from v2.24.0 to v2.25.0
- Update node-exporter from v1.0.1 to v1.1.1
- Update Grafana from v7.3.7 to v7.4.2
v1.20.2
Addons
- Update nginx-ingress from v0.41.2 to v0.43.0
- Update Prometheus from v2.23.0 to v2.24.0
- Update Grafana from v7.3.6 to v7.3.7
v1.20.1
- Kubernetes v1.20.1
Fedora CoreOS
- Fedora CoreOS 33 has stronger crypto defaults (notice, #915)
- Use a non-RSA SSH key or add the workaround provided in upstream Fedora docs as a snippet (action required)
Addons
- Update Grafana from v7.3.5 to v7.3.6
v1.20.0
- Kubernetes v1.20.0
- Add input variable validations (#880)
- Require Terraform v0.13+ (migration guide)
- Set output sensitive to suppress console display for some cases (#885)
- Add service account token volume projection (#897)
- Scope kube-scheduler and kube-controller-manager permissions (#898)
- Update etcd from v3.4.12 to v3.4.14
- Update Calico from v3.16.5 to v3.17.1 (#890)
- Enable Calico MTU auto-detection
- Remove workaround to Calico cni-plugin issue
- Update Cilium from v1.9.0 to v1.9.1
- Relax
terraform-provider-ct
version constraint to v0.6+ (#893)- Allow upgrading
terraform-provider-ct
to v0.7.x (warn)
- Allow upgrading
AWS
- Enable Network Load Balancer (NLB) dualstack (#883)
- NLB subnets assigned both IPv4 and IPv6 addresses
- NLB DNS name has both A and AAAA records
- NLB to target node traffic is IPv4 (no change)
Bare-Metal
- Remove iSCSI
/etc/iscsi
andiscsadm
mounts from Kubelet (#912)
Fedora CoreOS
AWS
- Fix AMI query for which could fail in some regions (#887)
Bare-Metal
- Promote Fedora CoreOS to stable
- Use initramfs and rootfs images as initrd's (#889)
- Requires Fedora CoreOS version with rootfs images (e.g. 32.20200923.3.0+)
Addons
- Update Prometheus from v2.22.2 to v2.23.0
- Update kube-state-metrics from v2.0.0-alpha.2 to v2.0.0-alpha.3
- Update Grafana from v7.3.2 to v7.3.5
v1.19.4
- Kubernetes v1.19.4
- Update Cilium from v1.8.4 to v1.9.0
- Update Calico from v3.16.3 to v3.16.5
- Remove
asset_dir
variable (defaulted off in v1.17.0, deprecated in v1.18.0)
Fedora CoreOS
- Improve
etcd-member.service
systemd unit (#868)- Allow a snippet with a systemd dropin to set an alternate image (e.g. mirror)
- Fix local node delete oneshot on node shutdown (#856)
AWS
- Add experimental Fedora CoreOS arm64 support (docs, #875)
- Allow arm64 full-cluster or mixed/hybrid cluster with worker pools
- Add
arch
variable to cluster module - Add
daemonset_tolerations
variable to cluster module - Add
node_taints
variable to workers module - Requires flannel CNI provider and use of experimental AMI (see docs)
Flatcar Linux
- Rename
container-linux
modules toflatcar-linux
(#858) (action required) - Change on-host system containers from rkt to docker
- Fix local node delete oneshot on node shutdown (#855)
- Remove CoreOS Container Linux Matchbox profiles (#859)
Addons
- Update nginx-ingress from v0.40.2 to v0.41.2
- Update Prometheus from v2.22.0 to v2.22.1
- Update kube-state-metrics from v2.0.0-alpha.1 to v2.0.0-alpha.2
- Update Grafana from v7.2.1 to v7.3.2
v1.19.3
- Kubernetes v1.19.3
- Update Cilium from v1.8.3 to v1.8.4
- Update Calico from v1.15.3 to v1.16.3 (#851)
- Update flannel from v0.13.0-rc2 to v0.13.0 (#219)
Flatcar Linux
Addons
- Update nginx-ingress from v0.35.0 to v0.40.2
- Update Grafana from v7.1.5 to v7.2.1
- Update Prometheus from v2.21.0 to v2.22.0
- Update kube-state-metrics from v1.9.7 to v2.0.0-alpha.1
v1.19.2
- Kubernetes v1.19.2
- Update flannel from v0.12.0 to v0.13.0-rc2 (#216)
- Update flannel-cni from v0.4.0 to v0.4.1
- Update CNI plugins from v0.8.6 to v0.8.7
Addons
- Refresh Prometheus rules/alerts and Grafana dashboards (#831)
- Reduce apiserver metrics cardinality for non-core APIs (#830)
v1.19.1
- Kubernetes v1.19.1
- Change control plane seccomp annotations to GA
seccompProfile
(#822)
- Change control plane seccomp annotations to GA
- Update Cilium from v1.8.2 to v1.8.3
- Promote Cilium from experimental to general availability (#827)
- Update Calico from v1.15.2 to v1.15.3
Fedora CoreOS
- Update Fedora CoreOS Config version from v1.0.0 to v1.1.0
- Require any snippets customizations to update to v1.1.0
Addons
- Update IngressClass resources to
networking.k8s.io/v1
(#824) - Update Prometheus from v2.20.0 to v2.21.0
- Remove Kubernetes node name labelmap
relabel_config
from etcd, Kubelet, and CAdvisor scrape config (#828)
- Remove Kubernetes node name labelmap
v1.19.0
Fedora CoreOS
- Fix race condition during bootstrap of multi-controller clusters (#808)
- Fix SELinux label of bootstrap-secrets on non-bootstrap controllers
Addons
- Introduce fleetlock for Fedora CoreOS reboot coordination (#814)
- Update nginx-ingress from v0.34.1 to v0.35.0
- Repository changed to
k8s.gcr.io/ingress-nginx/controller
- Repository changed to
- Update Grafana from v7.1.3 to v7.1.5
v1.18.8
- Kubernetes v1.18.8
- Migrate from Terraform v0.12.x to v0.13.x (#804) (action required)
- Recommend Terraform v0.13.x (migration guide)
- Support automatic install of poseidon's provider plugins (poseidon/ct, poseidon/matchbox)
- Require Terraform v0.12.26+ (migration compatibility)
- Require
terraform-provider-ct
v0.6.1 - Require
terraform-provider-matchbox
v0.4.1
- Update etcd from v3.4.9 to v3.4.10
- Update CoreDNS from v1.6.7 to v1.7.0
- Update Cilium from v1.8.1 to v1.8.2
- Update coreos/flannel-cni to poseidon/flannel-cni (#798)
- Update CNI plugins and fix CVEs with Flannel CNI (non-default)
- Transition to a poseidon maintained container image
AWS
- Allow
terraform-provider-aws
v3.0+ (#803)- Recommend updating
terraform-provider-aws
to v3.0+ - Continue to allow v2.23+, no v3.x specific features are used
- Recommend updating
DigitalOcean
- Require
terraform-provider-digitalocean
v1.21+ for Terraform v0.13.x (unenforced) - Require
terraform-provider-digitalocean
v1.20+ for Terraform v0.12.x
Fedora CoreOS
- Fix support for Flannel with Fedora CoreOS (#795)
- Configure
flannel.1
link to select its own MAC address to solve flannel pod-to-pod traffic drops starting with default link changes in Fedora CoreOS 32.20200629.3.0 (details)
- Configure
Addons
v1.18.6
Addons
- Update nginx-ingress from v0.33.0 to v0.34.1
- ingress-nginx will publish images only to gcr.io
- Update Prometheus from v2.19.1 to v2.19.2
- Update Grafana from v7.0.4 to v7.0.6
v1.18.5
- Kubernetes v1.18.5
- Add Cilium v1.8.0 as a (experimental) CNI provider option (#760)
- Set
networking
to "cilium" to enable
- Set
- Update Calico from v3.14.1 to v3.15.0
DigitalOcean
- Isolate each cluster in an independent DigitalOcean VPC (#776)
- Create droplets in a VPC per cluster (matches Typhoon AWS, Azure, and GCP)
- Require
terraform-provider-digitalocean
v1.16.0+ (action required) - Output
vpc_id
for use with an attached DigitalOcean loadbalancer
Fedora CoreOS
Google Cloud
- Promote Fedora CoreOS to stable
- Remove
os_image
variable deprecated in v1.18.3 (#777)- Use
os_stream
to select a Fedora CoreOS image stream
- Use
Flatcar Linux
Azure
- Allow using Flatcar Linux Edge by setting
os_image
to "flatcar-edge" (#778)
Addons
v1.18.4
- Kubernetes v1.18.4
- Update Kubelet image publishing (#749)
- Build Kubelet images internally and publish to Quay and Dockerhub
- quay.io/poseidon/kubelet (official)
- docker.io/psdn/kubelet (fallback)
- Continue offering automated image builds with an alternate tag strategy (see docs)
- Document use of alternate Kubelet images during registry incidents
- Build Kubelet images internally and publish to Quay and Dockerhub
- Update Calico from v3.14.0 to v3.14.1
- Fix CVE-2020-13597
- Rename controller NoSchedule taint from
node-role.kubernetes.io/master
tonode-role.kubernetes.io/controller
(#764)- Tolerate the new taint name for workloads that may run on controller nodes
- Remove node label
node.kubernetes.io/master
from controller nodes (#764)- Use
node.kubernetes.io/controller
(present since v1.9.5, #160) to node select controllers
- Use
- Remove unused Kubelet
-lock-file
and-exit-on-lock-contention
(#758)
Fedora CoreOS
Azure
- Use
strict
Fedora CoreOS Config (FCC) snippet parsing (#755) - Reduce Calico vxlan interface MTU to maintain performance (#767)
AWS
- Fix Kubelet service race with hostname update (#766)
- Wait for a hostname to avoid Kubelet trying to register as
localhost
- Wait for a hostname to avoid Kubelet trying to register as
Flatcar Linux
- Use
strict
Container Linux Config (CLC) snippet parsing (#755)- Require
terraform-provider-ct
v0.4+, recommend v0.5+ (action required)
- Require
Addons
- Update nginx-ingress from v0.32.0 to v0.33.0
- Update Prometheus from v2.18.1 to v2.19.0
- Update node-exporter from v1.0.0-rc.1 to v1.0.1
- Update kube-state-metrics from v1.9.6 to v1.9.7
- Update Grafana from v7.0.0 to v7.0.3
v1.18.3
- Kubernetes v1.18.3
- Use Kubelet TLS bootstrap with bootstrap token authentication (#713)
- Enable Node Authorization and NodeRestriction to reduce authorization scope
- Renew Kubelet certificates every 72 hours
- Update etcd from v3.4.7 to v3.4.9
- Update Calico from v3.13.1 to v3.14.0
- Add CoreDNS node affinity preference for controller nodes (#188)
- Deprecate CoreOS Container Linux support (no OS updates after May 2020)
- Use a
fedora-coreos
module for Fedora CoreOS - Use a
container-linux
module for Flatcar Linux
- Use a
AWS
- Fix Terraform plan error when
controller_count
exceeds AWS zones (e.g. 5 controllers) (#714)- Regressed in v1.17.1 (#605)
Azure
- Update Azure subnets to set
address_prefixes
list (#730)- Fix warning that
address_prefix
is deprecated - Require
terraform-provider-azurerm
v2.8.0+ (action required)
- Fix warning that
DigitalOcean
- Promote DigitalOcean to beta on both Fedora CoreOS and Flatcar Linux
Fedora CoreOS
- Fix Calico
install-cni
crashloop on Pod restarts (#724)- SELinux enforcement requires consistent file context MCS level
- Restarting a node resolved the issue as a previous workaround
AWS
- Support Fedora CoreOS image streams (#727)
- Add
os_stream
variable to set the stream tostable
(default),testing
, ornext
- Remove unused
os_image
variable
- Add
- Support Fedora CoreOS image streams (#723)
- Add
os_stream
variable to set the stream tostable
(default),testing
, ornext
- Deprecate
os_image
variable. Manual image uploads are no longer needed
- Add
Flatcar Linux
Azure
- Use the Flatcar Linux Azure Marketplace image
- Change
os_image
to use aflatcar-stable
default
- Promote Flatcar Linux to beta
Addons
- Update nginx-ingress from v0.30.0 to v0.32.0
- Add support for IngressClass
- Update Prometheus from v2.17.1 to v2.18.1
- Update kube-state-metrics from v1.9.5 to v1.9.6
- Update node-exporter from v1.0.0-rc.0 to v1.0.0-rc.1
- Update Grafana from v6.7.2 to v7.0.0
v1.18.2
- Kubernetes v1.18.2
- Choose Fedora CoreOS or Flatcar Linux (action required)
- Use a
fedora-coreos
module for Fedora CoreOS - Use a
container-linux
module for Flatcar Linux
- Use a
- Change Container Linux modules' defaults from CoreOS Container Linux to Flatcar Container Linux (#702)
- CoreOS Container Linux won't receive updates after May 2020
Fedora CoreOS
- Fix bootstrap race condition from SELinux unshared content label (#708)
Azure
- Add support for Fedora CoreOS (#704)
DigitalOcean
- Fix race condition creating firewall allow rules (#709)
Flatcar Linux
AWS
- Change
os_image
default fromcoreos-stable
toflatcar-stable
(#702)
Azure
- Change
os_image
to be required. Recommend uploading a Flatcar Linux image (action required) (#702) - Disable Flatcar Linux Azure Marketplace image support (breaking, #707)
- Revert to manual uploading until marketplace issue is closed (#703)
Bare-Metal
- Recommend changing os_channel from
coreos-stable
toflatcar-stable
- Change
os_image
to be required. Recommend uploading a Flatcar Linux image (action required) (#702)
DigitalOcean
- Change
os_image
to be required. Recommend uploading a Flatcar Linux image (action required) (#702) - Fix race condition creating firewall allow rules (#709)
v1.18.1
- Kubernetes v1.18.1
- Choose Fedora CoreOS or Flatcar Linux (action recommended)
- Use a
fedora-coreos
module for Fedora CoreOS - Use a
container-linux
module with OS set to Flatcar Linux
- Use a
- Update etcd from v3.4.5 to v3.4.7
- Change
kube-proxy
andcalico
orflannel
to tolerate specific taints (#682)- Tolerate master and not-ready taints, rather than tolerating all taints
- Update flannel from v0.11.0 to v0.12.0 (#690)
- Fix bootstrap when
networking
modeflannel
(non-default) is chosen (#689)- Regressed in v1.18.0 changes for Calico (#675)
- Rename Container Linux
controller_clc_snippets
tocontroller_snippets
for consistency (#688) - Rename Container Linux
worker_clc_snippets
toworker_snippets
for consistency - Rename Container Linux
clc_snippets
(bare-metal) tosnippets
for consistency - Drop support for gitRepo volumes (kubelet#3)
Azure
- Fix Azure worker UDP outbound connections (#691)
- Fix Azure worker clock sync timeouts
DigitalOcean
- Add support for Fedora CoreOS (#699)
Addons
- Refresh Prometheus rules/alerts and Grafana dashboards (#692)
- Update Grafana from v6.7.1 to v6.7.2
v1.18.0
- Kubernetes v1.18.0
- Update etcd from v3.4.4 to v3.4.5
- Switch from upstream hyperkube image to individual images (#669)
- Use upstream k8s.gcr.io
kube-apiserver
,kube-controller-manager
,kube-scheduler
, andkube-proxy
container images - Use poseidon/kubelet to package the upstream Kubelet binary and dependencies as a container image (checksummed, automated build)
- Add quay.io/poseidon/kubelet as a Typhoon distributed artifact in the security policy
- Update base images from debian 9 to debian 10
- Background: Kubernetes will stop releasing the hyperkube container image and provide the Kubelet as a binary for packaging
- Use upstream k8s.gcr.io
- Choose Fedora CoreOS or Flatcar Linux (action recommended)
- Use a
fedora-coreos
module for Fedora CoreOS - Use a
container-linux
module with OS set for Flatcar Linux (varies, see docs) - CoreOS Container Linux won't receive updates after May 2020
- Use a
- Add support for Fedora CoreOS snippets (
terraform-provider-ct
v0.5+) (#686) - Recommend updating
terraform-provider-ct
plugin from v0.4.0 to v0.5.0 - Set Fedora CoreOS log driver back to the default
journald
(#681) - Deprecate
asset_dir
variable and remove docs (#678) - Deprecate support for gitRepo volumes. A future release will drop support.
AWS
- Fix Fedora CoreOS AMI to filter for stable images (#685)
- Latest Fedora CoreOS
testing
orbodhi-update
images could be chosen depending on the region
- Latest Fedora CoreOS
Bare-Metal
- Update Fedora CoreOS default
os_stream
from testing to stable
Google Cloud
- Known: Use of stale Fedora CoreOS image may require terraform re-apply during bootstrap (#687)
DigitalOcean
- Rename
image
variable toos_image
for consistency (#677) (action required)
Addons
v1.17.4
- Kubernetes v1.17.4
- Update etcd from v3.4.3 to v3.4.4
- On Container Linux, fetch using the docker transport format (#659)
- Update CoreDNS from v1.6.6 to v1.6.7 (#648)
- Update Calico from v3.12.0 to v3.13.1
AWS
- Promote Fedora CoreOS to stable (#668)
- Allow VPC route table extension via reference (#654)
- Fix
worker_node_labels
on Fedora CoreOS (#651) - Fix automatic worker node delete on shutdown on Fedora CoreOS (#657)
Azure
- Upgrade to
terraform-provider-azurerm
v2.0+ (action required)- Change
worker_priority
fromLow
toSpot
if used (action required) - Switch to Azure's new Linux VM and Linux VM Scale Set resources
- Set controller's Azure disk caching to None
- Associate subnets (in addition to NICs) with security groups (aesthetic)
- Change
- Add support for Flatcar Container Linux (#664)
- Requires accepting Flatcar Linux Azure Marketplace terms
Bare-Metal
- Add
worker_node_labels
map variable for per-worker node labels (#663) - Add
worker_node_taints
map variable for per-worker node taints (#663)
DigitalOcean
- Add support for Flatcar Container Linux (#644)
Google Cloud
- Promote Fedora CoreOS to beta (#668)
- Fix
worker_node_labels
on Fedora CoreOS (#651) - Fix automatic worker node delete on shutdown on Fedora CoreOS (#657)
Addons
- Update nginx-ingress from v0.28.0 to v0.30.0
- Update Prometheus from v2.15.2 to v2.16.0
- Refresh Prometheus rules and alerts
- Add a BlackboxProbeFailure alert
- Update kube-state-metrics from v1.9.4 to v1.9.5
- Update node-exporter from v0.18.1 to v1.0.0-rc.0
- Update Grafana from v6.6.1 to v6.6.2
- Refresh Grafana dashboards
- Remove Container Linux Update Operator (CLUO) addon example (#667)
- CLUO hasn't been in active use in our clusters and won't be relevant beyond Container Linux. Requires patches for use on Kubernetes v1.16+
v1.17.3
- Kubernetes v1.17.3
- Update Calico from v3.11.2 to v3.12.0
- Allow Fedora CoreOS clusters to pass CNCF conformance suite
- Set Docker log driver to
json-file
as a workaround
- Set Docker log driver to
- Try Fedora CoreOS or Flatcar Linux alongside CoreOS Container Linux clusters (recommended)
AWS
- Promote Fedora CoreOS to beta (#645)
Bare-Metal
Google Cloud
Addons
- Update nginx-ingress from v0.27.1 to v0.28.0
- Update kube-state-metrics from v1.9.3 to v1.9.4
- Update Grafana from v6.5.3 to v6.6.1
v1.17.2
- Kubernetes v1.17.2
AWS
- Promote Fedora CoreOS from preview to alpha
Bare-Metal
- Promote Fedora CoreOS from preview to alpha
- Update Fedora CoreOS images location
- Use Fedora CoreOS production download streams
- Use live PXE kernel and initramfs images
Addons
- Update nginx-ingress from v0.26.1 to v0.27.1 (#625)
- Change runAsUser from 33 to 101 for alpine-based image
- Update kube-state-metrics from v1.9.2 to v1.9.3
v1.17.1
- Kubernetes v1.17.1
- Update CoreDNS from v1.6.5 to v1.6.6 (#602)
- Update Calico from v3.10.2 to v3.11.2 (#604)
- Inline Kubelet service on Container Linux nodes (#606)
- Disable unused Kubelet
127.0.0.1:10248
healthz listener (#607) - Enable kube-proxy metrics and allow Prometheus scrapes
- Allow TCP/10249 traffic with worker node sources
AWS
- Update Fedora CoreOS AMI filter for fedora-coreos-31 (#620)
- Allow
terraform-provider-google
v3.0+ (#617)- Only enforce
v2.19+
to ease migration, as no v3.x features are used
- Only enforce
Addons
- Update Prometheus from v2.14.0 to v2.15.2
- Add discovery for kube-proxy service endpoints
- Update kube-state-metrics from v1.8.0 to v1.9.2
- Reduce node-exporter DaemonSet tolerations (#614)
- Update Grafana from v6.5.1 to v6.5.3
v1.17.0
- Kubernetes v1.17.0
- Manage clusters without using a local
asset_dir
(#595)- Change
asset_dir
to be optional. Remove the variable to skip writing assets locally (action recommended) - Allow keeping cluster assets only in Terraform state (pluggable, encryption) and allow
terraform apply
from stateless automation systems - Improve asset unpacking on controllers
- Obtain kubeconfig from Terraform module outputs
- Change
- Replace usage of
template_dir
withtemplatefile
function (#587)- Require Terraform version v0.12.6+ (action required)
- Update CoreDNS from v1.6.2 to v1.6.5 (#588)
- Add health
lameduck
option to wait before shutdown
- Add health
- Update Calico from v3.10.1 to v3.10.2 (#599)
- Reduce pod eviction timeout for deleting pods on unready nodes from 5m to 1m (#597)
- Present since v1.13.3, but mistakenly removed in v1.16.0
- Add CPU requests for control plane static pods (#589)
- May provide slight edge case benefits and aligns with upstream
- Use new
google_compute_region_instance_group_manager
version block format- Fixes warning that
instance_template
is deprecated - Require
terraform-provider-google
v2.19.0+ (action required)
- Fixes warning that
Addons
- Update Grafana from v6.4.4 to v6.5.1
- Add pod networking details in dashboards (#593)
- Add node alerts and Grafana dashboard from node-exporter (#591)
- Reduce Prometheus high cardinality time series (#596)
v1.16.3
- Kubernetes v1.16.3
- Update etcd from v3.4.2 to v3.4.3 (#582)
- Upgrade Calico from v3.9.2 to v3.10.1
- Allow advertising service ClusterIPs to peer routers via a BGPConfiguration
- Switch
kube-proxy
from iptables to ipvs mode (#574)
Addons
- Update Prometheus from v2.13.0 to v2.14.0
- Refresh rules, alerts, and dashboards from upstreams
- Remove addon-resizer from kube-state-metrics (#575)
- Update Grafana from v6.4.2 to v6.4.4
v1.16.2
- Kubernetes v1.16.2
- Update etcd from v3.4.1 to v3.4.2 (#570)
- Update Calico from v3.9.1 to v3.9.2
- Default to using Calico and supporting NetworkPolicy on all platforms
Azure
- Change default networking provider from "flannel" to "calico" (#573)
Bare-Metal
- Add
controllers
andworkers
as typed lists of machine detail objects (#566)- Define clusters' machines cleanly and with Terraform v0.12 type constraints (action required, see PR example)
- Remove
controller_names
,controller_macs
, andcontroller_domains
variables - Remove
worker_names
,worker_macs
, andworker_domains
variables
DigitalOcean
- Change default networking provider from "flannel" to "calico" (#573)
Addons
- Update Grafana from v6.4.1 to v6.4.2
- Change CLUO label from "app" to "name"
v1.16.1
- Kubernetes v1.16.1
- Update etcd from v3.4.0 to v3.4.1
- Update Calico from v3.8.2 to v3.9.1
- Add Terraform v0.12 variables types (#553, #557, #560, #556, #562)
- Deprecate
cluster_domain_suffix
variable
- Deprecate
AWS
- Add
worker_node_labels
variable to set initial worker node labels (#550) - Add
node_labels
variable to internalworkers
pool module (#550) - For Fedora CoreOS, detect most recent AMI in the region
Azure
- Promote
networking
provider Calico VXLAN out of experimental (setnetworking = "calico"
) - Add
worker_node_labels
variable to set initial worker node labels (#550) - Add
node_labels
variable to internalworkers
pool module (#550) - Change
workers
module defaultvm_type
toStandard_DS1_v2
(followup to #539)
Bare-Metal
- For Fedora CoreOS, use new kernel, initrd, and raw paths (#563)
- Fix Terraform missing comma error (#549)
- Remove deprecated
container_linux_oem
variable (#562)
DigitalOcean
- Promote
networking
provider Calico VXLAN out of experimental (setnetworking = "calico"
) - Fix Terraform missing comma error (#549)
Google Cloud
- Add
worker_node_labels
variable to set initial worker node labels (#550) - Add
node_labels
variable to internalworkers
module (#550)
Addons
- Update Prometheus from v2.12.0 to v2.13.0
- Fix Prometheus etcd target discovery and scraping (#561, regressed with Kubernetes v1.16.0)
- Update kube-state-metrics from v1.7.2 to v1.8.0
- Update nginx-ingress from v0.25.1 to v0.26.1 (#555)
- Add lifecycle hook to allow draining for up to 5 minutes
- Update Grafana from v6.3.5 to v6.4.1
v1.16.0
- Kubernetes v1.16.0 (#543)
- Read about several Kubernetes API deprecations!
- Remove legacy node role labels (no longer shown in
kubectl get nodes
) - Rename node labels to
node.kubernetes.io/master
andnode.kubernetes.io/node
(migratory)
- Migrate control plane from self-hosted to static pods (#536)
- Run
kube-apiserver
,kube-scheduler
, andkube-controller-manager
as static pods on each controller kubectl
edits tokube-apiserver
,kube-scheduler
, andkube-controller-manager
are no longer possible (change)- Remove bootkube, self-hosted pivot, and
pod-checkpointer
- Run
- Update CoreDNS from v1.5.0 to v1.6.2 (#535)
- Update etcd from v3.3.15 to v3.4.0
- Recommend updating
terraform-provider-ct
plugin from v0.3.2 to v0.4.0
Azure
- Change default
controller_type
toStandard_B2s
(#539)B2s
is cheaper by $17/month and provides 2 vCPU, 4GB RAM
- Change default
worker_type
toStandard_DS1_v2
(#539)F1
is previous generation.DS1_v2
is newer, similar cost, and supports Low Priority mode
Addons
- Update Grafana from v6.3.3 to v6.3.5
v1.15.3
AWS
- Enable root block device encryption by default (#527)
- Require
terraform-provider-aws
v2.23+ (action required)
- Require
Addons
- Update Prometheus from v2.11.0 to v2.12.0
- Update kube-state-metrics from v1.7.1 to v1.7.2
- Update Grafana from v6.2.5 to v6.3.3
- Use stable IDs for etcd, CoreDNS, and Nginx Ingress dashboards (#530)
- Update nginx-ingress from v0.25.0 to v0.25.1
- Fix Nginx security advisories
v1.15.2
- Kubernetes v1.15.2
- Update Calico from v3.8.0 to v3.8.1
- Publish new load balancing, TCP/UDP, and firewall docs (#523)
Addons
- Add new Grafana dashboards for CoreDNS and Nginx Ingress Controller (#525)
v1.15.1
- Kubernetes v1.15.1
- Upgrade Calico from v3.7.3 to v3.8.0
- Enable CNI
bandwidth
plugin for traffic shaping
- Enable CNI
- Run
kube-apiserver
with lower privilege user (nobody) (#506) - Relax
terraform-provider-ct
version constraint (v0.3.2+)- Allow provider versions below v1.0.0 (e.g. upgrading to v0.4)
Azure
- Fix to add all controller nodes to the apiserver load balancer backend address pool (#518)
- kube-apiserver availability relied on the 0th controller
Google Cloud
- Allow controller nodes to span more than 3 zones if available in a region (#504)
- Eliminate extraneous controller instance groups in single-controller clusters (#504)
- Raise network deletion timeout from 4m to 6m (#505)
Addons
- Update Prometheus from v2.10.0 to v2.11.0
- Refresh rules, alerts, and dashboards from upstreams
- Update kube-state-metrics from v1.6.0 to v1.7.1
- Update Grafana from v6.2.4 to v6.2.5
- Update nginx-ingress from v0.24.1 to v0.25.0
- Support
networking.k8s.io/v1beta1
apiVersion
- Support
v1.15.0
- Kubernetes v1.15.0
- Migrate from Terraform v0.11 to v0.12.x (action required!)
- Migration instructions for Terraform v0.12
- Require
terraform-provider-ct
v0.3.2+ to support Terraform v0.12 (action required) - Update Calico from v3.7.2 to v3.7.3
- Remove Fedora Atomic modules (deprecated in March) (#501)
AWS
- Require
terraform-provider-aws
v2.7+ to support Terraform v0.12 (action required) - Allow using Flatcar Linux Edge by setting
os_image
to "flatcar-edge"
Azure
- Require
terraform-provider-azurerm
v1.27+ to support Terraform v0.12 (action required) - Avoid unneeded rotations of Regular priority virtual machine scale sets
- Azure only allows
eviction_policy
to be set for Low priority VMs. Supporting Low priority VMs meant when Regular VMs were used, eachterraform apply
rolled workers, to set eviction_policy to null. - Terraform v0.12 nullable variables fix the issue so plan does not produce a diff.
- Azure only allows
Bare-Metal
- Require
terraform-provider-matchbox
v0.3.0+ to support Terraform v0.12 (action required) - Allow using Flatcar Linux Edge by setting
os_channel
to "flatcar-edge"
DigitalOcean
- Require
terraform-provider-digitalocean
v1.3+ to support Terraform v0.12 (action required) - Change the default
worker_type
froms-1vcpu1-1gb
tos-1vcpu-2gb
Google Cloud
- Require
terraform-provider-google
v2.5+ to support Terraform v0.12 (action required)
Addons
- Update Grafana from v6.2.1 to v6.2.4
- Update node-exporter from v0.18.0 to v0.18.1
v1.14.3
- Kubernetes v1.14.3
- Update CoreDNS from v1.3.1 to v1.5.0
- Add
ready
plugin to improve readinessProbe
- Add
- Fix trailing slash in terraform-render-bootkube version (#479)
- Recommend updating
terraform-provider-ct
plugin from v0.3.1 to v0.3.2 (#487)
AWS
- Rename
worker
pool modulecount
variable toworker_count
(#485) (action required)count
will become a reserved variable name in Terraform v0.12
Azure
- Replace
azurerm_autoscale_setting
withazurerm_monitor_autoscale_setting
(#482) - Rename
worker
pool modulecount
variable toworker_count
(#485) (action required)count
will become a reserved variable name in Terraform v0.12
Bare-Metal
Google Cloud
- Rename
worker
pool modulecount
variable toworker_count
(#485) (action required)count
is a reserved variable in Terraform v0.12
Addons
- Update Prometheus from v2.9.2 to v2.10.0
- Update Grafana from v6.1.6 to v6.2.1
v1.14.2
- Kubernetes v1.14.2
- Update etcd from v3.3.12 to v3.3.13
- Upgrade Calico from v3.6.1 to v3.7.2
- Change flannel VXLAN port from 8472 (kernel default) to 4789 (IANA VXLAN)
AWS
- Only set internal VXLAN rules when
networking
is "flannel" (default: calico)
Azure
- Allow choosing Calico as the network provider (experimental) (#472)
- Add a
networking
variable accepting "flannel" (default) or "calico" - Use VXLAN encapsulation since Azure doesn't support IPIP
- Add a
DigitalOcean
- Allow choosing Calico as the network provider (experimental) (#472)
- Add a
networking
variable accepting "flannel" (default) or "calico" - Use VXLAN encapsulation since DigitalOcean doesn't support IPIP
- Add a
- Add explicit ordering between firewall rule creation and secure copying Kubelet credentials (#469)
- Fix race scenario if copies to nodes were before rule creation, blocking cluster creation
Addons
- Update Prometheus from v2.8.1 to v2.9.2
- Update kube-state-metrics from v1.5.0 to v1.6.0
- Update node-exporter from v0.17.0 to v0.18.0
- Update Grafana from v6.1.3 to v6.1.6
- Reduce nginx-ingress Role RBAC permissions (#458)
v1.14.1
- Kubernetes v1.14.1
Addons
- Update Grafana from v6.1.1 to v6.1.3
- Update nginx-ingress from v0.23.0 to v0.24.1
v1.14.0
- Kubernetes v1.14.0
- Update Calico from v3.6.0 to v3.6.1
- Add
enable_aggregation
option for CNCF conformance (#436)- Aggregation is disabled by default to retain our security stance
- Aggregation increases the security surface area. Extensions become part of the control plane and must be scrutinized carefully and trusted. Favor leaving aggregation disabled.
AWS
- Add ability to load balance TCP applications (#443)
- Output the network load balancer ARN as
nlb_id
- Accept a
worker_target_groups
(ARN) list to which worker instances should be added
- Output the network load balancer ARN as
Azure
- Add ability to load balance TCP/UDP applications (#447)
- Output the load balancer ID as
loadbalancer_id
- Output the load balancer ID as
- Output
worker_security_group_name
andworker_address_prefix
for extending firewall rules (#447)
DigitalOcean
- Harden internal (node-to-node) firewall rules to align with other platforms (#444)
- Add ability to load balance TCP applications (#444)
- Output
controller_tag
andworker_tag
for extending firewall rules (#444)
- Output
Google Cloud
- Add ability to load balance TCP/UDP applications (#442)
- Add worker instances to a target pool, output as
worker_target_pool
- Health check for workers with Ingress controllers. Forward rules don't support differing internal/external ports, but some Ingress controllers support TCP/UDP proxy as a workaround
- Add worker instances to a target pool, output as
- Remove Haswell minimum CPU platform requirement (#439)
Addons
- Update Prometheus from v2.8.0 to v2.8.1
- Update Grafana from v6.0.2 to v6.1.1
- Add dashboard for pods in a workload (deployment/daemonset/statefulset) (#446)
- Add dashboard for workloads by namespace
v1.13.5
- Kubernetes v1.13.5
- Resolve in-addr.arpa reverse DNS lookups (PTR) for pod IPv4 addresses (#415)
- Reverse DNS lookups for service IPv4 addresses unchanged
- Upgrade Calico from v3.5.2 to v3.6.0 (#430)
- Change pod IPAM from
host-local
tocalico-ipam
.pod_cidr
is still divided into/24
subnets per node, but managed asippools
andipamblocks
- Change pod IPAM from
- Recommend updating terraform-provider-ct from v0.3.0 to v0.3.1 (#434)
- Announce: Fedora Atomic modules will be not be updated beyond Kubernetes v1.13.x (#437)
- Thank you Project Atomic team and users, please see the deprecation notice
AWS
- Support
terraform-provider-aws
v2.0+ (#419)
Bare-Metal
- Change the default iPXE kernel and initrd download protocol from HTTP to HTTPS (#420)
- Require an iPXE-enabled network boot environment with support for TLS downloads. PXE clients must chainload to iPXE firmware compiled with
DOWNLOAD_PROTO_HTTPS
enabled. (action required) - Only affects Container Linux and Flatcar Linux install profiles that pull public images (default)
- Add
download_protocol
variable. Recognizing boot firmware TLS support is difficult in some environments, set the protocol to "http" for the old behavior (discouraged)
- Require an iPXE-enabled network boot environment with support for TLS downloads. PXE clients must chainload to iPXE firmware compiled with
DigitalOcean
- Fix kubelet hostname-override to set node metadata InternalIP correctly (#424)
- Uniquely, DigitalOcean does not resolve hostnames to instance private IPs. Kubelet auto-detect mechanisms require the internal IP be set directly.
- Regressed in v1.12.3 (#337) which aimed to provide friendly hostname-based node names on DigitalOcean
Addons
- Update Prometheus from v2.7.1 to v2.8.0
- Refresh rules based on upstreams (#426)
- Define NetworkPolicy to allow only traffic from the Grafana addon
- Update Grafana from v6.0.0 to v6.0.2
- Add liveness and readiness probes
- Refresh dashboards and organize to stay below ConfigMap size limit (#426)
- Remove heapster manifests from addons (#427)
- Heapster addon powers
kubectl top
(in early Kubernetes, running the addon was expected). Today, there are better monitoring options. kubectl top
reliance on a non-core extension means its not in-scope for minimal Kubernetes- Look to prior releases if you still wish to apply heapster
- Heapster addon powers
v1.13.4
- Kubernetes v1.13.4
- Update etcd from v3.3.11 to v3.3.12
- Update Calico from v3.5.0 to v3.5.2
- Assign priorityClassNames to critical cluster and node components (#406)
- Inform node out-of-resource eviction and scheduler preemption and ordering
- Add CoreDNS readiness probe (#410)
Bare-Metal
- Recommend updating terraform-provider-matchbox plugin from v0.2.2 to v0.2.3 (#402)
- Improve docs on using Ubiquiti EdgeOS with bare-metal clusters (#413)
Google Cloud
- Support
terraform-provider-google
v2.0+ (#407)- Require
terraform-provider-google
v1.19+ (action required)
- Require
- Set the minimum CPU platform to Intel Haswell (#405)
- Haswell or better is available in every zone (no price change)
- A few zones still default to Sandy/Ivy Bridge (shifts in April 2019)
Addons
- Modernize Prometheus rules and alerts (#404)
- Drop extraneous metrics (#397)
- Add
pod
name label to metrics discovered via service endpoints - Rename
kubernetes_namespace
label tonamespace
- Modernize Grafana and dashboards, see docs (#403, #404)
- Update nginx-ingress from v0.22.0 to v0.23.0
- Raise nginx-ingress liveness/readiness timeout to 5 seconds
- Remove nginx-ingess default-backend (#401)
Fedora Atomic
- Build Kubelet system container with buildah. The image is an OCI format and slightly larger.
v1.13.3
- Kubernetes v1.13.3
- Update etcd from v3.3.10 to v3.3.11
- Update CoreDNS from v1.3.0 to v1.3.1
- Switch from the
proxy
plugin to the fasterforward
plugin for upsteam resolvers
- Switch from the
- Update Calico from v3.4.0 to v3.5.0
- Update flannel from v0.10.0 to v0.11.0
- Reduce pod eviction timeout for deleting pods on unready nodes to 1 minute
- Respond more quickly to node preemption (previously 5 minutes)
- Fix automatic worker deletion on shutdown for cloud platforms
- Lowering Kubelet privileges in #372 dropped a needed node deletion authorization. Scale-in due to manual terraform apply (any cloud), AWS spot termination, or Azure low priority deletion left old nodes registered, requiring manual deletion (
kubectl delete node name
)
- Lowering Kubelet privileges in #372 dropped a needed node deletion authorization. Scale-in due to manual terraform apply (any cloud), AWS spot termination, or Azure low priority deletion left old nodes registered, requiring manual deletion (
AWS
- Add
ingress_zone_id
output with the NLB DNS name's Route53 zone for use in alias records (#380)
Azure
- Fix azure provider warning,
public_ip
allocation_method
replacespublic_ip_address_allocation
- Require
terraform-provider-azurerm
v1.21+ (action required)
- Require
Addons
- Update nginx-ingress from v0.21.0 to v0.22.0
- Update Prometheus from v2.6.0 to v2.7.1
- Update kube-state-metrics from v1.4.0 to v1.5.0
- Fix ClusterRole to collect and export PodDisruptionBudget metrics (#383)
- Update node-exporter from v0.15.2 to v0.17.0
- Update Grafana from v5.4.2 to v5.4.3
v1.13.2
- Kubernetes v1.13.2
- Add ServiceAccounts for
kube-apiserver
andkube-scheduler
(#370) - Use lower-privilege TLS client certificates for Kubelets (#372)
- Use HTTPS liveness probes for
kube-scheduler
andkube-controller-manager
(#377) - Update CoreDNS from v1.2.6 to v1.3.0
- Allow the
certificates.k8s.io
API to issue certificates signed by the cluster CA (#376)- Configure controller manager to sign CSRs that are manually approved by an administrator
AWS
- Change
controller_type
andworker_type
default from t2.small to t3.small (#365)- t3.small is cheaper, provides 2 vCPU (instead of 1), and 5 Gbps of pod-to-pod bandwidth!
Bare-Metal
- Remove the
kubeconfig
output variable
Addons
- Update Prometheus from v2.5.0 to v2.6.0
v1.13.1
- Kubernetes v1.13.1
- Update Calico from v3.3.2 to v3.4.0 (#362)
- Install CNI plugins with an init container rather than a sidecar
- Improve the
calico-node
ClusterRole
- Recommend updating
terraform-provider-ct
plugin from v0.2.1 to v0.3.0 (#363)- Migration instructions for upgrading
terraform-provider-ct
in-place for v1.12.2+ clusters (action required) - Require switching from
~/.terraformrc
to the Terraform third-party plugins directory~/.terraform.d/plugins/
- Require Container Linux 1688.5.3 or newer
- Migration instructions for upgrading
Google Cloud
- Increase TCP proxy apiserver backend service timeout from 1 minute to 5 minutes (#361)
- Align
port-forward
behavior closer to AWS/Azure (no timeout)
- Align
Addons
- Update Grafana from v5.4.0 to v5.4.2
v1.13.0
Addons
- Update Grafana from v5.3.4 to v5.4.0
- Disable Grafana login form, since admin user can't be disabled (#352)
- Example manifests aim to provide a read-only dashboard view
v1.12.3
- Kubernetes v1.12.3
- Add
enable_reporting
variable (default "false") to provide upstreams with usage data (#345) - Change kube-apiserver
--kubelet-preferred-address-types
to InternalIP,ExternalIP,Hostname - Update Calico from v3.3.0 to v3.3.1
- Disable Felix usage reporting by default (#345)
- Improve flannel manifests
- Update CoreDNS from v1.2.4 to v1.2.6
- Enable CoreDNS
loop
andloadbalance
plugins (#340)
- Enable CoreDNS
- Fix pod-checkpointer log noise and checkpointable pods detection (#346)
- Use kubernetes-incubator/bootkube v0.14.0
- Recommend switching from
~/.terraformrc
to the Terraform third-party plugins directory~/.terraform.d/plugins/
.- Allows pinning
terraform-provider-ct
andterraform-provider-matchbox
versions - Improves safety of later plugin version migrations
- Allows pinning
Azure
- Use eviction policy
Delete
forLow
priority virtual machine scale set workers (#343)- Fix issue where Azure defaults to
Deallocate
eviction policy, which required manually restarting deallocated instances.Delete
policy aligns Azure with AWS and GCP behavior. - Require
terraform-provider-azurerm
v1.19+ (action required)
- Fix issue where Azure defaults to
Bare-Metal
- Add Kubelet
/etc/iscsi
andiscsadm
mounts on bare-metal for iSCSI (#103)
Addons
- Update nginx-ingress from v0.20.0 to v0.21.0
- Update Prometheus from v2.4.3 to v2.5.0
- Update Grafana from v5.3.2 to v5.3.4
v1.12.2
- Kubernetes v1.12.2
- Update CoreDNS from 1.2.2 to 1.2.4
- Update Calico from v3.2.3 to v3.3.0
- Disable Kubelet read-only port (#324)
- Fix CoreDNS AntiAffinity spec to prefer spreading replicas
- Ignore controller node user-data changes (#335)
- Once all managed clusters use v1.12.2, it is possible to update
terraform-provider-ct
- Once all managed clusters use v1.12.2, it is possible to update
AWS
- Add
disk_iops
variable for EBS volume IOPS (#314)
Azure
- Use new
azurerm_network_interface_backend_address_pool_association
(#332)- Require
terraform-provider-azurerm
v1.17+ (action required)
- Require
- Add
primary
field toip_configuration
needed by v1.17+ (#331)
DigitalOcean
- Add AAAA DNS records resolving to worker nodes (#333)
- Hosting IPv6 apps requires editing nginx-ingress with
hostNetwork: true
- Hosting IPv6 apps requires editing nginx-ingress with
Google Cloud
- Add an IPv6 address and IPv6 forwarding rules for load balancing IPv6 Ingress (#334)
- Add
ingress_static_ipv6
output variable for use in AAAA DNS records - Allow serving IPv6 applications via Kubernetes Ingress
- Add
Addons
- Configure Heapster to scrape Kubelets with bearer token auth (#323)
- Update Grafana from v5.3.1 to v5.3.2
v1.12.1
- Kubernetes v1.12.1
- Update etcd from v3.3.9 to v3.3.10
- Update CoreDNS from 1.1.3 to 1.2.2
- Update Calico from v3.2.1 to v3.2.3
- Raise scheduler and controller-manager replicas to the larger of 2 or the number of controller nodes (#312)
- Single-controller clusters continue to run 2 replicas as before
- Raise default CoreDNS replicas to the larger of 2 or the number of controller nodes (#313)
- Add AntiAffinity preferred rule to favor spreading CoreDNS pods
- Annotate control plane and addon containers to use the Docker runtime seccomp profile (#319)
- Override Kubernetes default behavior that starts containers with
seccomp=unconfined
- Override Kubernetes default behavior that starts containers with
Azure
- Remove
admin_password
field (disabled) since it is now optional- Require
terraform-provider-azurerm
v1.16+ (action required)
- Require
Bare-Metal
- Add support for
cached_install
mode with Flatcar Linux (#315)
DigitalOcean
- Require
terraform-provider-digitalocean
v1.0+ (action required)
Addons
- Update nginx-ingress from v0.19.0 to v0.20.0
- Update Prometheus from v2.3.2 to v2.4.3
- Update Grafana from v5.2.4 to v5.3.1
v1.11.3
- Kubernetes v1.11.3
- Introduce Typhoon for Azure as alpha (#288)
- Special thanks @justaugustus for an earlier variant
- Update Calico from v3.1.3 to v3.2.1 (#278)
AWS
- Remove firewall rule allowing ICMP packets to nodes (#285)
Bare-Metal
- Remove
controller_networkds
andworker_networkds
variables. Use Container Linux Config snippets #277
Google Cloud
- Fix firewall to allow etcd client port 2379 traffic between controller nodes (#287)
- kube-apiservers were only able to connect to their node's local etcd peer. While master node outages were tolerated, reaching a healthy peer took longer than neccessary in some cases
- Reduce time needed to bootstrap the cluster
- Remove firewall rule allowing workers to access Nginx Ingress health check (#284)
- Nginx Ingress addon no longer uses hostNetwork, Prometheus scrapes via CNI network
Addons
- Update nginx-ingress from 0.17.1 to 0.19.0
- Update kube-state-metrics from v1.3.1 to v1.4.0
- Update Grafana from 5.2.2 to 5.2.4
v1.11.2
- Kubernetes v1.11.2
- Update etcd from v3.3.8 to v3.3.9
- Use kubernetes-incubator/bootkube v0.13.0
- Fix Fedora Atomic modules' Kubelet version (#270)
Bare-Metal
- Introduce Container Linux Config snippets on bare-metal
- Validate and additively merge custom Container Linux Configs during terraform plan
- Define files, systemd units, dropins, networkd configs, mounts, users, and more
- Require
terraform-provider-ct
plugin v0.2.1 (action required!)
Addons
- Update nginx-ingress from 0.16.2 to 0.17.1
- Add nginx-ingress manifests for bare-metal
- Update Grafana from 5.2.1 to 5.2.2
- Update heapster from v1.5.3 to v1.5.4
v1.11.1
- Kubernetes v1.11.1
Addons
- Update Prometheus from v2.3.1 to v2.3.2
Errata
- Fedora Atomic modules shipped with Kubelet v1.11.0, instead of v1.11.1. Fixed in #270.
v1.11.0
- Kubernetes v1.11.0
- Force apiserver to stop listening on
127.0.0.1:8080
- Replace
kube-dns
with CoreDNS (#261)- Edit the
coredns
ConfigMap to customize - CoreDNS doesn't use a resizer. For large clusters, scaling may be required.
- Edit the
AWS
- Update from Fedora Atomic 27 to 28 (#258)
Bare-Metal
- Update from Fedora Atomic 27 to 28 (#263)
- Promote Google Cloud to stable
- Update from Fedora Atomic 27 to 28 (#259)
- Remove
ingress_static_ip
module output. Useingress_static_ipv4
. - Remove
controllers_ipv4_public
module output.
Addons
- Update nginx-ingress from 0.15.0 to 0.16.2
- Update Grafana from 5.1.4 to 5.2.1
- Update heapster from v1.5.2 to v1.5.3
v1.10.5
AWS
- Switch
kube-apiserver
port from 443 to 6443 (#248) - Combine apiserver and ingress NLBs (#249)
- Reduce cost by ~$18/month per cluster. Typhoon AWS clusters now use one network load balancer.
- Ingress addon users may keep using CNAME records to the
ingress_dns_name
module output (few million RPS) - Ingress users with heavy traffic (many million RPS) should create a separate NLB(s)
- Worker pools no longer include an extraneous load balancer. Remove worker module's
ingress_dns_name
output - Disable detailed (paid) monitoring on worker nodes (#251)
- Favor Prometheus for cloud-agnostic metrics, aggregation, and alerting
- Add
worker_target_group_http
andworker_target_group_https
module outputs to allow custom load balancing - Add
target_group_http
andtarget_group_https
worker module outputs to allow custom load balancing
Bare-Metal
- Switch
kube-apiserver
port from 443 to 6443 (#248)- Users who exposed kube-apiserver on a WAN via their router/load-balancer will need to adjust its configuration (e.g. DNAT 6443). Most apiservers are on a LAN (internal, VPN-only, etc) so if you didn't specially configure network gear for 443, no change is needed. (possible action required)
- Fix possible deadlock when provisioning clusters larger than 10 nodes (#244)
DigitalOcean
- Switch
kube-apiserver
port from 443 to 6443 (#248)- Update firewall rules and generated kubeconfig's
Google Cloud
- Use global HTTP and TCP proxy load balancing for Kubernetes Ingress (#252)
- Switch Ingress from regional network load balancers to global HTTP/TCP Proxy load balancing
- Reduce cost by ~$19/month per cluster. Google bills the first 5 global and regional forwarding rules separately. Typhoon clusters now use 3 global and 0 regional forwarding rules.
- Worker pools no longer include an extraneous load balancer. Remove worker module's
ingress_static_ip
output - Allow using nginx-ingress addon on Fedora Atomic clusters (#200)
- Add
worker_instance_group
module output to allow custom global load balancing - Add
instance_group
worker module output to allow custom global load balancing - Deprecate
ingress_static_ip
module output. Addingress_static_ipv4
module output instead. - Deprecate
controllers_ipv4_public
module output
Addons
- Update CLUO from v0.6.0 to v0.7.0 (#242)
- Update Prometheus from v2.3.0 to v2.3.1
- Update Grafana from 5.1.3 to 5.1.4
- Drop
hostNetwork
from nginx-ingress addon- Both flannel and Calico support host port via
portmap
- Allows writing NetworkPolicies that reference ingress pods in
from
orto
. HostNetwork pods were difficult to write network policy for since they could circumvent the CNI network to communicate with pods on the same node.
- Both flannel and Calico support host port via
v1.10.4
- Kubernetes v1.10.4
- Update etcd from v3.3.5 to v3.3.6
- Update Calico from v3.1.2 to v3.1.3
Addons
- Update Prometheus from v2.2.1 to v2.3.0
- Add Prometheus liveness and readiness probes
- Annotate Grafana service so Prometheus scrapes metrics
- Label namespaces to ease writing Network Policies
v1.10.3
- Kubernetes v1.10.3
- Add Flatcar Linux (Container Linux derivative) as an option for AWS and bare-metal (thanks @kinvolk folks)
- Allow bearer token authentication to the Kubelet (#216)
- Require Webhook authorization to the Kubelet
- Switch apiserver X509 client cert org to satisfy new authorization requirement
- Require Terraform v0.11.x and drop support for v0.10.x (migration guide)
- Update etcd from v3.3.4 to v3.3.5 (#213)
- Update Calico from v3.1.1 to v3.1.2
AWS
- Allow Flatcar Linux by setting
os_image
to flatcar-stable (default), flatcar-beta, flatcar-alpha (#211) - Replace
os_channel
variable withos_image
to align naming across clouds- Please change values stable, beta, or alpha to coreos-stable, coreos-beta, coreos-alpha (action required!)
- Allow preemptible workers via spot instances (#202)
- Add
worker_price
to allow worker spot instances. Default to empty string for the worker autoscaling group to use regular on-demand instances - Add
spot_price
to internalworkers
module for spot worker pools
- Add
Bare-Metal
- Allow Flatcar Linux by setting
os_channel
to flatcar-stable, flatcar-beta, flatcar-alpha (#220) - Replace
container_linux_channel
variable withos_channel
- Please change values stable, beta, or alpha to coreos-stable, coreos-beta, coreos-alpha (action required!)
- Replace
container_linux_version
variable withos_version
- Add
network_ip_autodetection_method
variable for Calico host IPv4 address detection- Use Calico's default "first-found" to support single NIC and bonded NIC nodes
- Allow alternative methods for multi NIC nodes, like can-reach=IP or interface=REGEX
- Deprecate
container_linux_oem
variable
DigitalOcean
- Update Fedora Atomic module to use Fedora Atomic 28 (#225)
- Fedora Atomic 27 images disappeared from DigitalOcean and forced this early update
Addons
- Fix Prometheus data directory location (#203)
- Configure Prometheus to scrape Kubelets directly with bearer token auth instead of proxying through the apiserver (#217)
- Security improvement: Drop RBAC permission from
nodes/proxy
tonodes/metrics
- Scale: Remove per-node proxied scrape load from the apiserver
- Security improvement: Drop RBAC permission from
- Update Grafana from v5.04 to v5.1.3 (#208)
- Disable Grafana Google Analytics by default (#214)
- Update nginx-ingress from 0.14.0 to 0.15.0
- Annotate nginx-ingress service so Prometheus auto-discovers and scrapes service endpoints (#222)
v1.10.2
- Kubernetes v1.10.2
- Introduce Typhoon for Fedora Atomic (#199)
- Update Calico from v3.0.4 to v3.1.1 (#197)
- Update etcd from v3.3.3 to v3.3.4
- Update kube-dns from v1.14.9 to v1.14.10
Google Cloud
- Add support for multi-controller clusters (i.e. multi-master) (#54, #190)
- Switch from Google Cloud network load balancer to a TCP proxy load balancer. Avoid a bug in Google network load balancers that limited clusters to only bootstrapping one controller node.
- Add TCP health check for apiserver pods on controllers. Replace kubelet check approximation.
Addons
- Update nginx-ingress from 0.12.0 to 0.14.0
- Update kube-state-metrics from v1.3.0 to v1.3.1
v1.10.1
- Kubernetes v1.10.1
- Enable etcd v3.3 metrics endpoint (#175)
- Use
k8s.gcr.io
instead ofgcr.io/google_containers
(#180)- Kubernetes recommends using the alias to pull from the nearest regional mirror and to abstract the backing container registry
- Update etcd from v3.3.2 to v3.3.3
- Update kube-dns from v1.14.8 to v1.14.9
- Use kubernetes-incubator/bootkube v0.12.0
Bare-Metal
- Fix need for multiple
terraform apply
runs to create a cluster with Terraform v0.11.4 (#181)- To SSH during a disk install for debugging, SSH as user "core" with port 2222
- Remove the old trick of using a user "debug" during disk install
Google Cloud
- Refactor out the
controller
internal module
Addons
- Add Prometheus discovery for etcd peers on controller nodes (#175)
- Scrape etcd v3.3
--listen-metrics-urls
for metrics - Enable etcd alerts and populate the etcd Grafana dashboard
- Scrape etcd v3.3
- Update kube-state-metrics from v1.2.0 to v1.3.0
v1.10.0
- Kubernetes v1.10.0
- Remove unused, unmaintained
pxe-worker
internal module
AWS
- Add
disk_type
optional variable for setting the EBS volume type (#176)- Change default type from
standard
togp2
. Prometheus etcd alerts are tuned for fast disks.
- Change default type from
Digital Ocean
- Ensure etcd secrets are only distributed to controller hosts, not workers.
- Remove
networking
optional variable. Only flannel works on Digital Ocean.
Google Cloud
- Add
disk_size
optional variable for setting instance disk size in GB - Add
controller_type
optional variable for setting machine type for controllers - Add
worker_type
optional variable for setting machine type for workers - Remove
machine_type
optional variable. Usecontroller_type
andworker_type
.
Addons
v1.9.6
- Kubernetes v1.9.6
- Update Calico from v3.0.3 to v3.0.4
Addons
- Update heapster from v1.5.1 to v1.5.2
v1.9.5
- Kubernetes v1.9.5
- Fix
subPath
volume mounts regression (kubernetes#61076)
- Fix
- Introduce Container Linux Config snippets on cloud platforms (#145)
- Validate and additively merge custom Container Linux Configs during
terraform plan
- Define files, systemd units, dropins, networkd configs, mounts, users, and more
- Require updating
terraform-provider-ct
plugin from v0.2.0 to v0.2.1
- Validate and additively merge custom Container Linux Configs during
- Add
node-role.kubernetes.io/controller="true"
node label to controllers (#160)
AWS
Digital Ocean
Google Cloud
- Require updating
terraform-provider-ct
plugin from v0.2.0 to v0.2.1 (action required!) - Relax
os_image
to optional. Default to "coreos-stable".
Addons
- Update nginx-ingress from 0.11.0 to 0.12.0
- Update Prometheus from 2.2.0 to 2.2.1
v1.9.4
- Kubernetes v1.9.4
- Secret, configMap, downward API, and projected volumes now read-only (breaking, kubernetes#58720)
- Regressed
subPath
volume mounts (regression, kubernetes#61076) - Mitigated
subPath
CVE-2017-1002101
- Introduce worker pools for AWS and Google Cloud for joining heterogeneous workers to existing clusters.
- Use new Network Load Balancers and cross zone load balancing on AWS
- Allow flexvolume plugins to be used on any Typhoon cluster (not just bare-metal)
- Upgrade etcd from v3.2.15 to v3.3.2
- Update Calico from v3.0.2 to v3.0.3
- Use kubernetes-incubator/bootkube v0.11.0
- Recommend updating
terraform-provider-ct
plugin from v0.2.0 to v0.2.1 (action recommended)
AWS
- Promote AWS platform to stable
- Allow groups of workers to be defined and joined to a cluster (i.e. worker pools) (#150)
- Replace the apiserver elastic load balancer with a network load balancer (#136)
- Replace the Ingress elastic load balancer with a network load balancer (#141)
- AWS NLBs can handle millions of RPS with high throughput and low latency.
- Require
terraform-provider-aws
1.7.0 or higher
- Enable NLB cross-zone load balancing (#159)
- Requests are automatically evenly distributed to targets regardless of AZ
- Require
terraform-provider-aws
1.11.0 or higher
- Add kubelet
--volume-plugin-dir
flag to allow flexvolume plugins (#142) - Fix controller and worker launch configs to ignore AMI changes (#126, #158)
Digital Ocean
- Add kubelet
--volume-plugin-dir
flag to allow flexvolume plugins (#142) - Fix to pass
ssh_fingerprints
as a list to droplets (#143)
Google Cloud
- Allow groups of workers to be defined and joined to a cluster (i.e. worker pools) (#148)
- Add kubelet
--volume-plugin-dir
flag to allow flexvolume plugins (#142) - Add
kubeconfig
variable tocontrollers
andworkers
submodules (#147) - Remove
kubeconfig_*
variables fromcontrollers
andworkers
submodules (#147) - Allow initial experimentation with accelerators (i.e. GPUs) on workers (#161) (unofficial)
- Require
terraform-provider-google
v1.6.0
- Require
Addons
- Update Prometheus from 2.1.0 to 2.2.0 (#153)
- Scrape Prometheus itself to enable alerts about Prometheus itself
- Adjust KubeletDown rule to fire when 10% of kubelets are down
- Update heapster from v1.5.0 to v1.5.1 (#131)
- Use separate service account
- Update nginx-ingress from 0.10.2 to 0.11.0
v1.9.3
- Kubernetes v1.9.3
- Network improvements and fixes (#104)
- Switch from Calico v2.6.6 to v3.0.2
- Add Calico GlobalNetworkSet CRD
- Update flannel from v0.9.0 to v0.10.0
- Use separate service account for flannel
- Update etcd from v3.2.14 to v3.2.15
Digital Ocean
- Use new Droplet types which offer more CPU/memory, at lower cost. (#105)
- A small Digital Ocean cluster costs less than $25 a month!
Addons
- Update Prometheus from v2.0.0 to v2.1.0 (#113)
- Improve alerting rules
- Relabel discovered kubelet, endpoint, service, and apiserver scrapes
- Use separate service accounts
- Update node-exporter and kube-state-metrics
- Include Grafana dashboards for Kubernetes admins (#113)
- Add grafana-watcher to load bundled upstream dashboards
- Update nginx-ingress from 0.9.0 to 0.10.2
- Update CLUO from v0.5.0 to v0.6.0
- Switch manifests to use
apps/v1
Deployments and Daemonsets (#120) - Remove Kubernetes Dashboard manifests (#121)
v1.9.2
- Kubernetes v1.9.2
- Add Terraform v0.11.x support
- Add explicit "providers" section to modules for Terraform v0.11.x
- Retain support for Terraform v0.10.4+
- Add migration guide from Terraform v0.10.x to v0.11.x (action required!)
- Update etcd from 3.2.13 to 3.2.14
- Update calico from 2.6.5 to 2.6.6
- Update kube-dns from v1.14.7 to v1.14.8
- Use separate service account for kube-dns
- Use kubernetes-incubator/bootkube v0.10.0
Bare-Metal
- Use per-node Container Linux install profiles (#97)
- Allow Container Linux channel/version to be chosen per-cluster
- Fix issue where cluster deletion could require
terraform apply
multiple times
Digital Ocean
- Relax
digitalocean
provider version constraint - Fix bug with
terraform plan
always showing a firewall diff to be applied (#3)
Addons
- Update CLUO to v0.5.0 to fix compatibility with Kubernetes 1.9 (important)
- Earlier versions can't roll out Container Linux updates on Kubernetes 1.9 nodes (cluo#163)
- Update kube-state-metrics from v1.1.0 to v1.2.0
- Fix RBAC cluster role for kube-state-metrics
v1.9.1
- Kubernetes v1.9.1
- Update kube-dns from 1.14.5 to v1.14.7
- Update etcd from 3.2.0 to 3.2.13
- Update Calico from v2.6.4 to v2.6.5
- Enable portmap to fix hostPort with Calico
- Use separate service account for controller-manager
v1.8.6
- Kubernetes v1.8.6
- Update Calico from v2.6.3 to v2.6.4
v1.8.5
- Kubernetes v1.8.5
- Recommend Container Linux images with Docker 17.09
- Container Linux stable, beta, and alpha now provide Docker 17.09 (instead of 1.12)
- Older clusters (with CLUO addon) auto-update Container Linux version to begin using Docker 17.09
- Fix race where
etcd-member.service
could fail to resolve peers (#69) - Add optional
cluster_domain_suffix
variable (#74) - Use kubernetes-incubator/bootkube v0.9.1
Bare-Metal
- Add kubelet
--volume-plugin-dir
flag to allow flexvolume providers (#61)
Addons
- Discourage deploying the Kubernetes Dashboard (security)
v1.8.4
- Kubernetes v1.8.4
- Calico related bug fixes
- Update Calico from v2.6.1 to v2.6.3
- Update flannel from v0.9.0 to v0.9.1
- Service accounts for kube-proxy and pod-checkpointer
- Use kubernetes-incubator/bootkube v0.9.0
v1.8.3
- Kubernetes v1.8.3
- Run etcd on-host, across controllers
- Promote AWS platform to beta
- Use kubernetes-incubator/bootkube v0.8.2
Google Cloud
- Add required variable
region
(e.g. "us-central1") - Reduce time to bootstrap a cluster
- Change etcd to run on-host, across controllers (etcd-member.service)
- Change controller instances to automatically span zones in the region
- Change worker managed instance group to automatically span zones in the region
- Improve internal firewall rules and use tag-based firewall policies
- Remove support for self-hosted etcd
- Remove the
zone
required variable - Remove the
controller_preemptible
optional variable
AWS
- Promote AWS platform to beta
- Reduce time to bootstrap a cluster
- Change etcd to run on-host, across controllers (etcd-member.service)
- Fix firewall rules for multi-controller kubelet scraping and node-exporter
- Remove support for self-hosted etcd
Addons
- Add Prometheus 2.0 addon with alerting rules
- Add Grafana dashboard for observing metrics
v1.8.2
- Kubernetes v1.8.2
- Fixes a memory leak in the v1.8.1 apiserver (kubernetes#53485)
- Switch to using the
gcr.io/google_containers/hyperkube
- Update flannel from v0.8.0 to v0.9.0
- Add
hairpinMode
to flannel CNI config - Add
--no-negcache
to kube-dns dnsmasq - Use kubernetes-incubator/bootkube v0.8.1
v1.8.1
- Kubernetes v1.8.1
- Use kubernetes-incubator/bootkube v0.8.0
Digital Ocean
- Run etcd cluster across controller nodes (etcd-member.service)
- Remove support for self-hosted etcd
- Reduce time to bootstrap a cluster
v1.7.7
- Kubernetes v1.7.7
- Use kubernetes-incubator/bootkube v0.7.0
- Update kube-dns to 1.14.5 to fix dnsmasq vulnerability
- Calico v2.6.1
- flannel-cni v0.3.0
- Update flannel CNI config to fix hostPort
v1.7.5
- Kubernetes v1.7.5
- Use kubernetes-incubator/bootkube v0.6.2
- Add AWS Terraform module (alpha)
- Add support for Calico networking (bare-metal, Google Cloud, AWS)
- Change networking default from "flannel" to "calico"
AWS
- Add
network_mtu
to allow CNI interface MTU customization
Bare-Metal
- Add
network_mtu
to allow CNI interface MTU customization - Remove support for
experimental_self_hosted_etcd
v1.7.3
- Kubernetes v1.7.3
- Use kubernetes-incubator/bootkube v0.6.1
Digital Ocean
- Add cloud firewall rules (requires Terraform v0.10)
- Change nodes tags from strings to DO tags
v1.7.1
- Kubernetes v1.7.1
- Use kubernetes-incubator/bootkube v0.6.0
- Add Bare-Metal Terraform module (stable)
- Add Digital Ocean Terraform module (beta)
Google Cloud
- Remove
k8s_domain_name
variable,cluster_name
+dns_zone
resolves to controllers - Rename
dns_base_zone
todns_zone
- Rename
dns_base_zone_name
todns_zone_name
v1.6.7
- Kubernetes v1.6.7
- Use kubernetes-incubator/bootkube v0.5.1
v1.6.6
- Kubernetes v1.6.6
- Use kubernetes-incubator/bootkube v0.4.5
- Disable locksmithd on hosts, in favor of CLUO.
v1.6.4
- Kubernetes v1.6.4
- Add Google Cloud Terraform module (stable)
Earlier
Earlier versions, back to v1.3.0, used different designs and mechanisms.