CadolesKube
/
typhoon
mirror of https://github.com/puppetmaster/typhoon.git
7
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add new tutorial docs and links

This commit is contained in:
Dalton Hubble 2018-08-26 23:39:41 -07:00
parent c60ec642bc
commit 991a5c6cee
22 changed files with 383 additions and 53 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@ -25,6 +25,7 @@ Typhoon provides a Terraform Module for each supported operating system and plat
|---------------|------------------|------------------|--------| |---------------|------------------|------------------|--------|
| AWS | Container Linux | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable | | AWS | Container Linux | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
| AWS | Fedora Atomic | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha | | AWS | Fedora Atomic | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
| Azure | Container Linux | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
| Bare-Metal | Container Linux | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable | | Bare-Metal | Container Linux | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
| Bare-Metal | Fedora Atomic | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha | | Bare-Metal | Fedora Atomic | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
| Digital Ocean | Container Linux | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta | | Digital Ocean | Container Linux | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
@ -38,7 +39,7 @@ The AWS and bare-metal `container-linux` modules allow picking Red Hat Container
* [Docs](https://typhoon.psdn.io) * [Docs](https://typhoon.psdn.io)
* Architecture [concepts](https://typhoon.psdn.io/architecture/concepts/) and [operating systems](https://typhoon.psdn.io/architecture/operating-systems/) * Architecture [concepts](https://typhoon.psdn.io/architecture/concepts/) and [operating systems](https://typhoon.psdn.io/architecture/operating-systems/)
* Tutorials for [AWS](https://typhoon.psdn.io/cl/aws/), [Bare-Metal](https://typhoon.psdn.io/cl/bare-metal/), [Digital Ocean](https://typhoon.psdn.io/cl/digital-ocean/), and [Google-Cloud](https://typhoon.psdn.io/cl/google-cloud/) * Tutorials for [AWS](cl/aws.md), [Azure](cl/azure.md), [Bare-Metal](cl/bare-metal.md), [Digital Ocean](cl/digital-ocean.md), and [Google-Cloud](cl/google-cloud.md)
## Usage ## Usage
@ -71,15 +72,14 @@ module "google-cloud-yavin" {
} }
``` ```
Fetch modules, plan the changes to be made, and apply the changes. Initialize modules, plan the changes to be made, and apply the changes.
```sh ```sh
$ terraform init $ terraform init
$ terraform get --update
$ terraform plan $ terraform plan
Plan: 37 to add, 0 to change, 0 to destroy. Plan: 64 to add, 0 to change, 0 to destroy.
$ terraform apply $ terraform apply
Apply complete! Resources: 37 added, 0 changed, 0 destroyed. Apply complete! Resources: 64 added, 0 changed, 0 destroyed.
``` ```
In 4-8 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes. In 4-8 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes.

23
azure/container-linux/kubernetes/LICENSE Normal file
View File

@ -0,0 +1,23 @@
The MIT License (MIT)
Copyright (c) 2017 Typhoon Authors
Copyright (c) 2017 Dalton Hubble
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

22
azure/container-linux/kubernetes/README.md Normal file
View File

@ -0,0 +1,22 @@
# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
Typhoon is a minimal and free Kubernetes distribution.
* Minimal, stable base Kubernetes distribution
* Declarative infrastructure and configuration
* Free (freedom and cost) and privacy-respecting
* Practical for labs, datacenters, and clouds
Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
* Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
## Docs
Please see the [official docs](https://typhoon.psdn.io) and the Azure [tutorial](https://typhoon.psdn.io/cl/azure/).

4
azure/container-linux/kubernetes/variables.tf
View File

@ -61,7 +61,7 @@ variable "disk_size" {
variable "worker_priority" { variable "worker_priority" {
type = "string" type = "string"
default = "Regular" default = "Regular"
description = "Set worker priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time." description = "Set worker priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
} }
variable "controller_clc_snippets" { variable "controller_clc_snippets" {
@ -89,7 +89,7 @@ variable "asset_dir" {
} }
variable "host_cidr" { variable "host_cidr" {
description = "CIDR IPv4 range to assign to EC2 nodes" description = "CIDR IPv4 range to assign to instances"
type = "string" type = "string"
default = "10.0.0.0/16" default = "10.0.0.0/16"
} }

0
azure/ignore/.gitkeep Normal file
View File

2
digital-ocean/container-linux/kubernetes/README.md
View File

@ -13,7 +13,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
* Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube)) * Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking * Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/) * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
## Docs ## Docs

4
docs/advanced/customization.md
View File

@ -19,7 +19,7 @@ Clusters are kept to a minimal Kubernetes control plane by offering components l
Container Linux Configs (CLCs) declare how a Container Linux instance's disk should be provisioned on first boot from disk. CLCs define disk partitions, filesystems, files, systemd units, dropins, networkd configs, mount units, raid arrays, and users. Typhoon creates controller and worker instances with base Container Linux Configs to create a minimal, secure Kubernetes cluster on each platform. Container Linux Configs (CLCs) declare how a Container Linux instance's disk should be provisioned on first boot from disk. CLCs define disk partitions, filesystems, files, systemd units, dropins, networkd configs, mount units, raid arrays, and users. Typhoon creates controller and worker instances with base Container Linux Configs to create a minimal, secure Kubernetes cluster on each platform.
Typhoon AWS, Google Cloud, and Digital Ocean give users the ability to provide CLC *snippets* - valid Container Linux Configs that are validated and additively merged into the Typhoon base config during `terraform plan`. This allows advanced host customizations and experimentation. Typhoon AWS, Azure, bare-metal, DigitalOcean, and Google Cloud support CLC *snippets* - valid Container Linux Configs that are validated and additively merged into the Typhoon base config during `terraform plan`. This allows advanced host customizations and experimentation.
#### Examples #### Examples
@ -69,7 +69,7 @@ View the Container Linux Config [format](https://coreos.com/os/docs/1576.4.0/con
Write Container Linux Configs *snippets* as files in the repository where you keep Terraform configs for clusters (perhaps in a `clc` or `snippets` subdirectory). You may organize snippets in multiple files as desired, provided they are each valid. Write Container Linux Configs *snippets* as files in the repository where you keep Terraform configs for clusters (perhaps in a `clc` or `snippets` subdirectory). You may organize snippets in multiple files as desired, provided they are each valid.
[AWS](/cl/aws/#cluster), [Google Cloud](/cl/google-cloud/#cluster), and [Digital Ocean](/cl/digital-ocean/#cluster) clusters allow populating a list of `controller_clc_snippets` or `worker_clc_snippets`. [AWS](/cl/aws/#cluster), [Azure](/cl/azure/#cluster), [DigitalOcean](/cl/digital-ocean/#cluster), and [Google Cloud](/cl/google-cloud/#cluster) clusters allow populating a list of `controller_clc_snippets` or `worker_clc_snippets`.
``` ```
module "digital-ocean-nemo" { module "digital-ocean-nemo" {

3
docs/architecture/concepts.md
View File

@ -8,7 +8,7 @@ Let's cover the concepts you'll need to get started.
#### Nodes #### Nodes
Cluster nodes provision themselves from a declarative configuration upfront. Nodes run a `kubelet` service and register themselves with the control plane to join the higher order cluster. All nodes run `kube-proxy` and `calico` or `flannel` pods. All cluster nodes provision themselves from a declarative configuration upfront. Nodes run a `kubelet` service and register themselves with the control plane to join the cluster. All nodes run `kube-proxy` and `calico` or `flannel` pods.
#### Controllers #### Controllers
@ -77,6 +77,7 @@ infra/
└── terraform └── terraform
└── clusters └── clusters
├── aws-tempest.tf ├── aws-tempest.tf
├── azure-ramius.tf
├── bare-metal-mercury.tf ├── bare-metal-mercury.tf
├── google-cloud-yavin.tf ├── google-cloud-yavin.tf
├── digital-ocean-nemo.tf ├── digital-ocean-nemo.tf

4
docs/atomic/aws.md
View File

@ -24,7 +24,7 @@ $ terraform version
Terraform v0.11.7 Terraform v0.11.7
``` ```
Read [concepts](../architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`). Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
``` ```
cd infra/clusters cd infra/clusters
@ -184,7 +184,7 @@ kube-system pod-checkpointer-4kxtl-ip-10-0-12-221 1/1 Running 0
## Going Further ## Going Further
Learn about [maintenance](../topics/maintenance.md) and [addons](../addons/overview.md). Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
## Variables ## Variables

4
docs/atomic/bare-metal.md
View File

@ -190,7 +190,7 @@ providers {
} }
``` ```
Read [concepts](../architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`). Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
``` ```
cd infra/clusters cd infra/clusters
@ -389,7 +389,7 @@ kube-system pod-checkpointer-wf65d-node1.example.com 1/1 Running 0
## Going Further ## Going Further
Learn about [maintenance](../topics/maintenance.md) and [addons](../addons/overview.md). Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
## Variables ## Variables

4
docs/atomic/digital-ocean.md
View File

@ -24,7 +24,7 @@ $ terraform version
Terraform v0.11.7 Terraform v0.11.7
``` ```
Read [concepts](../architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`). Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
``` ```
cd infra/clusters cd infra/clusters
@ -179,7 +179,7 @@ kube-system pod-checkpointer-pr1lq-10.132.115.81 1/1 Running 0
## Going Further ## Going Further
Learn about [maintenance](../topics/maintenance.md) and [addons](../addons/overview.md). Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
## Variables ## Variables

6
docs/atomic/google-cloud.md
View File

@ -12,7 +12,7 @@ Controllers are provisioned to run an `etcd` peer and a `kubelet` service. Worke
## Requirements ## Requirements
* Google Cloud Account and Service Account * Google Cloud Account and Service Account
* Google Cloud DNS Zone (registered Domain Name or delegated subdomain) * Google Cloud DNS Zone (registered main Name or delegated subdomain)
* Terraform v0.11.x installed locally * Terraform v0.11.x installed locally
* `gcloud` and `gsutil` for uploading a disk image to Google Cloud (temporary) * `gcloud` and `gsutil` for uploading a disk image to Google Cloud (temporary)
@ -25,7 +25,7 @@ $ terraform version
Terraform v0.11.7 Terraform v0.11.7
``` ```
Read [concepts](../architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`). Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
``` ```
cd infra/clusters cd infra/clusters
@ -224,7 +224,7 @@ kube-system pod-checkpointer-l6lrt 1/1 Running 0
## Going Further ## Going Further
Learn about [maintenance](../topics/maintenance.md) and [addons](../addons/overview.md). Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
## Variables ## Variables

10
docs/cl/aws.md
View File

@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys
```sh ```sh
$ terraform version $ terraform version
Terraform v0.11.1 Terraform v0.11.7
``` ```
Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system. Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
@ -37,7 +37,7 @@ providers {
} }
``` ```
Read [concepts](../architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`). Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
``` ```
cd infra/clusters cd infra/clusters
@ -197,7 +197,7 @@ kube-system pod-checkpointer-4kxtl-ip-10-0-12-221 1/1 Running 0
## Going Further ## Going Further
Learn about [maintenance](../topics/maintenance.md) and [addons](../addons/overview.md). Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
!!! note !!! note
On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot. On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@ -245,8 +245,8 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
| disk_size | Size of the EBS volume in GB | "40" | "100" | | disk_size | Size of the EBS volume in GB | "40" | "100" |
| disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 | | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
| worker_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" | | worker_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
| controller_clc_snippets | Controller Container Linux Config snippets | [] | | | controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization.md) |
| worker_clc_snippets | Worker Container Linux Config snippets | [] | | | worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization.md) |
| networking | Choice of networking provider | "calico" | "calico" or "flannel" | | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
| network_mtu | CNI interface MTU (calico only) | 1480 | 8981 | | network_mtu | CNI interface MTU (calico only) | 1480 | 8981 |
| host_cidr | CIDR IPv4 range to assign to EC2 instances | "10.0.0.0/16" | "10.1.0.0/16" | | host_cidr | CIDR IPv4 range to assign to EC2 instances | "10.0.0.0/16" | "10.1.0.0/16" |

277
docs/cl/azure.md Normal file
View File

@ -0,0 +1,277 @@
# Azure
!!! danger
Typhoon for Azure is alpha. Expect rough edges and changes.
In this tutorial, we'll create a Kubernetes v1.11.2 cluster on Azure with Container Linux.
We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
Controllers are provisioned to run an `etcd-member` peer and a `kubelet` service. Workers run just a `kubelet` service. A one-time [bootkube](https://github.com/kubernetes-incubator/bootkube) bootstrap schedules the `apiserver`, `scheduler`, `controller-manager`, and `coredns` on controllers and schedules `kube-proxy` and `flannel` on every node. A generated `kubeconfig` provides `kubectl` access to the cluster.
## Requirements
* Azure account
* Azure DNS Zone (registered Domain Name or delegated subdomain)
* Terraform v0.11.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally
## Terraform Setup
Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your system.
```sh
$ terraform version
Terraform v0.11.7
```
Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
```sh
wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
```
Add the plugin to your `~/.terraformrc`.
```
providers {
ct = "/usr/local/bin/terraform-provider-ct"
}
```
Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
```
cd infra/clusters
```
## Provider
[Install](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) the Azure `az` command line tool to [authenticate with Azure](https://www.terraform.io/docs/providers/azurerm/authenticating_via_azure_cli.html).
```
az login
```
Configure the Azure provider in a `providers.tf` file.
```tf
provider "azurerm" {
version = "1.13.0"
alias = "default"
}
provider "local" {
version = "~> 1.0"
alias = "default"
}
provider "null" {
version = "~> 1.0"
alias = "default"
}
provider "template" {
version = "~> 1.0"
alias = "default"
}
provider "tls" {
version = "~> 1.0"
alias = "default"
}
```
Additional configuration options are described in the `azurerm` provider [docs](https://www.terraform.io/docs/providers/azurerm/).
## Cluster
Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.
```tf
module "azure-ramius" {
source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.11.3"
providers = {
azurerm = "azurerm.default"
local = "local.default"
null = "null.default"
template = "template.default"
tls = "tls.default"
}
# Azure
cluster_name = "ramius"
region = "centralus"
dns_zone = "azure.example.com"
dns_zone_group = "example-group"
# configuration
ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
asset_dir = "/home/user/.secrets/clusters/ramius"
# optional
worker_count = 3
host_cidr = "10.0.0.0/20"
}
```
Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/container-linux/kubernetes/variables.tf) source.
## ssh-agent
Initial bootstrapping requires `bootkube.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
```sh
ssh-add ~/.ssh/id_rsa
ssh-add -L
```
## Apply
Initialize the config directory if this is the first use with Terraform.
```sh
terraform init
```
Plan the resources to be created.
```sh
$ terraform plan
Plan: 86 to add, 0 to change, 0 to destroy.
```
Apply the changes to create the cluster.
```sh
$ terraform apply
...
module.azure-ramius.null_resource.bootkube-start: Still creating... (6m50s elapsed)
module.azure-ramius.null_resource.bootkube-start: Still creating... (7m0s elapsed)
module.azure-ramius.null_resource.bootkube-start: Creation complete after 7m8s (ID: 3961816482286168143)
Apply complete! Resources: 86 added, 0 changed, 0 destroyed.
```
In 4-8 minutes, the Kubernetes cluster will be ready.
## Verify
[Install kubectl](https://coreos.com/kubernetes/docs/latest/configure-kubectl.html) on your system. Use the generated `kubeconfig` credentials to access the Kubernetes cluster and list nodes.
```
$ export KUBECONFIG=/home/user/.secrets/clusters/ramius/auth/kubeconfig
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ramius-controller-0 Ready controller,master 24m v1.11.2
ramius-worker-000001 Ready node 25m v1.11.2
ramius-worker-000002 Ready node 24m v1.11.2
ramius-worker-000005 Ready node 24m v1.11.2
```
List the pods.
```
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7c6fbb4f4b-b6qzx 1/1 Running 0 26m
kube-system kube-apiserver-hxgsx 1/1 Running 3 26m
kube-system kube-controller-manager-5ff9cd7bb6-b942n 1/1 Running 0 26m
kube-system kube-controller-manager-5ff9cd7bb6-bbr6w 1/1 Running 0 26m
kube-system kube-flannel-bwf24 2/2 Running 2 26m
kube-system kube-flannel-ks5qb 2/2 Running 0 26m
kube-system kube-flannel-nghsx 2/2 Running 2 26m
kube-system kube-flannel-tq2wg 2/2 Running 0 26m
kube-system kube-proxy-j4vpq 1/1 Running 0 26m
kube-system kube-proxy-jxr5d 1/1 Running 0 26m
kube-system kube-proxy-lbdw5 1/1 Running 0 26m
kube-system kube-proxy-v8r7c 1/1 Running 0 26m
kube-system kube-scheduler-5f76d69686-s4fbx 1/1 Running 0 26m
kube-system kube-scheduler-5f76d69686-vgdgn 1/1 Running 0 26m
kube-system pod-checkpointer-cnqdg 1/1 Running 0 26m
kube-system pod-checkpointer-cnqdg-ramius-controller-0 1/1 Running 0 25m
```
## Going Further
Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
!!! note
On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
## Variables
Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/container-linux/kubernetes/variables.tf) source.
### Required
| Name | Description | Example |
|:-----|:------------|:--------|
| cluster_name | Unique cluster name (prepended to dns_zone) | "ramius" |
| region | Azure region | "centralus" |
| dns_zone | Azure DNS zone | "azure.example.com" |
| dns_zone_group | Resource group where the Azure DNS zone resides | "global" |
| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
| asset_dir | Path to a directory where generated assets should be placed (contains secrets) | "/home/user/.secrets/clusters/ramius" |
!!! tip
Regions are shown in [docs](https://azure.microsoft.com/en-us/global-infrastructure/regions/) or with `az account list-locations --output table`.
#### DNS Zone
Clusters create a DNS A record `${cluster_name}.${dns_zone}` to resolve a load balancer backed by controller instances. This FQDN is used by workers and `kubectl` to access the apiserver(s). In this example, the cluster's apiserver would be accessible at `ramius.azure.example.com`.
You'll need a registered domain name or delegated subdomain on Azure DNS. You can set this up once and create many clusters with unique names.
```tf
# Azure resource group for DNS zone
resource "azurerm_resource_group" "global" {
name = "global"
location = "centralus"
}
# DNS zone for clusters
resource "azurerm_dns_zone" "clusters" {
resource_group_name = "${azurerm_resource_group.global.name}"
name = "azure.example.com"
zone_type = "Public"
}
```
Reference the DNS zone with `"${azurerm_dns_zone.clusters.name}"` and its resource group with `"${azurerm_resource_group.global.name}"`.
!!! tip ""
If you have an existing domain name with a zone file elsewhere, just delegate a subdomain that can be managed on Azure DNS (e.g. azure.mydomain.com) and [update nameservers](https://docs.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns).
### Optional
| Name | Description | Default | Example |
|:-----|:------------|:--------|:--------|
| controller_count | Number of controllers (i.e. masters) | 1 | 1 |
| worker_count | Number of workers | 1 | 3 |
| controller_type | Machine type for controllers | "Standard_DS1_v2" | See below |
| worker_type | Machine type for workers | "Standard_F1" | See below |
| os_image | Channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha |
| disk_size | Size of the disk GB | "40" | "100" |
| worker_priority | Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Low |
| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
| host_cidr | CIDR IPv4 range to assign to instances | "10.0.0.0/16" | "10.0.0.0/20" |
| pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
| service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
| cluster_domain_suffix | FQDN suffix for Kubernetes services answered by coredns. | "cluster.local" | "k8s.example.com" |
Check the list of valid [machine types](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) and their [specs](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-general). Use `az vm list-skus` to get the identifier.
!!! warning
Unlike AWS and GCP, Azure requires its *virtual* networks to have unique, non-overlapping IPv4 CIDRs (yeah, go figure). Instead of each cluster just using `10.0.0.0/16` for instances, each Azure cluster's `host_cidr` must be non-overlapping (e.g. 10.0.0.0/20 for the 1st cluster, 10.0.16.0/20 for the 2nd cluster, etc).
!!! warning
Do not choose a `controller_type` smaller than `Standard_DS1_v2`. Smaller instances are not sufficient for running a controller.
#### Low Priority
Add `worker_priority=Low` to use [Low Priority](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-use-low-priority) workers that run on Azure's surplus capacity at lower cost, but with the tradeoff that they can be deallocated at random. Low priority VMs are Azure's analog to AWS spot instances or GCP premptible instances.

6
docs/cl/bare-metal.md
View File

@ -110,7 +110,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys
```sh ```sh
$ terraform version $ terraform version
Terraform v0.11.1 Terraform v0.11.7
``` ```
Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system. Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system.
@ -137,7 +137,7 @@ providers {
} }
``` ```
Read [concepts](../architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`). Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
``` ```
cd infra/clusters cd infra/clusters
@ -346,7 +346,7 @@ kube-system pod-checkpointer-wf65d-node1.example.com 1/1 Running 0
## Going Further ## Going Further
Learn about [maintenance](../topics/maintenance.md) and [addons](../addons/overview.md). Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
!!! note !!! note
On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot. On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.

10
docs/cl/digital-ocean.md
View File

@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys
```sh ```sh
$ terraform version $ terraform version
Terraform v0.11.1 Terraform v0.11.7
``` ```
Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system. Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
@ -37,7 +37,7 @@ providers {
} }
``` ```
Read [concepts](../architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`). Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
``` ```
cd infra/clusters cd infra/clusters
@ -191,7 +191,7 @@ kube-system pod-checkpointer-pr1lq-10.132.115.81 1/1 Running 0
## Going Further ## Going Further
Learn about [maintenance](../topics/maintenance.md) and [addons](../addons/overview.md). Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
!!! note !!! note
On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot. On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@ -254,8 +254,8 @@ Digital Ocean requires the SSH public key be uploaded to your account, so you ma
| controller_type | Droplet type for controllers | s-2vcpu-2gb | s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb, ... | | controller_type | Droplet type for controllers | s-2vcpu-2gb | s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb, ... |
| worker_type | Droplet type for workers | s-1vcpu-1gb | s-1vcpu-1gb, s-1vcpu-2gb, s-2vcpu-2gb, ... | | worker_type | Droplet type for workers | s-1vcpu-1gb | s-1vcpu-1gb, s-1vcpu-2gb, s-2vcpu-2gb, ... |
| image | Container Linux image for instances | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha | | image | Container Linux image for instances | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha |
| controller_clc_snippets | Controller Container Linux Config snippets | [] | | | controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advnaced/customization.md) |
| worker_clc_snippets | Worker Container Linux Config snippets | [] | | | worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](customization.md) |
| pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" | | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
| service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" | | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
| cluster_domain_suffix | FQDN suffix for Kubernetes services answered by coredns. | "cluster.local" | "k8s.example.com" | | cluster_domain_suffix | FQDN suffix for Kubernetes services answered by coredns. | "cluster.local" | "k8s.example.com" |

10
docs/cl/google-cloud.md
View File

@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys
```sh ```sh
$ terraform version $ terraform version
Terraform v0.11.1 Terraform v0.11.7
``` ```
Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system. Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
@ -37,7 +37,7 @@ providers {
} }
``` ```
Read [concepts](../architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`). Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
``` ```
cd infra/clusters cd infra/clusters
@ -199,7 +199,7 @@ kube-system pod-checkpointer-l6lrt 1/1 Running 0
## Going Further ## Going Further
Learn about [maintenance](../topics/maintenance.md) and [addons](../addons/overview.md). Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
!!! note !!! note
On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot. On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@ -249,8 +249,8 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
| os_image | Container Linux image for compute instances | "coreos-stable" | "coreos-stable-1632-3-0-v20180215" | | os_image | Container Linux image for compute instances | "coreos-stable" | "coreos-stable-1632-3-0-v20180215" |
| disk_size | Size of the disk in GB | 40 | 100 | | disk_size | Size of the disk in GB | 40 | 100 |
| worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true | | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
| controller_clc_snippets | Controller Container Linux Config snippets | [] | | | controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization.md) |
| worker_clc_snippets | Worker Container Linux Config snippets | [] | | | worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](customization.md) |
| networking | Choice of networking provider | "calico" | "calico" or "flannel" | | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
| pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" | | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
| service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" | | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |

7
docs/index.md
View File

@ -25,6 +25,7 @@ Typhoon provides a Terraform Module for each supported operating system and plat
|---------------|------------------|------------------|--------| |---------------|------------------|------------------|--------|
| AWS | Container Linux | [aws/container-linux/kubernetes](cl/aws.md) | stable | | AWS | Container Linux | [aws/container-linux/kubernetes](cl/aws.md) | stable |
| AWS | Fedora Atomic | [aws/fedora-atomic/kubernetes](atomic/aws.md) | alpha | | AWS | Fedora Atomic | [aws/fedora-atomic/kubernetes](atomic/aws.md) | alpha |
| Azure | Container Linux | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
| Bare-Metal | Container Linux | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable | | Bare-Metal | Container Linux | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
| Bare-Metal | Fedora Atomic | [bare-metal/fedora-atomic/kubernetes](atomic/bare-metal.md) | alpha | | Bare-Metal | Fedora Atomic | [bare-metal/fedora-atomic/kubernetes](atomic/bare-metal.md) | alpha |
| Digital Ocean | Container Linux | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta | | Digital Ocean | Container Linux | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta |
@ -37,7 +38,7 @@ The AWS and bare-metal `container-linux` modules allow picking Red Hat Container
## Documentation ## Documentation
* Architecture [concepts](architecture/concepts.md) and [operating-systems](architecture/operating-systems.md) * Architecture [concepts](architecture/concepts.md) and [operating-systems](architecture/operating-systems.md)
* Tutorials for [AWS](cl/aws.md), [Bare-Metal](cl/bare-metal.md), [Digital Ocean](cl/digital-ocean.md), and [Google-Cloud](cl/google-cloud.md) * Tutorials for [AWS](cl/aws.md), [Azure](cl/azure.md), [Bare-Metal](cl/bare-metal.md), [Digital Ocean](cl/digital-ocean.md), and [Google-Cloud](cl/google-cloud.md)
## Example ## Example
@ -70,10 +71,10 @@ module "google-cloud-yavin" {
} }
``` ```
Fetch modules, plan the changes to be made, and apply the changes. Initialize modules, plan the changes to be made, and apply the changes.
```sh ```sh
$ terraform get --update $ terraform init
$ terraform plan $ terraform plan
Plan: 64 to add, 0 to change, 0 to destroy. Plan: 64 to add, 0 to change, 0 to destroy.
$ terraform apply $ terraform apply

2
docs/topics/faq.md
View File

@ -8,7 +8,7 @@ Formats rise and evolve. Typhoon may choose to adapt the format over time (with
## Operating Systems ## Operating Systems
Typhoon supports Container Linux and Fedora Atomic 27. These two operating systems were chosen because they offer: Typhoon supports Container Linux and Fedora Atomic 28. These two operating systems were chosen because they offer:
* Minimalism and focus on clustered operation * Minimalism and focus on clustered operation
* Automated and atomic operating system upgrades * Automated and atomic operating system upgrades

17
docs/topics/performance.md
View File

@ -7,14 +7,15 @@ Provisioning times vary based on the operating system and platform. Sampling the
| Platform | Apply | Destroy | | Platform | Apply | Destroy |
|---------------|-------|---------| |---------------|-------|---------|
| AWS | 6 min | 5 min | | AWS | 6 min | 5 min |
| Azure | 7 min | 7 min |
| Bare-Metal | 10-15 min | NA | | Bare-Metal | 10-15 min | NA |
| Digital Ocean | 3 min 30 sec | 20 sec | | Digital Ocean | 3 min 30 sec | 20 sec |
| Google Cloud | 6 min | 4 min 30 sec | | Google Cloud | 7 min | 6 min |
Notes: Notes:
* SOA TTL and NXDOMAIN caching can have a large impact on provision time * SOA TTL and NXDOMAIN caching can have a large impact on provision time
* Platforms with auto-scaling take more time to provision (AWS, Google) * Platforms with auto-scaling take more time to provision (AWS, Azure, Google)
* Bare-metal POST times and network bandwidth will affect provision times * Bare-metal POST times and network bandwidth will affect provision times
## Network Performance ## Network Performance
@ -26,17 +27,19 @@ Network performance varies based on the platform and CNI plugin. `iperf` was use
| AWS (flannel) | ? | 976 MB/s | 900-999 MB/s | | AWS (flannel) | ? | 976 MB/s | 900-999 MB/s |
| AWS (calico, MTU 1480) | ? | 976 MB/s | 100-350 MB/s | | AWS (calico, MTU 1480) | ? | 976 MB/s | 100-350 MB/s |
| AWS (calico, MTU 8981) | ? | 976 MB/s | 900-999 MB/s | | AWS (calico, MTU 8981) | ? | 976 MB/s | 900-999 MB/s |
| Bare-Metal (flannel) | 1 GB/s | 934 MB/s | 903 MB/s | | Azure (flannel) | ? | 749 MB/s | 680 MB/s |
| Bare-Metal (calico) | 1 GB/s | 941 MB/s | 931 MB/s | | Bare-Metal (flannel) | 1 GB/s | ~940 MB/s | 903 MB/s |
| Bare-Metal (calico) | 1 GB/s | ~940 MB/s | 931 MB/s |
| Bare-Metal (flannel, bond) | 3 GB/s | 2.3 GB/s | 1.17 GB/s | | Bare-Metal (flannel, bond) | 3 GB/s | 2.3 GB/s | 1.17 GB/s |
| Bare-Metal (calico, bond) | 3 GB/s | 2.3 GB/s | 1.17 GB/s | | Bare-Metal (calico, bond) | 3 GB/s | 2.3 GB/s | 1.17 GB/s |
| Digital Ocean | ? | 938 MB/s | 820-880 MB/s | | Digital Ocean | ? | ~940 MB/s | 820-880 MB/s |
| Google Cloud (flannel) | ? | 1.94 GB/s | 1.76 GB/s | | Google Cloud (flannel) | ? | 1.94 GB/s | 1.76 GB/s |
| Google Cloud (calico) | ? | 1.94 GB/s | 1.81 GB/s | | Google Cloud (calico) | ? | 1.94 GB/s | 1.81 GB/s |
Notes: Notes:
* Calico and Flannel have comparable performance. Platform and configuration differences dominate. * Calico and Flannel have comparable performance. Platform and configuration differences dominate.
* Neither CNI provider seems to be able to leverage bonded NICs (bare-metal) * AWS and Azure node bandwidth (i.e. upper bound) depends greatly on machine type
* AWS and Digital Ocean network bandwidth fluctuates more than on other platforms.
* Only [certain AWS EC2 instance types](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) allow jumbo frames. This is why the default MTU on AWS must be 1480. * Only [certain AWS EC2 instance types](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) allow jumbo frames. This is why the default MTU on AWS must be 1480.
* Neither CNI provider seems to be able to leverage bonded NICs well (bare-metal)

10
docs/topics/security.md
View File

@ -12,7 +12,7 @@ Typhoon aims to be minimal and secure. We're running it ourselves after all.
* Workloads run on worker nodes only, unless they tolerate the master taint * Workloads run on worker nodes only, unless they tolerate the master taint
* Kubernetes [Network Policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) and Calico [Policy](https://docs.projectcalico.org/latest/reference/calicoctl/resources/policy) support [^1] * Kubernetes [Network Policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) and Calico [Policy](https://docs.projectcalico.org/latest/reference/calicoctl/resources/policy) support [^1]
[^1]: Requires `networking = "calico"`. Calico is the default on AWS, bare-metal, and Google Cloud. Digital Ocean is limited to `networking = "flannel"`. [^1]: Requires `networking = "calico"`. Calico is the default on AWS, bare-metal, and Google Cloud. Azure and Digital Ocean are limited to `networking = "flannel"`.
**Hosts** **Hosts**
@ -24,11 +24,13 @@ Typhoon aims to be minimal and secure. We're running it ourselves after all.
* Cloud firewalls limit access to ssh, kube-apiserver, and ingress * Cloud firewalls limit access to ssh, kube-apiserver, and ingress
* No cluster credentials are stored in Matchbox (used for bare-metal) * No cluster credentials are stored in Matchbox (used for bare-metal)
* No cluster credentials are stored in Digital Ocean metadata * No cluster credentials are stored in Digital Ocean metadata
* Cluster credentials are stored in Google Cloud metadata (for managed instance groups)
* Cluster credentials are stored in AWS metadata (for ASGs) * Cluster credentials are stored in AWS metadata (for ASGs)
* No account credentials are available to Google Cloud instances (no IAM permissions) * Cluster credentials are stored in Azure metadata (for scale sets)
* No account credentials are available to AWS EC2 instances (no IAM permissions) * Cluster credentials are stored in Google Cloud metadata (for managed instance groups)
* No account credentials are available to Digital Ocean droplets * No account credentials are available to Digital Ocean droplets
* No account credentials are available to AWS EC2 instances (no IAM permissions)
* No account credentials are available to Azure instances (no IAM permissions)
* No account credentials are available to Google Cloud instances (no IAM permissions)
## Precautions ## Precautions

1
mkdocs.yml
View File

@ -47,6 +47,7 @@ pages:
- 'Operating Systems': 'architecture/operating-systems.md' - 'Operating Systems': 'architecture/operating-systems.md'
- 'Container Linux': - 'Container Linux':
- 'AWS': 'cl/aws.md' - 'AWS': 'cl/aws.md'
- 'Azure': 'cl/azure.md'
- 'Bare-Metal': 'cl/bare-metal.md' - 'Bare-Metal': 'cl/bare-metal.md'
- 'Digital Ocean': 'cl/digital-ocean.md' - 'Digital Ocean': 'cl/digital-ocean.md'
- 'Google Cloud': 'cl/google-cloud.md' - 'Google Cloud': 'cl/google-cloud.md'