Introduction to:

Provisioning basic infrastructure on Google Cloud Platform with Terraform

Config Management Camp 2018

@steinim

stein.inge.morisbak@BEKK.no

Why GCP?

Broaden my horizon
Build competence
Know what I'm talking about
Have fun!

AWS Architecture to reproduce

GCP Architecture result

Terraform

What does it do?

Documents (infrastructure as code)
Plans (no surprises)
Graphs (parallelizes where possible)
Automates (takes the human out of the equation)
Provider/Cloud agnostic (sort of)

Why use it?

Repeatable
Testable
Predictable
Self-documenting

Terraform Components

Configurations (*.tf)
Providers (AWS, Google Cloud, Azure, DigitalOcean, OpenStack...)
Resources (building blocks) → provider specific
Variables (infrastructure as data)
Data sources (fetch or compute data)
Outputs (data "after the fact")
Modules (for reuseability)

The juicy part

https://github.com/steinim/gcp-terraform-workshop

Source code


.
├── helloworld-java-app
└── terraform
    ├── modules
    │   ├── db
    │   ├── instance-template
    │   ├── lb
    │   ├── network
    │   │   ├── bastion
    │   │   ├── subnet
    │   └── project
    ├── prod
    └── test

Remote state


cd terraform/test

gsutil mb -l ${TF_VAR_region} -p ${TF_ADMIN} gs://${TF_ADMIN}

cat > backend.tf <<EOF
terraform {
 backend "gcs" {
   bucket = "${TF_ADMIN}"
   prefix  = "terraform/state/test"
 }
}
EOF

Initialize the backend


terraform init

GCP provider and project


provider "google" {
 region = "${var.region}"
}

resource "random_id" "id" {
 byte_length = 4
 prefix      = "${var.name}-"
}

resource "google_project" "project" {
 name            = "${var.name}"
 project_id      = "${random_id.id.hex}"
 billing_account = "${var.billing_account}"
 org_id          = "${var.org_id}"
}

resource "google_project_services" "project" {
 project = "${google_project.project.project_id}"
 services = [
   "compute.googleapis.com",
   "sqladmin.googleapis.com"
 ]
}

Create a module for it


.
├── modules
│   └── project
│       └── main.tf
├── prod
└── test
    ├── backend.tf
    ├── main.tf
    └── vars.tf


# main.tf
module "project" {
  source          = "../modules/project"
  name            = "hello-${var.env}"
  region          = "${var.region}"
  billing_account = "${var.billing_account}"
  org_id          = "${var.org_id}"
}


# vars.tf
variable "env" { default = "test" }
variable "region" { default = "europe-west3" }
variable "billing_account" {}
variable "org_id" {}

`terraform plan`


Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

------------------------------------------------------------------------

Terraform will perform the following actions:

  + module.project.google_project.project
      id:                  <computed>
      billing_account:     "0000C2-2C8D36-F6184D"
      folder_id:           <computed>
      name:                "hello-test"
      number:              <computed>
      org_id:              "396389084239"
      policy_data:         <computed>
      policy_etag:         <computed>
      project_id:          "${random_id.id.hex}"
      skip_delete:         <computed>

  + module.project.google_project_services.project
      id:                  <computed>
      project:             "${google_project.project.project_id}"
      services.#:          "1"
      services.2240314979: "compute.googleapis.com"

  + module.project.random_id.id
      id:                  <computed>
      b64:                 <computed>
      b64_std:             <computed>
      b64_url:             <computed>
      byte_length:         "4"
      dec:                 <computed>
      hex:                 <computed>
      prefix:              "hello-test-"


Plan: 3 to add, 0 to change, 0 to destroy.

`terraform apply`


...
module.project.random_id.id: Creating...
  b64:         "" => "<computed>"
  b64_std:     "" => "<computed>"
  b64_url:     "" => "<computed>"
  byte_length: "" => "4"
  dec:         "" => "<computed>"
  hex:         "" => "<computed>"
  prefix:      "" => "hello-test-"
module.project.random_id.id: Creation complete after 0s (ID: 4r8CrQ)
module.project.google_project.project: Creating...
  billing_account: "" => "0000C2-2C8D36-F6184D"
  folder_id:       "" => "<computed>"
  name:            "" => "hello-test"
  number:          "" => "<computed>"
  org_id:          "" => "396389084239"
  policy_data:     "" => "<computed>"
  policy_etag:     "" => "<computed>"
  project_id:      "" => "hello-test-e2bf02ad"
  skip_delete:     "" => "<computed>"
module.project.google_project.project: Still creating... (10s elapsed)
module.project.google_project.project: Creation complete after 17s (ID: hello-test-e2bf02ad)
module.project.google_project_services.project: Creating...
  project:             "" => "hello-test-e2bf02ad"
  services.#:          "" => "1"
  services.2240314979: "" => "compute.googleapis.com"
module.project.google_project_services.project: Still creating... (10s elapsed)
...
module.project.google_project_services.project: Still creating... (1m30s elapsed)
module.project.google_project_services.project: Creation complete after 1m37s (ID: hello-test-e2bf02ad)

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Profit 💰

Networking and bastion host

Network module


modules/network/
├── vars.tf
├── main.tf
├── outputs.tf
├── bastion
│   ├── main.tf
│   ├── outputs.tf
│   └── vars.tf
├── subnet
│   ├── main.tf
│   ├── outputs.tf
│   └── vars.tf

    
...
        
module "network" {
  source                     = "../modules/network"
  name                       = "${module.project.name}"
  project                    = "${module.project.id}"
  region                     = "${var.region}"
  zones                      = "${var.zones}"
  webservers_subnet_name     = "webservers"
  webservers_subnet_ip_range = "${var.webservers_subnet_ip_range}"
  management_subnet_name     = "management"
  management_subnet_ip_range = "${var.management_subnet_ip_range}"
  bastion_image              = "${var.bastion_image}"
  bastion_instance_type      = "${var.bastion_instance_type}"
  user                       = "${var.user}"
  ssh_key                    = "${var.ssh_key}"
}


# modules/network
# main.tf
resource "google_compute_network" "network" {
  name    = "${var.name}-network"
  project = "${var.project}"
}

resource "google_compute_firewall" "allow-internal" {
  name    = "${var.name}-allow-internal"
  project = "${var.project}"
  network = "${var.name}-network"

  allow {
    protocol = "icmp"
  }

  allow {
    protocol = "tcp"
    ports    = ["0-65535"]
  }

  allow {
    protocol = "udp"
    ports    = ["0-65535"]
  }

  source_ranges = [
    "${module.management_subnet.ip_range}",
    "${module.webservers_subnet.ip_range}"
  ]
}

resource "google_compute_firewall" "allow-ssh-from-everywhere-to-bastion" {
  name    = "${var.name}-allow-ssh-from-everywhere-to-bastion"
  project = "${var.project}"
  network = "${var.name}-network"

  allow {
    protocol = "tcp"
    ports    = ["22"]
  }

  source_ranges = ["0.0.0.0/0"]

  target_tags = ["bastion"]
}

resource "google_compute_firewall" "allow-ssh-from-bastion-to-webservers" {
  name               = "${var.name}-allow-ssh-from-bastion-to-webservers"
  project            = "${var.project}"
  network            = "${var.name}-network"
  direction          = "EGRESS"

  allow {
    protocol = "tcp"
    ports    = ["22"]
  }

  target_tags        = ["ssh"]
}

resource "google_compute_firewall" "allow-ssh-to-webservers-from-bastion" {
  name          = "${var.name}-allow-ssh-to-private-network-from-bastion"
  project       = "${var.project}"
  network       = "${var.name}-network"
  direction     = "INGRESS"

  allow {
    protocol = "tcp"
    ports    = ["22"]
  }

  source_tags   = ["bastion"]
}

resource "google_compute_firewall" "allow-http-to-appservers" {
  name          = "${var.name}-allow-http-to-appservers"
  project       = "${var.project}"
  network       = "${var.name}-network"

  allow {
    protocol = "tcp"
    ports    = ["80"]
  }

  source_ranges = ["0.0.0.0/0"]

  source_tags   = ["http"]
}

resource "google_compute_firewall" "allow-db-connect-from-webservers" {
  name               = "${var.name}-allow-db-connect-from-webservers"
  project            = "${var.project}"
  network            = "${var.name}-network"
  direction          = "EGRESS"

  allow {
    protocol = "tcp"
    ports    = ["3306"]
  }

  destination_ranges = ["0.0.0.0/0"]

  target_tags        = ["db"]
}

module "management_subnet" {
  source   = "./subnet"
  project  = "${var.project}"
  region   = "${var.region}"
  name     = "${var.management_subnet_name}"
  network  = "${google_compute_network.network.self_link}"
  ip_range = "${var.management_subnet_ip_range}"
}

module "webservers_subnet" {
  source   = "./subnet"
  project  = "${var.project}"
  region   = "${var.region}"
  name     = "${var.webservers_subnet_name}"
  network  = "${google_compute_network.network.self_link}"
  ip_range = "${var.webservers_subnet_ip_range}"
}

module "bastion" {
  source        = "./bastion"
  name          = "${var.name}-bastion"
  project       = "${var.project}"
  zones         = "${var.zones}"
  subnet_name   = "${module.management_subnet.self_link}"
  image         = "${var.bastion_image}"
  instance_type = "${var.bastion_instance_type}"
  user          = "${var.user}"
  ssh_key       = "${var.ssh_key}"
}

---

# outputs.tf
output "name" {
  value = "${google_compute_network.network.name}"
}
output "bastion_public_ip" {
  value = "${module.bastion.public_ip}"
}
output "gateway_ipv4"  {
  value = "${google_compute_network.network.gateway_ipv4}"
}

---

# vars.tf
variable "name" {}
variable "project" {}
variable "region" {}
variable "zones" { type = "list" }
variable "webservers_subnet_name" {}
variable "webservers_subnet_ip_range" {}
variable "management_subnet_name" {}
variable "management_subnet_ip_range" {}
variable "bastion_image" {}
variable "bastion_instance_type" {}
variable "user" {}
variable "ssh_key" {}


# modules/network/subnet
# main.tf
resource "google_compute_subnetwork" "subnet" {
  name          = "${var.name}"
  project       = "${var.project}"
  region        = "${var.region}"
  network       = "${var.network}"
  ip_cidr_range = "${var.ip_range}"
}

---

# outputs.tf
output "ip_range" {
  value = "${google_compute_subnetwork.subnet.ip_cidr_range}"
}
output "self_link" {
  value = "${google_compute_subnetwork.subnet.self_link}"
}

---

# vars.tf
variable "name" {}
variable "project" {}
variable "region" {}
variable "network" {}
variable "ip_range" {}


# modules/network/bastion
# main.tf
resource "google_compute_instance" "bastion" {
  name         = "${var.name}"
  project      = "${var.project}"
  machine_type = "${var.instance_type}"
  zone         = "${element(var.zones, 0)}"

  metadata {
    ssh-keys = "${var.user}:${file("${var.ssh_key}")}"
  }

  boot_disk {
    initialize_params {
      image = "${var.image}"
    }
  }

  network_interface {
    subnetwork = "${var.subnet_name}"

    access_config {
      # Ephemeral IP - leaving this block empty will generate a new external IP and assign it to the machine
    }
  }

  tags = ["bastion"]
}

---

# outputs.tf
output "private_ip" {
  value = "${google_compute_instance.bastion.network_interface.0.address}"
}
output "public_ip" {
  value = "${google_compute_instance.bastion.network_interface.0.access_config.0.assigned_nat_ip}"
}

---

# vars.tf
variable "name" {}
variable "project" {}
variable "zones" { type = "list" }
variable "subnet_name" {}
variable "image" {}
variable "instance_type" {}
variable "user" {}
variable "ssh_key" {}

Profit 💰


ssh -i ~/.ssh/id_rsa $USER@$(terraform output --module=network bastion_public_ip)

[steinim@hello-test-bastion ~]$

Next: Database

Database module


modules/db
├── main.tf
├── outputs.tf
└── vars.tf


# main.tf
...

module "mysql-db" {
  source           = "../modules/db"
  db_name          = "${module.project.name}"
  project          = "${module.project.id}"
  region           = "${var.region}"
  db_name          = "${module.project.name}"
  user_name        = "hello"
  user_password    = "hello"
}

---

# vars.tf

...

variable "db_region" { default = "europe-west1" }


# main.tf
resource "google_sql_database_instance" "master" {
  name             = "${var.db_name}"
  project          = "${var.project}"
  region           = "${var.region}"
  database_version = "${var.database_version}"

  settings {
    tier                        = "${var.tier}"
    activation_policy           = "${var.activation_policy}"
    disk_autoresize             = "${var.disk_autoresize}"
    backup_configuration        = ["${var.backup_configuration}"]
    location_preference         = ["${var.location_preference}"]
    maintenance_window          = ["${var.maintenance_window}"]
    disk_size                   = "${var.disk_size}"
    disk_type                   = "${var.disk_type}"
    pricing_plan                = "${var.pricing_plan}"
    replication_type            = "${var.replication_type}"
    ip_configuration {
        ipv4_enabled = "true"

        authorized_networks {
          value           = "0.0.0.0/0"
          name            = "all"
        }
    }
  }

  replica_configuration = ["${var.replica_configuration}"]
}

resource "google_sql_database" "default" {
  name      = "${var.db_name}"
  project   = "${var.project}"
  instance  = "${google_sql_database_instance.master.name}"
  charset   = "${var.db_charset}"
  collation = "${var.db_collation}"
}

resource "google_sql_user" "default" {
  name     = "${var.user_name}"
  project  = "${var.project}"
  instance = "${google_sql_database_instance.master.name}"
  host     = "${var.user_host}"
  password = "${var.user_password}"
}

---

# outputs.tf

output instance_address {
  value       = "${google_sql_database_instance.master.ip_address.0.ip_address}"
}

---

# vars.tf

variable project { default = "" }

variable region { default = "europe-west1" }

variable database_version { default = "MYSQL_5_6" }

variable tier { default = "db-f1-micro" }

variable db_name { default = "default" }

variable db_charset { default = "" }

variable db_collation { default = "" }

variable user_name { default = "default" }

variable user_host { default = "%" }

variable user_password { default = "" }

variable activation_policy { default = "ALWAYS" }

variable disk_autoresize { default = false }

variable disk_size { default = 10 }

variable disk_type { default = "PD_SSD" }

variable pricing_plan { default = "PER_USE" }

variable replication_type { default = "SYNCHRONOUS" }

variable backup_configuration {
  type    = "map"
  default = {}
}

variable location_preference {
  type    = "list"
  default = []
}

variable maintenance_window {
  type    = "list"
  default = []
}

variable replica_configuration {
  type    = "list"
  default = []
}

Profit 💰


mysql --host=$(terraform output --module=mysql-db instance_address) --user=hello --password

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 57
Server version: 5.6.36-google (Google)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> quit
Bye

Next: Instance template

Instance template module


modules/instance-template
├── main.tf
├── outputs.tf
├── scripts
│   └── startup.sh
└── vars.tf


# main.tf
...

module "instance-template" {
  source        = "../modules/instance-template"
  name          = "${module.project.name}"
  env           = "${var.env}"
  project       = "${module.project.id}"
  region        = "${var.region}"
  network_name  = "${module.network.name}"
  image         = "${var.app_image}"
  instance_type = "${var.app_instance_type}"
  user          = "${var.user}"
  ssh_key       = "${var.ssh_key}"
  db_name       = "${module.project.name}"
  db_user       = "hello"
  db_password   = "hello"
  db_ip         = "${module.mysql-db.instance_address}"
}

---

# vars.tf
...

variable "appserver_count" { default = 2 }
variable "app_image" { default = "centos-7-v20170918" }
variable "app_instance_type" { default = "f1-micro" }


# main.tf
data "template_file" "init" {
  template = "${file("${path.module}/scripts/startup.sh")}"
  vars {
    db_name     = "${var.db_name}"
    db_user     = "${var.db_user}"
    db_password = "${var.db_password}"
    db_ip       = "${var.db_ip}"
  }
}

resource "google_compute_instance_template" "webserver" {
  name         = "${var.name}-webserver-instance-template"
  project      = "${var.project}"
  machine_type = "${var.instance_type}"
  region       = "${var.region}"

  metadata {
    ssh-keys = "${var.user}:${file("${var.ssh_key}")}"
  }

  disk {
    source_image = "${var.image}"
    auto_delete  = true
    boot         = true
  }

  network_interface {
    network            = "${var.network_name}"
    access_config {
      # Ephemeral IP - leaving this block empty will generate a new external IP and assign it to the machine
    }
  }

  metadata_startup_script = "${data.template_file.init.rendered}"

  tags = ["http"]

  labels = {
    environment = "${var.env}"
  }
}

---

# outputs.tf

output "instance_template" {
  value = "${google_compute_instance_template.webserver.self_link}"
}

---

# vars.tf

variable "name" {}
variable "project" {}
variable "network_name" {}
variable "image" {}
variable "instance_type" {}
variable "user" {}
variable "ssh_key" {}
variable "env" {}
variable "region" {}
variable "db_name" {}
variable "db_user" {}
variable "db_password" {}
variable "db_ip" {}


      
# scripts/startup.sh
#!/bin/bash

yum install -y nginx java

cat <<'EOF' > /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    server {
        listen      80 default_server;
        server_name _;
        location / {
            proxy_pass  http://127.0.0.1:1234;
        }
    }
}
EOF

setsebool -P httpd_can_network_connect true

systemctl enable nginx
systemctl start nginx

cat <<'EOF' > /config.properties
db.user=${db_user}
db.password=${db_password}
db.name=${db_name}
db.ip=${db_ip}
EOF

curl -o app.jar https://morisbak.net/files/helloworld-java-app.jar

java -jar app.jar > /dev/null 2>&1 &

Profit 💰

Browse to the public ip's of the webservers.

Auto scaling and load balancing

😰 What you'll need 😰

google_compute_global_forwarding_rule
google_compute_target_http_proxy
google_compute_url_map
google_compute_backend_service
google_compute_http_health_check
google_compute_instance_group_manager
google_compute_target_pool
google_compute_instance_group
google_compute_autoscaler

Load balancer module


modules/lb
├── main.tf
└── vars.tf


# main.tf

...

module "lb" {
  source            = "../modules/lb"
  name              = "${module.project.name}"
  project           = "${module.project.id}"
  region            = "${var.region}"
  count             = "${var.appserver_count}"
  instance_template = "${module.instance-template.instance_template}"
  zones             = "${var.zones}"
}


# main.tf
resource "google_compute_global_forwarding_rule" "global_forwarding_rule" {
  name       = "${var.name}-global-forwarding-rule"
  project    = "${var.project}"
  target     = "${google_compute_target_http_proxy.target_http_proxy.self_link}"
  port_range = "80"
}

resource "google_compute_target_http_proxy" "target_http_proxy" {
  name        = "${var.name}-proxy"
  project     = "${var.project}"
  url_map     = "${google_compute_url_map.url_map.self_link}"
}

resource "google_compute_url_map" "url_map" {
  name            = "${var.name}-url-map"
  project         = "${var.project}"
  default_service = "${google_compute_backend_service.backend_service.self_link}"
}

resource "google_compute_backend_service" "backend_service" {
  name                  = "${var.name}-backend-service"
  project               = "${var.project}"
  port_name             = "http"
  protocol              = "HTTP"
  backend {
    group                 = "${element(google_compute_instance_group_manager.webservers.*.instance_group, 0)}"
    balancing_mode        = "RATE"
    max_rate_per_instance = 100
  }

  backend {
    group                 = "${element(google_compute_instance_group_manager.webservers.*.instance_group, 1)}"
    balancing_mode        = "RATE"
    max_rate_per_instance = 100
  }

  health_checks = ["${google_compute_http_health_check.healthcheck.self_link}"]
}

resource "google_compute_http_health_check" "healthcheck" {
  name         = "${var.name}-healthcheck"
  project      = "${var.project}"
  port         = 80
  request_path = "/"
}

resource "google_compute_instance_group_manager" "webservers" {
  name               = "${var.name}-instance-group-manager-${count.index}"
  project            = "${var.project}"
  instance_template  = "${var.instance_template}"
  base_instance_name = "${var.name}-webserver-instance"
  count              = "${var.count}"
  zone               = "${element(var.zones, count.index)}"
  named_port {
    name = "http"
    port = 80
  }
}

resource "google_compute_autoscaler" "autoscaler" {
  name    = "${var.name}-scaler-${count.index}"
  project = "${var.project}"
  count   = "${var.count}"
  zone    = "${element(var.zones, count.index)}"
  target  = "${element(google_compute_instance_group_manager.webservers.*.self_link, count.index)}"

  autoscaling_policy = {
    max_replicas    = 2
    min_replicas    = 1
    cooldown_period = 90

    cpu_utilization {
      target = 0.8
    }
  }
}

---

# vars.tf

variable "name" {}
variable "project" {}
variable "region" {}
variable "count" {}
variable "instance_template" {}
variable "zones" { type = "list" }

Profit 💰

Jump through bastion


ssh -i ~/.ssh/id_rsa -J $USER@$(terraform output --module=network bastion_public_ip) \
$USER@<webserver-private-ip> \
-o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no

[steinim@hello-test-webserver-instance-hr80 ~]$

and ... 🎉

💰 🚀🚀🚀🚀 💰

What did I learn?

Switching to a different provider is not 1:1
Terraform is less mature on GCP than on AWS
Very few resources on the web (wish I knew about registry.terraform.io)
Especially dependencies is troublesome (asynchrony)
- Run it multiple times or write resources that waits
- Tainting resources is not always possible (instances and auto scaling)
- Destruction is a mess
Would be useful to know more about GCP on beforehand
GCP is in my opinion very different from AWS
I'm not done (first take on this talk)
- More refactoring is needed!
- More security is needed!
It was fun 😁
... but a lot of work 😩
I should probably be focusing on a PaaS

Thank you!

Slides:
http://steinim.github.io/slides/terraform-on-gcp/

Workshop:
https://github.com/steinim/gcp-terraform-workshop

@steinim

stein.inge.morisbak@BEKK.no