Continuous Security in the Cloud

DevOpsCon 2017

@steinim

stein.inge.morisbak@BEKK.no

Why?

Why now?

A shift in focus

Competitive advantage

• Innovate more
• Experiment more
• Faster time to market
• Build → Measure → Learn → Repeat ↺

How?

Continuous Delivery

What is Continuous Delivery?

...a software development discipline where you build software in such a way that the software can be released to production at any time.

Martin Fowler

Continuous delivery is about putting the release schedule in the hands of the business, not in the hands of IT.

Jez Humble

Continuous Delivery ♥ DevOps

What is DevOps?

Definition #1

But we have no Ops!

NoOps != No Operations

Definition #2

Definition #3

Our process:

Get 💩 done!

DevOps ♥ Cloud

Cloud reduces ops complexity

Cloud is secure

Principles for harnessing the cloud

Let the cloud vendor cover your *aaS

On premises

•  Applications
•  Data
•  Runtime
•  OS
•  Virtualization 
•  Servers
•  Storage
•  Network

IaaS (host)

•  Applications
•  Data
•  Runtime
•  OS
•  Virtualization 
•  Servers
•  Storage
•  Network

PaaS (build)

•  Applications
•  Data
•  Runtime
•  OS
•  Virtualization 
•  Servers
•  Storage
•  Network

SaaS (consume)

•  Applications
•  Data
•  Runtime
•  Os
•  Virtualization 
•  Servers
•  Storage
•  Network

Use public cloud!

The juicy part

Source code

./infrastructure          #1
./app-infrastructure...   #2
./app1                    #3
./app2
...
./appN
     

IaaS

Networking
• VPC
• Availability zones
• Subnets
• Security groups (FW rules)
• Bastion host
• Management server

Terraform

What does it do?

• Documents (infrastructure as code)
• Plans (no surprises)
• Graphs (parallelizes where possible)
• Automates (takes the human out of the equation)

IaaS

./infrastructure

Layout

./infrastructure
├── <environment>
│   ├── main.tf
│   └── vars.tf
└── modules
    ├── vpc
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── vars.tf
    ├── ig-route-table
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── vars.tf
    ├── subnet
    │   ├── main.tf
    │   └── vars.tf
    ├── bastion
    │   ├── main.tf
    │   └── vars.tf
    ├── mgmtserver   
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── vars.tf
    ├── ...
     

Networking

VPC

provider "aws" {
  region = "${var.aws_region}"
}

resource "aws_vpc" "vpc" {
    cidr_block           = "${var.vpc_cidr}"
    enable_dns_hostnames = true
    tags { Name = "${var.vpc_name}" }
}

resource "aws_internet_gateway" "ig" {
    vpc_id = "${aws_vpc.vpc.id}"
    tags { Name = "${var.ig_name}" }
}

# Grant the VPC internet access on its main route table
resource "aws_route" "internet_access_route" {
  route_table_id         = "${aws_vpc.vpc.main_route_table_id}"
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = "${aws_internet_gateway.ig.id}"
}
     

Subnets

resource "aws_subnet" "subnet" {
  vpc_id                  = "${var.vpc_id}"
  count                   = "${var.number_of_subnets}"
  cidr_block              = "${lookup(var.cidr_blocks, "zone_${count.index}")}"
  availability_zone       = "${lookup(var.zones, "zone_${count.index}")}"
  map_public_ip_on_launch = "${var.map_public_ip_on_launch}"

  tags { Name = "${var.name}_subnet_${lookup(var.zones, "zone_${count.index}")}" }
}
     

DB subnet group

resource "aws_db_subnet_group" "db_subnet_group" {
  name        = "${var.db_subnet_group_name}"
  description = "${var.db_subnet_group_name}"
  subnet_ids  = ["${var.private_subnet_ids}"]
}
     

Instances

resource "aws_instance" "instance" {
  ami                         = "${var.ami}"
  instance_type               = "t2.micro"
  key_name                    = "${var.key_pair_id}"
  subnet_id                   = "${var.subnet_id}"
  associate_public_ip_address = true
  source_dest_check           = "${var.source_dest_check}"
  vpc_security_group_ids = [ "${var.security_group_ids} ]
  tags { Name = "${var.instance_name}" }
}
     

Security groups

resource "aws_security_group_rule" "mgmtserver_ssh_ingress_from_bastion_security_rule" {
  type = "ingress"
  from_port = 22
  to_port = 22
  protocol = "tcp"
  source_security_group_id = "${var.bastion_egress_ssh_security_group_id}"
  security_group_id = "${aws_security_group.mgmtserver_sg.id}"
}

resource "aws_security_group_rule" "bastion_ssh_egress_to_mgmtserver_security_rule" {
  type = "egress"
  from_port = 22
  to_port = 22
  protocol = "tcp"
  security_group_id = "${var.bastion_egress_ssh_security_group_id}"
  source_security_group_id = "${aws_security_group.mgmtserver_sg.id}"
}

...
     

Automating SSH Audit

Tools

Passwords, secrets and keys in an automated pipeline

cloud-config.yml

vars:
  - name: AWS_DEFAULT_REGION
    value: eu-central-1

  - name: TF_VAR_env
    value: prod

  - name: TF_VAR_vpc_name
    value: prod

  - name: TF_VAR_public_key
    value: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzwGCC+nMTL6QIg+Z7SXtIXhiJJ5caa5tZkWQ1E6jDgHp7NqDAb21ZzjK3mGFlugF81h3OBF8uZPE5E+A0mRCB/pMzEMi/SBv0nvuPhijT81OeJxiF11Zxejc6gk8YiJIywQcYD3OcmmWvP2gK7MU1VIf3SQjHEROMaz+4uNJlC6QpDJfepxevQVJ4GVk7uq71NJjXV91gyT4/smOz5dP6tT7dYuP5Zn3lr5VN/BmHmtpklK9AWhwoIyRi+t97T1ihgiDvrmg7QSH9hIM+zKH2oxWi0RGV99t+ac6DV54ys4XD7OLAhHdlL7qV2G1TSIFqXXOaV98Cj2Mkxd2vJhhZ"

secret-vars:
  - name: TF_VAR_db_root_password
    key: nsb/aws/nsbno/prod/db_root_password

  - name: TF_VAR_private_key
    key: nsb/aws/nsb.pem

     

pass

~/src/nsb/.password-store
├── nsb.pem.gpg
├── nsbno
│   ├── prod
│   │   ├── bucket_user_secret_key.gpg
│   │   ├── db_password.gpg
│   │   └── vault_password.gpg
│   ├── test1
│   │   ├── ...
├── push
│   ├── prod
│   │   ├── ...
│   └── test1
│       ├── ...
├── nexus
│   └── users
│       ├── admin.gpg
│       └── deployment.gpg
├── ...
│   ├── maven
│       └── settings.xml.gpg
│   └── travis
│       └── nsbno-buildserver.gpg
├── enonic
│      └── cms.licenseKey.gpg
...
     

✗ pass show -c nsbno/prod/db_password
     

┌────────────────────────────────────────────────────────────────────────────────────┐
│ Please enter the passphrase to unlock the secret key for the OpenPGP certificate:  │
│ "Stein Inge Morisbak <stein.inge.morisbak@BEKK.no>"                                │
│ 2048-bit RSA key, ID 38380D80,                                                     │
│ created 2014-05-15 (main key ID 0CB573BF).                                         │
│                                                                                    │
│                                                                                    │
│ Passphrase ***********____________________________________________________________ │
│                                                                                    │
│            <OK>                                                  <Cancel>          │
└────────────────────────────────────────────────────────────────────────────────────┘
     

✗ pass show -c nsbno/prod/db_password
Copied nsbno/prod/db_password to clipboard. Will clear in 45 seconds.
     

envchain

✗ envchain --set aws AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
aws.AWS_ACCESS_KEY_ID: AKIAXXXXXXXXXXXXXXXX
aws.AWS_SECRET_ACCESS_KEY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
     

✗ envchain aws env | grep AWS_
AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXXXXXXX
AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
     

✗ envchain aws terraform-wrapper apply
     

You are provisioning PROD. Type PROD to continue... PROD
     

Started terraform operation at: 2017-09-08 12:46:57.870942455 +0200 CEST m=+6.052644916

null_resource.upload_efs_backup_script: Refreshing state... (ID: 8456414560025152219)
aws_route53_record.environment_route53_record: Refreshing state... (ID: Z1FHZIDFG100FA_prod.cloud.nsb.no_CNAME)
aws_route53_record.environment_route53_record: Refreshing state... (ID: Z1FHZIDFG100FA_app1.cloud.nsb.no_CNAME)
aws_route53_record.environment_route53_record: Refreshing state... (ID: Z1FHZIDFG100FA_app1.prod.cloud.nsb.no_CNAME)
aws_security_group.app_security_group: Refreshing state... (ID: sg-5f731534)
aws_security_group.efs_backup_security_group: Refreshing state... (ID: sg-279d864c)
aws_route53_record.environment_route53_record: Refreshing state... (ID: Z1FHZIDFG100FA_cloud.nsb.no_A)
aws_security_group.efs_security_group: Refreshing state... (ID: sg-a43d30cf)
aws_iam_user.user: Refreshing state... (ID: nsbno-prod)
aws_iam_role.role: Refreshing state... (ID: prod-nsbno-s3-full-access)
aws_iam_policy.policy: Refreshing state... (ID: arn:aws:iam::635004941268:policy/AmazonS3FullAccess-ElasticBeanstalk-NSBNO-prod)
aws_security_group.db_sg: Refreshing state... (ID: sg-817214ea)
aws_efs_file_system.efs: Refreshing state... (ID: fs-985cb8c1)
...
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
     

SaaS

./app-infrastructure

• RDS - Relational Database Service
• Route53 (DNS)
• Certificate Manager (SSL/TLS certificates)
• EFS - Elastic File System (NFS)
• S3 (Scalable storage)
• CloudWatch (monitoring, events, logs)
• CloudTrail (audit)
• IAM (Identity & Access Management)

Layout

./app-infrastructure
└── modules
    ├── database
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── vars.tf
    ├── efs
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── vars.tf
    ├── route53_alias_record
    │   ├── main.tf
    │   └── vars.tf
    ├── route53_cname_record
    │   ├── main.tf
    │   └── vars.tf
    └── ssl_cert
        ├── main.tf
        ├── outputs.tf
        └── vars.tf
     

resource "aws_db_instance" "db" {
  name                    = "${var.db_name}"
  identifier              = "${var.db_identifier}"
  engine                  = "${var.db_engine}"
  engine_version          = "${var.db_engine_version}"
  instance_class          = "${var.db_instance_class}"
  username                = "${var.db_username}"
  password                = "${var.db_password}"
  vpc_security_group_ids  = ["${aws_security_group.db_sg.id}"]
  db_subnet_group_name    = "${var.db_subnet_group_id}"
  parameter_group_name    = "${var.db_parameter_group_name}"
  backup_retention_period = "${var.backup_retention_period}"
  availability_zone       = "${var.availability_zone}"
  multi_az                = "${var.multi_az}"
  backup_window           = "${var.backup_window}"
  maintenance_window      = "${var.maintenance_window}"
  allocated_storage       = "${var.allocated_storage}"
  storage_type            = "${var.storage_type}"
  apply_immediately       = "${var.apply_immediately}"
  skip_final_snapshot     = "${var.skip_final_snapshot}"
}

resource "aws_security_group" "db_sg" {
    vpc_id      = "${var.vpc_id}"
    name        = "${var.db_sg_name}"
    description = "${var.db_sg_name}"
    tags { Name = "${var.db_sg_name}" }
}
     

AWS CLI

NAME
       aws -

DESCRIPTION
       The  AWS  Command  Line  Interface is a unified tool to manage your AWS
       services.

SYNOPSIS
          aws [options] <command> <subcommand> [parameters]

       Use aws command help for information on a  specific  command.  Use  aws
       help  topics  to view a list of available help topics. The synopsis for
       each command shows its parameters and their usage. Optional  parameters
       are shown in square brackets.
     

PaaS

{./app1,./app2,...}

From pets to cattle

Elastic Beanstalk

Easy to begin, Impossible to outgrow

Automates and manages:
configuration
deploy
capacity provisioning
load balancing
auto scaling
monitoring
Without extra cost beyond the resources you use

.<app1>
├──.ebextensions
|  ├── 00-set-timezone.config
|  ├── 03-secure-listener.config
|  ├── 05-cw-logging.config
|  └── ...
├── Procfile
├── create.sh
├── deploy.sh
├── terminate.sh
     

.ebextensions/00-set-timezone.config

commands:
  link_Oslo:
    command: "ln -sf /usr/share/zoneinfo/Europe/Oslo /etc/localtime"
     

.ebextensions/03-securelistener.config

option_settings:
  aws:elb:listener:443:
    SSLCertificateId: arn:aws:acm:eu-central-1:635004941268:certificate/6502eb35-99a7-4004-8e47-a5a0ec194c6a
    ListenerProtocol: HTTPS
    InstancePort: 80
  aws:elb:listener:80:
    ListenerEnabled: false
  aws:elasticbeanstalk:application:
    Application Healthcheck URL: /health
     

.ebextensions/05-cw-logging.config

option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
    StreamLogs: true
    DeleteOnTerminate: false
    RetentionInDays: 7
     

Procfile

web: java -jar app.jar
     

3 commands

eb create
eb deploy
eb terminate
     

./create.sh

#!/bin/bash
...
eb create ${beanstalk_env} \
  --keyname ${vpc_name} \
  --vpc.id ${vpc_id} \
  --vpc.dbsubnets ${private_subnet_ids} \
  --vpc.ec2subnets ${private_subnet_ids}  \
  --vpc.elbpublic \
  --vpc.elbsubnets ${public_subnet_ids} \
  --vpc.securitygroups ${app_security_group_id} \
  --envvars RDS_HOSTNAME=${rds_hostname},RDS_PORT=${rds_port},ENV=${app_env} \
  --platform "${platform}" \
  --instance_type ${instance_type} \
  --cname ${beanstalk_env} \
  --scale ${scale}
...
     

Summing it up

Thank you!

Slides:
http://steinim.github.io/slides/continuous-security-in-the-cloud-devopscon/

@steinim

stein.inge.morisbak@BEKK.no