Zero Downtime Deployment with Ansible
Slides
http://steinim.github.io/slides/fosdem/zero-downtime-ansible
Source code
git@github.com:steinim/zero-downtime-ansible.git
Full tutorial
http://steinim.github.io/slides/fosdem/zero-downtime-ansible/tutorial.html
What's a provisioning framework?
- Automated setup of servers
- Configuration as code
- Not immutable (in itself)
Examples
- Create users
- Install software
- Generate and manipulate config files
- Start/stop/restart processes
- Set up dependencies between operations
Describe what to do (imperative)
#!/bin/bash
if $( command -v vim >/dev/null 2>&1 ); then
echo "vim is already installed."
else
apt-get install vim
fi
if $( grep -Fxq "filetype indent off" /etc/vim/vimrc ); then
echo "set filetype indent off is already in /etc/vim/vimrc."
else
echo "filetype indent off" >> /etc/vim/vimrc
# TODO: Do not continue if this fails.
fi
# TODO: Rollback if something fails.
Describe state (declarative)
- name: Ensure installed vim
apt:
pkg: vim
state: present
update_cache: no
tags:
- vim
- name: Set filetype indent off
lineinfile:
dest: /etc/vim/vimrc
line: 'filetype indent off'
state: present
tags:
- vim
Pros
- In source control.
- Self documenting (it's code!).
- Refactoring.
- Less differences between environments.
- Deterministic.
- Prevents manual steps.
- Fast and easy to configure up a new environment.
- Easier to test server setup.
Cons
- Isn't really immutable
Ansible
- SSH-based
- Client only (no server)
- YAML configuration
- Push (and pull)
- Supports more than setup and provisioning:
- Application deployment
- Remote command execution
Bring up the boxes
vagrant up
Layout (convention over configuration)
├── ansible.cfg
├── hosts
├── site.yml
├── group_vars
│ └── <group name>
├── host_vars
│ └── <host name>
├── roles
│ ├── <role>
│ │ ├── files
│ │ └── <file>
│ │ └── templates
│ │ └── <template>.j2
│ │ ├── handlers
│ │ │ └── main.yml
│ │ ├── tasks
│ │ │ └── main.yml
│ │ ├── vars
│ │ │ └── main.yml
Play!
ansible-playbook site.yml
Facts
- Ansible by default gathers “facts” about the machines under management.
- These facts can be accessed in Playbooks and in templates.
ansible -m setup app1.local
The task
- An app user 'devops', with:
- Home directory: /home/devops
- ssh-key
- A PostgresSQL database.
- Nginx as a reverse proxy and loadbalancer.
- An init script installed as a service.
- Deploy an application that uses the provisioned infrastructure.
Help!
http://docs.ansible.com/list_of_all_modules.htmlTask1: Install and configure software
git checkout start
- Modify roles/common/tasks/main.yml.
- Install Vim.
- Insert the line 'filetype indent off' in /etc/vim/vimrc
Help: http://docs.ansible.com/apt_module.html http://docs.ansible.com/lineinfile_module.html
Task1: Solution
- name: Ensure installed vim
apt:
pkg: vim
state: present
update_cache: no
tags:
- vim
- name: Set filetype indent off
lineinfile:
dest: /etc/vim/vimrc
line: 'filetype indent off'
state: present
tags:
- vim
Run it!
ansible-playbook site.yml --tags vim
ProTip: Use '--tags', '--skip-tags', '--limit' and/or 'gather_facts: False'
to reduce execution time.
Progress
- Installed software
- Manipulated files
Variables
- Use variables! → Infrastructure as data.
- Where should variables be defined? Ansible has very many options.
- http://docs.ansible.com/playbooks_variables.html
- Inventory (./hosts)
- group_vars and host_vars
- Playbook (site.yml)
- Facts (local or server)
- Command line (as arguments)
- Access variables from playbooks: "{{ variable }}"
Task2: Create an application user
- Create roles/users/tasks/main.yml
- Home directory: /home/devops
- ssh-key
- Use variables! (group_vars)
Help: http://docs.ansible.com/group_module.html http://docs.ansible.com/user_module.html http://docs.ansible.com/file_module.html (create a directory) http://docs.ansible.com/lineinfile_module.html (.ssh/authorized_keys) http://docs.ansible.com/playbooks_best_practices.html#group-and-host-variables
Define some variables
# group_vars/appservers
user: devops
group: devops
user_name: "Devops app user"
Task2: Solution
# roles/users/tasks/main.yml
- name: Create group
group:
name: "{{ group }}"
state: present
tags:
- users
- name: Create user
user:
name: "{{ user }}"
comment: "{{ user_name }}"
group: "{{ group }}"
createhome: yes
home: "/home/{{ user }}"
shell: /bin/bash
state: present
tags:
- users
- name: Ensure ssh directory exists for user
file:
dest: "/home/{{ user }}/.ssh"
owner: "{{ user }}"
group: "{{ group }}"
state: directory
mode: 0700
tags:
- users
- name: Ensure public key is in authorized_keys
lineinfile:
dest: "/home/{{ user }}/.ssh/authorized_keys"
state: present
line: "{{ lookup('file','~/.ssh/id_rsa.pub') }}"
insertafter: EOF
create: yes
owner: "{{ user }}"
group: "{{ group }}"
mode: 0600
regexp: ^ssh-rsa
tags:
- users
Run it!
ansible-playbook site.yml --limit appservers --skip-tags apt,vim,java
Try it!
ssh devops@app1.local
Progress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
Task3: Install and configure PostgreSQL
roles/postgresql
├── files
│ └── postgresql.conf
├── handlers
│ └── main.yml
├── tasks
│ ├── main.yml
│ └── ...
└── templates
└── pg_hba.conf.j2
Use variables (group_vars/all and/or group_vars/dbservers).
Use handler to restart postgresql upon notification
Template: git checkout task3 -- roles/postgresql/templates/pg_hba.conf.j2
Help: http://docs.ansible.com/template_module.html (pg_hba.conf.j2) http://docs.ansible.com/postgresql_user_module.html http://docs.ansible.com/postgresql_db_module.html http://docs.ansible.com/playbooks_intro.html#handlers-running-operations-on-change http://docs.ansible.com/playbooks_best_practices.html#group-and-host-variables
Define some variables
# group_vars/dbservers
postgresql:
version: 9.4
repo: 'deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main'
user: postgres
group: postgres
data_dir: /var/lib/pgsql/data
address: 192.168.101.0/24
db:
name: devops
user: devops
password: devops123
Create a template
# roles/postgresql/templates/pg_hba.conf.j2
# TYPE DATABASE USER ADDRESS METHOD
local all all trust
host {{ db.name }} {{ db.user }} {{ postgresql.address }} md5
Create a handler
# roles/postgresql/handlers/main.yml
- name: Restart postgresql
service:
name: postgresql
state: restarted
Task3: Solution
# roles/postgresql/tasks/postgresql.yml
- name: Install Postgresql and dependencies
apt:
pkg: "{{ item }}"
state: installed
with_items:
- "postgresql-{{ postgresql.version }}"
- python-psycopg2
tags:
- pg_install
- name: Install postgresql.conf
copy:
src: postgresql.conf
dest: "/etc/postgresql/{{ postgresql.version }}/main/postgresql.conf"
owner: "{{ postgresql.user }}"
group: "{{ postgresql.group }}"
notify:
- Restart postgresql
tags:
- pg_install
- name: Install pg_hba.conf
template:
src: pg_hba.conf.j2
dest: "/etc/postgresql/{{ postgresql.version }}/main/pg_hba.conf"
owner: "{{ postgresql.user }}"
group: "{{ postgresql.group }}"
notify:
- Restart postgresql
tags:
- pg_install
- name: Ensure that postgresql is started
service:
name: postgresql
state: started
tags:
- pg_install
- name: Create database
postgresql_db:
name: "{{ db.name }}"
become_user: postgres
tags:
- pg_install
- name: Create db user
postgresql_user:
db: "{{ db.name }}"
name: "{{ db.user }}"
password: "{{ db.password }}"
become_user: postgres
tags:
- pg_install
- name: Ensure that postgresql is started
service:
name: postgresql
state: started
tags:
- pg_install
Run it!
ansible-playbook site.yml --limit dbservers --tags pg_install
Try it!
$ vagrant ssh db
vagrant@db:~$ psql -d devops -U devops -W
devops=> \q
Progress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
- Installed and configured a database and a db user
Task4: Deploy!
roles/app
├── files
│ └── init.sh
├── tasks
│ └── main.yml
└── templates
└── config.properties.j2
NB! Use variables (./hosts).
Set 'serial: 1' for appservers in the playbook (site.yml).
Help: http://docs.ansible.com/service_module.html
Run it!
ansible-playbook site.yml --limit appservers --tags deploy
Try it!
Browse to http://app1.local:1234/What just happened?
/home/devops
├── config.properties
├── current -> /home/devops/devops_1416228023.jar
├── previous -> /home/devops/devops_1416221573.jar
├── devops_1416221573.jar
├── devops_1416228023.jar
└── logs
├── stderr.log
└── stdout.log
/etc/init.d
└── devops
Progress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
- Installed and configured a database and a db user
- Deployed an application to two appservers and enabled it as a service
Task5: Deploy database
roles/db
├── files
│ └── migrate_db.sql
└── tasks
└── main.yml
Help: http://docs.ansible.com/command_module.html
psql -d {{ db.name }} -q -f /tmp/migrate_db.sql
become_user: postgres
Task5: Solution
- name: Copy db migration script
copy:
src: migrate_db.sql
dest: /tmp/migrate_db.sql
become_user: postgres
tags:
- deploy
- name: Run db migration script
command: psql -d {{ db.name }} -q -f /tmp/migrate_db.sql
become_user: postgres
tags:
- deploy
Run it!
ansible-playbook site.yml --limit dbservers --tags deploy
Try it!
$ vagrant ssh db
vagrant@db:~$ psql -d devops -U devops -W
devops=> \dt
devops=> select * from hello;
devops=> \q
Progress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
- Installed and configured a database and a db user
- Deployed an application to two appservers and enabled it as a service
- Migrated the database schema and fetched data from it through the application
Task6: Set up proxy
roles/nginx
├── handlers
│ └── main.yml
├── tasks
│ ├── config_nginx.yml
│ ├── install_nginx.yml
│ └── main.yml
└── templates
└── devops.conf.j2
Help: http://jinja.pocoo.org/docs/latest/templates/#for
Task6: Solution
upstream backend {
{% for appserver in groups.appservers %}
server {{ appserver }}:{{ app_port }} fail_timeout=1s;
{% endfor %}
}
server {
listen 80;
server_name _;
access_log /var/log/nginx/devops-access.log;
location / {
proxy_pass http://backend;
proxy_redirect off;
proxy_next_upstream error timeout invalid_header http_500 http_502;
proxy_connect_timeout 2;
}
}
Run it!
ansible-playbook site.yml --limit proxies --tags nginx
Try it!
Browse to http://proxy.local/ # refresh me many timesProgress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
- Installed and configured a database and a db user
- Deployed an application to two appservers and enabled it as a service
- Migrated the database schema and fetched data from it through the application
- Set up a reverse proxy for automatic failover between the two appservers
The Expand/Contract pattern
| Expand | Contract |
|---|---|
|
|
Play time :-)
- Suggestions:
- Change database table name from HELLO to MESSAGES and deploy a new version without downtime.
- Implement automated rollback.
Thank you!
Bonus
Secrets using Ansible Vault
What Can Be Encrypted With Vault?
- Any structured data file used by Ansible
- group_vars/, host_vars/, inventory
- variables loaded by "include_vars" or "vars_files"
- variable files passed on the ansible-playbook command line with "-e @file.yml" or "-e @file.json"
- Also
- tasks, handlers, and so on
- arbitrary files, even binary files
All will be decrypted on the target host
(assuming a valid vault password is supplied when running the play)
Task7: Put your secrets in the vault
ansible-vault create group_vars/vault
ansible-playbook site.yml --ask-vault-pass
Help: http://docs.ansible.com/ansible/playbooks_vault.html
What did we forget?
To test!
ServerSpec
- RSpec
- Tests your servers' actual state
- SSH access
- No agent software on your servers
- Use any configuration management tool
- The one we have talked about
- Even manual setup
What are you waiting for?
- Describe your existing infrastructure with ServerSpec
- Use Vagrant and VirtualBox to play around in a "real" environment.
- Use a provisioning framework to make your tests go green
- Go live!
Thank you!