Zero Downtime Deployment with Ansible
Tutorial Slides
http://steinim.github.io/slides/fosdem/zero-downtime-ansible/tutorial.html
Presentation Slides
http://steinim.github.io/slides/fosdem/zero-downtime-ansible/
Source code
What's a provisioning framework?
- Automated setup of servers
- Configuration as code
- Not immutable (in itself)
Examples
- Create users
- Install software
- Generate and manipulate config files
- Start/stop/restart processes
- Set up dependencies between operations
Describe what to do (imperative)
#!/bin/bash
if $( command -v vim >/dev/null 2>&1 ); then
echo "vim is already installed."
else
apt-get install vim
fi
if $( grep -Fxq "filetype indent off" /etc/vim/vimrc ); then
echo "set filetype indent off is already in /etc/vim/vimrc."
else
echo "filetype indent off" >> /etc/vim/vimrc
# TODO: Do not continue if this fails.
fi
# TODO: Rollback if something fails.
Describe state (declarative)
- name: Ensure installed vim
apt:
pkg: vim
state: present
update_cache: no
tags:
- vim
- name: Set filetype indent off
lineinfile:
dest: /etc/vim/vimrc
line: 'filetype indent off'
state: present
tags:
- vim
Pros
- In source control.
- Self documenting (it's code!).
- Refactoring.
- Less differences between environments.
- Deterministic.
- Prevents manual steps.
- Fast and easy to configure up a new environment.
- Easier to test server setup.
Cons
- Unfamiliar to "old school operators".
- Operations is unfamiliar to developers.
- Startup cost.
- Training
- Migrating
- Isn't really immutable
Ansible
- SSH-based
- Client only (no server)
- YAML configuration
- Push (and pull)
- Supports more than setup and provisioning:
- Application deployment
- Remote command execution
Bring up the boxes
vagrant up
Layout (convention over configuration)
├── ansible.cfg
├── hosts
├── site.yml
├── group_vars
│ └── <group name>
├── host_vars
│ └── <host name>
├── roles
│ ├── <role>
│ │ ├── files
│ │ └── <file>
│ │ └── templates
│ │ └── <template>.j2
│ │ ├── handlers
│ │ │ └── main.yml
│ │ ├── tasks
│ │ │ └── main.yml
│ │ ├── vars
│ │ │ └── main.yml
Play!
ansible-playbook site.yml
Facts
- Ansible by default gathers “facts” about the machines under management.
- These facts can be accessed in Playbooks and in templates.
ansible -m setup app1.local
The task
- An app user 'devops', with:
- Home directory: /home/devops
- ssh-key
- A PostgresSQL database.
- Nginx as a reverse proxy and loadbalancer.
- An init script installed as a service.
- Deploy an application that uses the provisioned infrastructure.
Help!
http://docs.ansible.com/list_of_all_modules.htmlTask1: Install and configure software
git checkout start
- Modify roles/common/tasks/main.yml.
- Install Vim.
- Insert the line 'filetype indent off' in /etc/vim/vimrc
Help: http://docs.ansible.com/apt_module.html http://docs.ansible.com/lineinfile_module.html
Task1: Solution
git diff HEAD origin/task1
git checkout task1 # or keep your own solution
Check before
ansible app1.local -a "grep 'filetype indent off' /etc/vim/vimrc"
Run it!
ansible-playbook site.yml --tags vim
ProTip: Use '--tags', '--skip-tags', '--limit' and/or 'gather_facts: False'
to reduce execution time.
Check again
ansible app1.local -a "grep 'filetype indent off' /etc/vim/vimrc"
Progress
- Installed software
- Manipulated files
Variables
- Use variables! → Infrastructure as data.
- Where should variables be defined? Ansible has very many options.
- http://docs.ansible.com/playbooks_variables.html
- Inventory (./hosts)
- group_vars and host_vars
- Playbook (site.yml)
- Facts (local or server)
- Command line (as arguments)
- Access variables from playbooks: "{{ variable }}"
Task2: Create an application user
- Create roles/users/tasks/main.yml
- Home directory: /home/devops
- ssh-key
- Use variables! (group_vars)
Help: http://docs.ansible.com/group_module.html http://docs.ansible.com/user_module.html http://docs.ansible.com/lineinfile_module.html (.ssh/authorized_keys) http://docs.ansible.com/playbooks_best_practices.html#group-and-host-variables
git checkout task2_help
Task2: Solution
git diff HEAD origin/task2
git checkout task2 # or keep your own solution
Run it!
ansible-playbook site.yml --limit appservers --skip-tags vim,java
Try it!
ssh devops@app1.local
Progress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
Task3: Install and configure PostgreSQL
roles/postgresql
├── files
│ └── postgresql.conf
├── handlers
│ └── main.yml
├── tasks
│ ├── main.yml
│ └── ...
└── templates
└── pg_hba.conf.j2
Use variables (group_vars/all and/or group_vars/dbservers).
Use handler to restart postgresql upon notification
Template: git checkout task3 -- roles/postgresql/templates/pg_hba.conf.j2
Help: http://docs.ansible.com/template_module.html (pg_hba.conf.j2) http://docs.ansible.com/postgresql_user_module.html http://docs.ansible.com/postgresql_db_module.html http://docs.ansible.com/playbooks_intro.html#handlers-running-operations-on-change http://docs.ansible.com/playbooks_best_practices.html#group-and-host-variables
Task3: Solution
git diff HEAD origin/task3
git checkout task3 # or keep your own solution
Run it!
ansible-playbook site.yml --limit dbservers --tags pg_install
Try it!
$ vagrant ssh db
vagrant@db:~$ psql -d devops -U devops -W
devops=> \q
Progress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
- Installed and configured a database and a db user
Task4: Deploy!
roles/app
├── files
│ └── init.sh
├── tasks
│ └── main.yml
└── templates
└── config.properties.j2
NB! Use variables (./hosts).
Set 'serial: 1' for appservers in the playbook (site.yml).
Help: http://docs.ansible.com/service_module.html
Task4: Solution
git diff HEAD origin/task4
git checkout task4 # or keep your own solution
Run it!
ansible-playbook site.yml --limit appservers --tags deploy
Try it!
Browse to http://app1.local:1234/What just happened?
/home/devops
├── config.properties
├── current -> /home/devops/devops_1416228023.jar
├── previous -> /home/devops/devops_1416221573.jar
├── devops_1416221573.jar
├── devops_1416228023.jar
└── logs
├── stderr.log
└── stdout.log
/etc/init.d
└── devops
Progress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
- Installed and configured a database and a db user
- Deployed an application to two appservers and enabled it as a service
Task5: Deploy database
roles/db
├── files
│ └── migrate_db.sql
└── tasks
└── main.yml
Help: http://docs.ansible.com/command_module.html
psql -d {{ db.name }} -q -f /tmp/migrate_db.sql
become_user: postgres
Task5: Solution
git diff HEAD origin/task5
git checkout task5 # or keep your own solution
Run it!
ansible-playbook site.yml --limit dbservers --tags deploy
Try it!
$ vagrant ssh db
vagrant@db:~$ psql -d devops -U devops -W
devops=> \dt
devops=> select * from hello;
devops=> \q
Progress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
- Installed and configured a database and a db user
- Deployed an application to two appservers and enabled it as a service
- Migrated the database schema and fetched data from it through the application
Task6: Set up proxy
roles/nginx
├── handlers
│ └── main.yml
├── tasks
│ ├── config_nginx.yml
│ ├── install_nginx.yml
│ └── main.yml
└── templates
└── devops.conf.j2
Help: http://jinja.pocoo.org/docs/latest/templates/#for
Task6: Solution
git diff HEAD origin/task6
git checkout task6 # or keep your own solution
Run it!
ansible-playbook site.yml --limit proxies --tags nginx
Try it!
Browse to http://proxy.local/ # refresh me many timesProgress
- Installed software
- Manipulated files
- Created a user and set up a ssh-key
- Installed and configured a database and a db user
- Deployed an application to two appservers and enabled it as a service
- Migrated the database schema and fetched data from it through the application
- Set up a reverse proxy for automatic failover between the two appservers
The Expand/Contract pattern
| Expand | Contract |
|---|---|
|
|
Play time :-)
- Suggestions:
- Change database table name from HELLO to MESSAGES and deploy a new version without downtime.
- Implement automated rollback.
I have been playing :-)
git checkout play
ansible-playbook site.yml --limit appservers,dbservers --tags deploy
ansible-playbook site.yml --limit appservers,dbservers --tags rollback
Thank you!
Bonus tasks
Secrets using Ansible Vault
What Can Be Encrypted With Vault?
- Any structured data file used by Ansible
- group_vars/, host_vars/, inventory
- variables loaded by "include_vars" or "vars_files"
- variable files passed on the ansible-playbook command line with "-e @file.yml" or "-e @file.json"
- Also
- tasks, handlers, and so on
- arbitrary files, even binary files
All will be decrypted on the target host
(assuming a valid vault password is supplied when running the play)
Task7: Put your secrets in the vault
ansible-vault create group_vars/vault
ansible-playbook site.yml --ask-vault-pass
Help: http://docs.ansible.com/ansible/playbooks_vault.html
What did we forget?
To test!
ServerSpec
- RSpec
- Tests your servers' actual state
- SSH access
- No agent software on your servers
- Use any configuration management tool
- The one we have talked about
- Even manual setup
What are you waiting for?
- Describe your existing infrastructure with ServerSpec
- Use Vagrant and VirtualBox to play around in a "real" environment.
- Use a provisioning framework to make your tests go green
- Go live!
Thank you!