Automating Django Deployments with Fabric and Ansible

In the last post, we covered all the steps required to successfully develop and deploy a Django app on a single server. In this tutorial we will automate the deployment process with Fabric (v1.12.0) and Ansible (v2.1.3) to address these issues:

Scaling: When it comes to scaling a web app to handle thousands of daily requests, relying on a single server is not a good approach. Put simply, as the server approaches maximum CPU utilization, it can cause slow load times which can eventually lead to server failure. To overcome this, the app must be scaled to run on more than one server so that the servers can cumulatively handle the incoming concurrent requests.
Redundancy: Deploying a web app manually to a new server means a lot of repeated work with more chances of human error. Automating the process is key.

Specifically, we will automate:

Adding a new, non-root user
Configuring the server
Pulling the Django app code from a GitHub repo
Installing the dependencies
Daemonizing the app

Setup and Config

Start by spinning up a new Digital Ocean droplet, making sure to use the Fedora 25 image. Do not set up a pre-configured SSH key; we will be automating this process later via a Fabric script. Since the deployment process should be scalable, create a separate repository to house all the deployment scripts. Make a new project directory locally, and create and activate a virtualenv using Python 2.7x.

Why Python 2.7? Fabric does NOT support Python 3. Don’t worry: We’ll be using Python 3.5 when we provision the server.

$ mkdir automated-deployments
$ cd automated-deployments
$ virtualenv env
$ source env/bin/activate

Fabric Setup

Fabric is a tool used for automating routine shell commands over SSH, which we will be using to:

Set up the SSH keys
Harden user passwords
Install Ansible dependencies
Upgrade the server

Start by installing Fabric:

$ pip install fabric==1.12.0

Create a new folder called “prod”, and add a new file called fabfile.py to it to hold all of the Fabric scripts:

# prod/fabfile.py

import os
from fabric.contrib.files import sed
from fabric.api import env, local, run
from fabric.api import env

# initialize the base directory
abs_dir_path = os.path.dirname(
    os.path.dirname(os.path.abspath(__file__)))


# declare environment global variables

# root user
env.user = 'root'

# list of remote IP addresses
env.hosts = ['<remote-server-ip>']

# password for the remote server
env.password = '<remote-server-password>'

# full name of the user
env.full_name_user = '<your-name>'

# user group
env.user_group = 'deployers'

# user for the above group
env.user_name = 'deployer'

# ssh key path
env.ssh_keys_dir = os.path.join(abs_dir_path, 'ssh-keys')

Take note of the inline comments. Be sure to add you remote server’s IP address to the env.hosts variable. Update env.full_name_user as well. Hold off on updating env.password; we will get to that shortly. Look over all the env variables - they are completely customizable based on your system setup.

Set up the SSH keys

Add the following code to fabfile.py:

def start_provision():
    """
    Start server provisioning
    """
    # Create a new directory for a new remote server
    env.ssh_keys_name = os.path.join(
        env.ssh_keys_dir, env.host_string + '_prod_key')
    local('ssh-keygen -t rsa -b 2048 -f {0}'.format(env.ssh_keys_name))
    local('cp {0} {1}/authorized_keys'.format(
        env.ssh_keys_name + '.pub', env.ssh_keys_dir))
    # Prevent root SSHing into the remote server
    sed('/etc/ssh/sshd_config', '^UsePAM yes', 'UsePAM no')
    sed('/etc/ssh/sshd_config', '^PermitRootLogin yes',
        'PermitRootLogin no')
    sed('/etc/ssh/sshd_config', '^#PasswordAuthentication yes',
        'PasswordAuthentication no')

    install_ansible_dependencies()
    create_deployer_group()
    create_deployer_user()
    upload_keys()
    set_selinux_permissive()
    run('service sshd reload')
    upgrade_server()

This function acts as the entry point for the Fabric script. Besides triggering a series of functions, each explained in further steps, it explicitly-

Generates a new pair of SSH keys in the specified location within your local system
Copies the contents of the public key to the authorized_keys file
Makes changes to the remote sshd_config file to prevent root login and disable password-less auth

Preventing SSH access for the root user is an optional step, but it is recommended as it ensures that no one has superuser rights.

Create a directory for your SSH keys in the project root:

├── prod
│   └── fabfile.py
└── ssh-keys

Harden user passwords

This step includes the addition of three different functions, each executed serially to configure SSH password hardening…

Create deployer group

def create_deployer_group():
    """
    Create a user group for all project developers
    """
    run('groupadd {}'.format(env.user_group))
    run('mv /etc/sudoers /etc/sudoers-backup')
    run('(cat /etc/sudoers-backup; echo "%' +
        env.user_group + ' ALL=(ALL) ALL") > /etc/sudoers')
    run('chmod 440 /etc/sudoers')

Here, we add a new group called deployers and grant sudo permissions to it so that users can carry out processes with root privileges.

Create user

def create_deployer_user():
    """
    Create a user for the user group
    """
    run('adduser -c "{}" -m -g {} {}'.format(
        env.full_name_user, env.user_group, env.user_name))
    run('passwd {}'.format(env.user_name))
    run('usermod -a -G {} {}'.format(env.user_group, env.user_name))
    run('mkdir /home/{}/.ssh'.format(env.user_name))
    run('chown -R {} /home/{}/.ssh'.format(env.user_name, env.user_name))
    run('chgrp -R {} /home/{}/.ssh'.format(
        env.user_group, env.user_name))

This function-

Adds a new user to the deployers user group, which we defined in the last function
Sets up the SSH directory for keeping SSH key pairs and grants permission to the group and the user to access that directory

Upload SSH keys

def upload_keys():
    """
    Upload the SSH public/private keys to the remote server via scp
    """
    scp_command = 'scp {} {}/authorized_keys {}@{}:~/.ssh'.format(
        env.ssh_keys_name + '.pub',
        env.ssh_keys_dir,
        env.user_name,
        env.host_string
    )
    local(scp_command)

Here, we-

Upload the locally generated SSH keys to the remote server so that non-root users can log in via SSH without entering a password
Copy the public key and the authorized keys to the remote server in the newly created ssh-keys directory

Install Ansible dependencies

Add the following function to install the dependency packages for Ansible:

def install_ansible_dependencies():
    """
    Install the python-dnf module so that Ansible
    can communicate with Fedora's Package Manager
    """
    run('dnf install -y python-dnf')

Keep in mind that this is specific to the Fedora Linux distro, as we will be using the DNF module for installing packages, but it could vary by distro.

Set SELinux to permissive mode

The next function sets SELinux to permissive mode. This is done to overcome any potential Nginx 502 Bad Gateway errors.

def set_selinux_permissive():
    """
    Set SELinux to Permissive/Disabled Mode
    """
    # for permissive
    run('sudo setenforce 0')

Again, this is specific to the Fedora Linux distro.

Upgrade the server

Finally, upgrade the server:

def upgrade_server():
    """
    Upgrade the server as a root user
    """
    run('dnf upgrade -y')
    # optional command (necessary for Fedora 25)
    run('dnf install -y python')
    run('reboot')

Sanity check

With that, we’re done with the Fabric script. Before running it, make sure you SSH into the server as root and change the password:

$ ssh root@<server-ip-address>
You are required to change your password immediately (root enforced)
Changing password for root.
(current) UNIX password:
New password:
Retype new password:

Be sure to update env.password with the new password. Exit the server and return to the local terminal, then execute Fabric:

$ fab -f ./prod/fabfile.py start_provision

If all went well, new SSH keys will be generated, and you will be asked to create a password (make sure to do this!):

Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

A number of tasks will run. After the deployer user is created, you will be prompted to add a password for the user-

[104.236.66.172] out: Changing password for user deployer.

-which you will then have to enter when the SSH keys are uploaded:

deployer@104.236.66.172s password:

After this script exits successfully, you will NO longer be able to log into the remote server as a root user. Instead, you will only be able to use the non-root user deployer.

Try it out:

$ ssh root@<server-ip-address>
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

This is expected. Then, when you run-

$ ssh -i ./ssh-keys/104.236.66.172_prod_key deployer@104.236.66.172

-you should be able to log in just fine:

[deployer@fedora-512mb-nyc2-01 ~]$

Ansible Primer

Ansible is a configuration management and provisioning tool used to automate deployment tasks over SSH.

You can fire individual Ansible tasks against the app servers from your shell remotely and execute tasks on the go. Tasks can also be combined into Playbooks - a collection of multiple plays, where each play defines certain specific tasks that are required during the deployment process. They are executed against the app servers during the deployment process. Playbooks are written in YAML.

Playbooks

Playbooks consist of a modular architecture as follows:

Hosts specify all the IP addresses or domain names of our remote servers that need to be orchestrated. Playbooks always run on a targeted group of hosts.
Roles are divided into sub parts. Let’s look at some sample roles:
- Tasks are a collection of multiple tasks that need to be carried out during the deployment process.
- Handlers provide a way to trigger a set of operations when a module makes a change to the remote server (best thought of as hooks).
- Templates, in this context, are generally used for specifying some module-related configuration files - like nginx.
Variables are simply a list of key-value pairs where every key (a variable) is mapped to a value. Such variables can be used in the Playbooks as placeholders.

Sample Playbook

Now let’s look at a sample single-file Playbook:

---
# My Ansible playbook for configuring Nginx
- hosts: all

  vars:
    http_port: 80
    app_name: django_bootstrap

  tasks:
    - name: Install nginx
      dnf: name=nginx state=latest

    - name: Create nginx config file
      template: src=django_bootstrap.conf dest=/etc/nginx/conf.d/{{ app_name }}.conf
      become: yes
      notify:
        - restart nginx

  handlers:
    - name: Restart nginx
      service: name=nginx state=restarted enabled=yes
      become: yes

Here, we defined:

Hosts as hosts: all, which indicates that the Playbook will run on all of the servers that are listed in the inventory/hosts file
Variables http_port: 80 and app_name: django_bootstrap for use in a template
Tasks in order to install nginx, set up the nginx config (become indicates that we need admin privileges), and trigger the restart handler
Handler in order to restart the nginx service

Playbook Setup

Now let’s set up a Playbook for Django. Add a deploy.yml file to the “prod” directory:

##
# This playbook deploys the whole app stack
##
- name: apply common configuration to server
  hosts: all
  user: deployer
  roles:
    - common

The above snippet glues together the Ansible hosts, users, and roles.

Hosts

Add a hosts (plain text format) file to the “prod” directory and list the servers under their respective role names. We are provisioning a single server here:

[common]
<server-ip-address>

In the above snippet, common refers to the role name. Under the roles we have a list of IP addresses that need to be configured. Make sure to add your remote server’s IP address in place of <server-ip-address>.

Variables

Now we define the variables that will be used by the roles. Add a new folder inside “prod” called “group_vars”, then create a new file called all (plain text format) within that folder. Here, specify the following variables to start with:

# App Name
app_name: django_bootstrap

# Deployer User and Groups
deployer_user: deployer
deployer_group: deployers

# SSH Keys Directory
ssh_dir: <path-to-your-ssh-keys>

Make sure to update <path-to-your-ssh-keys>. To get the correct path, within the project root, run:

$ cd ssh-keys
$ pwd
/Users/michael.herman/repos/realpython/automated-deployments/ssh-keys

With these files in place, we are now ready to coordinate our deployment process with all the roles that need be carried out on the server.

Playbook Roles

Again, Playbooks are simply a collection of different plays, and all these plays are run under specific roles. Create a new directory called “roles” within “prod”.

Did you catch the name of the role in the deploy.yml file?

Then within the “roles” directory add a new directory called “common” - the role. Roles consists of “tasks”, “handlers”, and “templates”. Add a new directory for each.

Once done your file structure should look something like this:

├── prod
│   ├── deploy.yml
│   ├── fabfile.py
│   ├── group_vars
│   │   └── all
│   ├── hosts
│   └── roles
│       └── common
│           ├── handlers
│           ├── tasks
│           └── templates
└── ssh-keys
    ├── 104.236.66.172_prod_key
    ├── 104.236.66.172_prod_key.pub
    └── authorized_keys

All the plays are defined in a “tasks” directory, starting with a main.yml file. This file serves as the entry point for all Playbook tasks. It’s simply a list of multiple YAML files that need to be executed in order.

Create that file now within the “tasks” directory, then add the following to it:

##
# Configure the server for the Django app
##
- include: 01_server.yml
- include: 02_git.yml
- include: 03_postgres.yml
- include: 04_dependencies.yml
- include: 05_migrations.yml
- include: 06_nginx.yml
- include: 07_gunicorn.yml
- include: 08_systemd.yml
# - include: 09_fix-502.yml

Now, let’s create each task. Be sure to add a new file for each task to the “tasks” directory and add the accompanying code to each file. If you get lost, refer to the repo.

01_server.yml

##
# Update the DNF package cache and install packages as a root user
##
- name: Install required packages
  dnf: name={{item}} state=latest
  become: yes
  with_items:
    - vim
    - fail2ban
    - python3-devel
    - python-virtualenv
    - python3-virtualenv
    - python-devel
    - gcc
    - libselinux-python
    - redhat-rpm-config
    - libtiff-devel
    - libjpeg-devel
    - libzip-devel
    - freetype-devel
    - lcms2-devel
    - libwebp-devel
    - tcl-devel
    - tk-devel
    - policycoreutils-devel

Here, we list all the system packages that need to be installed.

02_git.yml

##
# Clone and pull the repo
##
- name: Set up git configuration
  dnf: name=git state=latest
  become: yes

- name: Clone or pull the latest code
  git: repo={{ code_repository_url }}
        dest={{ app_dir }}

Add the following variables to the group_vars/all file:

# Github Code's Repo URL
code_repository_url: https://github.com/realpython/django-bootstrap

# App Directory
app_dir: /home/{{ deployer_user }}/{{app_name}}

Make sure to fork then clone the django-bootstrap repo, then update the code_repository_url variable to the URL of your fork.

03_postgres.yml

##
# Set up and configure postgres
##
- name: Install and configure db
  dnf: name={{item}} state=latest
  become: yes
  with_items:
    - postgresql-server
    - postgresql-contrib
    - postgresql-devel
    - python-psycopg2

- name: Run initdb command
  raw: postgresql-setup initdb
  become: yes

- name: Start and enable postgres
  service: name=postgresql enabled=yes state=started
  become: yes

- name: Create database
  postgresql_db: name={{ app_name }}
  become_user: postgres
  become: yes

- name: Configure a new postgresql user
  postgresql_user: db={{ app_name }}
                                name={{ db_user }}
                                password={{ db_password }}
                                priv=ALL
                                role_attr_flags=NOSUPERUSER
  become: yes
  become_user: postgres
  notify:
    - restart postgres

Update group_vars/all with the database configuration needed for the playbook:

# DB Configuration
db_url: postgresql://{{deployer_user}}:{{db_password}}@localhost/{{app_name}}
db_password: thisissomeseucrepassword
db_name: "{{ app_name }}"
db_user: "{{ deployer_user }}"

Update the db_password variable with a secure password.

Did you notice that we restart the postgres service within the main.yml file in order to apply the changes after the database is configured? This is our first handler. Create a new file called main.yml in the “handlers” folder, then add the following:

- name: restart postgres
  service: name=postgresql state=restarted
  become: yes

04_dependencies.yml

##
# Set up all the dependencies in a virtualenv required by the Django app
##
- name: Create a virtualenv directory
  file: path={{ venv_dir }} state=directory

- name: Install dependencies
  pip:    requirements={{ app_dir }}/requirements.txt
          virtualenv={{ venv_dir }}
          virtualenv_python=python3.5

- name: Create the .env file for running ad-hoc python commands in our virtualenv
  template: src=env.j2 dest={{ app_dir }}/.env
  become: yes

Update group_vars/all like so:

# Application Dependencies Setup
venv_dir: '/home/{{ deployer_user }}/envs/{{ app_name }}'
venv_python: '{{ venv_dir }}/bin/python3.5'

Add a template called env.j2 to the “templates” folder, and add the following environment variables:

#!/bin/bash
export DEBUG="True"
export DATABASE_URL="postgresql://deployer:thisissomeseucrepassword@localhost/django_bootstrap"
export DJANGO_SECRET_KEY="changeme"
export DJANGO_SETTINGS_MODULE="config.settings.production"

Be very careful with the environment variables and their values in env.j2 since these are used to get the Django Project up and running.

05_migrations.yml

##
# Run db migrations and get all static files
##
- name: Make migrations
  shell: ". {{ app_dir }}/.env; {{ venv_python }} {{ app_dir }}/manage.py makemigrations "
  become: yes

- name: Migrate database
  django_manage: app_path={{ app_dir }}
                                 command=migrate
                                 virtualenv={{ venv_dir }}

- name: Get all static files
  django_manage: app_path={{ app_dir }}
                                 command=collectstatic
                                 virtualenv={{ venv_dir }}
  become: yes

06_nginx.yml

##
# Configure nginx web server
##
- name: Set up nginx config
  dnf: name=nginx state=latest
  become: yes

- name: Write nginx conf file
  template: src=django_bootstrap.conf dest=/etc/nginx/conf.d/{{ app_name }}.conf
  become: yes
  notify:
    - restart nginx

Add the following variable to group_vars/all:

# Remote Server Details
server_ip: <remote-server-ip>
wsgi_server_port: 8000

Don’t forget to update <remote-server-ip>. Then add the handler to handlers/main.yml:

- name: restart nginx
  service: name=nginx state=restarted enabled=yes
  become: yes

Then we need to add the django_bootstrap.conf template. Create that file within the “templates” directory, then add the code:

upstream app_server {
    server 127.0.0.1:{{ wsgi_server_port }} fail_timeout=0;
}

server {
    listen 80;
    server_name {{ server_ip }};
    access_log /var/log/nginx/{{ app_name }}-access.log;
    error_log /var/log/nginx/{{ app_name }}-error.log info;

    keepalive_timeout 5;

    # path for staticfiles
    location /static {
            autoindex on;
            alias {{ app_dir }}/staticfiles/;
    }

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;

        if (!-f $request_filename) {
            proxy_pass http://app_server;
            break;
        }
    }
}

07_gunicorn.yml

##
# Set up Gunicorn and configure systemd to execute gunicorn_start script
##
- name: Create a deploy directory
  file: path={{ deploy_dir }} state=directory
  become: yes

- name: Create the gunicorn_start script for running our app from systemd service
  template: src=gunicorn_start
                    dest={{ deploy_dir }}/gunicorn_start
  become: yes

- name: Make the gunicorn_start script executable
  raw: cd {{ deploy_dir }}; chmod +x gunicorn_start
  become: yes

Add more variables to groups_vars/all:

# Deploy Dir in App Directory
deploy_dir: '{{ app_dir }}/deploy'

# WSGI Vars
django_wsgi_module: config.wsgi
django_settings_module: config.settings.production
django_secret_key: 'changeme'
database_url: '{{ db_url }}'

Add the gunicorn_start template:

#!/bin/bash

### Define script variables

# Name of the app
NAME='{{ app_name }}'
# Path to virtualenv
VIRTUALENV='{{ venv_dir }}'
# Django Project Directory
DJANGODIR='{{ app_dir }}'
# The user to run as
USER={{ deployer_user }}
# The group to run as
GROUP={{deployer_group }}
# Number of worker processes Gunicorn should spawn
NUM_WORKERS=3
# Settings file that Gunicorn should use
DJANGO_SETTINGS_MODULE={{django_settings_module}}
# WSGI module name
DJANGO_WSGI_MODULE={{ django_wsgi_module }}


### Activate virtualenv and create environment variables

echo "Starting $NAME as `whoami`"
# Activate the virtual environment
cd $VIRTUALENV
source bin/activate
cd $DJANGODIR
# Defining the Environment Variables
export DJANGO_SECRET_KEY='{{ django_secret_key }}'
export DATABASE_URL='{{ db_url }}'
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH


### Start Gunicorn

exec gunicorn ${DJANGO_WSGI_MODULE}:application \
        --name $NAME \
        --workers $NUM_WORKERS \
        --user=$USER --group=$GROUP \
        --log-level=debug \
        --bind=127.0.0.1:8000

08_systemd.yml

##
# Set up systemd for executing gunicorn_start script
##
- name: write a systemd service file
  template: src=django-bootstrap.service
                    dest=/etc/systemd/system
  become: yes
  notify:
    - restart app
    - restart nginx

Add the template - django-bootstrap.service:

#!/bin/sh

[Unit]
Description=Django Web App
After=network.target

[Service]
PIDFile=/var/run/djangoBootstrap.pid
User={{ deployer_user }}
Group={{ deployer_group }}
ExecStart=/bin/sh {{ deploy_dir }}/gunicorn_start
Restart=on-abort

[Install]
WantedBy=multi-user.target

Add the following to the handlers:

- name: restart app
  service: name=django-bootstrap state=restarted enabled=yes
  become: yes

09_fix-502.yml

##
# Fix the 502 nginx error post deployment
#
- name: Fix nginx 502 error
  raw: cd ~; cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M mynginx; semodule -i mynginx.pp
  become: yes

Sanity Check (final)

With the virtualenv activated, install Ansible locally:

$ pip install ansible==2.1.3

Create a new file called deploy_prod.sh in the project root to run the playbook, making sure to update <server-ip>:

#!/bin/bash

ansible-playbook ./prod/deploy.yml --private-key=./ssh_keys<server-ip>_prod_key -K -u deployer -i ./prod/hosts -vvv

Then run following command to execute the playbook:

$ sh deploy_prod.sh

If any errors occur, consult the terminal for info on how to correct them. Once fixed, execute the deploy script again. When the script is done visit the server’s IP Address to verify your Django web app is live and running!

Make sure to uncomment this line in prod/roles/common/tasks/main.yml if you see the 502 error, which indicates that there is a problem with communication between nginx and Gunicorn:

# - include: 09_fix-502.yml

Then execute the playbook again.

If you execute the playbook more than once, make sure to comment out the Run initdb command found in 03_postgres.yml since it needs run only once. Otherwise, it will throw errors when trying to reinitialize the DB server.

Conclusion

This post provides a basic understanding of how you can automate the configuring of a server with Fabric and Ansible. Ansible Playbooks are particularly powerful since you can automate almost any task on the server via a YAML file. Hopefully, you can now start writing your own Playbooks and even use them in your workplace to configure production-ready servers.

Please add questions and comments below. The full code can be found in the automated-deployments repository.

TechiCrunch

Search This Blog