Shipping With Confidence: Continuous Integration, Continuous Delivery, And You

When it comes to software development, getting your product into the hands of the customer is critical to the success of the business. Long release cycles, often measured in years, will not help your business or your customers.

Several years ago, there was a revolution/evolution in the software industry called Extreme Programming (XP). The initial buzz of the movement has quieted down, but several valuable practices embedded within Extreme Programming still remain popular — Continuous Integration (CI) and Continuous Delivery (CD).

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily — leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.” — Martin Fowler

“Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time.[1] It aims at building, testing, and releasing software faster and more frequently. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.” — Wikipedia

The goal of both practices is to ship more code, faster, and with more confidence.

At TextNow we aim to deploy our API servers every day of the week, and we’ve had good success with that schedule. The code changes being deployed are small, and if critical bugs end up affecting our customers it’s relatively easy to determine what changed and what broke that day.

The Problem

Ok, so the issues with Continuous Integration are solved with hundreds of builds, unit tests and integration tests every day. But the Continuous Delivery part needed some help, as shipping our code to production was tedious and would run into issues, like:

• A twenty-two-step manual process where developers followed steps outlined on a wiki page.
• Developers’ laptops need to be setup with the correct tools and ssh keys.
• Releases get tagged in Github but, too often, the wrong branch would be tagged.
• It wasn’t clear who was doing the deploy that day.
• It’s not immediately clear who is responsible for a deploy on any given day.
• Communication around deployment is not as rigorous as it should be. Developers would paste messages into Slack to show the status of the deployment.
• No logs are created, making it difficult to troubleshoot issues.
• The entire process is time consuming and can require an entire morning of a developer’s time.

‍

The TL;DR version: The whole process takes too long, and can be riddled with errors.

The Rise of ChatOps

We maybe could have solved the problem with an external tool, but our infrastructure is very stable due to the use of puppet. We already had a solid and working process that was fitting our needs, so really all we needed was a way to automate it and bring the process together.

The nicest and most visible way to automate this process was through Slack. We were already announcing our releases in Slack, so why not just do everything there? A Slack bot would be the user interface to the system and would provide the desired level of operational visibility.

Now, instead of the twenty-two-step error-prone process we previously had, we now use a five-step process where developers just message our Slack bot.

The Slack bot will send messages to different Slack channels based on the audience of that particular channel. Our general release channel is widely used and sees basic release status messages:

But if the operations team wants to see more details of what’s happening with the deploy, they can go to another Slack channel:

Under the Hood

The technology piecing everything together isn’t overly complicated [Says you, Curt — The Editor] but some extra tooling was required to make everything work together nicely:

Hubot

Hubot is a robot that connects to Slack and listens for messages. Hubot is what’s being used to create the help command in the “release help” image above. It listens for a message, like “release pr” for example, and then executes a shell command.

Golang

Golang is an open-source programming language created by Google in 2009. (Google language, in other words.) Golang is my language of choice for building automation or deployment systems. It’s simple language and has large amounts of library/package support.

Slack API

Slack provides an API that allows you to send messages to channels or users. That “Deploy Notice” message in the image above comes from a Slack tool written in Golang. It uses the nlopes/slack package and is quite nice to work with, in my opinion.

Github API

The Github API allows you to create branches, pull requests (PRs), releases, add incremental release numbers, and tag releases. The tool replaced the manual steps of creating a release branch, creating a pull request to merge into master, assigning reviewers based on commits, filling in release notes, tagging the release, and publishing the release. The added advantage of creating a separate tool is that it has give us the ability to use it with our Jenkins jobs, which has added intelligence to our automation systems. The tool is built using the Google Go-Github package.

Ansible

Ansible is being used for deployments to several server pools.

Puppet

Puppet’s automation software is what keeps our servers running.

Shell

Shell scripts are used to piece everything together. Hubot uses a shell script and that shell script calls the Slack and Github tools. Version 2 of the system will be written in Golang and this will allow us to use reusable pieces of code amongst several products and microservices.

‍

If you’re interested in working with Curt making Futurama references with Slack bots, or in any of our other awesome teams and departments, check out our job openings!