Development within agencies, continuous integration, and Gitlab CI

There’s a lot of excellent discussion about continuous integration and how useful it is in modern web development. In this post, E2 Digital’s Tim Oliver explains how to get started with GitLab CI.

There’s a lot of excellent discussion about continuous integration (CI) and how useful it is in modern web development. If you’re relatively new to the concept, this post by Robbie Averill is a good starting point. Below, I’ll describe the use case for CI from an agency perspective and explain practically how we’re using CI with GitLab at E2 Digital.

The CI use case from a different angle

The reality of an agency environment is that you have a constant flow of projects and updates going on at all times. It’s a constant whirlwind. The websites you’re looking after were built at different times and by different people and subteams. Sometimes, you’re updating an experimental project from a few years ago, and other times you’re taking a subcontractor’s site and updating it. Budgets and timelines are often tight, there are compromises involved, and many things competing for attention. As you improve your tooling, it’s often impossible to apply those improvements to your older work.

One of the biggest changes over the last 10 years of web development has been the shift in complexity from backend to frontend development. This article sums it up pretty well. Once upon a time you could just edit CSS, but any modern website requires some kind of build process and tooling to generate your frontend assets. The rate of change in this area of the industry is high, so how do you get started working on a project? You probably have source control, you probably have some documentation, but it’s easy for things to get out of date.

Enter GitLab CI

GitLab has an integrated CI tool that’s really easy to get started with. Because it’s right there and straightforward we were able to get going without committing to setting up extra services and build servers. It’s fully integrated with GitLab so you can see the status of the build attached to each merge request and commit.

The CI configuration for a site lives right in the source code with the rest of your code. That’s been the best part for us at E2 Digital – there’s no separate service to configure, which cuts down the maintenance load and the learning curve for people coming in.

How are we using GitLab CI at E2 Digital?

Test that our composer dependencies are able to be installed successfully.
Build our frontend assets and store them as build artifacts to be used in the deploy stages.
Deploy to the staging environment on commits to master.
Create a release branch when the repository is tagged and deploy to a UAT environment on the SilverStripe Platform using the Deploynaut API.

Why a release branch?

When using preprocessors such as Sass, we don’t want to commit the compiled CSS and other build artifacts to our Git repository as we have to deploy to the SilverStripe Platform through a repository. We therefore need the compilation to happen somewhere. What we do through GitLab CI is commit the build artifacts to our separate release branches when we create a tag on our repository. But thanks to package deployments, the release branches approach is no longer required.

A quick introduction

All you need to do to get started with GitLab CI is add a file named .gitlab-ci.yml to your GitLab repo, push your code to GitLab, and it’ll start running your tests or build. This works with their free private repos too and although you have a limited amount of processing time per month, it’s completely fine for getting started.

Your build runs on a Docker container on Github’s infrastructure by default. The build itself is a list of arbitrary shell commands. If you have a series of commands that you type to build your code, you can copy them into the file and get started that way.

.gitlab-ci.yml

This is a YAML file which lives in the root of your repository. The .gitlab-ci.yml file tells the GitLab runner what to do. By default, it runs a pipeline with three stages: build, test, and deploy. However, stages with no jobs are simply ignored so you don’t need to use all three stages.

There’s no other setup to get started. If you have a .gitlab-ci.yml file in your project, GitLab will process it.

Example .gitlab-ci.yml file

The image tag allows you to specify a Docker image to use to run your builds. There are a variety of images available by default and you can generate your own custom one with any requirements to reduce installation work. For example, we install SilverStripe dependencies and composer in our image at https://hub.docker.com/r/e2digital/ss-php71/~/dockerfile/.

If you have any more specific dependencies, you can include them in a “before_script” tag which allows you to bring in any extra packages, but generally you’ll want to include them in the Docker image.

“build_assets” is an actual job. The only required element there is a script section, which are the actual commands run on the Docker image. In this case, we have Yarn configured to run the rest of our build process.

Example .gitlab-ci.yml file

The “only” tag lets you run jobs only for specific git branches. In this case, we run “staging_deploy” only when code passes review and is merged into master, and “platform_deploy” when we tag code matching a pattern to deploy our code to the SilverStripe Platform.

The agency advantage

CI ensures your work can actually be built. For example, with local development it’s possible for a dev to have packages installed that are part of the process of producing a site’s assets, but that aren’t defined anywhere in source control, locally installed programs and the like. This can be a nightmare for another developer to pick up, but this CI approach makes that problem easily visible. This is particularly helpful when developers have to pick up the site for the first time or return to it after a period of time.

Some downsides and caveats

There is some turnaround time spent waiting for your changes to build before you can merge and deploy them. This is something that can be optimised though, there’s a lot of scope for caching Docker runners and the like, and you can build custom Docker images to reduce setup overhead.

Having your CI setup in source control means that you can’t make changes to your CI setup without doing commits and pushing them to GitLab.

GitLab provides a limited amount of free runner time – you get 2000 CI minutes per month for free. We found that we only started running out with multiple active projects going on. However, after that point you can set up your own runners on your own infrastructure to do the CI work, and it’s very straightforward.

Tips and tricks

You can test .gitlab-ci.yml changes on a branch – a CI block can run for only a particular branch name.
Private CI variables let you handle all the site specific credentials and keys.
CI_DEBUG_TRACE – enable debug tracing for your jobs.
Setup Slack webhooks to alert your team to events such CI failures.
Custom and tailored Docker images will help reduce dependency installation.
Validate your .gitlab-ci.yml file with GitLab’s linting tool.

Where next?

Our primary focus now is switching to package deployments instead of release branches. Other things on the roadmap are experimenting with ZAP for automated security scanning and getting performance testing into the pipeline.

Read part two of this post: Automating deployments to SilverStripe Platform with GitLab