CI/CD For Microservices Using Monorepos

- By Ambarish Chitnis on October 12, 2017

We wrote a very popular blog a little over a year ago, detailing the reasone behind our choice of organizing our microservices codebase in a single repository called a mono repo. 

Since then, we've often been asked - how do you set up a CI/CD pipeline for a mono repo? When a code change to the repository triggers CI, how does your CI know which microservice changed so that it can rebuild and test just that service?

In this blog, we will demonstrate how the Shippable platform makes it simple to independently build, test and deploy microservices from a mono repo. For simplicity,  we will use a monorepo sample (that you can fork) that consists of just two microservices, and create a CI/CD pipeline with Amazon ECR and ECS. 

Scenario

  • Our Node.js application has two microservices: a front-end microservice www that makes API calls to a backend API microservice called api. The source code for both microservices is in separate folders in a mono repo.

  • Each microservice is packaged as a Docker image during the build process and has its own independent unit tests.

  • Each Docker image is pushed to its own Amazon ECR repository. Both images get deployed to a common Amazon ECS cluster.   

  • Both these microservice share some common code that is maintained in a separate folder in the mono repo.

  • A commit to a microservice builds, tests and deploys that specific microservice.

  • A commit to the common code builds, tests and deploys both microservices.

 

Shippable Workflow

This is a pictorial representation of the workflow we're going to configure to implement our use case. The green boxes are jobs and the grey boxes are the input resources for the jobs. The workflow is defined across 3 configuration files: shippable.yml, shippable.jobs.yml and shippable.resources.yml

 

monorepo-2.png

Resources (grey boxes)

  • www_img and api_img are image resources that represents the docker images of the www and api microservices respectively.
  • ecs_cluster is a cluster resource that represents the orchestration platform (ECS) where the microservices aare deployed to.
Jobs (green boxes)
  • app_mono_runCI is the CI job that tests and builds a docker image for the www and api microservices and.pushes the docker images to dedicated ECR repostiroies. This job is authored in the CI stage of the workflow in a CI config file shippable.yml.
  • micro_api_def and micro_www_def are jobs that build the service definition of each micro services that translate to tasks in ECS.
  • app_deploy_job is a job that automatically deploys the service definitions to an ECS cluster.

Our workflow thus comprises of 3 distinct connected stages:

  • Stage 1Build, test, package and push microservices to ECR
  • Stage 2: Microservice service definition
  • Stage 3: Microservice deployment

 

Sample project

The code for this monorepo example can be found in our public GitHub repositoryThis repository has all the Shippable configuration files to setup the entire workflow. You can fork the repository to try out this sample yourself or just follow the instructions below to configure your own use case. 

 

Stage 1: Build, test, package and push microservices to ECR

 

1. Create an AWS Keys Integration

  • This integration is used by the app_mono_runCI job a for pushing the docker image to ECR.
  • Steps to create this integration are documented here.
  • The name for this integration should be dr-aws-keys. If you change the integration name, please also change it in your shippable.resources.yml file.

2. Create app_mono_runCI

app_mono_runCI is a runCI job that represents your CI workflow. This  job is automatically created after you enable Shippable CI on your monorepo in your shippable account by following the steps below:

  • Sign in with your GitHub or Bitbucket credentials.
  • Create the CI config file shippable.yml and commit to the root of your repository. The next section will go into the details of how to author shippable.yml file. If you're using a fork of our sample repository, you do not need to do this step since the repo already contains the config file. 
  • Enable your repository so that we can set up webhooks on your behalf. 

 

 3. Author CI config file shippable.yml

 

# Language setting
language: node_js

# Version number
node_js:
- 8.2.1

env:
global:
- XUNIT_FILE=$SHIPPABLE_BUILD_DIR/shippable/testresults/result.xml API_PORT=80

build:
ci:
# create directories needed for unit tests and code coverage
- mkdir -p $SHIPPABLE_BUILD_DIR/shippable/testresults
- mkdir -p $SHIPPABLE_BUILD_DIR/shippable/codecoverage
# run a script that detects if any microservice code has changed and tests,
# builds and publishes the docker image to ECR
- |
if [ "$IS_PULL_REQUEST" != true ]; then
./detect-changed-services.sh
else
echo "skipping because it's a PR"
fi

# ECR subscription integration that automatically does a docker login into ECR
# using credentials specified in the account integration
integrations:
hub:
- integrationName: "dr-aws-keys"
type: ecr
region: us-east-1
branches:
only:
- master 

 

The overall flow for testing and building the microservices is as follows:

  • detect-changed-services.sh detects the changed folders and copies the common code folder into the target microservice folder.
  • The common code folder also has a common packaging script package-service.sh that tests, builds and pushes the docker image for the microservice to the ECR repository.
  • detect-changed-services.sh invokes the packaging script at the root folder of each microservice.
  • This organization allows a Dockerfile to be defined at the root folder of each microservice. If you want to avoid copying your common code to every microservice folder, simply place your Dockerfiles at the root of your directory and tweak the packaging script.

#!/bin/bash -e

detect_changed_services() {
echo "----------------------------------------------"
echo "detecting changed folders for this commit"

# get a list of all the changed folders only
changed_folders=`git diff --name-only $SHIPPABLE_COMMIT_RANGE | grep / | awk 'BEGIN {FS="/"} {print $1}' | uniq`
echo "changed folders "$changed_folders

changed_services=()
for folder in $changed_folders
do
if [ "$folder" == '_global' ]; then
echo "common folder changed, building and publishing all microservices"
changed_services=`find . -maxdepth 1 -type d -not -name '_global' -not -name 'shippable' -not -name '.git' -not -path '.' | sed 's|./||'`
echo "list of microservice "$changed_services
break
else
echo "Adding $folder to list of services to build"
changed_services+=("$folder")
fi
done

# Iterate on each service and run the packaging script
for service in $changed_services
do
echo "-------------------Running packaging for $service---------------------"
# copy the common code to the service so that it can be packaged in the docker image
cp -r ./_global $service
pushd "$service"
# move the build script to the root of the service
mv ./_global/package-service.sh ./.
./package-service.sh "$service"
popd
done
}

detect_changed_services

 

Stage 2Microservice definition

 

1. Define image respositories and seed versions of images .

Add the api_img and www_img resources to your shippable.resource.yml file

resources:

  - name: api_img
    type: image
    integration: dr-aws-keys
    pointer:
      sourceName: "679404489841.dkr.ecr.us-east-1.amazonaws.com/api"
    seed:
      versionName: "master.1"

  - name: www_img
    type: image
    integration: dr-aws-keys
    pointer:
      sourceName: "679404489841.dkr.ecr.us-east-1.amazonaws.com/www"
    seed:
      versionName: "master.1"

 api_img and www_img are image resources that represent the Docker images of the api and www microservices. These use the AWS Keys integration that we created in Step 1. The location and seed version of the ECR repository are specified in the pointer attribute.


 2. Define microservices

Add app-mono_runCI, micro_api_def and micro_www_def jobs to your shippable.jobs.yml file.

jobs:

  - name: app-mono_runCI
    type: runCI
    steps:
       - OUT: api_img
       - OUT: www_img

  - name: micro_api_def
    type: manifest
    steps:
     - IN: api_img

  - name: micro_www_def
    type: manifest
    steps:
     - IN: www_img

app-mono_runCI which represents the CI workflow, specifies OUT on each image resource. This is needed to trigger the manifest jobs downstream whenever a new docker image is built for the microservices.

micro_api_def and micro_www_def are manifest jobs used to create a service definitions for the microservices. The service definition consists of the image and can optionally include docker options as well as environment variables. The definition is also versioned (any change to the inputs of the manifest creates a new semantic version of the manifest) and is immutable.

 

Stage 3: Deploying Microservices to ECS

1. Create An AWS IAM Integration

  • This integration is used by the app_deploy_job for deploying the ECR image to the ECS cluster.
  • Steps to create this integration are documented here.
  • The name for this integration should be dr-aws. If you change the integration name, please also change it in your shippable.resources.yml file.

 

2. Define ECS cluster .

Add ecs_cluster resource to your shippable.resource.yml file

resources:

  - name: ecs_cluster
    type: cluster
    integration: dr-aws
    pointer:
      sourceName : "mono-repo" #name of the cluster to which we are deploying
      region: "us-east-1"

 ecs_cluster is a cluster resource that represents the cluster in your orchestration platform where your application is deployed to. In our example, the cluster points to a cluster on Amazon ECS. The name and region of the cluster are specified in the pointer attribute.

 

3. deploy to ECS

Add app_deploy job to your shippable.jobs.yml file.

jobs:
- name: app_deploy type: deploy method: replace steps: - IN: micro_api_def - IN: micro_www_def - IN: ecs_cluster
app_deploy is a deploy job that actually deploys the micro services to the cluster and starts the container instances on ECS.

 

Import Config into your Shippable account

Once you have defined the shippable resources and jobs  configuration files as described above, commit them to your repository. The repository containing your jobs and resources ymls is called a Sync repository and basically represents your workflow configuration.

Follow these instructions to import your configuration files into your Shippable account.

Your pipeline in the SPOG view should look like this:

Screen Shot 2017-10-11 at 10.20.57 PM.png

Triggering and testing the pipeline

Now we come to the most interesting part of this blog. 

A. Test a commit to a single monorepo, the WWW service.

The screeshots below show the CI job testing and building the image only the www microservice.

ci-job-queued.png

api-unit-tests.png

api-build-docker-image.png

 

The deploy job then deploys on the www microservice.

deploy-api-service.png

B. Test a commit to non-microservice code.

You will see CI not building anything.

Screen Shot 2017-10-12 at 6.17.21 PM.png

 

C. Test a commit to the common code folder _global.

You will see both the microservices getting pushed to Amazon ECR and deployed to Amazon ECS.

Try it for yourself!

Sign in for FREE to try out this mono repo approach today: 

Try Shippable