Build, test and deploy applications independently from a monorepo

- By Chetan Tarale

BACK TO BLOG HOME

In our previous blog posts, Our journey to microservices: mono repo vs multiple repositories, we shared our thoughts and experiences on our approach with monorepo. We received a few questions after that blog on how CI and deploys go with the monorepo.

In this article we will learn how to run CI, build and deploy applications independently from a monorepo. On each PR/commit we will run tests on the service which has changed build a docker image from it and push it to a registry. This image can then be deployed to to a cluster on any supported Container ServiceWe will use Shippable for this scenario.

 

Setup

Our monorepo has two projects. Both of them are simple node js apps, www is a front-end app and will run selenium tests, api is the back-end app which we will test using superagent and chai. The initial setup can be a bit tricky and depends on how projects in your repository are structured. Detecting what has changed in the repository is important.  We will call a shell script from shippable.yml which will detect what has changed and trigger tests. When all our test pass successfully we will build a docker image and then push it to the registry.

shippable.yml

language: node_js

services:
- selenium

build:
ci:
- ./detectChangedFolders.sh

This yml will run the below shell script which will detect changes folders(i.e services in our repository and run tests). Also note that we are specifying all the services required in shippable.yml. 

detectChangedFolders.sh

#!/bin/bash -e
export IGNORE_FILES=$(ls -p | grep -v /)

detect_changed_folders() {
echo "detecting changes for this build"
folders=`git diff --name-only $SHIPPABLE_COMMIT_RANGE | sort -u | awk 'BEGIN {FS="/"} {print $1}' | uniq`
export changed_components=$folders
}

run_tests() {
for component in $changed_components
do
echo "----------------------------------------------"
echo "$component has changed"
if ! [[ " ${IGNORE_FILES[@]} " =~ "$component" ]]; then
echo "-------------------Running tests for $component---------------------"
cd "$component"
./run.sh "$component"
cd ..
else
echo "Ignoring this change"
fi
done
}

if [ "$IS_PULL_REQUEST" != true ]; then
detect_changed_folders
run_tests
else
echo "skipping because it's a PR"
fi

 

Step 1. Enable the project on shippable.

Now on navigating to the pipelines tab you can view your project on SPOG.

Selection_006.png 

On every build triggered the changed folders are detected and a image is built and pushed to the registry. In our example the run.sh file that we have in every folder is responsible for triggering tests, building an image and pushing it to the registry.

Step 2.  Now that our images are build we will use pipelines to deploy these images. We will begin our resources and jobs in shippable.resources.yml and shippable.jobs.yml respectively.

Here is how we will define resources and jobs for our pipeline.

resources:
- name: api-img
type: image
integration: dockerhub
pointer:
sourceName: "shippableSamples/sampleapp"
seed:
versionName: master.3

- name: www-img
type: image
integration: dockerhub
pointer:
sourceName: "shippableSamples/sampleapp"
seed:
versionName: master.3

- name: ddc-cluster
type: cluster
integration: ddc-cluster

- name: www-options #required
type: dockerOptions #required
version:
portMappings: #optional
- "51000:51000"

- name: api-options #required
type: dockerOptions #required
version:
portMappings: #optional
- "50000:50000"

- name: dockerhub
type: integration
integration: "dockerhub"

www-img and api-img are the two images that will be built when the ci job runs. These images are the OUT's from the ci job.

dockerhub is an resource of type  integration and this points to the regisrty where we will push our image. ddc-cluster is an integration of type cluster and this is where we will deploy our app. www-options and api-options are the options that can be appended to a docker image to be pushed.

We will define the following jobs in our jobs yml

  • A runCI job that builds our  images
  • Two manifest jobs to generate a manifests
  • Two deploy jobs that deploys our app
jobs:
- name: monorepo_sample_runCI
type: runCI
steps:
- OUT: api-img
- OUT: www-img
on_success:
- script: echo 'on success !!!!!'
on_failure:
- script: echo 'Failed job .... :('

- name: www-man
type: manifest
steps:
- IN: www-options
- IN: www-img
force: true

- name: www-deploy
type: deploy
steps:
- IN: www-man
- IN: ddc-cluster

- name: api-man
type: manifest
steps:
- IN: api-options
- IN: api-img
force: true

- name: api-deploy
type: deploy
steps:
- IN: api-man
- IN: ddc-cluster

 Now push these files and add this repo as syncRepo from Shippable UI.Shippable will read the jobs and resources ymls and create your pipeline.

Selection_007.png

Selection_010.png

We have successfully set up our pipeline. For any changes to a project within in the monorepo, CI for that corresponding project is triggered, a docker image is built and deployed.

Links to repositories used in this tutorial

Topics: Docker, devops, continuous delivery, microservices