How to build, deploy and test applications with Shippable pipelines.

- By Chetan Tarale on March 22, 2017

When appropriate tools & technology are available in abundance, automation is considered the primary resource to improve the efficiency of any software development process. Faster innovation cycles and speed to market are crucial to a business. With the very same principle in mind Shippable was founded so that software powered organizations could accelerate innovation. Shippable provides an integrated platform, using which you can automate your software delivery from source control to production, without needing to write complex, imperative code.

In this tutorial we will learn how to create an end to end deployment for a single service, with source code maintained in a repository on github. We will also see how to trigger tests every time the deployed environment is updated or your test cases are updated.


Before you begin

Here are the links for the repositories used in this tutorial.

All our tasks, task definitions and their dependencies are defined as resources and jobs. For this setup we will use a separate repository and define resources and jobs there. This repository will act as syncRepo.On adding a syncRepo, Shippable will read the jobs and resources ymls and create your pipeline.  

We will need the following resources to successfully set up a pipeline

  • syncRepo                                              # to define our jobs and images
  • gitRepo                                                  # this is where our app code resides
  • An Integration of type hub                     # to push the app image
  • An Integration of type deploy                 # to deploy our app

Before we begin with our code, we will create integrations through shippable UI. We will create a dockerhub integration where we will push our image. We will also create a docker data center integration and this is where we will deploy our app. You can push and deploy your image to many other integrations that are part of the platform.



  •  Create a dockerhub integration from the UI.

  •   Create a docker data center integration from the UI.

Defining resources and jobs

Next define your resources and jobs in shippable.resources.yml and respectively. Here is how we will define resources and jobs  for our pipeline.

# Pointer to a repository on source control
 - name: sampleapp_repo
   type: gitRepo
   integration: github
     sourceName: shipppableSamples/build-test-deploy-with-pipelines
     branch: master

# Points to docker registry where we will push our image  
- name: dockerhub
   type: integration
   integration: "dockerhub"

# Points to the image that we will push
 - name: app-img
   type: image
   integration: dockerhub
     sourceName: "library/sampleapp"
     versionName: latest

# A cluster to deploy our app
 - name: ddc-cluster
   type: cluster
   integration: ddc-cluster

# docker options that can be appended to a docker image to be pushed
 - name: options                  
   type: dockerOptions        
       - "51000:51000"


We will define the following 3 jobs in our jobs yml

  • A runSh job that builds our app image
  • A manifest job to generate a manifest
  • A deploy job that deploys our app
 - name: build-app
   type: runSh
     - IN: sampleapp_repo
     - IN: dockerhub
     - OUT: app-img
     - TASK:
       - script: ./IN/sampleapp_repo/gitRepo/
     - script: echo 'on success !!!!!'
     - script: echo 'Failed job .... :('

 - name: sampleapp-man
   type: manifest
     - IN: options
     - IN: app-img
       force: true
       versionName: latest

 - name: sampleapp-deploy
   type: deploy
     - IN: sampleapp-man
     - IN: ddc-cluster

The first job is build-app, this job will take the resource sampleapp_repo as input and dockerhub integration. The git repository will be cloned into the build machine and the dockerhub credentials are also available in the build machine. We have defined a task section which will run the script which the present at the specified location Shippable clones the repository at the location ./IN/<RESOURCE_NAME>/gitRepo/. So is the script present on the source control. This job builds a docker image and pushes it to dockerhub. app-img is the output of this job.

Manifest job is triggered when the build-app job runs successfully. The image is input to the manifest job and Manifest jobs generate a new version of the manifest each time anything in the manifest changes. Every time a new image is built a new manifest gets generated or when you change the docker Options which is also an IN to this job.

When a manifest a generated successfully the deploy rob is triggered and the app is deployed to the cluster which is an IN to the deploy job. The ddc-cluster in this case which is defined in shippable.resources.yml.

Adding a syncRepo

Now push these files and add this repo as syncRepo from Shippable UI.Shippable will read the jobs and resources ymls and create your pipeline.





Running Jobs

On triggering the build job from the UI, the file will run and a docker image is built and pushed to docker hub. The instructions to build and push the image are in the file. Here is how your pipeline will look when all the jobs run successfully. At this point out app is deployed to Docker data center.  On visiting the url for the we can see that our app has been deployed.






Running tests in pipelines

Now that our app is deployed we will hook in another job into the pipelines that will test our app. All the tests are written and pushed to another repo called bat(build acceptance test). We will define another job called bat which is of type runCI. runCI jobs allow us  to use CI jobs that will do acceptance tests in the pipelines.We will first go and enable this project on Shippable.

Then define this job in our yml.

  - name: bat_runCI
   type: runCI
     - IN: bat_params
     - IN: sampleapp-deploy


This job takes sampleapp-deploy job as input and another resource called bat_params. bat_params are some parameters that you want in your runCI job. Tokens can be encrypted and passed as params from here. We will define bat params in resources yml as follows.


 - name: bat_params
   type: params
       url: ""
       port: 51000


If you are already using CI to run tests you can simply hook in our already existing CI job to your pipeline. Also another reason to use CI jobs to run tests is that the build images comes with preinstalled services which can be used instead of setting up yourself. Now on pushing these changes the sync job will run and the runCI job is hooked into the pipeline. Now every time you deploy some changes to your app or commit some changes to your repo where you have defined your test(the repo bat in this case). The bat_runCI job is triggered and your changes are tested on your app.




Topics: continuous integration (CI), devops, Docker Hub, continuous delivery, Docker Datacenter