The Shippable Blog

Spinning up Custom Nodes Programmatically

Most customers use nodes provided by Shippable to run their CI/CD workflows. However, you also have another option called Custom Nodes, which allows you to attach your own build nodes, which can be inside your VPC and/or behind your firewall, to run your CI/CD workflows

Custom nodes provide many key advantages:

  • Security: Your build machines can be inside your VPC and/or behind your firewall, which gives you the ability to configure access, IAM, etc. We even have a way of configuring these machines so that you do not have to grant Shippable SSH access! This means your code never leaves your firewall and no external entity can access your machines.
  • Faster build times: You can leave your build machines running all the time, which eliminates the occasional 2-3 mins per build that is added when new machines are spun up on Shippable's hosted infrastructure.
  • Docker caching: If you use Docker for your build workflows like pulling Docker images from a registry or building Docker images, your build machines will already have these images and this will speed up your builds.

Using NPM Private Modules In Your CI Workflow

In addition to publicly available packages, npm also allows users to publish and manage packages in a private namespace. If you want your npm install command in the CI workflow to install your private dependencies, there are a couple of setup steps that are required.

Unless you follow these steps, your npm install command will throw an error, such as 

  npm ERR! 404 Not found : @manishas/my-private-module   

 

Step 1: Get your npm auth token

First, you need to find your npm token, which is stored in the .npmrc file in your home directory. The file will contain a line which looks like this:

  //registry.npmjs.org/:_authToken=GFYR$E8D-89DG-GH54-HJDR-TYDGOSYT785T  

Copy the value of the token, which is everything after authToken=

CI For Gitflow Or Feature Branch Workflows

Many Shippable customers create branches on their source control repository in order to work on new features.

Let's take an example of two developers, Ted and Mary,  who're going to work on Feature #42 for the FooApp application. Their feature branch workflow will look something like this:

  1. Ted creates a feature branch off the master branch. He calls the new branch feature-42
  2. Ted and Mary commit code changes to feature-42
  3. When coding for Feature #42 is complete, Mary sends a pull request to merge feature-42 into master 
  4. A dev lead reviews the pull request.and if everything looks good, merges the code into master
  5. If the merged code passes all checks, feature-42 is deleted

Fig 1: Gitflow workflow

The Atlassian team has written a more detailed description of the feature branch and Gitflow workflow which explains how to handle it from a source control provider perspective: 

Let us look into how you can configure your Continuous Integration (CI) platform to handle these scenarios. In this example, we'll use Shippable's native CI functionality. 

The Difference Between CI Pipelines and DevOps Assembly Lines

What's in a name? wrote the greatest Bard that ever lived.

He was wrong. If someone ever asked me a hypothetical question about which dead person I'd want to have a conversation with, it'll definitely be Shakespeare. Let me explain why.

When we launched Shippable Pipelines last year, we wanted to highlight the difference between plain vanilla CI and the capability to put together a deployment "pipeline" that spans orchestration across multiple environments and supports all tasks involved in software delivery like CI, infrastructure provisioning, test automation, deployments, security patching, release management, config mgmt, service discovery, etc. 

However, shortly after we launched, other CI providers launched their own interpretations of "pipelines"! For example,

How did everyone come to the same exact place so rapidly? As we found out, they didn't. Pipelines was being used as a fancy name for CI, with features that Shippable had supported for over two years, like Matrix builds for splitting tests or testing against multiple environments for example. 

We needed a way to explain why Shippable is different. This blog explains why we landed on DevOps Assembly Lines as the perfect way to describe our approach to DevOps and CI/CD. 

Why DevOps Efforts Fail

[This article was originally published on SDTimes as a contributed article.]

The main goal of DevOps is simple: ship software updates frequently, reliably, and with better quality. This goal is somewhat 'motherhood and apple pie', and every organization will agree that they want to get there. Many will tell you they've already embarked on the DevOps journey by following some commonly followed frameworks, such as CALMS

However, very few will express satisfaction with the results. After speaking to 200+ DevOps professionals at various stages of the adoption lifecycle, we found that organizations generally fall in one of 3 categories:

We were most interested in groups 2 and 3 since they were actually in the middle of their DevOps journey. When asked to better explain the challenges and roadblocks, here is what we found:

  • 68% said the lack of connectivity between the various DevOps tools in their toolchain was a very frustrating aspect.
  • 52% said a large portion of their testing was still manual, slowing them down 
  • 38% had a mix of legacy and modern applications, i.e. brownfield environments. This created complexity in terms of deployment strategies and endpoints, toolchain, etc.
  • 27% were still struggling with silo'ed teams that could not collaborate as expected
  • 23% had limited access to self-service infrastructure.
  • Other notable pain points included finding the right DevOps skillset, difficulty managing the complexity of multiple services and environments, lack of urgency/budget, and limited support from executive leadership.

Let us look at each of these challenges in greater detail.