Migrating from Shippable to JFrog Pipelines

Thank you for being a Shippable customer! As you might be aware, Shippable was acquired by JFrog in February 2019. Since the acquisition, we have been hard at work integrating CI/CD functionality into the JFrog platform to provide a superior integrated experience that goes beyond just CI/CD.

As part of this effort, we will retire the Shippable service after May 3, 2021. After this date, you will no longer be able  to access your Shippable subscriptions.

We encourage you to try the much improved evolution of Shippable - JFrog Pipelines. It carries forward popular features from Shippable with a bunch of additional improvements based on customer feedback and the ability to leverage the advantage of being part of  a more complete DevOps platform.     

Shippable vs JFrog Pipelines 

Let is look at key differences between Shippable and JFrog Pipelines

1. Standalone CI/CD vs an Enterprise-grade DevOps Platform with integrated Artifactory and Xray

Shippable is a standalone CI/CD tool which integrates with a host of other providers to automate your software delivery workflows. 

With the JFrog platform, you get much more than CI/CD. You also get Artifactory, the universal binary repository manager that is used by 70% of Fortune 100 companies, and Xray, the security scanning tool which helps detect vulnerabilities and license violations in your artifacts. Both these products are seamlessly integrated with JFrog Pipelines and you can easily create automation workflows across these tools. 

That being said, JFrog Pipelines also integrates with many other tools and services, so you still have the flexibility to choose the toolchain that works best for you.

2.  Security  

Shippable authentication was tightly coupled with your source control provider -  users signed in with their SCM credentials. Permissions were managed in source control and you could grant different levels of access by setting permissions on the source control repository that contained your CI/pipeline configuration.

The JFrog Platform has several authentication mechanisms as described here: JFrog  Platform Security. Permissions are managed through the platform UI and the model allows for fine grained access control. Pipeline permissions  are managed at a pipeline source (file(s) containing pipeline definitions) level and you can set read/trigger/manage access as needed. Pattern based permissions are supported so you don't have to update when new sources are added.

3. YAML definition

Shippable had inconsistent YAML structures for CI and CD, making it difficult to seamlessly include both into a single workflow. There were also weird idiosyncrasies such as capitalizing IN and OUT tags, random usage of camelCase, "-", and "_" in config, etc. 

With JFrog Pipelines, we have standardized the YAML and combined CI and CD functionality into one format. You just define the workflow you want, without worrying whether it's CI or CD.

4. Extensibility

Shippable had embarked on the journey to unscripted pipelines with managed jobs for common actions.

With JFrog Pipelines, we have created a library of Native (i.e. pre-packaged) steps to help you create pipeline definitions without custom scripting. However, we have also gone several steps beyond with two key features that will get you much closer to the goal of having pipelines be 100% unscripted - instead, they will just be assembled using pre-packaged steps:

• Extensions: You can create your own pre-packaged steps and resources and reuse them across pipelines
• Templates:  You can create standardized pipeline templates, which can be used in combination with a values.yml to create pipeline definitions

These features provide reusable components that help you scale automation very easily without reinventing the wheel each time.

5. Pricing

Shippable subscriptions were priced based on number of build nodes, which dictated how many builds could be run in parallel.

JFrog Pipelines is included as part of a JFrog Cloud subscription. Each tier includes a quota for build minutes with overage charges for additional build minutes. As an example, the free tier includes 2000 CI/CD build minutes every month. Parallel capacity for each tier is preset and can be changed by moving to a higher tier.

Example: Migration of Shippable CI to JFrog Pipelines

Let's look at a simple example of how Shippable CI (runCI) jobs can be migrated to a pipeline in the JFrog world. For this walkthrough, we will use the standard Shippable sample - basic_node - which builds a simple node application and runs some tests.

The Shippable configuration for basic_node is here.

As discussed before, JFrog Pipelines supports CI use cases through a combination of a Bash step and GitRepo resource. We have simplified this to make CI  workflows more flexible and consistent with pipelines configuration. You can split your CI across several steps, define complex workflows with parallel and serial execution, and get richer information at each step. We also show a visualization graph that helps you detect failures at a single glance.

The Pipelines configuration for the basic_node example is as follows:

resources:
  - name: basic_node_repo        # Pointer to app source code repository.    
    type: GitRepo
    configuration:
      gitProvider: manishas_github        
      path: devops-recipes/basic-node

pipelines:
  - name: basic_node_pipeline
    steps:
      - name: build
        type: Bash
        configuration:
          inputResources:
            - name: basic_node_repo        # Trigger this step automatically on commits to this repo
        execution:
          onExecute:
            - cd $res_basic_node_repo_resourcePath         # Env variable pointing to location where git repository contents are cloned
            - npm install
            - $res_basic_node_repo_resourcePath/node_modules/.bin/mocha --recursive "tests/**/*.spec.js" -R mocha-junit-reporter --reporter-options mochaFile=testresults.xml
            - save_tests testresults.xml        # Utility function to save tests and show results in the UI

The building blocks of this config are:

• resources: Resources are entities that hold information that is consumed or created by a pipeline step. The GitRepo resource is a pointer to the source control  repository where your application code is stored. We automatically create webhooks based on your config and each time the repository has a commit/PR/tag/release, dependent steps are triggered and contents are cloned to the build node where the step executes.
  • gitProvider points to the integration which has credentials for your source control provider
  • path points to the source control repository where your application code is stored.
  For a complete  reference to GitRepo config click here: GitRepo.
• Pipeline definition:  A pipeline is a collection of interconnected steps that defines your workflow logic. You can configure a set of properties at a pipeline level, and every step in the pipeline will use this config unless it is overridden in the step definition.  Pipeline names need to be unique to avoid  conflicts.
• Step definition: The Bash step can be custom scripted and is very versatile. This is the basic building block of JFrog Pipelines and supports advanced configuration to help you control your CI/CD workflow . We are using the following  config in this example:
  • inputResources: Any resources included in this section will trigger the build step. In our case, we use the GitRepo resource pointing to the app code repository. 

  • execution: The execution section includes the commands you want to run as part of this step. We support several subsections onStart, onExecute, onSuccess, onFailure, and onComplete. In our case, we are using the main onExecute section to include CI commands.

    • The first command  cd $res_basic_node_repo_resourcePath changes the working directory to where the source control repository contents are cloned.  A complete list of env variables for the GitRepo resource is available here.

    • The second command is an npm install, which installs dependencies from package.json.
    • The third command $res_basic_node_repo_resourcePath/node_modules/.bin/mocha --recursive "tests/**/*.spec.js" -R mocha-junit-reporter --reporter-options mochaFile=testresults.xml runs a tests using mocha test framework and outputs results to testresults.xml.  
    • The fourth command save_tests testresults.xml calls a utility function which parses and uploads the test report so that it is displayed in the Pipelines UI. A complete list of utility functions available to use during step execution is here.

Now that we have the pipeline definition nailed down, we can run this sample on JFrog Pipelines. To do so, follow the steps below:

1. Sign up for a free cloud subscription here: https://jfrog.com/artifactory/start-free/#saas This includes 2000 CI/CD mins per month, which should be enough to get you started.
2. Once your subscription is activated, sign in to the JFrog Platform using the link provided in the activation email.
3. Fork the basic_node repository.
4. In the left navigation bar, expand the Pipelines menu and navigate to Integrations. This is a central dashboard where you can create secrets that will be securely stored in Vault and can be used across your pipelines. For our example, we will click on Add an integration to create an integration with your source control provider.
5. Replace the value for gitProvider in your pipeline configuration with the name of your integration, and the path value with the path to your forked basic_node repo.
6. Next, we will add add the pipeline definition to the platform to create a pipeline. In the left navbar, click on Pipeline Sources. To add your pipeline definition, follow instructions here. 
   • Choose Add Pipeline Source ->  From YAML
   • SCM provider integration is the integration you created in step 4
   • Repository Full Name and Branch: Choose your forked repository, then choose the branch (master)
   • Pipeline config file filter: pipelines.yml is at the root  of your forked repo, so the default is fine
7. Your pipeline should be visible on the My Pipelines page.
8. You can trigger the pipeline manually through the UI, or commit to your basic_node repo to trigger automatically.

And that's it! You have successfully created a pipeline.  Example of a successful run is shown below:

Mapping of Shippable to JFrog Pipelines concepts

The following table provides a mapping of Shippable to JFrog Pipelines concepts + Documentation.

Next Steps

The Shippable service will no longer be available after May 3, 2021. We encourage you to try JFrog Pipelines by signing up for the free tier and determine which subscription plan works best for your CI/CD needs.   

We will host a Webinar on February 11, 2021 that will include a demo of JFrog Pipelines, as well as dig in deeper into Artifactory and Xray,

Once again, thank you for being a Shippable customer and your patience for the last couple of years.

Questions? Feel free to contact us and ask away.