Announcing CI/CD pipelines with Amazon EC2 Container Service and Amazon EC2 Container Registry

- By Tom Trahan on October 08, 2015

This blog is deprecated

DEPRECATED BLOG:

We have significantly updated the Shippable platform with several new features. Hence most of the content in this blog is deprecated. Checkout a similar blog of deploying your first Continuous Deployment pipeline using Docker Hub and Amazon EC2 Container Service. For the latest information, refer to our documentation and/or open a support issue, if you have questions.


Today's a big day!  We're thrilled to announce the integration of our Shippable CI/CD platform with Amazon EC2 Container Service (ECS) and the coming integration with Amazon EC2 Container Registry (ECR). With this integration, Shippable customers can easily deploy the Docker images generated from their Shippable CI processes into Amazon's services for managing and running containers.

As announced at Amazon re:Invent 2015 earlier today, AWS is all-in on containers. Amazon ECS is a container service that allows customers to run Docker applications on a managed cluster of EC2 instances, while Amazon ECR (coming later this year) makes it easy for developers to store, manage, and deploy their Docker container images. Shippable makes it easy to seamlessly integrate these services into your software delivery pipeline.

In this first beta release, you can create and run Amazon ECS services and tasks on your ECS clusters from within the Shippable Formations module. Shippable will automatically update your ECS task definitions with the latest image information built by Shippable CI and either automatically deploy or deploy with a single-click. In upcoming releases, soon to be released, you'll be able to fully configure, provision and deploy your entire container-based application environment on AWS from your Shippable pipeline including load balancing, cluster instances, and clusters, as well as push and pull images from Amazon EC2 Container Registry repos.

Let's take a look at how this is enabled.

Sample CI/CD pipeline powered by Shippable and Amazon Web Services

 

You can enable the above workflow with the following steps:

     OPTIONAL: Fork the sample application code
  1. Set up your ECS cluster and container instance in AWS
  2. Set up AWS Integration in Shippable Account Settings
  3. Create a Shippable Environment for your Amazon ECS cluster
  4. Set up deployment config in Shippable Formations 
  5. Execute a build in Shippable CI, push image
  6. Trigger deployment (auto-deploy or manual deploy)

OPTIONAL: Fork the sample application code

  • If you'd like to try this out yourself, you can do so by forking the sample application code used for the walkthrough below and enabling these repos as projects in Shippable CI. This walkthrough uses a sample two-tier NodeJS application from this GitHub repository:

    micro-www (sample app front-end) and micro-api (sample app back-end): https://github.com/shippableSamples/micro-sample

    For more detailed instructions on how to do this, please see pages 6-14 in this white paper.

Step 1: Set up your ECS cluster and container instance in AWS

  • Amazon has great instructions on how to create your Amazon ECS cluster and the related container instances.  Use these instructions to create an Amazon cluster and one container instance that's registered to that cluster.
  • When complete, you should see your cluster and container instance available in your Amazon ECS console

    Amazon EC2 Container Service Cluster

Step 2: Set up AWS Integration in Shippable Account Settings

  • Sign in to Shippable at shippable.com
  • Click the ‘Settings’ icon in the top-nav (upper right) and choose the 'Integrations' tab

Account Settings for AWS integration

  • Click 'Add Integration' icon on the right and select "AWS" from the ‘Master Integration’ drop-down list. Enter a name for the integration (e.g. 'AWS) The AWS integration will contain your AWS credential information to access the ECS instance. Enter your access and secret keys provided by AWS. See here for info on how to generate them.
Adding AWS integration

 

 Step 3: Create a Shippable Environment

  • From the Shippable Home screen, select your 'Subscription Account', Go to 'Pipelines' tab and choose 'Add Environment' in the 'Environments' section
  • Adding a new Environment
    • Provide a name for your Environment 'AWS-ENV'
    • Using the 'Select a Deployment Integration' dropdown, select AWS (created from Step 2)
    • Click 'Confirm'
    • In the 'Select a Cluster' drop down, select the cluster configured in Step 1 and click 'Confirm'
    • Click 'Done'

Creating an environment for Amazon ECS cluster

Step 4: Set up Pipelines

Having created the environment with the AWS integration & ECS cluster, next let us create a Pipeline. By default, you have one free pipeline that we'll use to deploy the application to ECS cluster.

 

  • From the Shippable Home screen, select your 'Subscription Account', Go to 'Pipelines' tab and choose 'Add Pipeline' in the 'Pipelines' section
  • Provide a name for your Environment 'AWS-Pipeline'
  • In the 'Cell Manifest' section, click on 'Add Image'
    1. Click on the 'Select an Image' drop down
    2. Click 'Create an image'
    3. Give the image name '<docker_registry_username>/micro-api'. For this tutorial, I'll go with Docker Hub.
    4. Under 'Hub Integration' section, select your Docker Integration.
      • If you don't have one, click on 'Create an Integration' and follow the prompts to create a Docker Integration providing your Docker Hub credentials
        Adding a Docker Image
    5. Click 'Create Image'
    6. In the 'Image Tag' section, select the tag you want to deploy, 'master.13' in this case
    7. 'Add Port' for your container. Give the value '3001'
    8. Leave all other fields as 'Default' values & click 'Save Image'
  • Back in the 'Add a Pipeline' page, repeat the above steps (a - h) for the image micro-www
  • For the micro-www port number, enter 80 and click 'Save Image'
  • Next, let's add Environment Variables. Click 'Add Variable' and add the following variables - API_PORT, API_URL, WWW_PORT, LOG_LEVEL and SHUD_LOG_TO_FILE.
    Adding environment variables
  • In the 'Auto Increment' section, check the 'Auto Increment' box and specify 'master.*' as the values for both 'micro-api' & 'micro-www' images. By doing this, new cell manifest versions are automatically created when new tags are detected for images in your pipeline.
  • In the 'Pipeline Triggers' section, select the enabled project 'micro-sample'
  • Click 'Save' to create the Pipeline

Step 5: Configure and deploy the cell

You should see the new pipeline as shown below:AWS Pipeline prior to deployment

Next, let's configure and deploy the cell.

  • Click the 'AWS-ENV' under the 'Environments' section and go to the 'Configuration' tab
  • Check the 'Auto Deploy' option under the 'Automatic Deployment' for this tutorial
  • Under the 'Environment Configs' section, enter the following values for the environment variables
    Env_variables.png
  • Leave number of Replicas as 1. This indicates that only one instance of this cell runs in the environment
  • Skip the 'Volumes' section for this tutorial
  • Next, click the 'Routing' tab
  • Click the 'Select Load Balancer' drop down under the 'Load Balancer' section
  • Select the load balancer created in Step 1
  • Enter '80' for 'Port HTTP' and '' for 'Path'

 

 

 

 

 



as well as running in your ECS cluster.


Repeat these deployment steps for the 'api' service and you'll be ready for auto-deployments.

 

Step 5: Execute a build in Shippable CI, push image

  • Initiate a Shippable CI build for both the radar-api and radar-ww projects.  You'll see the CI build processes complete and push new Docker images to Docker Hub.  Here's what the CI console looked like for my successful CI build of the radar-www project:



  • This successfully completed the CI build and pushed the radar-api/master.58 image to Docker Hub.


Step 6: Trigger a deployment

  • Nothing to do here!  Since you set up auto-deploy in Step 4, your newly generated image (radar-api/master.58) has already triggered a deployment to ECS.
  • Navigate to your ECS cluster in the AWS console to validate this

    The new Task Definition was automatically created in ECS for the new master.58 image:


    And it was automatically deployed as a new Task in ECS:


    ...and ECS gracefully shut down the previously running task before successfully bring the new one to RUNNING status:

     

As you can see, with Shippable, you can quickly and easily configure continuous delivery of your software changes into Amazon ECS.

Check back soon for more announcements regarding upcoming releases on Shippable's AWS integration!

  • Automated provisioning and configuration of clusters, container instances and load balancers, including creation and management of security groups, roles, and networking
  • One-click scale-out of your Amazon EC2 Container Service environment
  • Push and pull from Amazon EC2 Container Registry

 

 

 

Topics: continuous deployment (CD), continuous delivery, EC2 container services, Amazon ECS