Kubernetes Tutorial: Deploying a load-balanced Docker application

- By Ambarish Chitnis on October 27, 2017

Kubernetes is a Production grade container orchestration platform with automated scaling and management of containerized applications. It is also open source, so you can install Kubernetes on any cloud like AWS, Digital Ocean, Google Cloud Platform, or even just on your own machines on premises. Kubernetes was started at Google and is also offered as a hosted Container Service called GKE. With Shippable, you can easily hook up your automated DevOps pipeline from source control to deploy to your Kubernetes pods and accelerate innovation.

In this blog, we demonstrate how to deploy a load balanced, multi-container application to multiple Kubernetes environments on GKE. The deployment occurs in multiple stages in a Shippable defined workflow.

 

Kubernetes Deployment spec

The pods and services (load balancer) for the application are created using a deployment spec. Instead of creating and maintaining a deployment spec per environment which is a common practice, we create a single deployment spec template. This template has placeholders for the image and service/pod labels. When we deploy the application to a specific environment, we use powerful yet simple Shippable platform functions and resources to  replace these placeholders at run time when the deployment actually happens.

The deployment spec template (located here in our public repository) defines the label selectors placeholders in the .spec.selector section and the labels for the pods in the .spec.template.metadata.labels section. Labels are defined for both the front end voting application (FE_LABEL) as well as the Redis service (BE_LABEL) which the front ends makes API calls on via another load balancer.   

 

Application architecture

The application is a simple voting web app with the web-tier authored in Python / Flask. The front-end runs its own pod behind a front end load balancer.  The votes are stored in a Redis cache, which runs in another pod. The Redis container runs behind another backend load balancer. Our deplyoment workflow thus creates two load balancers and two pods, each of which can have its own set of replicas. 

 

Multi-stage deployment workflow

In our multi-stage deployment scenario, we deploy the application in several stages. 

  • Stage 1: Stage 1 runs on every commit to the application source code, builds the docker image and pushes it to GCR.
  • Stage 2: Stage 2 runs on successful completion of Stage 1. In Stage 2, we deploy the application to a test environment on the cluster using labels and create a load balancer for the application.
  • Stage 3In Stage 3, we deploy the application to a production environment on the cluster using labels and create a load balancer for the application. Stage 3 is configured to run manually.

This is a pictorial representation of the workflow we're going to configure. The green boxes are jobs and the grey boxes are the input resources for the jobs. The workflow is defined across two configuration files: shippable.jobs.yml and shippable.resources.yml.

  • build_image is a runSh job that builds the voting app image called front_img and pushes the image  to GCR. It executes Stage 1 of our workflow.
  • deploy_test is a runSh job that takes the deployment template, replaces the image and labels placeholders in the template with the actual values from the front_img and test_params resources, and deploys both front_img and redis images. It also outputs relstate which is an updated deployment spec with immutable image tags, used in Stage 3. It executes Stage 2 of our workflow.
  • deploy_prod is a runSh job that takes the deployment spec from relstate, replaces the labels needed for prod deployment using the prod_params resource, and deploys both frontend_img and redis  images to production.  It executes Stage 3 of our workflow.
  • These jobs have input and output resources that are described in detail in other sections below. 

 

Prerequisites

  • A Google Cloud account at https://cloud.google.com/
    • A Project that has Google Container Engine (GKE) and Google Container Registry (GCR) enabled.
    • An existing Kubernetes cluster on GKE where you will deploy this sample application.
  • A GitHub account where you will fork and run this sample
  • Sign in with GitHub to create a Shippable account

If you're not familiar with Shippable, it is also recommended that you read the Platform overview doc to understand the overall structure of Shippable's DevOps Assembly Lines platform.

 

Sample project

The code for this example is in GitHub at devops-recipes/ship-voting-app-redis. You can fork the repository to try out this sample yourself or just follow instructions to configure your own use case

  • The voting app source code and Dockerfile can be found here in the repository.
  • This repository also has the Shippable configuration files to setup the multi-stage pipeline. 
  • The Kubernetes deployment spec ship_vote_all_in_one_redis.yml.template used to create the pods and services can be found here in the repository.

 

1. Build application Docker image

A. Add the subscription integration for your SCM account integration

By default, you will already have an account integration with whichever SCM provider you've used to log into Shippable. If your source code repository is on that SCM account, you should use it as is. You might have to add it to your Subscription by following the steps here: Adding an account integration to a Subscription.

If your source code repository is on another SCM account, create an account integration for it first by using one of the following supported SCM docs:

 

B. Create account integration for Google Cloud.

Since your deployment config will interact with GCR to push the front-end image and GKE to deploy the application, you will need to create an integration for Google Cloud in the Shippable UI.

  • Create a Google Cloud integration called drship_gcloud using instructions found here
  • Ensure that you have set the Subscription Scopes in the account integration to the subscription where your respository is located.
     

C. Define resources used to build the application Docker image

Add the following to your shippable.resources.yml file: 

resources:

  - name: ship_voting_app_redis_gitRepo
    type: gitRepo
    # replace dr_github with your GitHub integration name
    integration: dr_github
    pointer:
      # replace with source code location (e.g. GitHub) where you cloned this
      # sample project.
      sourceName: devops-recipes/ship-voting-app-redis
      branch: master

  - name: gcloud_cliConfig
    type: cliConfig
    integration: drship_gcloud
    pointer:
      # replace us-central1-a with your availability zone
      region: us-central1-a

  - name: front_img
    type: image
    integration: drship_gcloud
    pointer:
      # replace devops-samples with your google cloud project name
      sourceName: "gcr.io/devops-samples/vote"
    seed:
      versionName: "master.1"
  • ship_voting_app_redis_gitRepo is a gitRepo resource which is a pointer to the git repository that contains the source code for this application. 
  • gke_cliConfig is a cliConfig resource which is a pointer to the private key of your GKE service account needed to initialize the gcloud CLI.
  • front_img is an image resource that represent the Docker image of the vote application. This uses the Google Cloud integration that we created earlier.

 

D. Define build_image 

Add the build_image job to your shippable.jobs.yml file.

It is a runSh job that lets you run any shell script. It takes app_gitRepo as an input, builds the voting app image and pushes the docker image to GCR.

It sets the image tag in the front_img resource and outputs that resource so that a new version of the resource can be created. This in turn triggers Stage 2 of our pipeline since front_img is an input to the Stage 2 job.

jobs:

  - name: build_image
    type: runSh
    steps:
      - IN: ship_voting_app_redis_gitRepo
      - IN: gcloud_cliConfig
      - OUT: front_img
      - TASK:
        # build the ship-vote image and push it to GCR
        - script: |
pushd $(shipctl get_resource_state "ship_voting_app_redis_gitRepo")/ship-vote export IMG_TAG=master.$BUILD_NUMBER

# Replace devops-samples with your google cloud project name export IMG_REF=gcr.io/devops-samples/vote:$IMG_TAG docker build -t $IMG_REF . gcloud docker -- push $IMG_REF # Persist the image tag in the front_img resource so that the next stage # of the workflow knows which tag to deploy shipctl put_resource_state front_img "versionName" "$IMG_TAG" popd

 

E. Commit config files and add them to your Shippable account.

Once you have these configuration files as described above, commit them to your repository. The shippable.jobs.yml and shippable.resources.yml can be committed to the same app repository, or to a separate repository.

The repository containing your jobs and resources ymls is called a Sync repository and basically represents your workflow configuration.

Follow these instructions to import your configuration files into your Shippable account.

Stage 1 is complete at this point

 

2. Deploy application to Test

A. Define resources used in the second stage

Add the following to your shippable.resources.yml file: 

resources:
 
 - name: test_params
    type: params
    version:
      params:
        FE_LABEL: "vote-front-test"
        BE_LABEL: "vote-back-test"
- name: relstate type: state - name: teststate type: state - name: kube_cluster
type: cluster
integration: drship_gcloud
pointer:
# replace devops-test-cluster with your kubernetes cluster name
sourceName: "devops-test-cluster"
# replace us-central1-a with your availability zone
region: us-central1-a
  • test_params is a params resource that specifies the labels for the load balancer service and the pods that run our images. 
  • relstate and teststate are state resources where we store shared state.
    •  relstate stores the deployment spec template that is used in Stage 3.
    • teststate stores the deployment spec that was used to create the pods and services in Stage 2 for the test environment.
  • kube_cluster is a cluster resource that points to your Kubernetes cluster on GKE. 

 

B. Define deploy_test

Add the deploy_test job to your shippable.jobs.yml file.

It is a runSh job that takes the deployment template, replaces the image and labels placeholders in the template with the actual values from the front_img and test_params resources, and deploys both front_img and redis images.

It also outputs relstate which is an updated deployment spec with immutable image tags, used in Stage 3. It executes Stage 2 of our workflow.

jobs:

  - name: deploy_test
    type: runSh
    steps:
      - IN: ship_voting_app_redis_gitRepo
        switch: off
      - IN: gcloud_cliConfig
      - IN: front_img
      - IN: test_params
      - IN: kube_cluster
      - OUT: relstate
      - OUT: teststate
      - TASK:
        # Create a Kubernetes deploy spec and service for the test environment for an existing cluster and deploy them
        - script: |
            pushd $(shipctl get_resource_state "ship_voting_app_redis_gitRepo")/kubernetes-manifests
              # Get the image tag from the front_img resource that we persisted earlier
              export FE_TAG=$(shipctl get_resource_version_name "front_img")

              # Save the image tag in the shared state resource relstate
              # so that we can also use it in the next stage for prod deployment.
              shipctl post_resource_state "relstate" "FE_TAG" $FE_TAG

              # Save the template file in the shared state resource relstate
              # so that we can also use it in the next stage for prod deployment.
              cp ./ship_vote_all_in_one_redis.yml.template ./release.template.yml
              shipctl refresh_file_to_out_path ./release.template.yml relstate

              # Replace the image and label placeholder values in our template
              cp ./ship_vote_all_in_one_redis.yml.template ./test_deploy_spec.yml
              shipctl replace ./test_deploy_spec.yml
              shipctl refresh_file_to_out_path ./test_deploy_spec.yml teststate

              # Init the kubeconfig file for the cluster since we are going to use kubectl
              gcloud container clusters get-credentials $KUBE_CLUSTER_POINTER_SOURCENAME --zone $KUBE_CLUSTER_POINTER_REGION

              # Delete previous deployments and services
              kubectl delete deployment $FE_LABEL 2>/dev/null || echo ""
              kubectl delete deployment $BE_LABEL 2>/dev/null || echo ""
              kubectl delete service $FE_LABEL 2>/dev/null || echo ""
              kubectl delete service $BE_LABEL 2>/dev/null || echo ""

              # Create the service and deployment object in the test environment
              kubectl create -o json -f ./test_deploy_spec.yml > kube_output.json
              cat kube_output.json
            popd

 

C. Commit the shippable.resources.yml and shippable.jobs.yml file to your repository. This completes Stage 2 of our workflow.

 

3. Deploy application to Prod

A. Define resources used in the third stage

Add the following to your shippable.resources.yml file: 

resources:

  - name: prodstate
    type: state

prodstate is a state resource that stores the deployment spec that was used to create the pods and services in Stage 3 for the prod environment.

 

B. Define deploy_prod

Add the deploy_prod job to your shippable.jobs.yml file.

It is a runSh job that takes the deployment template, replaces the labels placeholders in the template with the actual values from the prod_params resource. It deploys both front_img and redis images to the prod environment and also outputs prodstateIt executes Stage 3 of our workflow.

jobs:

  - name: deploy_prod
      type: runSh
      steps:
        - IN: deploy_test
          switch: off
        - IN: gke_cliConfig
        - IN: prod_params
        - IN: kube_cluster
        - IN: relstate
        - OUT: prodstate
        - TASK:
          # Create a Kubernetes deploy spec and service for the Prod environment for an existing cluster and deploy them
          - script: |
              # Extract the image tag from the previous stage of the workflow
              export FE_TAG=$(eval echo "$"$(shipctl get_resource_version_key "relstate" "FE_TAG"))

              # Get the deployment spec template from relstate and replace the
              # image and label placeholder values.
              shipctl copy_resource_file_from_state relstate release.template.yml .
              cp ./release.template.yml ./prod_deploy_spec.yml
              shipctl replace ./prod_deploy_spec.yml

              # Copy the final deployment spec to prodstate
              shipctl refresh_file_to_out_path ./prod_deploy_spec.yml prodstate

              # Init the kubeconfig file for the cluster since we are going to use kubectl
              gcloud container clusters get-credentials $KUBE_CLUSTER_POINTER_SOURCENAME --zone $KUBE_CLUSTER_POINTER_REGION

              # Delete previous deployments and services
              kubectl delete deployment $FE_LABEL 2>/dev/null || echo ""
              kubectl delete deployment $BE_LABEL 2>/dev/null || echo ""
              kubectl delete service $FE_LABEL 2>/dev/null || echo ""
              kubectl delete service $BE_LABEL 2>/dev/null || echo ""

              # Create the service and deployment object in the prod environment
              # using the prod deployment spec
              kubectl create -o json -f ./prod_deploy_spec.yml > kube_output.json
              cat kube_output.json

 

C. Commit the shippable.resources.yml and shippable.jobs.yml file to your repository. This completes Stage 2 of our workflow.

Your pipeline should now look like this in the SPOG view

pipeline.png

4. Trigger your pipeline

To test the pipeline, commit a change to the voting app. This will trigger the first two stages of the workflow automatically after which you can manually trigger the third stage of the workflow.

 

Grid view of all the jobs

Screen Shot 2017-10-25 at 11.32.36 AM.png 

 

Stage 1 screenshot

Screen Shot 2017-10-25 at 11.30.35 AM.png

 

Stage 2 screenshot

Screen Shot 2017-10-25 at 11.31.43 AM.png

 

Stage 3 screenshot

Screen Shot 2017-10-25 at 11.32.11 AM.png

 

 

Topics: Kubernetes, continuous delivery, multi-stage-ci