Setting up code coverage for tests in Continuous Integration

- By Abhijit Kini on May 26, 2016

I set up Continuous Integration (CI) test results visualization, in the previous tutorial. In this one, I'll set up code coverage visualization for the CI project.

What is Code coverage and why is it useful?

Code coverage is a measure used to describe the degree to which a source code of a program is tested using a particular test suite. In other words, it tells you the percentage of your code that is being tested. It is used as a guide to write more comprehensive tests to parts of your code. 

In practice, developers use it as part of exit criteria for milestons, to write tests for untested parts of  code, review developers have written good unit tests, etc.

While it is useful, one should remember not to use it as the only value to determine the quality of your code. In the words of Martin Fowler "Test coverage (also called code coverage) is a useful tool for finding untested parts of a codebase. Test coverage is of little use as a numeric statement of how good your tests are."

Before you begin

Ensure you have:

Fork this Node.js project, available on GitHub, and follow along.

Configure test

I've used Grunt to run tests (execute mochaTest) against an Express server, which generates reports with Xunit and Istanbul (a code coverage report for node.js). I've also ensured that Istanbul is installed as part of devDependencies in the package.json file. See a snapshot of both files below

Grunt file for continuous integration code coverage

package.json nodejs file for continuous integration code coverage

Configure the shippable.ymlfile

In order to setup code coverage, I have to ensure:

  1. Run code coverage commands as part of the CI work flow, configuring the shippable.yml file.
  2. Make sure the code coverage output is in Cobertura XML format.
  3. The code coverage output is stored in shippable/codecoverage folder.

Shown below is my shippable.yml file where, I've:

  • Added an env variable called XUNIT_FILE, and use it to pipe out the test results to the shippable/testresults folder.
  • Including 'npm test' as part of my CI workflow.
  • Configured code coverage in the post_ci: section, which runs only if the ci section succeeds.
  • Ensured the code coverage output is stored in the shippable/codecoverage folder.
language: node_js

  - 0.12

  - XUNIT_FILE=shippable/testresults/result.xml

    - npm install
    - npm test

- ./node_modules/.bin/istanbul cover grunt -- -u tdd
- ./node_modules/.bin/istanbul cobertura --dir shippable/codecoverage/

View Code Coverage Results

Having made these changes to the shippable.yml file, I'll now run the git add. ; git commit -m "configured code coverage" ; git push origin master ; commands to push this code change to my GitHub repository. This will trigger a build on Shippable that executes the tests and shows the code coverage results as shown below.

Successful continuous integration code coverage result

Few things to remember:

  • The above example has one test configured and it covers the entire code for the node.js application. For complex projects with multiple tests, the code coverage percentages can vary.
  • This example uses Istanbul for Node.js for the code coverage report. It includes show casing both branch & sequence coverage. Few other code coverage reports for other languages, may show only of branch or sequence coverage reports.


Bonus visualization

I'll take this a step further to configure a build badge for this project's code coverage report in the ReadMe file hosted on my GitHub repository. To do this:

  • Go to the project's 'Settings' tab in Shippable portal.
    • For this tutorial, the project is 'sample_nodejs'.
  • Scroll down to the 'Coverage Badge' section.
  • Under the 'Branch' section, select the branch you want to show code coverage
    • For this tutorial, I choose the default 'Master' branch
  • Choose the format for build badge - Image URL or Markdown
    • I'll choose Markdown
  • Copy the code and paste it in the ReadMe file.

Showing the code coverage result using Markdown


Run the git add. ; git commit -m "Updated ReadMe" ; git push origin master ; commands to push this code change to the GitHub repository. This in turn triggers a build on Shippable which executes the tests and updates the code coverage reports. 

Review your ReadMe file to see the code coverage badge.

Code coverage result displayed using markdown



While the above example uses Markdown option to update the ReadMe file, you can get creative and embed the Image URL in any of your applications. In addition to code coverage badge, you can configure your test results as a build badge. Read this tutorial to know more about build badges and configure it.

Now that I've configured test and code coverage for my Continuous Integration project, next, I'll set up the notification of failure/success CI project status on email & Slack.

Try Shippable

Topics: continuous integration (CI), nodejs, tutorial, tips, code coverage, unit test