I set up Continuous Integration (CI) test results visualization, in the previous tutorial. In this one, I'll set up code coverage visualization for the CI project.
What is Code coverage and why is it useful?
Code coverage is a measure used to describe the degree to which a source code of a program is tested using a particular test suite. In other words, it tells you the percentage of your code that is being tested. It is used as a guide to write more comprehensive tests to parts of your code.
In practice, developers use it as part of exit criteria for milestons, to write tests for untested parts of code, review developers have written good unit tests, etc.
While it is useful, one should remember not to use it as the only value to determine the quality of your code. In the words of Martin Fowler "Test coverage (also called code coverage) is a useful tool for finding untested parts of a codebase. Test coverage is of little use as a numeric statement of how good your tests are."
Before you begin
Ensure you have:
- Have signed into Shippable using your GitHub or BitBucket credentials
- Have an enabled project on Shippable platform, to run continuous integration
- Gone through the tutorial of configuring test results visualization
Fork this Node.js project, available on GitHub, and follow along.
I've used Grunt to run tests (execute mochaTest) against an Express server, which generates reports with Xunit and Istanbul (a code coverage report for node.js). I've also ensured that Istanbul is installed as part of devDependencies in the package.json file. See a snapshot of both files below
In order to setup code coverage, I have to ensure:
- Run code coverage commands as part of the CI work flow, configuring the
- Make sure the code coverage output is in Cobertura XML format.
- The code coverage output is stored in shippable/codecoverage folder.
Shown below is my
shippable.yml file where, I've:
- Added an env variable called XUNIT_FILE, and use it to pipe out the test results to the shippable/testresults folder.
- Including 'npm test' as part of my CI workflow.
- Configured code coverage in the
post_ci:section, which runs only if the
- Ensured the code coverage output is stored in the shippable/codecoverage folder.
language: node_js node_js: - 0.12 env: - XUNIT_FILE=shippable/testresults/result.xml build: ci: - npm install - npm test
- ./node_modules/.bin/istanbul cover grunt -- -u tdd
- ./node_modules/.bin/istanbul cobertura --dir shippable/codecoverage/
View Code Coverage Results
Having made these changes to the
shippable.yml file, I'll now run the
git add. ;
git commit -m "configured code coverage" ;
git push origin master ; commands to push this code change to my GitHub repository. This will trigger a build on Shippable that executes the tests and shows the code coverage results as shown below.
Few things to remember:
- The above example has one test configured and it covers the entire code for the node.js application. For complex projects with multiple tests, the code coverage percentages can vary.
- This example uses Istanbul for Node.js for the code coverage report. It includes show casing both branch & sequence coverage. Few other code coverage reports for other languages, may show only of branch or sequence coverage reports.
I'll take this a step further to configure a build badge for this project's code coverage report in the ReadMe file hosted on my GitHub repository. To do this:
- Go to the project's 'Settings' tab in Shippable portal.
- For this tutorial, the project is 'sample_nodejs'.
- Scroll down to the 'Coverage Badge' section.
- Under the 'Branch' section, select the branch you want to show code coverage
- For this tutorial, I choose the default 'Master' branch
- Choose the format for build badge - Image URL or Markdown
- I'll choose Markdown
- Copy the code and paste it in the ReadMe file.
git add. ;
git commit -m "Updated ReadMe" ;
git push origin master ; commands to push this code change to the GitHub repository. This in turn triggers a build on Shippable which executes the tests and updates the code coverage reports.
Review your ReadMe file to see the code coverage badge.
While the above example uses Markdown option to update the ReadMe file, you can get creative and embed the Image URL in any of your applications. In addition to code coverage badge, you can configure your test results as a build badge. Read this tutorial to know more about build badges and configure it.
Now that I've configured test and code coverage for my Continuous Integration project, next, I'll set up the notification of failure/success CI project status on email & Slack.