NEW: Debugging with SSH (And Why We Don't Like It)

- By Manisha Sahasrabudhe on July 27, 2017

I am very happy to announce that we've launched a frequently requested (and internally controversial) feature: the ability to SSH into build machines to better debug your CI/CD jobs. I say controversial because being a team of over 20 strong minded people, each person had a distinct opinion on the merits and demerits of debugging with SSH. 

The merit is obvious, debugging is much faster when you can see what's happening on the machine. You can print out the environment, or see what happened to a particular container, look at CPU/memory utilization, or even change the configuration of a test database to see if that helps your failing tests. 

However, our team has made a conscious decision to disallow SSH access for our developers for two main reasons:

  • It doesn't encourage a mentality of 'designing for failure', for example, adding sufficient logging or making sure you can collect the information you need without needing access to the machine. You're not going to have access to Production machines, so avoiding SSHing into Test machines will build the discipline of designing for failure, rather than just reacting to failure.
  • Making ad-hoc changes to the machine environment results in environment drift and often, these changes aren't documented or really applied in all environments. Making it work on one Test machine isn't the goal, the discipline needed to then propagate those changes to all environments and reconfigure them is sometimes missed, which means your job will just fail again when your code change reaches the next environment.

Regardless, many of our customers still wanted this feature, so we're happy to launch it today. 

Before we dive in deeper, you need to know that SSH access is only available for paid accounts. 

Step 1: Log in to Shippable and go to the Subscription or Project page of the job you want to debug.

Screen Shot 2017-07-27 at 3.26.55 PM.png  










Step 2: Click on the Build or Rebuild button for the job. This will launch the following window.

Screen Shot 2017-07-27 at 3.33.36 PM.png

Click on Rebuild with SSH.

Step 3: Your build will start as usual, but you'll also see a new tab called Debug on the build page. Clicking on it while the node is being provisioning will show a Node not available message. Wait until the node is available for SSH instructions. 

Screen Shot 2017-07-27 at 3.36.08 PM.png


Step 4: When the node is available, you will see the message below in the debug tab. Run the command in a terminal window to SSH into your job node.

Screen Shot 2017-07-27 at 3.40.56 PM.png

Remember that the node will only be up for 30 mins, so this isn't the time to grab that cup of coffee.

Step 5: Once you are on the job node, run the command docker ps to see the containers running:

Screen Shot 2017-07-27 at 3.43.13 PM.png


The shippable-exec container is the Shippable agent running on the node. If you're having problems with anything in the pre_ci, pre_ci_boot or push sections of your yml, you can run commands inside shippable-exec using docker exec.

The c.exec container is the container running the job. If you're having trouble with commands in the ci, post_ci, on_success, on_failure sections of your yml, you can run the problematic commands inside c.exec using docker exec.

To all the non-Shippable developers out there: Happy debugging!   

Topics: features, CI/CD, troubleshooting, DevOps Automation