Last week, we announced the General Availability of Shippable Server, the behind-the-firewall version of our hosted platform. We also articulated our vision around where DevOps is today and why Assembly Lines is the future of DevOps.
As I think about our journey from CI to Assembly Lines, it mirrors the journey of most organizations as they mature their DevOps efforts. In a nutshell, DevOps has created an awareness of the need to automate and be more efficient in terms of software delivery. However, most of the focus around the how has been around cultural changes and tools that help automate bits and pieces of the end-to-end software delivery workflow. This has led to the formation of "islands of automation" that are optimized for specific tasks but do not enable the holy grail of Continuous Delivery or Continuous Deployment. To achieve those goals, you need an Assembly Line platform that takes all these tools and connects them into end-to-end workflows with complete visibility, traceability, and auditability.
So let's take a look at this journey, and dig in a bit deeper into why Assembly Lines are the essential factor to DevOps success.
Starting Out With Continuous Integration (CI)
We started Shippable in early 2013 with a simple mission: Making CI effortless. Avi and I had spent several years working in large companies where everything moved at a snail's pace. The process of shipping code took six months to a year, with endless triage meetings where bugs were bounced around between teams, 2am build breaks, manual handoffs, sync meetings, and so on. We both believed there had to be a better way and the first step was solving CI.
I was always unhappy with Jenkins as a CI tool, it required too much time to set up and manage, plugins were clunky and inconsistent, and all the configuration was in the UI. We started working on providing a much improved CI experience. Barely three months after we started, Docker was open sourced and we immediately recognized it was going to revolutionize software development, including CI.
We went all in on Docker and built Shippable to be the best CI, with a special focus on Docker-based workflows.
Maturing Beyond CI With Pipelines
As our CI platform gained traction, we realized that CI wasn't enough to help developers identify code defects. CI is about running unit tests, which help identify basic problems within a component, but misses most defects that are caused by the component running in a live environment. Customers needed a way to package their application and deploy it into a series of live environments so they could trigger different test suites.
To address these needs, we built the first version of Pipelines in 2015. Pipelines was called Formations back then and we got great feedback from customers, along with pain points. While Pipelines supported creating workflows, they also offered very little control: once triggered, all pipeline steps executed in sequence without any way to pause, wait for a specific event, or stop at any step. Also, Pipelines, just like CI, were primarily focused on developer workflows and did not solve problems for other teams as gracefully.
We addressed most of the feedback around Pipelines over the next year, but there was a lingering feeling that we needed to do more to solve software delivery as a whole and help teams achieve Continuous Delivery or Continuous Deployment.
The Audacious Vision Of Assembly Lines
Software delivery goes much beyond developers and involves multiple teams like Test, Operations, Security, SecOps, and Release Management. This means that any platform or solution that claims to help achieve Continuous Delivery or Deployment needs to address the workflows of all these teams, as well as manage the interactions between them.
DevOps has identified the need for a cultural change within most organization, and teams have started to automate their tasks and improve efficiency. Several tool vendors have emerged to aid with these efforts.
Each of the tools above solves a portion of the software delivery workflow really well, but just looking at it begs the question: If my deployment pipeline consists of all these disconnected tools developed by different vendors, how can I achieve Continuous Deployment?
We created Assembly Lines to solve precisely this problem. The tools available today optimize a specific task or set of tasks and are usually focused on the workflows for a single team. This approach has led to "Islands of Automation", where individual tasks are automated, but the connections between them are still missing, so there is no visibility, traceability or auditability across your entire end-to-end workflow.
Most organizations today take one of two approaches: they either try to compensate through cultural cooperation like meetings, chat rooms, or spreadsheets, or they create a central DevOps team that builds custom point-to-point integrations between the tools. These are both problematic since the first creates a dependence on manual processes, and the latter needs a whole new team which itself becomes the bottleneck as the number of applications, services, and tools increase, leading to spaghetti code that is difficult to manage and scale.
Assembly Lines solve this problem by borrowing concepts from hyper-efficient car manufacturing processes to achieve Continuous Deployment (or Delivery). There are four key aspects to Assembly Lines:
- Connecting islands of automation into interconnected end-to-end stateful workflows that enable toolchain collaboration.
- Defining your workflows as-code with a common, declarative specification language that is easy to learn and templatize, and versioned.
- Gaining powerful end-to-end visibility with rich logistics that support management, insight, and audit capability.
- Integrating with technical stacks and application architectures of today and the future with a simple plug-and-play interface.
The first version of Assembly Lines was launched in September 2016. We have refined it tremendously over the last few months as we've received helpful feedback from customers. If you're interested in learning more about Assembly Lines and the vision behind it, you can download our whitepaper:
Shipping Shippable Server
Last week, we launched Shippable Server for organizations that want to run their Assembly Lines behind the firewall. While many organizations are running their workloads in the cloud, they often have security and compliance restrictions that require them to operate behind a firewall. Shippable Server can run on any infrastructure and a Starter version running on one machine is available in the AWS Marketplace with a 30-day free trial. Check it out today and get started with your own Assembly Lines for DevOps!
I would love to hear your thoughts on our journey, as well as any feedback or comments on our Assembly Lines approach. Onwards with DevOps!