There has been a tectonic shift in the way we build, deploy and run applications. What started with cloud infrastructure has now created a revolution in IT organizations. As organizations realized the importance of continuous feedback loops, it created a need to ship software updates as fast as possible and brought in a culture of collaboration where everyone works as a team and not as a member of a silo.
This is fundamentally due to the DevOps movement. The word itself derives from “Development” and “Operations” and is all about removing barriers between Software Developers and Operations engineers and enabling better collaboration between the two. It also focuses on automation as the primary way to make software delivery lean and measurable.
DevOps is gaining rapid adoption due to some macro trends in the software universe:
- Software Defined Hybrid Cloud
- Web-scale Requirements
- Modern application architectures (micro services, functions)
1. Software-Defined Hybrid Cloud
In the past few years, Datacenters have gone through a massive transformation predominantly driven by Cloud Computing. In the pre-DevOps era, the barrier to entry for setting up a datacenter was pretty high. Organizations struggled with simple things like scaling infrastructure as it took months to obtain new machines and configure them. Updating all machines was an even bigger pain point, since the Ops team has to manually run patches and updates on every machine.
Amazon made it very easy to obtain on-demand compute in the form of VMs. Almost all public and private clouds today have APIs that allow IT to make changes very quickly, dropping the SLA from weeks to a few seconds for many tasks like provisioning and networking. Besides cloud native tools and APIs, several tools like Terraform, Ansible, Chef, and Puppet enable infrastructure-as-code. As a result, virtually the entire datacenter can be programmed as code, versioned and deployed automatically to reflect the change.
As more of the traditionally manual tasks began to move towards "as-code", the DevOps movement gained momentum by encouraging Ops teams to be self-sufficient by learn scripting techniques so they could write the scripts themselves.
2. Web scale requirements
Every organization is now a software organization. Companies like Amazon, Netflix, and Uber have disrupted traditional business models and made SaaS the defacto way of delivering software. As a result, applications have transformed from being systems of record to systems of engagement, such as mobile apps or interactive self-service portals.
Since consumers directly interact with systems of engagement, they need to have the following attributes:
- 24x7 availability with no downtime
- Massive scale to support hundreds or thousands of concurrent active workflows
- Ability to scale up or down to manage the demand ebb and flow both economically and efficiently
- Higher quality since the system is judged by every single consumer
- Complex infrastructure with load balancers, firewalls, and security to avoid being hacked
- Frequent updates with continuous feedback loops to reduce risk and innovate faster at the same time.
Almost all these attributes create more work for Dev and Ops teams and it became critical for them to work in lock-step to avoid buggy releases or less-than-perfect deployments. Also, as the frequency of releases increased, all tasks in the software delivery process needed to be performed much more often. This created the necessity for efficiency, further fueling interest in DevOps principles.
3. Modern application architectures
The move to Systems of engagement forced software teams to find innovative ways to handle scale, be more agile in building and shipping software, while also being efficient in managing COGS. This gave birth to a modern application architecture known as Microservices. In this approach, you break down the application into small decoupled units called microservices that can be independently developed, deployed, and scaled. The application business logic is implemented by composing the relevant microservices into a workflow. Some benefits of microservices are improved speed, agility, quality, and availability, as well as cost efficiency.
Microservices also affect organizational structure since each decoupled microservice is developed independenty. So instead of the old Triad model, the new structure is composed of small cross-functional teams working independently, one on each microservice.
Let's stop and digest this for a moment. In the old world, most applications consisted of 3-5 tiers which were often deployed and managed together. In the new world of microservices, the same-sized application can be broken down into 150+ microservices, each of which is deployed and managed independently. Theoretically, each team can choose their stack, right from programming language to deployment tools. Can you imagine the exponential extra burden on Ops?
It is almost impossible to imagine any organization adopting microservices without a DevOps mindset and the right automation in place. The need for microservices is definitely a big boost for the DevOps movement.
While containers have been around for some time, Docker created a tremendous momentum and in just 4 years, container adoption has done from negligible to ubiquitous. Docker provides much needed decoupling between the application runtime and the host OS on the virtual machine. This makes containers lightweight and portable across clouds. There was already interest in leveraging multiple clouds within an organization for scenarios such as disaster recovery or cloud bursting. Docker has made these scenarios more plausible.
As organizations explore these new areas, it creates new requirements for development and operations teams, and also increases the complexity of software delivery workflows. Organizations invest in DevOps principles and automation in order to handle this added complexity and create synergy across cross-functional teams.
So what's next?
While the DevOps movement is gaining rapid adoption, it is not easy to change many years of ingrained organizational processes and human behaviour. Also, the DevOps toolchain is relatively immature with a lot of scripting and automation tools that target a specific task or silo. With the initial focus of the movement was encouraging a cultural change in Dev and Ops mindsets, they also need mature tooling in order to avoid the inefficiency caused by manual handoffs and excessive number of custom scripts.
In the next blog in this series, we will look at the challenges that an organization faces while trying to adopt DevOps. Stay tuned!