Introduction
The challenges of developing software centric systems are not new –
they have been with us for decades. Traditional systems were hard to
build, often complex and fragile once they are built, and are
difficult to change. In today’s digital world the rate of change
exponentially higher than it was just a few years ago. Change is
inevitable, change is constant, change is necessary.
DevOps is a collection of technologies and disciplines which the key
to enabling rapid change to software centric ecosystems large and
small. In its simplest form DevOps is the practice of bringing all
development and operations teams together. In reality its impacts are
much broader and harder to define – we believe that in the most
successful organizations the business and its customers and partners
will directly participate in the rapid evolution of software features.
The DevOps Efficiency Matrix
This website presents the DevOps Efficiency Matrix, an open source tool
for evaluating the effectiveness of your product delivery process. The
matrix focuses on the 24 key capabilities defined in Accelerate, The
Science of DevOps by Nicole Forsgren PhD, Jez Humble, Gene Kim
(ISBN: 978-1942788331) which are known to drive high performing DevOps
teams. This open framework provides a way to measure where your
organization is on its DevOps journey.
Contributors
August 2019, Kirkland, Washington, USA
Continuous Delivery Capabilities
1: Version Control
| Level | Qualifier |
|---|
| 1 | None / Manual code backups. |
| 2 | System to Track Changes (SCCS, SVN, Git etc.) |
| 3 | Version control system does not lock files but provides tools for merging overlapping changes. May also include configuration and or tests (unit test code). |
| 4 | Code includes configuration, tests and deployment automation. Changes are checked in at least daily. Use of feature flags is beginning. |
| 5 | Pair Approval of application code, tests, configuration, feature flags and code for infrastructure (compute, storage, network). Feature flags are common. Code is checked in multiple times per hour. |
2: Deployment Automation
| Level | Qualifier |
|---|
| 1 | None / Manual Deployments. |
| 2 | Use of ad-hoc scripts or use of manual checklists. |
| 3 | Comprehensive, 70 to 95%, automation (may or may no include config). Either development team or operations team driven. |
| 4 | 95% or better automation. Includes test checks and configuration. Automation is driven by the team. |
| 5 | Fully automatic with check, configuration and monitoring. May include creation removal of test environments. |
3: Continuous Integration
| Level | Qualifier |
|---|
| 1 | None / manual. |
| 2 | Scheduled or triggered builds from common code repository. |
| 3 | Builds include significant basic unit testing. |
| 4 | Builds have quality gates for at least code test coverage and security analysis. |
| 5 | Test gates are comprehensive including integration and regression testing. |
4: Trunk-Based Development
| Level | Qualifier |
|---|
| 1 | Unorganized branches (missing documentation). Or, no branches and branch process (rules or guidelines). |
| 2 | Pet-based branches. Branches created as backups or checkpoints |
| 3 | Project-based branches. Or environment-based branches (dev, test, stage, prod). |
| 4 | Long lived feature / story branches (with a product focus). May or may not include process. |
| 5 | Short lived story branches that are merged to trunk master. |
5: Test Automation
| Level | Qualifier |
|---|
| 1 | None / Manual Testing. |
| 2 | Checklists ; Written scripts for manual testing. |
| 3 | Unit Testing ; Feature Testing ; Developers primarily create and maintain acceptance tests. |
| 4 | Coverage analysis drives test plans ; Security testing ; Service virtualization used for integration ; Tests are stable and accurate. |
| 5 | Full end-to-end regression testing continuous on dynamically created environments. |
6: Test Data Management
| Level | Qualifier |
|---|
| 1 | None ; Manual. |
| 2 | Test data can be copied. |
| 3 | Have adequate test data available ; Test data is checked in as code. |
| 4 | Test data does limit the range of tests that can be run ; Test data is minimized to reduce test run times ; Test data mimics real world scenarios. |
| 5 | Test Data generated by automation on demand ; Ability to condition data in the pipeline. |
7: Shift Left on Security
| Level | Qualifier |
|---|
| 1 | No planning or testing for security Manual. |
| 2 | Security assessments manual and are post development and/or late in testing. |
| 3 | Irregular scanning with tools for security vulnerabilities. Product design include security requirements and collaboration with security experts. |
| 4 | Recurring scanning with tools for security vulnerabilities. Team understands security requirements and take ownership. |
| 5 | Automated scanning included in build pipeline; both static and dynamic. Team monitors product for real-time security impacts. |
8: Continuous Delivery (CD)
| Level | Qualifier |
|---|
| 1 | None / manual / waterfall big releases / big design up front. |
| 2 | Working in small batches (or smaller batches). Time boxing work. |
| 3 | Uses a pull-based system. Use of Kanban to allow development to flow. |
| 4 | Fully empowered teams. Teams building/leveraging automation. Last column in Kanban means item is fully in production. |
| 5 | Automation of all processes not related to coding and code reviews. (Build, Testing, Test Data, Configuration, Deployment, Monitoring) |
Architecture Capabilities
9: Loosely Coupled Architecture
| Level | Qualifier |
|---|
| 1 | Monoliths and systems tightly coupled via point to point data transfers. Downstream systems must change when upstream systems modify data structures. |
| 2 | Service Orientation for systems and encapsulation of systems. Abstraction of data sources are emergent. |
| 3 | Use of APIs for communication between systems. Messages for independent events. |
| 4 | Abstractions of data vs business rules vs presentation layers. |
| 5 | Use of patterns for design of distribute systems, .e.g. sagas for long running transactions. |
10: Empowered Teams
| Level | Qualifier |
|---|
| 1 | Teams are told what to work on and have their schedules set by outside authorities. |
| 2 | Participate in decisions related to schedule and tooling. Write user stories and define requirements. |
| 3 | Flexibility to choose tools that fit the desired ecosystem. Have reasonable slack time to deal with unplanned work. |
| 4 | Chooses when to release software. Interact with stakeholders and sponsors. |
| 5 | Can raise issues and experiment with new features. |
Product And Process Capabilities
11: Customer Feedback
| Level | Qualifier |
|---|
| 1 | None. |
| 2 | Ad-hoc gathering of customer feedback ; late feedback, as from a UAT phase. |
| 3 | Regularly gathering customer metrics. |
| 4 | Actively seeking customer insights on quality and features ; Feedback sought early in the development process. |
| 5 | Using feedback to inform the design of products and features ; Empowering teams to respond to customer feedback ; Use of minimum viable products. |
12: Value Stream
| Level | Qualifier |
|---|
| 1 | No ties, or undocumented ties, to underlying business value. |
| 2 | Organization structure loosely determines product business value. |
| 3 | Developers understand impact of changes for the larger organization. |
| 4 | Regular consultation with business owners with insight into organizational goals. |
| 5 | Full synergy between business owners and developers via integrated product management teams. |
13: Working in Small Batches
| Level | Qualifier |
|---|
| 1 | Work planned for weeks or months at a time in a project oriented approach. |
| 2 | Use of Scrum to break large projects and features into sprint sized increments. |
| 3 | Decompose work in to small features using a product orientation ; Use of Kanban to manage shorter lead times and faster feedback loops. |
| 4 | Use of feature flags to allow combining small increments of work into larger gated features. |
| 5 | Working changes are deployed multiple times per day ; work is naturally sized in very small units. |
14: Team Experimentation
| Level | Qualifier |
|---|
| 1 | None. |
| 2 | A - B Test or other testing framework available for experimentation. |
| 3 | Experiments can be measure via multiple tools and viewpoints. |
| 4 | Seek to update business specifications during development to create value. |
| 5 | Developer experimentation without outside approval ; Close consultation with. |
Lean Management Capabilities
15: Change Approval Processes
| Level | Qualifier |
|---|
| 1 | None / Manual. |
| 2 | Little to no process around tracking or approving changes. External change advisory board has oversight for changes. |
| 3 | Processes for approval of changes to code. May also include configuration and or tests (unit test code). |
| 4 | Peer or Team approval of code. May also include tests (unit test code) and/or deployment automation (scripts) |
| 5 | Pair Approval of code, tests, configuration. Also including code for infrastructure (network, disk and compute). |
16: Monitoring
| Level | Qualifier |
|---|
| 1 | Minimal or no monitoring. Monitoring is adhoc and not related to business value. |
| 2 | Batched based monitoring via manual processes. Monitoring is difficult to tie to business value. Monitoring is focused only one delivery process or application usage, but not both. |
| 3 | Visual display of work and process limits are defined and enforced. Monitoring of both delivery process and application are emerging. |
| 4 | Defined metrics for work, e.g. cycle times, defect rates. Metrics are visible to all, not just the team. Monitoring of both delivery process and application have equal weight. |
| 5 | Metrics are aligned to operational goals, and business value. Monitoring is used for feedback to delivery processes. |
17: Proactive Notification
| Level | Qualifier |
|---|
| 1 | None. |
| 2 | Use of infrastructure or systems monitoring tools. Process metrics are captured. |
| 3 | Key metrics identified, including system, business and process metrics. Process metrics include lead time, release frequency and mean time to repair (MTTR). |
| 4 | Dashboards setup and alerting in place for all metrics. Metrics are reviewed regularly. |
| 5 | Data is used to inform business decisions and feature and work planning processes. Metrics collection and usage is continuously refined for better insights. |
18: WIP Limits
| Level | Qualifier |
|---|
| 1 | None. |
| 2 | Standard Work Processes. |
| 3 | Kanban (Either Sprint or Continuous Delivery). Process is being optimized for flow of work |
| 4 | Limits Identified and Enforced (without data or by outside entity). Flow is defined. Overload is limited. |
| 5 | Work in small batches. Team uses data to define their own WIP limits (not an outside entity). No overload. |
19: Visualizing Work
| Level | Qualifier |
|---|
| 1 | No process / ad-hoc or manual reports. |
| 2 | Defined process (either lightweight or comprehensive). Recurring reports (either manual or automatic). |
| 3 | Storyboards / Kanban. Visible dashboards or wall displays. Product hierarchy. |
| 4 | Real time updates on defect, defect rates, etc. Worked items linked to code updates. |
| 5 | All information is publicly available. Shared understanding with senior management. Advanced techniques (AR, VR). |
Cultural Capabilities
| Level | Qualifier |
|---|
| 1 | Low cooperation and siloed organizations with minimal interaction between them. |
| 2 | Novelty discouraged. Silos have become well defined with narrow responsibilities and have bureaucratic interfaces. |
| 3 | Modest cooperation between teams and across the organization; Information sharing is limited. |
| 4 | Root cause analysis is objective. Experimentation embraced or encouraged. Responsibilities are discussed in terms of overall risk. |
| 5 | High cooperation. Good information Flow. Failures are opportunities to improve the system. Risks are shared among teams. |
21: Supporting Learning
| Level | Qualifier |
|---|
| 1 | Learning is invisible or non-existent. |
| 2 | Learning is a cost. Time for learning is considered un-productive and interferes with real work. |
| 3 | Learning is budgeted and visible. Learning is considered a perk or is reserved for select resources. Resources have limited ability to choose topics for learning. |
| 4 | Learning from customer and actual usage of the system. Learning is an investment broadly available. Mentoring in supported. |
| 5 | Leaning is continuous. Learning incorporates views from all stakeholders. Learning is encouraged as a normal course of work. Mentoring is seen as a learning practice. |
22: Collaboration Among Teams
| Level | Qualifier |
|---|
| 1 | Little trust and little interaction between teams. Collaboration when it occurs, occurs late in the process. |
| 2 | Teams require overly formal processes for collaboration. |
| 3 | Teams build and support knowledge transfer processes, and good documentation. |
| 4 | Leadership supports assistance across teams, rather than sticking to defined silos. Teams exhibit trust in other teams. |
| 5 | Resources are encouraged to move between departments. Successes are shared and celebrated at the organizational level. Actively and continuously rewarding work that facilitates collaboration. |
23: Job Satisfaction
| Level | Qualifier |
|---|
| 1 | Negative words are used to describe the work items and work environment. Tasks oriented work is assigned to team members by others. |
| 2 | Work assigned takes advantage of the employeesÕ skill set. The right tools are available to do the work required. |
| 3 | Acquisition of tools is supported. All members feel welcome and valued. Team members choose work from a prioritized list of features created by the business. |
| 4 | Works loads are consistent and continuous. Teams have spontaneous discussions and members are eager to contribute. Judgement of the team is supported across the organization. |
| 5 | When issues are raised they are seen as opportunities to solve problems. Employees identify with the organization and have a sense of contribution. Diverse backgrounds are encouraged to collaborate. |
| Level | Qualifier |
|---|
| 1 | No clear direction or understanding of the vision for the organization. Positive feedback is rare. |
| 2 | Transactional leadership focused on detailed measurements and little recognition for different approaches. Supports status quo. |
| 3 | Teams and leadership agree about alignment and progress toward goals. Broader and more holistic views are encouraged. |
| 4 | Long term vision is articulated through motivating communication about shared goals and direction. |
| 5 | Leadership trusts the team and supports team experimentation to realize the vision. Supports individuals as well as the broader goals. |
