The DevOps Handbook

This award-winning and bestselling business handbook for digital transformation is now fully updated and expanded with the latest research and new case studies!

“[The DevOps Handbook] remains a must-read for any organization seeking to scale up its IT capability and expand DevOps practices across multiple departments or lines of business.” —Mike Perrow, TechBeacon

For years, The DevOps Handbook has been the definitive guide for taking the successes laid out in the bestselling The Phoenix Project and applying them in any organization. Now, with this fully updated and expanded edition, it's time to take DevOps out of the IT department and apply it across the full business.

Technology is now at the core of every company, no matter the business model or product. The theories and practices laid out in The DevOps Handbook are tools to be used by anyone from across the organization to create joy and succeed in the marketplace.

The second edition features 15 new case studies, including stories from Adidas, American Airlines, Fannie Mae, Target, and the US Air Force. In addition, renowned researcher and coauthor of Accelerate, Dr. Nicole Forsgren, provides her insights through new and updated material and research. With over 100 pages of new content throughout the book, this expanded edition is a must read for anyone who works with technology.

“[The DevOps Handbook is] a practical roadmap to improving IT in any organization. It's also the most valuable book on software development I've read in the past 10 years.” —Adam Hawkins, software developer and host of the podcast SmallBatches

CONTENTS

Figures & Tables xii

Note from the Publisher on the Second Edition xv

Foreword to the Second Edition: Nicole Forsgren xix

Foreword to the First Edition: John Allspaw xxi

Preface: Aha! xxiii

Introduction xxxi

Part I—The Three Ways

Part I Introduction 3

01 Agile, Continuous Delivery, and the Three Ways 7

NEW Case Study: Approaching Cruising Altitude:

American Airlines' DevOps Journey Part 1 (2020) 15

02 The First Way: The Principles of Flow 19

NEW Case Study: Flow and Constraint Management in

Healthcare (2021) 29

03 The Second Way: The Principles of Feedback 33

NEW Case Study: Pulling the Andon Cord at

Excella (2019) 39

04 The Third Way: The Principles of Continual Learning and

Experimentation 45

NEW Case Study: The Story of Bell Labs 54

Part 1 Conclusion 57

Part II—Where to Start

Part II Introduction 61

05 Selecting Which Value Stream to Start With 63

Case Study: Nordstrom's DevOps Transformation 63

NEW Case Study: Kessel Run: The Brownfield

Transformation of a Mid-Air Refueling System (2020) 69

NEW Case Study: Scaling DevOps Across the Business:

American Airlines' DevOps Journey (Part 2) (2020) 74

viii CONTENTS

NEW Case Study: Saving the Economy From Ruin (With a

Hyperscale PaaS) (2021) 77

06 Understanding the Work in Our Value Stream, Making it

Visible, and Expanding it Across the Organization 81

Case Study: Nordstrom's Experience with Value Stream

Mapping 81

Case Study: Operation InVersion at LinkedIn (2011) 91

07 How to Design Our Organization and Architecture

with Conway's Law in Mind 97

Case Study: Conway's Law at Etsy 98

Case Study: API Enablement at Target (2015) 112

08 How to Get Great Outcomes by Integrating Operations

into the Daily Work of Development 115

Case Study: Big Fish Games 115

NEW Case Study: Better Ways of Working at Nationwide

Building Society 124

Part II Conclusion 129

Part III—The First Way: The Technical Practices of Flow

Part III Introduction 133

09 Create the Foundations of Our Deployment Pipeline 135

Case Study: Enterprise Data Warehouse 135

NEW Case Study: How a Hotel Company Ran $30B of

Revenue in Containers (2020) 143

10 Enable Fast and Reliable Automated Testing 147

Case Study: Google Web Server 148

11 Enable and Practice Continuous Integration 167

Case Study: HP LaserJet Firmware 168

Case Study: Continuous Integration of Bazaarvoice (2012) 173

12 Automate and Enable Low-Risk Releases 177

Case Study: Daily Deployments at CSG International (2013) 181

Case Study: Etsy—Self-Service Developer Deployment:

An Example of Continuous Deployment (2014) 186

Case Study: Dixons Retail—Blue-Green Deployment for

Point-of-Sale System (2008) 193

Case Study: Dark Launch of Facebook Chat (2008) 198

CONTENTS ix

NEW Case Study: Creating a Win-Win for Dev & Ops

at CSG (2016) 201

13 Architect for Low-Risk Releases 207

Case Study: Evolutionary Architecture at Amazon (2002) 202

Case Study: Strangler Pattern at Blackboard Learn (2011) 215

Part III Conclusion 219

Part IV—The Second Way: The Technical Practices of Feedback

Part IV Introduction 223

14 Create Telemetry to Enable Seeing and Solving Problems 225

Case Study: DevOps Transformation at Etsy (2012) 226

Case Study: Creating Self-Service Metrics at

LinkedIn (2011) 237

15 Analyze Telemetry to Better Anticipate Problems and

Achieve Goals 245

Case Study: Telemetry at Netflix (2012–2020) 245

Case Study: Auto-Scaling Capacity at Netflix (2012) 251

Case Study: Advanced Anomaly Detection (2014) 255

16 Enable Feedback So Development and Operations Can

Safely Deploy Code 259

Case Study: Right Media (2006) 259

Case Study: The Launch and Hand-Off Readiness

Review Google (2010) 269

17 Integrate Hypothesis-Driven Development and

A/B Testing into Our Daily Work 273

Case Study: Hypothesis-Driven Development

at Intuit, Inc. (2012) 273

Case Study: Doubling Revenue Growth through Fast

Release Cycle Experimentation at Yahoo! Answers (2010) 278

18 Create Review and Coordination Processes to Increase

Quality of Our Current Work 281

Case Study: Peer Review at GitHub (2011) 286

NEW Case Study: From Six-Eye Principle to Release at

Scale at Adidas (2021) 286

Case Study: Code Reviews at Google (2010) 290

Case Study: Pair Programming Replacing Broken Code

Review Processes at Pivotal Labs (2011) 293

Part IV Conclusion 299

x CONTENTS

Part V—The Third Way: The Technical Practices of Continual Learning

and Experimentation

Part V Introduction 303

19 Enable and Inject Learning into Daily Work 305

Case Study: AWS US-EAST and Netflix (2011) 305

NEW Case Study: Turning an Outage into a Powerful

Learning Opportunity at CSG (2020) 318

20 Convert Local Discoveries into Global Improvements 321

Case Study: Standardizing a New Technology Stack

at Etsy (2010) 332

NEW Case Study: Crowdsourcing Technology Governance at

Target (2018) 333

21 Reserve Time to Create Organizational Learning

and Improvement 335

Case Study: 30-Day Challenge at Target (2015) 335

Case Study: Internal Technology Conferences at

Nationwide Insurance, Capital One, and Target (2014) 342

Part V Conclusion 347

Part VI—The Technological Practices of Integrating Information

Security, Change Management, and Compliance

Part VI Introduction 351

22 Information Security Is Everyone's Job Every Day 353

Case Study: Static Security Testing at Twitter (2009) 360

Case Study: 18F Automating Compliance for the Federal

Government with Compliance Masonry (2016) 369

Case Study: Instrumenting the Environment at

Etsy (2010) 373

NEW Case Study: Shifting Security Left at Fannie

Mae (2020) 376

23 Protecting the Deployment Pipeline, and Integrating into

Change Management and Other Security and Compliance

Controls 379

Case Study: Automated Infrastructure Changes as

Standard Changes at Salesforce.com (2012) 383

CONTENTS xi

Case Study: PCI Compliance and a Cautionary Tale of

Separating Duties at Etsy (2014) 385

NEW Case Study: Biz and Tech Partnership toward 10 "No Fear

Releases" Per Day at Capital One (2020) 387

Case Study: Proving Compliance in Regulated

Environments (2015) 389

Case Study: Relying on Production Telemetry for

ATM Systems 392

Part VI Conclusion 395

A Call to Action: Conclusion to the DevOps Handbook 397

Afterword to the Second Edition 401

Appendices 409

Bibliography 423

Notes 441

Index 461

Acknowledgments 479

Author Biographies 482