Book: DevOps Handbook

“DevOps Handbook: How to Create World-Class Agility, Reliability, & Security in Technology Organizations” By Gene Kim, Jez Humble, Patrick Debois, & John Willis

• Two conflicting goals:
  • “Respond to the rapidly changing competitive landscape”
  • “Provide stable, reliable, and secure service to the customer”

• “organizations adopting DevOps are able to linearly increase the number of deploys per day as they increase their number of developers, just as Google, Amazon, and Netflix have done.”
• lead time: “clock starts when the request is made and ends when it is fulfilled”
• process time: “starts only when we being work on the customer request—specifically, it omits the time that the work is in queue, waiting to be processed”
• rework: “percent complete and accurate (%C/A)”, “the %C/A can be obtained by asking downstream customers what percentage of the time they receive work that is ‘usable as is,’ meaning that they can do their work without having to correct the information that was provided, add missing information that should have been supplied, or clarify information that should have and could have been clearer.”
• Three Ways:
• • Flow: “flow of work from Development to Operations to the customer. In order to maximize flow, we need to make work visible, reduce our batch sizes and intervals of work, build in quality by preventing defects from being passed to downstream work centers, and constantly optimize for the global goals.”
  • Feedback: “the fast and constant flow of feedback”
  • Continual Learning and Experimentation: “the creation of a generative, high-trust culture that supports a dynamic, disciplined, and scientific approach to experimentation and risk-taking, facilitating the creation of organizational learning, both from our successes and failures. Furthermore, by continually shortening and amplifying our feedback loops, we create ever-safer systems of work and are better able to take risks and perform experiments that help us learn faster than our competition and win in the marketplace.”
• “when we limit WIP, we find that we may have nothing to do because we are waiting on someone else. Although it may be tempting to start new work (i.e. ‘It’s better to be doing something than nothing’), a far better action would be to find out what is causing the delay and help fix that problem.”
• “we strive to reduce the number of handoffs, either by automating significant portions of the work or by reorganizing teams so they can deliver value to the customer themselves, instead of having to be constantly dependent on others. As a result, we increase flow by reducing the amount of time that our work spends waiting in queue, as well as the amount of non-value-added time.”
• waste and hardship:
• • “Partially done work”: “Partially done work becomes obsolete and loses value as time progresses.”
  • “Extra processes: Any additional work that is being performed in a process that does not add value to the customer.”
  • “Extra features: Features built into the service that are not needed by the organization or the customer (e.g., “gold plating”).”
  • “Task switching”
  • “Waiting”
  • “Motion: The amount of effort to move information or materials from one work center to another. Motion waste can be created when people who need to communicate frequently are not colocated. Handoffs also create motion waste and often require additional communication to resolve ambiguities.”
  • “Defects”
  • “Nonstandard or manual work”
  • “Heroics: In order for an organization to achieve goals, individuals and teams are put in a position where they must perform unreasonable acts, which may even become a part of their daily work (e.g. nightly 2:00 a.m. problems in production, creating hundreds of work tickets as part of every software release).”
• Capabilities to work safely in a complex system:
• • “Complex work is managed so that problems in design and operations are revealed”
  • “Problems are swarmed and solved, resulting in quick construction of new knowledge”
  • “New local knowledge is exploited globally throughout the organization”
  • “Leaders create other leaders who continually grow these type of capabilities”
• “According to Dr. Spear, the goal of swarming is to contain problems before they have a chance to spread, and to diagnose and treat the problem so that it cannot secure.”
• “It prevents the problem from progressing downstream, where the cost and effort to repair it increases exponentially and technical debt is allowed to accumulate.”
• “In complex systems, adding more inspection steps and approval processes actually increases the likelihood of future failures. The effectiveness of approval processes decreases as we push decision-making further away from where the work is performed. Doing so not only lowers the quality of decisions but also increases our cycle time, thus decreasing the strength of the feedback between cause and effect, and reducing our ability to learn from successes and failures.”
• “Instead, we need everyone in our value stream to find and fix problems in their area of control as part of our daily work. By doing this, we push quality and safety responsibilities and decision-making to where the work is performed, instead of relying on approvals from distant executives.”
• “When accidents and failures occur, instead of looking for human error, we look for how we can redesign the system to prevent the accident from happening again.”
• “As Bethany Macri, an engineer at Easy who led the creation of the Morgue tool to help with recording of postmortems, stated, ‘By removing blame, you remove fear; by removing fear, you enable honesty; and honesty enables prevention.”
• “Even if we have the highest levels of executive sponsorship, we will avoid the big bang approach (i.e. starting everywhere all at once), choosing instead to focus our efforts in a few areas of the organization, ensuring that those initiatives are successful, and expand from there.”
• “value stream mapping exercise—a process (described later in this chapter) designed to help capture all the steps required to create value.”
• “Reserve 20% of cycles for non-functional requirements and reducing technical debt”
• “Cagan notes that when organizations do not pay their ’20% tax,’ technical debt will increase to the point where an organization inevitably spends all of its cycles paying down technical debt.”
• “Conway’s Law, which states that ‘organizations which design systems…are constrained to produce designs which are copies of the communication structures of these organizations…The larger an organization is, the less flexibility it has and the more pronounced the phenomenon.”
• “Functional-oriented organizations optimize for expertise, division of labor, or reducing cost. These organizations centralize expertise, which helps enable career growth and skill development, and often have tall hierarchical organization structures. This has been the prevailing method of organization for Operations (i.e., server admins, network admins, database admins, and so forth are all organized into separate groups).”
• “Matrix-oriented organizations attempt to combine functional and market orientation. However as many who work in or manage matrix organizations observe, matrix organizations often result in complicated organizational structures, such as individual contributors reporting to two managers or more, and sometimes achieving neither of the goals of functional or market orientation.”
• “Market-oriented organizations optimize for responding quickly to customer needs. These organizations tend to be flat, composed of multiple, cross functional disciplines (e.g., marketing, engineering, etc.), which often lead to potential redundancies across the organization. This is how many prominent organizations adopting DevOps operate in extreme examples, such as at Amazon or Netflix, each service team is simultaneously responsible for feature delivery and service support.”
• “marke-orientöd teams are responsible not only for feature development, but also for testing, securing, deploying, and supporting their service in production, from idea conception to retirement.”
• “By cross-training and growing engineering skills, generalists can do orders of magnitude more work than their specialist counterparts, and it also improves our overall flow of work by removing queues and wait time.”
• “By encouraging everyone to learn, as well as providing training and support, we create the most sustainable and least expansive way to create greatness in our teams—by investing in the development of the people we already have.”
• “Ideally, our software architecture should enable small teams to be independently productive, sufficiently decoupled from each other so that work can be done without excessive unnecessary communication and coordination.”
• “When we have a tightly-coupled architecture, small changes can result in large scale failures. As a result, anyone working in one part of the system must constantly coordinate with anyone else working in another part of the system they may affect, including navigating complex and bureaucratic change management processes.”
• “The idea is that developers should be able to understand and update the code of a service without knowing anything about the internals of its peer services. Services interact with their peers strictly through APIs and thus don’t share data structures, database schemata, or other internal representations of objects.”
• “Create self-service capabilities to enable developers in the service teams to be productive.”
• “Embed Ops engineers into the service teams”
• “Assign Ops liaisons to the service teams when embedding ops is not possible.”
• “All the platforms and services we provide should (ideally) be automated and available on demand, without requiring a developer to open up a ticket and wait for someone to manually perform work. This ensures that Operations doesn’t become a bottleneck for their customers (e.g. ‘We received your work request, and it will take six weeks to manually configure those test environments.’).”
• “Our goal is to ensure that we a re-create the entire production environment based on what’s in version control.”
• “When we can quickly rebuild and re-create our applications and environments on demand, we can also quickly rebuild them instead of repairing them when things go wrong.”
• “immutable infrastructure, where manual changes to the production environment are no longer allowed—the only way production changes can be made is to put the changes into version control and re-create the code and environments from scratch. By doing this, no variance is able to creep into production.”
• “A culture that ‘stops the entire production line’ when our validation tests fail.”
• “Developers working in small batches on trunk rather than long-lived feature branches.”
• “a small number of reliable, automated tests are almost always preferable over a large number of manual or unreliable automated tests.”
• “We prioritize the team goals over individual goals—whenever we help someone move their work forward, we help the entire team. This applies whether we’re helping someone fix the build or an automated test, or even performing a code review for them. And of course, we know that they’ll do the same for us, when we need help.” —Randy Shoup, former engineering director for Google App Engine
• “We may even configure our deployment pipeline to reject any commits (e.g., code or environment changes) that take us out of a deployable state. This method is called gated commits, where the deployment pipeline first confirms that the submitted change will successfully merge, build as expected, and pass all the automated tests before actually being merged into trunk. If not, the developer will be notified, allowing corrections to be made without impacting anyone else in the value stream.”
• Definition of done: “At end of each development interval, we must have integrated, tested, working, and potentially shippable code, demonstrated in a production-like environment, created from trunk using a one-click process, and validated with automated tests.”
• “Deployment is the installation of a specified version of software to a given environment (e.g. deploying code into an integration test environment or deploying code into production). Specifically, a deployment may or may not be associated with a release of a feature to customers.”
• “Release is when we make a feature (or set of features) available to all our customers or a segment of customers (e.g., we enable the feature to be used by 5% of our customer base). Our code and environments should be architected in such a way that the release of functionality does not require changing our application code.”
• “when we conflate deployment and release, it makes it difficult to create accountability for successful outcomes—decoupling these two activities allows us to empower Development and Operations to be responsible for the success of fast and frequent deployments, while enabling product owners to be responsible for the successful business outcomes of the release (i.e., was building and launching the feature worth our time).”
•  “Environment-based release patterns: This is where we have two or more environments that we deploy into, but only one environment is receiving live customer traffic (e.g., by configuring our load balancers). New code is deployed into a non-live environment, and the release is performed moving traffic to this environment.”
• “Application-based release patterns: This is where we modify our application so that we can selectively release and expose specific application functionality by small configuration changes. For instance, we can implement feature flags that progressively expose new functionality in production to the development team, all internal employees, 1% of our customers, or, when we are confident that the release will operate as designed, our entire customer base. As discussed earlier, this enables a technique called dark launching, where we stage all the functionality to be launched in production and test it with production traffic before our release.”
• For environment-based release pattern, there are two approaches to database changes:
• • “Create two databases (i.e. blue and green database): Each version-blue (old) and green (new)-of the application has its own database. During the release, we put the blue database into read-only mode, perform a backup of it, restore onto the green database, and finally switch traffic to the green environment.”
  • Decouple database changes from application changes: Instead of supporting two databases, we decouple the release of database changes from the release of application changes by doing two things: First, we make only additive changes to our database, we never mutate existing database objects, and second we make no assumptions in our application about which database version will be in production.”
• “The challenge is how to keep migrating from the architecture we have to the architecture we need.”
• “What Shoup’s team did at eBay is a textbook example of evolutionary design, using a technique called the strangler application pattern—instead of ‘ripping out and replacing’ old services with architecture that no longer support our organizational goals, we put the existing functionality behind an API and avoid making further changes to it. All new functionality is then implemented in the new services that use the new desired architecture, making calls to the old system when necessary.”
• “we seek to access all services through versioned APIs, also called versioned services or immutable services.”
• “Versioned APIs enable us to modify the service without impacting the callers, which allows the system to be more loosely-coupled—if we need to modify the arguments, we create a new API version and migrate teams who depend on our service to the new version.”
• “By researching the user, we can often re-engineer the process so that we can design a far simpler and more streamlined means to achieving the business goal.”
• “telemetry, widely defined as ‘an automated communications process by which measurements and other data are collected at remote points and are subsequently transmitted to receiving equipment for monitoring.’”
• “StatsD can generate timers and counters with one line of code (in Ruby, Perl, Python, Java, and other languages) and is often used in conjunction with Graphite or Grafana, which renders metric events into graphs and dashboards.”
• “information radiator, defined by the Agile Alliance as ‘the generic term for any of a number of handwritten, drawn, printed, or electronic displays which a team places in a highly visible location, so that all team members as well as passers-by can see the latest information at a glance: count of automated tests, velocity, incident reports, continuous integration status, and so on.’”
• “By putting information radiators in highly visible places, we promote responsibility among team members, actively demonstrating the following value:
• • The team has nothing to hide from its visitors (customers, stakeholders, etc.)
  • The team has nothing to hide from itself: it acknowledge and confronts problems”
• levels of metrics
• • “Business level: Examples include the number of sales transactions, revenue of sales, transactions, user signups, churn rate, A/B testing results, etc.”
  • “Application level: Examples include transaction times, user response times, application faults, etc.”
  • “Infrastructure level (e.g, database, operating system, networking storage): Examples include web server traffic, CPU load, disk usage, etc.”
  • “Client software level (e.g., JavaScript on the client browser, mobile application): Examples include application errors and crashes, user measured transaction times, etc.”
  • “Deployment pipeline level: Examples include build pipeline status (e.g. red or green for our various automated test suites), change deployment lead times, deployment frequencies, test environment promotions, and environment status.”
• “At the application level, our goal is to ensure that we are generating telemetry not only around application health (e.g., memory usage, transaction counts, etc.), but also to measure to what extend we are achieving our organizational goals (e.g., number of new users, user login events, user session lengths, percent of users active, how often certain features are being used, and so forth).”
• “We should define and link each metric to a business outcome metric at the earliest stage of feature definition and development, and measure the outcomes after we deploy them in production.”
• outlier detection: detecting “abnormal running conditions from which significant performance degradation may well result, such as an aircraft engine rotation defect or a flow problem in a pipeline.”
• Netflix “used outlier detection in a very simple way, which was to first compute what was the ‘current normal’ right now, given population of nodes in a compute cluster. And then we identified which nodes didn’t fit that pattern, and removed those nodes from production.”
• “We can automatically flag misbehaving nodes without having to actually define what the ‘proper’ behavior is in any way. And since we’re engineered to run resiliently int he cloud, we don’t tell anyone in Operations to do something—instead, we just kill the sick or misbehaving compute node, and then log it or notify the engineers in whatever form they want.”
• Netflix has “massively reduced the effort of finding sick servers, and, more importantly, massively reduced the time require Rapport states d to fix them, resulting in improved service quality. The benefit of using these techniques to preserve employee sanity, work/life balance, and service quality cannot be overstated.”
• “When people ask me for recommendations on what to monitor, I joke that in an ideal world, we would delete all the alerts we currently have in our monitoring system. Then, after each user-visible outage, we’d ask what indicators would have predicted that outage and then add those to our monitoring system, alerting as needed. Repeat. Now we only have alerts that prevent outages, as opposed to being bombarded by alerts after an outage already occurred.”
• “leading indicators that could have warned us earlier that we were starting to deviate from standard operations, such as:
• • Application level: increasing web page load time, etc.
  • OS level: sever free memory running low, disk space running low, etc.
  • Database level: database transaction times taking longer than normal, etc.
  • Network level: number of functioning servers behind the load balancer dropping, etc.”
• “And because everyone now knows that the smaller our production changes, the fewer problems we will have, developers start checking ever-smaller increments of code more frequently into the deployment pipeline, ensuring that their change is working successfully in production before moving to their next change.”
• “We are now deploying code more frequently than ever, and service stability is better than ever too. We have re-discovered that the secret to smooth and continuous flow is making small, frequent changes that anyone can inspect and easily understand.”
• Techniques for handling bugs in production:
• • “turn off broken features with feature toggles (which is often the easiest and least risky option since it involves no deployment to production)”
  • fix forward: “make code changes to fix the defect, which are then pushed into production through the deployment pipeline”
  • • “Although fixing forward can often be dangerous, it can be extremely safe when we have automated testing and fast deployment processes, and sufficient telemetry that allows us to quickly confirm whether everything is functioning correctly in production.”
  • roll back: “switch back to the previous release by using feature toggles or by taking broken servers out of rotation using the blue-green or canary release patterns, etc.”
• “Developers want to follow their work downstream—by seeing customer difficulties firsthand, they make better and moe informed decisions in their daily work.”
• “By doing this, we create feedback on the non-functional aspects of our code—all elements that are not related to the customer-facing features—and identify ways that we can improve deployability, manageability, operability, and so on.”
• “For decades, especially in retailing, the risk of revenue-impacting outages during the holiday season were so high that we would often put into place a change freeze from mid-October to mid-January. However, by making software deployments and releases fast and safe, the TurboTax team made online user experimentation and any required production changes a low-risk activity that could be performed during the highest traffic and revenue generating periods.”
• “The faster we can experiment, iterate, and integrate feedback into our product or service, the faster we can learn and out-experiment the competition. And how quickly we can integrate our feedback depends on our ability to deploy and release software.”
• “If we are not performing user research, the odds are that two-thirds of the features we are building deliver zero or negative value to our organization, even as they make our codebase ever more complex, thus increasing our maintenance costs over time and making our software more difficult to change.”
• “Our countermeasure is to integrate A/B testing into the way we design, implement, test, and deploy our features. Performing meaningful user research and experiments ensures that our efforts help achieve our customer and organizational goals, and help us win in the marketplace.”
• “Traditional change controls can lead to unintended outcomes, such as contributing to long lead times, and reading the strength and immediacy of feedback from the deployment process.”
• “As has been proven time and again, the further the distance between the person doing the work (i.e. the change implementer) and the person deciding to do the work (i.e., the change authorizer), the worse the outcome.”
• “In Puppet Labs’ 2014 State of DevOps Report, one of the key findings was that high-performing organizations relied moe on peer review and less on external approval of changes.”
• “There is a non-linear relationship between the size of the change and the potential risk of integrating that change—when you go from a ten line code change to a one hundred line code, the risk of something going wrong is more than ten times higher, and so forth.”
• “Furthermore, our ability to meaningfully critique code changes goes down as the change size goes up.”
• “paired programmers are 15% slower than two independent individual programmers, while ‘error-free’ code increase from 70% to 85%.”
• “As Adrian Cockcroft observed, ‘A great metric to publish widely is how many meetings and work tickets are mandatory to perform a release—the goal is to relentlessly reduce the effort required for engineers to perform work and deliver it to the customer.”
• “As one might expect, when they first ran Chaos Monkey in their production environments, services failed in ways they never could have predicted or imagined—by constantly finding and fixing these issues during normal working hours, Netflix engineers quickly and iteratively created a more resilient service, while simultaneously creating organizational learning (during normal working hours!) that enable them to evolve their systems far beyond their competition.”
• “Dr. Dekker calls this notion of eliminating error by eliminating the people who caused the errors the Bad Apple Theory. He asserts that this is invalid, because ‘human error is not our cause of troubles; instead human error is a consequence of the design of the tools that we give them.”
• “instead of ‘naming, blaming, and shaming’ the person who caused the failure, our goal should always be to maximize opportunities for organizational learning, continually reinforcing that we value actions that expose and share more widely the problem in our daily work.”
• “When engineers make mistakes and feel safe when giving details about it, they are not only willing too be held accountable but they are also enthusiastic in helping the rest of the company avoid the same error in the future.”
• “Two effective practices that help create a just, learning-based culture are blameless postmortems and the controlled introduction of failures into production to create opportunities to practice for the inevitable problems that arise within complex system.”
• During a blameless postmortems meeting: “Counterfactual statements, such as ‘I could have…’ or ‘If I had known about that, I should have…,’ frame the problem in terms of the system as imagined instead of in terms of the system that actually exists, which is the context we need to restrict ourselves to.”
• “a standardized model, where routine and systems govern everything, including strict compliance with timeliness and budgets, or and experimental model, where every day every exercise and every piece of new information is evaluated and debated in a culture that resembles a research and design (R&D) laboratory.”
• “DevOps must allow this sort of innovation and resulting risks of people making mistakes. Yes, you’ll have more failures in production. But that’s a good thing, and should not be punished.”
• “Robbins defines resilience through large-scale fault injection across critical systems.”
• “First, we schedule a catastrophic event, such as the simulated destruction of an entire data center, to happen at some point in the future.  We then give themes time to prepare, to eliminate all the single points of failure, and to create the necessary monitoring procedures, failover procedures, etc.”
• Examples of failures: “such as conducting database failover (i.e., simulating a database failure and ensuring that the secondary database works) or turning off an important network connection to expose problems in the defined process.”
• “By doing this, we start to expose the latent defects in our system, which are the problems that appear only because of having injected faults into the system. Robbins explains, ‘You might discover that certain monitoring or management systems crucial to the recovery process end up getting turned off as part of the failure you’ve orchestrated. [Or] you would find some single points of failure you didn’t know about that way.’”
• “By creating failure in a controlled situation, we can practice and crate the playbook we need. One of the other outputs of Game Days is that people actually know who to call and know who to talk to — by doing this, they develop relationships with people in other departments so they can work together during an incident, turning conscious actions into unconscious actions that are able to become routine.”
• Benefits of automating in the chat room (instead of running scripts in command line):
• • “Everyone saw everything that was happening.”
  • “Engineers on their first day of work could see what daily work looked like and how it was performed.”
  • “People were more apt to ask for help when they saw others helping each other.”
  • “Rapid organizational learning was enabled and accumulated.”
  • “chat rooms inherently record and make all communications public”
• “In many cost-focused organizations, engineers are often discouraged from attending conferences and learning from their peers. To help build a learning organization, we should encourage our engineers (both from Development and Operations) to attend conferences give talks at them, and, when necessary, create and organize internal or external conferences themselves.”
• Security
• • Static analysis: “Example of tools include Brakeman, Code Climate, and search for banned code functions (e.g., ‘exec()’).”
  • Dynamic analysis: “Examples include Arachni and OWASP ZAP (Zed Attach Proxy).”
  • Dependency scanning: “Examples include Gymnasium and bundler audit for Ruby, maven for Java, and the OWASP Dependency-Check.”
  • Source code integrity and code signing: “All developers should have their own PGP key, perhaps created and managed in a system such as key base.io. All commits to version control should be signed—that is straightforward to configure using the open source tools gig and git. Furthermore, all packages created by the CI process should be signed, and there hash recorded in the centralized logging service for audit purposes.”
  • Environment: “Examples of tools for this include Nmap to ensure that only expected ports are open and Metasploit to ensure that we’ve adequately hardened our environment against known vulnerabilities, such as scanning with SQL injection attacks.”
  • Creating security telemetry in applications: “Examples may include: Successful and unsuccessful user logins, User password resets, User email address resets, User credit card changes”
  • Creating security telemetry in environments: Examples includes: “OS changes (e.g., in production, in our build infrastructure), Security group changes, Change to configurations (e.g., OSSEC, Puppet, Chef, Tripwire), Cloud infrastructure changes (e.g., VPC, security groups, users and privileges), XSS attempts (i.e., ‘cross-site scripting attacks’), SQLi attempts (i.e., ‘SQL injection attacks’), Web server errors (e.g., 4xx and 5xx errors)”
  • Examples of errors that might relate to security:
  • • “Abnormal production program terminations (e.g., segmentation faults, core dumps, etc.)”
    • “Database syntax error”
    • “’UNION ALL’ show up in user-input fields”
  • To protect build, integration and deployment pipeline:
  • • “Hardening continuous build and integration servers and ensuring we can reproduce them in and automated manner”
    • “Reviewing all changes introduced into version control”
    • “Instrumenting our repository to detect when test code contains suspicious API calls (e.g., unit tests accessing the filesystem or network) is checked into he repository, perhaps quarantining it and triggering an immediate code review”
    • “Ensuring every CI process runs on its own isolated container or VM”
    • “Ensuring the version control credentials used by the CI system are read-only”
• DevOps Auditor Defense Toolkit: https://docs.google.com/document/d/1yGSBeKkqhavqk1I21f8UtJBWxDY_Q28nl3SEtcWNFPU/edit#
• “no matter what role you play in your organization, start finding people around you who want to change how work is performed.”
• “Don’t fight stupid, make more awesome.”
• “The wait time is the ‘percentage of time busy’ divided by the ‘percentage of time idle.’ In other words, if a resource is fifty percent busy, then it’s fifty percent idle. The wait time is fifty percent divided by fifty percent, so one unit of time.”
• “On the other hand, if a resource is ninety percent busy, the wait time is ‘ninety percent divided by ten percent,’ or nine hours. In other words, our task would wait in queue nine times longer than if the resource were fifty percent idle.”
• The Facebook Release Process: https://www.infoq.com/presentations/Facebook-Release-Process
• Feature Toggles: http://martinfowler.com/articles/feature-toggles.html
• Splitting the Check on Compliance and Security: Keeping Developers and Auditors Happy in the Cloud https://www.youtube.com/watch?v=Io00_K4v12Y&feature=youtu.be
• The Top Ten DevOps “Operational Requirements” https://www.devopsguys.com/2013/12/19/the-top-ten-devops-operational-requirements/

Advertisement