Building a DevOps Culture

Chapter 4. Ready, Set, Change

One thing is for sure: we cannot “greenfield” human beings the way we can tear down an application and rebuild it in another environment. We can’t theorize about spherical cultures in a vacuum to simplify our equations. We see this every day in society; the residual effects of cultural change writ large in our day-to-day lives. The Civil Rights movement in the United States didn’t wipe out the previous culture and start over; there are still people whose opinions about such things are “from another era.”

People come with baggage: preconceived notions, past experiences, prejudices. When these individual characteristics are at odds with the culture that we want to foster in our organizations, we create stress. Alleviating that stress means either changing the person or changing the organization they belong to.

Changing the values of our organization is a challenging task, one that has been under study by management scholars for decades. There are numerous interesting case studies about disastrous cultural change initiatives, failed cultural integrations from mergers, and investigations into what makes cultures change. We are not alone in our struggles to better serve our businesses by improving our work cultures and processes. Searching on the Harvard Business Review website for “cultural change” will get you 60+ publications going back nearly 30 years.

One recent anecdote that I found particularly interesting was buried in a story from The Atlantic about recent trends in repatriating manufacturing work to the United States after years of moving those tasks abroad for labor cost savings.

The story of GE’s GeoSpring water heater is fascinating, regardless of the underlying drama of globalization and outsourcing. The product was designed in the U.S. and the specifications were sent to Asia for production; the manufacturing process became a black box that the design team sent instructions to. When looking at how to manufacture this particular product in the U.S., they brought together “design engineers,…but also manufacturing engineers, line workers, staff from marketing and sales” to discuss how best to build it, and discovered the design was “terrible” and no one actually wanted to try to build it.

It sounds like Development wrote some code and “threw it over the wall” to Operations to figure out, doesn’t it?

GE’s integrated team sat down and redesigned the product with input from everyone, and ended up building a better product, at lower cost. It’s what we want for our technology products too, the ability to best serve the business and create value. The cultural shift is acknowledging that everyone has valuable input and communicates. This was a DevOps cultural shift. It is a change in the value of some new behaviors versus the old ones.

Know Where You Are Starting

When we first join new organizations, we go through a learning process, discovering the mores and acceptable behaviors of that organization, be it a new company, a sports team, or the PTA. It’s common to get different answers to questions like “Why are we doing X this way?” from different people. When we hope to change the prevailing culture of the group, we need to assess the current state of that culture, and ask all of the questions about why we’re doing X the way we are.

The one question you must ask is “What is our primary business goal?” You need to know what people have decided is the motivating factor for the business as they know it. If the organization’s leadership has done a good job of communicating the organization’s goals, you’ll find most people are on point. It’s not uncommon, though, to hear some folks claim that the business goal is “We’re a content company; we produce awesome content for our users” and others “We’re an ads-placement company; we sell ads to make money for our advertisers.” Or maybe, “We provide a stable application for our users” versus “We are driving innovation and leading the competition in new features.” Are these goals mutually exclusive? No, but they represent a difference in viewpoint that will color people’s actions and decisions when it comes to the products they work on.

In an ideal situation, you’ll have some sort of consensus. Even if you have two related goals, you’re doing mostly OK. The comedy of errors starts when you have several divergent ideas of what the company is actually trying to accomplish. The reasons for this often include poor communication from management as to what the company is driving toward. It’s also possible that fuzzy goals like “serving the customer in the best way possible” are different for different teams in your organization, especially if they have different “customers.”

Now that you know the goal of your business, you can take this knowledge and focus your DevOps cultural change toward enabling this business and working toward this goal.

Know Why You Are Changing

Proving change is necessary requires some legwork. It’s fine to want to change your organization because “everyone” is doing DevOps now, but you’re looking at months of work, new tools to learn and implement, teams to restructure. These costs must be outweighed by the benefits, so you have to be able to put real value on your processes.

Articulating upfront what your goals are will help you with other phases of your DevOps roll out. Some common measurable goals are:

• Reduce time-to-market for new features.
• Increase overall availability of the product.
• Reduce the time it takes to deploy a software release.
• Increase the percentage of defects detected in testing before production release.
• Make more efficient use of hardware infrastructure.
• Provide performance and user feedback to the product manager in a more timely manner.

These goals have monetary impacts on your business. It costs you people-hours when your software releases take days. It costs you revenue when the product isn’t available for use by your customers. It increases your operating costs to run your infrastructure in a non-efficient way. Taking what you know about these values, you can put a monetary value on improving any of them.

Additionally, all of these goals can have a numerical threshold attached to them. “Reduce deployment time from 10 hours to two hours.” “Increase percentage of defects detected in testing from 25% to 50%.” You can reason about these numbers and judge the success of your transition process.

These goals also give you a stable platform for obtaining support from the rest of your organization. Starting from metrics such as “It currently requires six team members 10 hours to deploy a new release of our product. This costs us $X for every release” provides a concrete number for comparison as you implement your DevOps changes. It attaches real value to a real pain point in a way that is more constructive than “Our deployments suck.”

Executive Sponsorship

Getting most folks on the same page is a job best sent up the org chart as far as you can push it. In order to make changes to things like compensation and incentives, you need to get management buy-in at a high enough level that any necessary changes can be facilitated without a battle. This is one distinct advantage small organizations have over large ones, the ability to make tweaks to core human resources functions in order to influence the behavior of individuals.

Having an executive sponsor is also helpful when spreading the good news about your cultural adjustments. It gives your teams a feeling that they aren’t going off on a hunt for DevOps nirvana only to have the plans changed on them a few months down the road. Resist the urge to start a grassroots movement; in this case, getting permission comes with more resources than asking forgiveness later.

Your executive sponsor is going to be particularly interested in the measurable goals you’ve developed, and will likely have their own that might fit in with your DevOps goals.

Technical Leadership

In the process of implementing a DevOps culture, you will likely be asking your technical staff to change their tools, behaviors, and workflows. You will want someone on hand to be your go-to person to talk about DevOps to your teams. It might be you! And that is awesome! It could also be an architect or other senior person who is well liked and well respected, or a few of these folks so you don’t overburden a single person.

This person, or team of people, will serve an important role, a combination of evangelist, tools expert, process subject-matter expert (SME), and buddy. They’re like your camp counselors. They’ll teach a little bit, answer questions, reassure the reluctant, and bring the marshmallows at the end of a long day. These folks are hard to find, and often aren’t who you think they are. The last person you want for a job like this is someone who’s smart but a complete jerk everyone hates but who happens to know how to use the tools. Find the people who everyone wishes were on their team, borrow some of their time, and form a working group to help other teams.

Without these folks to shepherd the various factions of your organization, it’s easy to lose a few sheep along the way. Your SMEs can be there to build interpersonal relationships and open communication paths with outside teams, cutting down on friction and politics.

Pilot Projects

At this point you know you’re going DevOps. Your management is on board and ready to help as necessary. Your tiger team is only a couple people so far, but they’re enthusiastic and well respected. Now what?

Now you need a pilot project. You have to find somewhere to start in your project portfolio. You are going to transform not just the product that the pilot team is building, but the processes, workflows, and team dynamics. As you’ve been talking to people in your company, getting feedback on things like goals and who’s awesome, you’ve probably piqued the curiosity of more than a few folks. You might have multiple projects that want to try out your fancy new DevOps processes that you can choose from to start a pilot.

What makes a good pilot candidate? Lots of different project characteristics can make a good pilot. Most importantly, you don’t want to start too big the first time out. Rolling out the DevOps machine for a key central service is going to be hugely frustrating while you’re figuring out which tools work best for your environment and how to improve your communications and processes.

You also don’t want something that is too small. It’s easy to dismiss a tiny pilot project with “Oh, sure, that team can do DevOps because they’ve only got five people. We have six in QA alone; it will never work!”

The first time out, try for a project team that is collocated. Changing up process and communication is hard; doing it across five timezones is a good hurdle to avoid if you possibly can. If all your teams are distributed, look for teams with a long history of working successfully across long distances. Their team culture may already reflect some of the characteristics of a DevOps culture that you’re looking for.

Set Goals

Remember those goals from earlier? Now you’re going to apply them directly to your pilot project.

Choose the goals that most make sense for the individual project, be it reducing time to market, increasing efficiency, reducing deployment time, or whatever. Talk with the project team. They have problems they want to improve, pain that needs to be alleviated.

Finally, give these goals a timeline. If your organization is using an agile methodology, work with that, do sprints, reassess, set new goals, keep improving incrementally. You’ll be taking time to install and learn new tools and processes, so at the beginning work may seem slower than you think it should be.