Who Is in Charge of Developing a DevOps Culture? – InApps 2022

CloudBees sponsored this story, as part of an ongoing series on “Cloud Native DevOps.” Check back through the month on further editions.

Want to understand the cultural barriers to your DevOps transformation? It often helps to have an outside perspective. InApps talked with two consultants who are brought in once a company’s C-levels starts to realize that they aren’t moving as fast as they would like to. Organizations are struggling with outages and delivering customers joy, fighting to get sysadmins to release the reigns, and they just don’t know what development and operations are doing, who in turn don’t know what value they are delivering.

In short, they want a DevOps transformation, but their culture is nowhere near ready yet. What kind of culture do you need to build? Who leads that culture? And who simply can’t be a part of that fast-moving, autonomous future? Today we dive into that and more.

DevOps Is Not for Everyone. And That’s OK.

Consultant Liz Warner, currently with fintech consultancy 11:FS, comes from the unique perspective of an interim CTO. She often comes into companies to help kick off or re-energize a DevOps transformation.

At what point does she usually come in?

She says, “People say ‘Hey we’re struggling. We hit a wall. We’re growing very quickly and our current process doesn’t scale. Or we have a process and it should be working but we have a lot of outages, our customers hate us’,” Warner told InApps.

She’s come onto startups as small as three people, and to companies with tech teams of about a 100, where their current CTO might not yet have the skills for running a larger org.

With DevOps, “a key part of the culture shift is how you approach technical change. Change is scary. Making changes to a live system is scary. Things can break. The DevOps way of working is to make change less scary by changing constantly.” — Liz Warner

She pointed out that the DevOps approach, where in some cases everyone can deploy to production, is diametrically opposed to the norm at many companies, where you keep technical systems “safe” by keeping technical change rare, and entrusting change only to a small, privileged group.

Warner says her job involves a lot of talking to people, listening:

• Can you trust other people on your team?
• Are you willing to do some introspection, and to improve based on what you learn?
• Do you believe automation processes can make change safer?

In the very traditional financial space, if even senior or middle management aren’t committed, Warners says you’re destined to hit a wall.

“Somebody will want to put controls in that slow you down and interfere your journey to rapid change, and you can end up in the worst of both worlds,” she said. “Any company can change their culture if they’re committed.”

She paraphrased agilest Mike Cohn saying a good test of agile or DevOps maturity is what choices are made under pressure. “Do you trust this way of working enough to lean into it in a crisis when you really need to, or are you kind of trying it?”

The Three Commitments You Need from any DevOps Teammate

Warner says there are three non-negotiable needs to a DevOps organization that must be made clear to everyone in the org and even key customers. There must be a company-wide commitment to:

1. Transparency
2. Autonomy
3. Automation

Warner called companies that even move toward radical transparency “really healthy.”

“People should understand shared goals. The sort of closed, need-to-know thinking gets in the way of product and technical delivery. Everyone should know how systems work if they are interested.” — Liz Warner

Warner talked of one contract she worked on in which only two people — “the priesthood of sysadmin” — knew how key systems worked, for fear of reverse engineering.

This gatekeeper culture means no one can improve things, no one can point out scenarios that won’t work, and, when those two sysadmins eventually leave the company, it’s really in trouble.

Warner says Jira and Confluence are solid examples of tools that promote such information radiation in a wiki-like way, where anyone can comment and make suggestions for improvements.  But the process is more important than the tools.

“Open by Default is a good starting place for DevOps culture — especially combined with introspection. What can we do better? How did we do last month? Last week?” Warner said, suggesting you carve out space in people’s work cycles for feedback and improvement.

Next for Warner comes a commitment to autonomy, saying DevOps orgs need to hire the right mixture of people, and make sure engineering teams understand the business. This includes making sure engineers know regulatory constraints and what customers and partners are expecting from the code. Make sure technical teams understand business problems, then give them freedom to come up with solutions to those problems.

She says part of the job is continuous learning. There is always a better tool or library and so many different solutions to existing problems that you get better results with more individuals who are continuously learning toward collective objectives.

Challenge everyone in a DevOps org with continuous feedback loops and actually making changes based on the feedback. — Liz Warner

Finally, of course, Warner sees that DevOps transformation requires a commitment to automation. Warner says if you want to have loads of tiny safe changes in a process where any engineer can trigger a code release, you need to make sure your automation — which includes infrastructure as code, continuous integration, and test automation — is in excellent shape, and that you are using automation at every stage, including production.

When someone says “I’ll automate everything up to production but then I don’t feel safe until I do things manually,” they aren’t ready for DevOps style rapid change.

Just remember, as Warner said, “If tech is driving for a collaborative culture but the rest of the company isn’t interested, it’s not going to work. It has to be across senior management and product.”

Pair Programming Builds Empathy Across Departments

In her role as engagement director at Pivotal, Ashley Hathaway works primarily with operations teams at very large banks, engaging with dev teams regularly. Time and time again, she sees upper-level management as the major hindrance to a DevOps culture change.

She says you need executive support from the top down, with “buy-in of leadership and management saying we will not hold you to shipping by a certain date.”

Hathaway explained that, with DevOps, you remove the deadline drive and ship with test-driven development (TDD). She said this removes the need for shipping code really fast, as you are officially choosing quality over speed.

At Pivotal, every engineer pair programs all day, every day. She recommends her clients kick off DevOps with pair programming or, at minimum user interviews between ops and devs, so each side can see what the other is doing.

“For our clients, it [pair programming] seems counterintuitive. ‘Now I’m cutting my time in half.’ But it actually strengthens code checks. You write better code. Then the domain knowledge doesn’t leave if someone is sick. Everyone deploys on the same pipelines, writes their own pipelines with continuous delivery,” Hathaway said.

Of course, DevOps is all about breaking down silos, but some silos are necessary, especially in banking security. Hathaway says, with banking regulations, the person that writes the code can’t ship the code. This is a bonus of Pivotal’s pair programming policy as two people together can continuously ship and test the code, just with checks along the way.

She told InApps of an app development team that saw great success when they talked to an ops team.

“Just the interaction of that one application group, those devs giving their feedback to the ops team, was a lightbulb moment for them,” she said, pointing out that ops wasn’t talking to their developer users before.

She says putting the devs who are using the infrastructure, system rules and integrations every day in front of the ops team is a logical first DevOps conversation.

 Middle Management Will Make or Break It

Hathaway says that CTOs and CIOs have usually bought into DevOps, but it’s one chain of command down that doesn’t interact with the dev and ops teams every day, but who oversees the success of the product, that can be the DevOps downfall. These are the roles who need to get on board with DevOps early, helping to form the product statement and drive toward actionable user feedback.

This upper middle management need to be “thinking about the platform as a product and these applications as products not projects. It gets away from shipping deadlines [and moves toward] interacting with each other and getting feedback from each other even when you are big groups that are separate.”

Hathaway says this helps everyone start to prioritize the work more clearly.

This is especially true in the transformations she is assisting because many of the people at this level have worked at these banks for at least 20 years.

She says it’s not so much about the fear of change, but rather the fear of failing that you need to get over to embrace DevOps. And she said devs and ops are the most open to change, it’s the higher up, that have to be convinced.

“It’s the people above them — that are most scared to miss a deadline or to go over budget,” Hathaway said, naming common culprits such as waterfall and middle management.

“It’s that group that makes the devs and operators in my experience miserable. And they then craft a story to their upper-level managers.”

Hathaway says C-level has to intercede before it’s too late. These are the ones lauding DevOps culture while still pushing too hard on those reporting to them.

“The middle managers are tasked with change the culture and make this successful, but ‘Oh, by the way, we need this yesterday’.”

She gave the example of one line of business at one of her big banking clients which was worth millions of dollars. The CTO came in and said, “I don’t care if we ship late. I don’t care if things break. We have to change things from the ground up and slowly and it’s going to be painful way more than a win.”

Hathaway said that is now the most innovative line of that large bank, and they act as an example for the rest of the company. They were OK not hitting budgets or deadlines for a couple months, and instead they are now successful, with lower turnover and happier employees.

“From a grassroots perspective, it’s devs actually having fun and learning new things every day. A healthy culture will draw people to DevOps.” — Ashley Hathaway

It all comes down to middle and senior management buying in for the change, and then setting expectations for a bumpy, but worthwhile ride.

What if middle management won’t change? Even trying to do pair managing, says Hathaway.

“Link up your weaker people with your really strong performers to see what different thinking looks like because it is a practice in thinking. They need to know it’s OK from their management.”

Feature image via Pixabay.