How KeyBank Used DevOps to Overcome Cost, Complexity, and Legacy Constraints – InApps Technology 2022

Two hundred years. That’s a lot of legacy to deal with. Now it’s only possible to have legacy software dating back for maybe a quarter of that time, but KeyBank — with close to 20,000 employees, a ranking among the top 15 banks in the US, and $137 billion in assets to juggle — has felt the pain of legacy and has leveraged DevOps to drive business into the twenty-first century. KeyBank’s Senior Vice President of Technology Infrastructure, John Rzeszotarski shared with this year’s European DevOps Enterprise Summit his observations of how the bank tackled the constraints of cost, complexity, and legacy.

One Year of Acquisitions, Containers and Weekly Releases

Sudden architectural changes within companies usually happen for two reasons — a cultural imperative or a business one. At KeyBank, it was certainly the latter. Back in the Autumn of 2015, KeyBank announced a deal to acquire First Niagara. Rzeszotarski says this demand to put a new digital application in place and then migrate First Niagara’s customers over quickly and securely saw the tech side of KeyBank evolve rapidly.  The project began in December 2015 and, by July 2016, the bank had online and mobile banking up on a new, somewhat daring infrastructure.

“Back in 2016 no bank was running [its] online banking on Kubernetes and Docker,” Rzeszotarski said in his talk.

This acquisition acted as an impetus that took the bank through a year of rapid change through to October 2016, where it was able to securely and stably make 12 changes in the first four days of the app releases, allowing for fast fixes on things like user experience. Kubernetes enabled it to go to rolling deployments. Rzeszotarski says the main benefit was that Kubernetes allowed them to install new versions of the applications, while still managing the other applications.

Rzeszotarski’s talk built on KeyBank’s story that had been shared during previous talks, this time focusing on, after a rapid period of growth, how it figured out the new business as usual.

After a Period of Rapid Change, How Do You Decide Next Steps?

After the First Niagara acquisition was completed and these apps were released, Rzeszotarski said: “Since we didn’t have these catalysts we find it challenging to move fast.”

The bank still had many applications that could benefit from these best practices, and it still had legacy systems, which it relied on integration with that needed some attention.

According to Rzeszotarski, it all comes down to “It’s difficult for us to see what has to change with different groups having dominant strategies.”

For him, in banking, that comes back to the need for operations,  security, development, risk, and architecture to move ahead, and to find compromises wherever possible, but just to work to avoid holding each other back.

With this in mind, Rzeszotarski and his team chose these areas of constraints within KeyBank’s tech operations:

• Cost — How do you do more with less?
• Complexity — How do you manage large enterprises which can be highly segmented with many different complex integrations?
• Legacy — How do you modernize and deal with legacy systems, including some that do not have roadmaps?

By organizing how the bank viewed its constraints, it was better able to organize its priorities in overcoming them, while still making the security side happy.

How to Change the Game of Costs

OK, this is the logical constraint that all companies have. Rzeszotarski said that most enterprise infrastructure budgets are going down, but that’s at least in part because infrastructure resources are usually getting cheaper over time, including cost-cutting virtualization and containers, so the bank is able to get more volume on existing resources.

In addition, open source began to dominate new tech adoption decisions.

“Not because open source is free — because it’s not free by any means,” Rzeszotarski explained. “It does allow you to start innovation without a significant investment. It does let us pay for what we need for the enterprise — not more than we need.”

The bank still pays for the enterprise support, but he says the typical open source subscription model is much more fair market value for the products it is purchasing “ versus long-term unfair licensing agreements” that usually comes with proprietary software.

How to Change the Game of Complexity

The number one game changer here was definitely containers and Kubernetes.

Rzeszotarski emphasized the importance of containers as “Rather than have to have 15 teams have to put their hands into the solution to release it, it’s all built within code. It’s reusable. You’re putting that responsibility back to the app team to manage it and we are making it testable and we are using Kubernetes to put the rails up to guard it and make sure it’s highly available and highly reliable.”

He continued that “For application frameworks and platforms, you’ve eliminated patching, upgrading, and hot-fixing solutions because you’re just deploying on top of the applications. It eliminates the complexity of cross-organizational teams.

Rzeszotarski told InApps Technology about the benefits of containers and Kubernetes: “It makes it more accurate — whatever you build once, we can test at a lower environment and then that image can’t change in other steps.”

He did say that, while that combination was great, it still hasn’t given them a fool-proof way of managing that infrastructure. The next step in overcoming the complexity constraint was definitely addressing integration with the legacy infrastructure.

How to Change the Game of Legacy Systems

For this, the team at KeyBank went the traditional route of following Martin Fowler’s advice, with evolutionary database design, the Circuit Breaker, and the Strangler Application.

An off-shoot of agile methodologies, evolutionary database design is built for continuous delivery and to allow for both database and application changes without any downtime. The rules of evolutionary database design are as follows:

• Database administrators collaborate closely with developers.
• All database artifacts are version-controlled alongside the application code.
• All database changes are migrations.
• Everyone gets their own database instance.
• Developers continuously integrate database changes.
• A database consists of schema and data.
• All database changes are database refactorings.
• These refactorings are automated.
• Developers can update their databases on demand.
• All database access code is clearly separated.
• Releases happen frequently.

At KeyBank, following Fowler’s practices means “We’ve now successfully tested a database change before we release it just how we use it in the container world,” Rzeszotarski said.

Now, this is an ongoing process at KeyBank, which means a lot of database environments aren’t meeting these standards yet and Rzeszotarski says it doesn’t always work in banking, which involves integrations with so many external partners.

“What if I have to integrate with a vendor who doesn’t version their services?”

He says the bank is often defending themselves against its partners. In order to do this, the bank follows a Circuit Breaker Pattern — like when something is off, we usually flip the switch on only one circuit, not cutting the electrical supply to the whole house. KeyBank’s applications integrate with a lot of external partners, including bill paying, person-to-person payments, and credit card services.

“Rather than an app starting to fail or even slowing down, we defend our application in circuit breaker,” Rzeszotarski said.

The bank automated this with Netflix’s open-source tool Hystrix. He says this tool looks for closed and open circuits, and, if requests are failing, it will turn off the connection with that piece. It will continue to try a request every second until it’s safe to flip it back on. This means that while one service may be temporarily down, customers can still check their balances and make transfers.

The bank is also following Fowler’s Strangler Pattern to safely and efficiently replace legacy systems. Named after a kind of Australian vine that eventually chokes trees to death, this involves slowly chipping away with the legacy app until it’s gone — not rewriting it. Rzeszotarski says if you try to do it all at once, you create way too much risk that you may not understand and you’ll never be finished with any legacy piece because you’re always going to want to update it with new features.

On the problem partner side, he says to apply the Strangler Pattern “It’s a little bit of phasing it out with the partner and coordinating on timelines and give and take with both. There’s no doubt that partnerships with our vendors are important to us.”

Rzeszotarski pointed out his views were his own and did not represent the views of KeyBank.

Photos: European DevOps Enterprise Summit Twitter

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.