Intuit’s Developer Experience with Kubernetes – InApps Technology 2022

Intuit is fundamentally a technology company — one that has been around for over 35 years. “If you think about any kind of legacy product, it probably exists here in some fashion,” says Pratik Wadher, vice president of product development at Intuit. When the SaaS versions of the financial service giant’s QuickBooks and TurboTax came out 10 years ago, it was all hosted in on-premises data centers. But around six years ago, as the cloud market maturated — and, for a financial services company like Intuit, started having more robust security and compliance controls — the company started moving to Amazon Web Services.

“It’s been mostly a lift and shift approach,” Wadher says. “As we migrated, we saw that our availability definitely improved, our ability to manage infrastructure and isolate infrastructure from developers improved. But what was not improving was our developer velocity, our ability to release faster.”

Intuit’s release cycles, after moving to the cloud, were still between three weeks and three months. The Mean Time to Repair (MTTR) after an incident improved slightly but was still in the 45 minutes to an hour range. Knowing that this was because the application architecture was essentially the same as it had been on-premises, the company started looking to re-architect the applications. The Kubernetes journey started around two years ago when Intuit started moving test infrastructure and other pre-production workloads into Kubernetes. 

Even though there was some immediate improvement in developer productivity, it became clear that Kubernetes was not well integrated into the software development lifecycle. In fact, Wadher came to Intuit when his company, Applatix, was acquired specifically to help Intuit manage the CI/CD process in Kubernetes.

We chatted with Wadher about Intuit’s Kubernetes journey, the challenges on the way and what the company hopes to see change in the future in terms of developer experience with Kubernetes. Here’s a condensed version of our conversation.

Intuit has 5,000 developers that need to stay productive every day while also adapting to major changes in technology and organizational structure. How did that process go? 

One of the things we wanted to do was to free up our developers so that they themselves could make the decisions about scaling their infrastructure, rolling back, doing failovers. We were essentially adopting DevOps, but that’s easier said than done. We had questions like, how do we migrate? How do we take our SRE teams and embed them into the service team so that they have the necessary expertise? How do we enhance our development pipeline so that it’s very easy for developers, so developers don’t need to know Kubernetes or containers? We wanted the rollover or failover process to be possible with the push of a single button. We wanted upgrades to be as simple as perhaps checking something into Git.

There was also an education component. Even though we want developers to mostly stay away from handling Kubernetes directly, they still need to have an innate understanding of what they’re running on. We had 1,500 developers who did our Kubernetes training.

We also built a whole new development pipeline, which cut the amount of time it took to create a new service from weeks down to ten minutes. Building this paved road for service development involved cobbling together existing products, creating new products and adapting some open source tools.

That sounds like quite a Kubernetes journey. Are you there yet? 

It’s not quite at the level we would like. For example, if services have a hiccup or a Kubernetes pod goes down, developers still need the level of knowledge to look at the logging and understand what happened. But they don’t need to understand how to manage clusters or namespaces, they don’t have to deal with auto-scaling.

What’s the biggest unsolved issue you have at Intuit related to Kubernetes? 

One of our main issues at the moment is around logging and monitoring, what I call mean time to detect (MTTD) and mean time to identify (MTTI). When there’s an issue, it’s still very difficult for us to identify where the problem is and which service owner needs to get called to address this. We’ve been spending a lot of time and energy recently trying to get the same clarity and visibility that we have at the cluster and infrastructure level and extend that to the service level.

For us, the developer experience journey also continues. This process I’ve been describing was only without existing applications, most of which were written in Spring Boot for Java. So the developer paved road was built on that model, and everything, from the sample applications to the pipelines was built on Spring Boot assumptions.

Even though we want developers to mostly stay away from handling Kubernetes directly, they still need to have an innate understanding of what they’re running on.

But now, one of our focuses is on expanding our use of artificial intelligence and using Kubernetes to build machine learning platforms. So that’s a completely new journey for us that we’re still perfecting.

As a FinTech company, we’re obviously also very interested in everything related to security and compliance. As a project, Kubernetes has never really focused on these areas, but we’ve been working with the broader community to step up the focus on security and governance. If more enterprises are going to adopt Kubernetes, security and governance will be key technologies.

Can you talk about some positives about the developer experience using Kubernetes at Intuit?

Because of our industry, we’re paranoid about data security and privacy. But security is one of the things that developers hate. So we have this pipeline of containers and workflows, and we have been able to integrate the security tools we need into the developer pipeline. So when they do the build, the security analysis is already done for them, when they deploy, the runtime tools are already integrated. That’s a big win from a developer perspective.

We’ve also built a number of other products in-house so that we could operate at scale. For example, Intuit Kubernetes Service Manager, which lets us update an entire suite of clusters with a new AMI image in one click.

I’m also definitely starting to see more focus on the developer experience in the broader community, projects that focus on ease of use and integration with lifecycle management tools.

AWS is a sponsor of InApps Technology.

Feature image by SashSegal from Pixabay.