Bringing DevOps to the Full Stack at Colonial Life – InApps Technology 2022

Colonial Life (a subsidiary of Unum Group) is a nearly 80-year-old voluntary benefits provider that found itself where many established organizations find themselves of late: in the middle of a critical effort to improve software delivery. This initiative was driven by pressure from the business to deliver software more quickly and with higher quality to remain competitive in the market. In response, we looked to Agile and DevOps tools and practices, and we knew that automation was going to be essential to our modernization effort.

Prior to the transformation, we made in our software lifecycle and delivery, building and deploying new code across our Development, Acceptance, and Production environments was a largely manual process. If anything went wrong, all the build and deployment work had to be redone manually.

We have a Microsoft-centric software stack, meaning that on the application code side, dynamic-link libraries (DLLs) were built from source and the deployment to each environment was done manually using a script that would copy over the DLLs. The database side was even worse, as SQL Server database administrators (DBAs) had to manually deploy SQL scripts to each environment. All of this manual intervention made the entire process risky, unreliable, and slow.

To get a sense of scale, our largest deployment had over 900 different application version scripts that had to be executed by a DBA against a database. Given the extent of the manual effort, it was common for steps in the database deployment process to be accidentally skipped or left incomplete, causing production issues that had to be triaged and addressed. To make matters worse, deployment windows were often late at night or early in the morning, and DBAs may have had to support multiple releases for other applications simultaneously, so they were incredibly strained. Things needed to change.

In 2016, we began an Agile transformation with the goals of enabling the business to release more frequently, providing value to end users with greater speed, quality and reliability, and giving our engineers more time to spend on feature development and delivery.

We started by transforming our largest revenue-generating app — the app that was responsible for enrolling users into our insurance programs. Not only was this app critical to our business, but it was also one of the most challenging in our entire portfolio to manage — typically requiring 15 to 17 hours of manual effort over the weekend for a release. Though it may seem counterintuitive to start with a large legacy app that wasn’t built for the age of automated test and deployment, we felt that if this one could be modernized, then every other app in our portfolio could be as well.

With all this automation in place across the entire stack, we were able to triple our number of builds and do an order of magnitude more deployments in lower environments. For overall business results, we’ve gone from three painful, risky, manual releases to seven automated deployments per year.

While undertaking the gargantuan task of modernizing our enrollment app, we faced a lot of skepticism. After all, we were changing well-established processes that had been in place for years, and many engineers who were comfortable with the existing process were reluctant to change. To build confidence in the new tools and process, we managed to find a small greenfield application team about to begin on a completely separate project and decided to apply the new technology and process to that team in parallel to the work we were doing for the enrollment app.

This approach served as our testing ground and it paid off well — while the greenfield team was unsure about doing things so differently, at first, they quickly realized substantial productivity and quality improvements. This small victory earned us a group of internal champions, which helped us earn the necessary buy-in from the rest of the organization to use these new tools and processes for the larger project of updating the behemoth enrollment application.

Regarding the actual continuous integration (CI) process, we invested in Jenkins, Datical, JFrog Artifactory, Octopus Deploy, and SonarQube. Developers continued to use Visual Studio and SQL Server Management Studio for application and database development respectively, and we continued to use Team Foundation Server (TFS) for source code management and Agile Central from CA for backlog management.

Our CI process now has two parts. The first part is a nightly Jenkins build that allows teams could get daily feedback on the entire stack — both application and database code changes. The second part is a triggered Jenkins build that occurs whenever application code is checked in. Since application code is faster to build, these triggered builds provide developers with more immediate feedback on application code changes during the course of the workday.

A diagram of Colonial Life’s nightly CI process

With all this automation in place across the entire stack, we were able to triple our number of builds and do an order of magnitude more deployments in lower environments. For overall business results, we’ve gone from three painful, risky, manual releases to seven automated deployments per year.

Having gone through this journey, my best advice is to first bring automated build and deploy steps into your software pipeline and defer the bigger re-architecture projects that would require a heavier lift. You will get much faster time to value with pipeline automation than if you were to rewrite your legacy apps using a cutting edge microservices and container architecture.

That said, I also advise taking stock of your company’s expectations and needs — for Colonial Life, tripling the number of builds delivered enough value to justify deferring the re-architecture work, but the necessary results and mileage may differ for others. The point is to tackle the smaller project of pipeline automation first, assess the lift — you might be pleasantly surprised by the ROI, and then commit to bigger re-architecture projects afterward.

What’s next for us at Colonial Life? To expand the pattern we’ve validated on one of our largest applications across the entire enterprise and bring the same lift to all the other apps in our portfolio. Onward and upward!

CloudBees is a sponsor of InApps Technology.

Feature image via Pixabay.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: JFrog, Octopus Deploy.