A/B Testing and Business Success – InApps Technology 2022

A/B testing can bring many benefits to an organization. Basic A/B testing at the user interface level can have an initial impact on business, but full stack experimentation that goes deep into the workings of an application to continuously adjust and optimize code and performance can really move the needle forward for business.

The easiest way to understand the impact of full stack experimentation is to see it in action in a business environment. This is what a group of researchers from Harvard and Duke set out to do, and they recently shared their findings in a report titled “A/B Testing and Firm Performance.” The research combed through data on more than 64,000 firms — evaluating the use of A/B testing against KPI’s like online traffic, VC funding and other metrics.

Overall the research found that A/B testing has a positive impact on business performance as a whole, however, there are some specific areas where the difference can be felt the most.

Hitting the Testing Sweet Spot

During the course of their research, the Harvard Business School’s Rembrand Koning and Duke University Fuqua School of Business professors Sharique Hasan and Aaron Chatterji pinpointed that bigger, younger, online-focused firms were both the biggest adopters and the most likely to benefit from full stack A/B testing.

Although this may feel like an obvious conclusion, the age of the company has the smallest correlation to the impact of A/B testing. However, the intersection of young companies who have grown quickly is where A/B testing is at its most impactful. But why are these two factors so important?

The answer is in the way that these businesses are built. A/B testing is a go-to strategy in product-based businesses, and product-centricity happens to be the most intense online because it is the most significant source of value for these companies. From a size perspective, when you are operating with a large customer base, even the slightest competitive advantage is magnified by the scale of operations. For these reasons, the biggest benefits from A/B testing can been seen among big, online companies.

Winning the War of Inches Through Testing

The research done by the team at Harvard and Duke puts product-centricity, obviously, at the center for its study. The researchers found that firms with product managers benefit twice as much from using A/B testing software when compared to those without product-focused teams.

This is reaffirmed by the fact that product development is complex, ongoing and incremental, making these teams hard to staff and stand up. Just like most football games are won more often than not by small, disciplined plays up the field, not long yardage passes, product supremacy is most often a war of inches. Full stack A/B testing is perfectly positioned to help win this type of war. So, when products are your bread and butter, driving consistent, small improvements that combine and scale over many users can create game-changing results.

Testing Brings Home the Bacon

Another key statistic in the study is that three-quarters of technology startups do rigorous A/B testing. It also found that embracing A/B testing and the tools that enable it brings modest yet statistically significant increases in the probability of raising initial funding and the number of subsequent funding rounds a company can expect.

This attractiveness to VC funds has a common factor — confidence in agile and DevOps practices. Incremental testing, automated provisioning and continuous build, integration and delivery are all DevOps principles that define startup operations generally and full stack experimentation in particular. In other words, when you embrace full stack A/B testing, you’re speaking the language of agile/DevOps, and VC rainmakers are currently responding to that.

In fact, whenever the researchers are ready to dive deeper into this topic, it would be interesting to understand how the maturity of a company’s A/B testing corresponds with its overall agile/DevOps continuous delivery model. I would predict a strong positive correlation since it would be an impressive feat and a missed opportunity to be mature in one but not the other.

For technologists, who are often knee-deep in the minutiae of data infrastructure, execution logic and engineering complexities, it can be extremely rewarding to see how all their work impacts the whole business. The more we understand and communicate how product and performance testing impacts the overall business, the more our efforts will pay off in providing a competitive advantage and bring concrete value to all kinds of businesses.

Feature image via Pixabay.