Business research is a critical part of how management learns to guide business growth. The more productive our Business Intelligence activity is the more we learn, and thus the better positioned we are to be successful. “BI Driven Growth”, simply put, is the part of a business’ growth rate that is a result of improvements in our know-how due to research.
This article will address some of the common choke points that keep business research from driving real company growth and in so doing make these two diametric points:
- Two out of three of the necessary requirements for research driven growth are non-technical
- Improving the methodology alone of the typical A/B Testing can yield 200%-1,000% more research driven growth
The 3 components of business research that drive company growth
To achieve anything via business research you need 1) good ideas to test, 2) effective evaluation of those ideas, and 3) the findings have to be implemented. It’s often difficult to bring together those three basic ingredients due to the divergent personal goals, the hand-offs between groups, and sometimes the lack of effective communication about the business imperatives driving the desired change.
Whatever the causes, it’s true that many useful findings do not get implemented in businesses. That reality makes analytic leader’s with good skills at building alliances, persuasion, and high credibility a must, because otherwise the majority of the benefit stream can be forfeited.
Similarly, good ideas are neither the sole domain of the researcher, nor should implementation depend upon whose idea it was. Hiring out of the box thinkers, which are tenacious and thick skinned enough to keep contributing potential ideas, even if not every idea tests well, is key to generating a constant stream of productive innovation and improvement.
A/B Tests are Reliable, But Limit Findings
What is an A/B Test and Why are they Popular?
A/B tests attempt to form two equivalent groups, and make a single, or set of changes, to one of them, and then make one comparison. (A vs B, Test vs Control, Champion vs Challenger, other equivalent names) It’s a reliable methodology because everything, except what’s being tested, remains the same. It’s also simple and easy to explain. Those strengths have driven its popularity and for good reason, since having hard to challenge results and being able to communicate those results are key factors to BI productivity.
How do they limit findings?
The simplicity of the A/B test is also its weakness. Inevitably the researcher will find himself in one of two scenarios: 1) There are important interactions – Mitigating factors that limit when and how the findings apply, or 2) There are no important interactions – Test findings are independent of other factors. In either of those situations an orthogonal multivariate test would produce more detailed results using the same data. In the first case it could account for the interactions and provide a fuller understanding, and in the second case it could identify several times as many relationships using the same data. Because the benefits are distinct in the two situations, I’ll cover each, but the net takeaway is that it’s possible to increase the number and quality of findings by orders of magnitude.
Multivariate Tests vs. A/B Tests in the Presence of Interactions
One of the problems with simple test results is that people tend to think of the result as universally true. Any test that comes out as “better” tends to become the general rule. Thus if no caveats are given, users tend to think of the finding as always true. This can lead to confusion as more tests are run and what has been generally understood is called into question. Take the following credit card example.
Price increases can result in higher total profits if the number of sales lost is small compared with the increased per unit profit, but as prices go higher too many lost sales will result in reduced profit. Consequently taking the two price increases together does not necessarily result in the sum of the first two test results. Additionally, there are groups of consumers who are more sensitive to one type of increase than the other. Being less competitive on both dimensions may simply be unattractive to everyone.
These types of situational, or conditional findings, in statistical terms are referred to as variable interactions, but whatever you call them, the net result is that tests that only account for a single dimension at a time will miss these interrelationships and can lead to seemingly conflicting results and misinterpretations.
The primary advantage of a multivariate test in the presence of variable interactions is to provide a more complete understanding in a single test. This avoids partial results that can appear conflicting and potentially lead to mistakes.
Multivariate Tests vs. A/B Tests in the Absence of Interactions
When no interactions exist (tested changes don’t influence one another) a multivariate test is arguably even more advantageous. Let’s look at another credit card example of a situation where that might be true. Note how the samples add up to the “N per level of interest”.
A/B Test of Interest Rate Multivariate Test
The important fact of interest is that, if the tested factors do not influence the impact of the other tests, then you can effectively use the sample multiple times if the tests are arranged in a balanced orthogonal design as shown above. Note the “N per level of interest” is always 2,000.
The net benefit in this situation is that the test offers 3x the findings of the A/B test using the same data. The 3x is arbitrary to this example and could be much larger.
When are A/B Tests Superior?
One of the points of the article was to point out how improved testing methodology alone could offer higher quality, or simply many more, findings to generate greater BI driven growth. That’s clearly true. However, having said that, I don’t want to imply that it’d never be more advantageous to use an A/B test. A/B tests are simple and reliable and a number of circumstance favor them. When sample costs are low, repeated quick tests can make sense. Perhaps more importantly, if the right infrastructure is in place, they might be used throughout the organization and thereby harness the ideas of many people which could favor the two BI Pillars of “Idea Generation” and “Implementation/Buy-In”.
Business Intelligence driven growth depends critically on a mix of technical and non-technical factors. The most successful companies will be those who are able to bring together the three critical components of 1) Idea Generation, 2) Testing Efficiency, and 3) Effective Implementation.
The prize to those successful companies is a measurable BI driven growth that is several magnitudes greater than that of competitors. In today’s information based economy, with the ever greater access to data and the ability to connect ideas and people, the companies with the best processes in place to turn data into information and information into good managerial practice will be the winners earning above market returns.
David Young has worked in Marketing Analytics 20+ years and lives in Vienna, VA