How Growth Hackers Get A/B Testing Wrong

A/B testing is amazing. It’s changed the face of online marketing, made software companies millions of dollars, and played a major role in presidential elections.  But amidst the 100’s of blog posts extolling the virtues of testing and the countless articles on how to analyze tests, there has been far too little focus on what to test, and this has meant that A/B testing has been dramatically under-utilized.

The problem with how so many “growth hackers” use A/B testing is that they aren’t testing the big questions. Testing has come to focus so purely on optimization that it isn’t being used to define overall strategy.

Tests focused on helping unearth the right major strategic choices aren’t the type that have a 20% impact – they can have a completely redefining impact that might mean the difference in a company succeeding vs failing or increasing your growth rate by 10X. Optimization is really, really important – but it isn’t enough.

When a marketer or business owner lists the three most important questions they need answered in the next six months, they should make sure they are using A/B testing to help get answers to those questions where they can.

Types of Testing

In broad strokes, there are two types of testing:

  • Optimization Testing: This testing is aimed at optimizing an existing process – think button placement on a website, changing the text on a headline on a page, seeing which subject line generates more opens, or figuring out whether a video or an image converts better.
  • Strategic Testing: These tests are aimed at changing something completely – think pricing or whether you use discounts, your overall brand positioning, your marketing channels, etc.

90% of the tests you read about are optimization tests – like this great write-up of a test run by to see whether “Get Started” or “Create Free Account” worked better.  With tools like Optimizely for website A/B testing or Klaviyo for all types of email A/B testing, these tests can be run incredibly quickly and cheaply, and analyzed with minimal effort (although per the test example above, it is important to think through what outcome you’re really aiming for).

While the average test doesn’t do much, some tests will have significant impacts and establishing a culture of testing is a great way to continually learn.  With email, we recommend testing subject lines (long vs short, casual vs formal, etc), style of layout, etc. Because of the large sample size and abundance of data, email A/B testing lets you move quick to see what’s working and what’s not – on conversions, not on opens or clicks.

Strategic tests are different. Their entire goal is not to make small changes to something you are already doing, but instead to start from the major strategic questions you are facing, then figuring out how to create tests to gather data to help make that decision.

4 Examples of Strategic Testing

To help explain strategic testing a bit further, it’s helpful to look at the offline world – because the costs of tests are so much higher when you’re dealing with more than changes to a website, strategic testing is one of the few areas where the offline world is ahead of the online.

A few examples of major strategic tests that helped completely redefine strategy:

Malnutrition and International Development

In Poor Economics, Abhijit Banerjee and Esther Duflo describe a test where under-nourished families were given more money to buy food with.  Counter-intuitively, when families’ caloric intake was analyzed after the test, they found that families were still under-nourished – but that they were spending more money on food to buy better tasting food.  While the test happens on small-scale level (in one village), this type of test helps governments fundamentally shape the type of food aid given to the poor. 

Subway and the $5 Footlong

In 2008, amidst increasing competition from dollar menus at other fast food chains nationwide, a Subway franchise owner tried lowering the price of his subs to $5 on the weekends temporarily to drive new business. It worked – really, really well. He extended it to every day of the week, and while his margins fell, he way more than made up for it on volume. What started as a temporary promotion became a permanent fixture – and nearby Subway stores copied the idea. From this success, corporate Subway ultimately got involved and the nationwide $5 footlong was born. While not a formal or planned test, it ended up serving as a way to test a major pricing shift for Subway to compete against competitor’s dollar menus – having not just a remarkable impact on their business but also redefining their image as a brand and their strategy for the last 5 years.

Capital One and Product Launches

Capital One has stated that they run over 28,000 tests a year – of completely new products, of new markets, apparently even a massive test of TV advertising where they held out multiple states over a period of months to see how effective their spending was.  Another example was introducing a new product that allowed customers to transfer debt to them when they activated a card. Both the product and the method of cross-selling were new for Capital One, but it allowed them to put hard numbers behind a major new initiative.

Cancer and Chemotherapy

One of the most established testing environments is how we figure out which drugs or procedures effectively cure diseases. Chemotherapy is one example – after World War I, doctors suspected mustard gas might have positive effects in treating certain diseases.  When an analysis of victims later showed it had a profound impact on lymphoid suppression, researchers were able to adapt it into a stable and safer compound – and then to design tests to confirm it had the impact they thought it did. These tests took decades – but ultimately, this meant that the learnings from a dangerous weapon were repurposed to provide major life-saving technique around the world.

The Power of Strategic Testing Online

Ecommerce businesses and web apps can take advantage of testing in similar ways. Rather than starting with a current process and thinking about optimizing it, the best approach is to start with a current question or strategy, then thinking through how to test it.

For example:

  • Pricing: By using discounts, targeted campaigns or partnerships, it’s possible to test dramatically different pricing strategies. For example, to understand whether an Ecommerce store should be heavily promotional with its newsletter, one test would be to take a subset of new newsletter sign-ups and to not send them discounts for several months and then to compare the impact to other customers.
  • Products: For every product you consider building, is there a way to test it?  This is the core idea behind the minimum viable product idea for startups, but as with the Capital One example, it really applies to all companies. For someone starting an Ecommerce store, this might mean not holding any inventory and buying from a competitor before setting up your own drop shipping arrangement. For a web startup, this might mean finding 5 customers who you consult for to solve the same problem that your app will before building any thing.
  • Positioning and Brand: Because being consistent in branding is so important, its key that it’s something you get right (or prove out before changing).  While difficult to test, one way to do this is to use landing pages to present a new positioning strategy to part of a channel.
  • Customer Support: How you choose to support customers both defines your brand and has a key impact on your costs.  Using both surveys to collect data after the fact and analyzing usage data, you can test the impact of the support strategy you choose

Next steps

Here’s the question: What are the most important strategic choices you’ll make this year as a business and can you test them?

While it’s easy to see how A/B testing will be used to see whether one email subject line works better than another, figuring out how to test these big questions isn’t easy.  Whether it’s landing pages, segmenting your newsletter list for months, or isolating pricing / messaging to a single channel, you’ll probably have to think creatively about how to run the test.

You don’t read much about strategic tests, both because any given company should be running a much smaller number of them but also because too many companies don’t run them. Keep optimizing – but also take the time to test your fundamental strategies.

Have you used testing to answer fundamental strategic questions? Add examples in the comments.

.yuzo_related_post img{width:260px !important; height:250px !important;}
.yuzo_related_post .relatedthumb{line-height:16px;background: !important;color:!important;}
.yuzo_related_post .relatedthumb:hover{background:#ffffff !important; -webkit-transition: background 0.2s linear; -moz-transition: background 0.2s linear; -o-transition: background 0.2s linear; transition: background 0.2s linear;;color:!important;}
.yuzo_related_post .relatedthumb a{color:#323b43!important;}
.yuzo_related_post .relatedthumb a:hover{ color:}!important;}
.yuzo_related_post .relatedthumb:hover a{ color:!important;}
.yuzo_related_post .yuzo_text {color:!important;}
.yuzo_related_post .relatedthumb:hover .yuzo_text {color:!important;}
.yuzo_related_post .relatedthumb{ margin: 0px 0px 0px 0px; padding: 5px 5px 5px 5px; }

jQuery(document).ready(function( $ ){
//jQuery(‘.yuzo_related_post’).equalizer({ overflow : ‘relatedthumb’ });
jQuery(‘.yuzo_related_post .yuzo_wraps’).equalizer({ columns : ‘> div’ });

Back to Blog Home


  • Good article. Better is the Ed Nash classic “Direct Marketing: Strategy, Planning, Execution.” Written before the rise of the “growth hacker”–show me somebody with that title, and I’ll show you a charlatan–and the Internet, Nash’s book is a classic on the science and discipline of quantitative marketing.

    • Mark – Thanks for the reading suggestion. Great point that the real roots of modern web marketing / testing lie in direct marketing, and there’s still plenty of classic quantitative marketing rigor that’s missing on the web. Capital One’s an incredible example of how valuable a complete and quickly moving testing process can be.

Comments are closed.