Why You Should Use Cohort Analysis and Control Groups

One of my favorite articles on analysis is a piece by Eric Reis called Vanity Metrics vs. Actionable Metrics. Having worked in business intelligence for six years, he nails one of the most common problems I saw (and still see) with the majority of web analytics / business intelligence software in the world today: they give you lots of numbers, but at the end of the day they don’t tell you what to do and just lead to lots of debate. The story of James Lind and scurvy has some important nuggets for software and Ecommerce companies.

Test your Ideas: Scurvy and the First Clinical Trial

In 1747, James Lind was a Scottish physician in the Royal Navy on a ship where 12 sailors were infected with scurvy.  He had theories about the cause of scurvy (he thought proteins in the the body were decaying) and hypothesized that acids might counteract scurvy.  Where this gets interesting is in what he did next – namely, he broke the 12 sailors into 6 groups of two and gave each group a different treatment.

He tried six treatments:

  • A daily quart of cider (the traditional remedy)
  • Sulfuric acid droplets
  • Six spoonfuls of Vinegar
  • Half a pint of seawater
  • Spicy paste and barley water
  • Two oranges and one lemon

The oranges and the lemons worked – not because Lind’s hypothesis was right, but because scurvy is caused by a Vitamin C deficiency (as an aside, eating oranges and lemons sounds a lot more pleasant than his other hypotheses). By most accounts, this was the birth of the modern medical clinical trial that most nearly all drug research is based on.

3 Lessons for Identifying your Oranges and Lemons

While this experiment is hardly revolutionary today, carefully controlled experiments have dramatically advanced medicine over the last 200 years. While the web has started to take note, web businesses can learn a tremendous amount from medical trials. Perhaps the most important lessons for us from Lind’s experiment:

  • Your ideas aren’t always good ones (even if backed up by data) – but if you try them out and measure them, you’ll suddenly have actionable metrics to either implement them (if they were right) or to stop debating them (if they were wrong).  Lind’s theory about acid was way off-base – but he still discovered the right action to take to cure survy for thousands of sailors. In short – stop debating, and start testing.
  • You need a control to compare results with. Lind did two smart things to make sure he could attribute the improvement in scurvy to the oranges and lemons. First, he didn’t give all the patients the same treatment (so he could tell when one group was noticeably improving over another). Second, he only implemented a single treatment on each group – so he was confident it was indeed the orange and lemon – and not living conditions, sunlight, meals, exercise, etc.  In short – use a control and you’ll have a much higher chance of generating results you trust. Good options are A/B testing, cohort analysis and broader experimentation on price, offer, marketing approach, etc. A personal favorite article of mine is this piece by Josh Porter on cohort analysis.
  • Agree to believe in the experiment outcome before you run it. When Lind published his results, it took the Royal Navy another 48 years to start provisioning lemon juice to all sailors to prevent scurvy. This delay speaks to a crucial lesson: running experiments is useless if you don’t follow the results, even if you don’t like them or they go against what you thought you knew. To that end, agree on the methodology ahead of time and talk through the next steps of each outcome of the experiment. The goal should be taking action – not just learning.

Taking Action

Let’s come back to the vanity vs actionable metrics for a second. When we implement an idea, we have a tendency to want to see it work. If we focus too much on our vanity metrics (overall site traffic, engagement rate of users this month, average purchase amount, number of people using a feature, etc), we are at risk of reading the result we want to see (by attributing broader growth or seasonal trends to our action).

However – if we experiment, we then have the actionable metrics we need to implement. The more disciplined you get about this process, the more nimble you can be, the happier your customers are, and the less pointless debate you’ll have to engage in. Go forth and test!

Follow us on Twitter.


How to Build Your Email List



How to Use Preheader Text



Feature Launch: List Growth Reports


.yuzo_related_post img{width:260px !important; height:250px !important;}
.yuzo_related_post .relatedthumb{line-height:16px;background: !important;color:!important;}
.yuzo_related_post .relatedthumb:hover{background:#ffffff !important; -webkit-transition: background 0.2s linear; -moz-transition: background 0.2s linear; -o-transition: background 0.2s linear; transition: background 0.2s linear;;color:!important;}
.yuzo_related_post .relatedthumb a{color:#323b43!important;}
.yuzo_related_post .relatedthumb a:hover{ color:}!important;}
.yuzo_related_post .relatedthumb:hover a{ color:!important;}
.yuzo_related_post .yuzo_text {color:!important;}
.yuzo_related_post .relatedthumb:hover .yuzo_text {color:!important;}
.yuzo_related_post .relatedthumb{ margin: 0px 0px 0px 0px; padding: 5px 5px 5px 5px; }

jQuery(document).ready(function( $ ){
//jQuery(‘.yuzo_related_post’).equalizer({ overflow : ‘relatedthumb’ });
jQuery(‘.yuzo_related_post .yuzo_wraps’).equalizer({ columns : ‘> div’ });

Back to Blog Home
Get email marketing insights delivered straight to your inbox.