Today’s business landscape is evolving at a breakneck pace and technology has made it possible to measure almost everything.
This is a double-edged sword because there’s so much data. If you’re not careful, the information bias can creep in and lead to drawing the wrong conclusions about what’s really happening.
You may kill campaigns that were actually yielding fruit, discard tests that were working, take down lead magnets that can be optimized, or push marketing initiatives that hurt you.
It seems unlikely but in this post, I’ll show you real-life examples of the information bias at work, the common places it hurts your marketing, and what to do about it.
What is information bias?
The information bias is an error that stems from a measurement error (the difference between the observed value and the true value). It’s a flaw in the measuring exposure, covariate, or outcome variables that result in different quality and accuracy of information between comparison groups.
Put another way, the information bias happens when the inputs used to draw conclusions aren’t measured or observed properly. This leads to the wrong decisions.
John Hopkins defines it simply as “when information is collected differently between two groups, leading to an error in the conclusion of the association.”
These definitions may not be immediately clear so let me illustrate with an example that cost the company billions of dollars.
The cost of the information bias
Gillette has been a household name for decades. Until recently, it held over 70% market share in the US for its flagship products. Due to many factors like new competitors and attitudes around grooming, that market share has dwindled to roughly 50%. Gillette decided to tap into a social issue to rebrand and increase sales.
Gillette released it’s now infamous short film – We Believe: The Best Men Can Be. It has been viewed 33,000,000 times and has been disliked 1.5 million times.
For perspective, promotional videos get less than 1% of viewers to like or dislike them. The We Believe video had 4.5% of its audience give a negative reaction.
It was polarizing.
How did a multibillion-dollar brand drop the ball in such a massive way?
Information bias.
Gillette has been championing masculinity for 30 years but in the face of declining sales, they listened to a loud minority. Those people make toxic masculinity seem like the norm.
The reality is that only a few men can be categorized as perpetuating toxic masculinity. The ad alienated their core demographic.
In short, they listened to the people who were shouting from the rooftops that modern men are like this:
When in reality, they’re more like this:
Needless to say, the people who buy Gillette products and are part of the second group didn’t like being lumped into the first group. It contributed to Gillette’s parent company, P&G, writing off $8 billion.
This is an extreme example of information bias but let’s look at how it can affect your business every single day.
Areas where information bias affects marketing
As mentioned earlier, there’s a lot of data – a lot of information – in modern business. You can measure everything from the number of people who visit your website and how long they stayed right down to whether or not the red or blue button gets more clicks.
All this data creates silos in organizations. The marketing team knows certain things almost intuitively. Even though it would positively impact the customer service team, they have no idea about it. That’s why an internal knowledge base and proactive knowledge sharing are so important.
If the knowledge isn’t shared, it brings up countless opportunities for information bias to impact your business in unintended ways. I’ll focus on the most common.
Conversion optimization
We do conversion optimization for everything from our lead generation forms and quizzes to sales pages and email subject lines. When done properly, they can have significant positive effects for your business.
It may be due to the methodology of the test, what’s being tested, or the number of people who go through the test.
There are two main problems with conversion optimization. The first is inadequate traffic levels which means a lot of tests don’t get to statistical significance. The second one is what the test is measuring.
A dirty little secret in the conversion optimization world is that the majority of split tests which focus on superficial items like the button color or button text revert back to the median over time. So if that red button increased conversions by 50% today, those results may not hold over time.
For example, there’s a case study about how changing the button color increased CTR by 21%.
I don’t doubt those claims. What I doubt is whether they were able to sustain those gains over time.
This is information bias at its finest. The sample size for tests is too small so we’re using the wrong information to draw conclusions. Conversely, we’re testing the wrong things over a short period of time. When we roll the changes out, they don’t stand up over time.
What to do about it
The solution is dependent on how mature your CRO efforts are right now. If you’re just getting started, the most important thing you can do is to start with best practices and create meaningful tests.
Instead of focusing on the button colors and other inconsequential elements, test things like the layout of a page, the overall messaging, etc. These changes, though larger, have the potential to create long-lasting conversion boosts and prevent you from being forced into a local maximum.
Finally, create benchmarks for your tests such as the number of visitors needed before a decision can be made or the length of time before it can be adopted across the board. Once created, stick with those benchmarks religiously so you have standard datasets to work with.
Surveys
There has been a shift towards optimizing web experiences for humans which is a good thing. Part of this process is interacting with the humans who use your website and products.
To that end, we use surveys which are incredibly versatile for understanding customer perception about different aspects of your business. A few things you find out are:
- Understand why people stop buying
- Figure out what makes people buy
- Get deep insights about willingness to pay
- Brand perception
A common problem is treating all survey respondents alike. For example, if you send out a survey to your entire email list, some of them are customers and some of them aren’t.
It would make sense to give more weight to the opinions of customers over the opinions of casual browsers or email subscribers. At the same time, you can further divide your customer groups into segments.
Information bias occurs when you send out a survey to everyone who you can contact and give every answer the same weight.
What to do about it
The name of the game here is segmentation. There are countless ways to segment people who receive your survey. You can do it between customers and non-customers. You can segment by purchase history. You can segment based on demographic markers like age or gender.
It’s important to make sure the way you segment is in line with the goal of the survey. For example, if it’s market research, you may want to segment by buyer persona or demographic information. Here are a few more ways to think about segmenting your survey respondents.
- Customers and non-customers
- Frequency of purchase
- Customer value (what pricing tier are they on or how much have they spent)
- Source of the respondent (EG did they come from your mailing list or social follower)
- Length of answer (for open-ended questions)
Advertising initiatives
This happened to us when we were testing Facebook ads for KyLeads. We tried to market like a SaaS company and were creating ads that were asking for a direct sign up.
We’d basically show up in your feed, ask you to sign up for a trial, and let our onboarding emails do the rest.
This technically worked. We got signups but it cost us $125 to get a customer. Our ads looked a bit like this:
(bonus points if you can guess the company).
Many people would consider this a success because with this acquisition cost, our LTV:CAC ratio is better than 3:1. There’s just one problem, we don’t have a lot of money in the bank from our VC backers (IE we don’t have ANY money from VCs).
If we tried to scale that acquisition strategy, we’d be out of business even though we’re acquiring customers who would eventually yield a profit.
We put a halt to Facebook because the data was telling us it wasn’t working. We were looking at Facebook ads the wrong way – information bias – and suffering because of it.
Other ways people fall prey to information bias with advertising initiatives include:
- Not letting the campaigns run long enough (not giving it enough data)
- Incorrect targeting
- Using broken funnels/sales processes and amplifying them with paid traffic.
In these instances, the information you receive will have errors which lead to the wrong conclusions. In the end, you may write off ad platforms that have the potential to change the game for you.
What to do about it
The solution here is manifold because there can be many reasons you’re getting the information bias. First, it’s important to run your campaigns long enough to get the right data. Make sure you allocate enough budget to your advertising campaigns before you kill the losers off.
Before you start running ad campaigns, research your market thoroughly so you can make sure your targeting is correct. This depends on the platform you’re using. With Facebook, that’s interests but with Google that’s Keywords.
If you don’t have the time, energy, or skills to do the research then be prepared to test many different variations of your targeting until you’re able to hone in on the one that works for you.
Finally, optimize your pages, sales process, or funnel before you write an ad platform off. Many times, you may get clicks on your ads but no one is converting. Is it because you’re being sent junk traffic? Maybe. At the same time, it may be because of message mismatch or poor landing pages. You won’t know until you test and optimize it.
That process takes time and energy. Sorry, no shortcut.
Conclusion
The information bias, like other cognitive biases, happens whether we like it or not. The key is to be aware of it and actively work to minimize it.
In this post, I’ve touched on a few ways information bias affects businesses and what to do about it. Don’t try to tackle everything at once so you’re not overwhelmed.
Instead, focus on one or two areas such as CRO and determine whether or not you’ve been affected by information bias. If you have, take steps to reverse or minimize the effects of it. When you’ve put processes in place to ensure it won’t repeat itself, tackle the next one until you’ve reduced the information bias in your business.
It’s important to note that you can’t get rid of it completely, you can only manage it. Let me know how you’re tackling the information bias in your business and don’t’ forget to share.