Price testing: 3 reasons to never a/b test your pricing strategy

What is price testing?

Price testing involves comparing how your market responds to different price points for your products. Instead of using optimization tools, you let the customer vote for the best pricing with their dollars.

3 Reason Why You Should Never A/B Test Your 

1. Statistical significance is rarely possible for pricing

There’s a concept in statistics called statistical significance. The idea can get pretty complicated, but at its root, you essentially need a certain amount of people in your A/B test in order to ensure your end results are legitimate and aren’t just due to random chance.

When you A/B test, each of the different scenarios you run needs a statistically significant amount of test subjects in order to make sure the results are valid. This means you need a lot of customers willing to purchase your service to even consider A/B testing your pricing, and an even greater amount when testing for multiple customer personas. For nascent businesses, as well as companies who focus on completing bigger projects for a small customer base, A/B testing is an impossible method to implement.

Think you’re in the clear with your thousands of visitors per month? Well, assume that you can somehow correctly identify each visitor or lead coming to your pricing page (big assumption). Further assume that each of those groups are large enough for statistical significance across both options you’re putting forth (even more if you’re testing more than two things - bigger assumption). You still would only be shooting yourself in the foot, because you would be tracking those visitors over the course of a month or more, when evidence suggests price sensitivity can shift dramatically in even a matter of days. Plus, if you decide to change the format of your pricing page, you will need to redo the entire A/B test to eliminate any effects from the change, which will require even more customers.

To summarize, in general, A/B testing is better suited for places that generate a lot of traffic, but don’t ask for large commitments (think landing pages and SEO).  

2. You price anchor your potential customers and will upset them

Ok, so with statistical significance we’ve eliminated maybe 80% of you reading this (I’d argue higher, but let’s be generous), but let’s say you fall into the 20% who generate enough traffic (looking at you 37Signals and HubSpot). A/B testing is still a horrible idea even if you have a large enough customer base, because you’ll more often than not anchor certain customers (and not in the good way!) or leave them fuming when the prices change or their boss sees a different price when trying to make the purchase.

If you offer your service to a group of potential customers for $10, but discover that a better price to sell it for is $50, you’ll have a hard time justifying the price increase to the customers who originally wanted to purchase the product (and now are price anchored) at $10. You can’t exactly tell them you’re raising prices by 400% because it will help you maximize your revenue - say for example, you've made the switch to prestige pricing overnight. Even if you somehow cookie those individuals to show them the anchored price, word gets out eventually (customers will typically revisit your site for something after they’ve cleared their cookies).

Things get worse because at the time of the A/B test you’re offering the exact same product at very different price points to different customers. This is a huge ethical gray area that we haven’t quite figured out yet, but before we do, the practice can still enrage customers and cause a PR nightmare. In the age of tweets, shares, and multiple co-workers getting involved in a decision, you’re almost guaranteed to get discovered with an even higher likelihood you’ll lose multiple sales.  

3. A/B testing is relative, not absolute

The worst thing about A/B testing your pricing though, is that it simply doesn’t work. A/B testing will not divulge customer valuations for your product. Instead, it will simply reveal how one permutation wins over another. To put it bluntly, just because A was more successful than B doesn’t mean A is the optimal setup for your business.

The crucial flaw in A/B testing is that it is relative instead of absolute. This isn’t such a big deal when you’re able to continually iterate your testing like you can with landing pages or lead flow through your site to find the optimal result. However, it isn’t feasible to constantly A/B test (change) your pricing, which means the tactic, especially when mixed with the above reasoning, is useless.  

A/B testing doesn’t work: how to implement a value-based pricing strategy

The best alternative to A/B testing (and the best technique for value-based pricing in general) is to survey your customer base. This avoids all of the pitfalls of A/B testing and will also provide you with better data. Here’s why:

Surveying doesn’t require as many people as A/B testing to execute, because you can target the subsets of customer personas you need. Plus, the number of respondents you can source will be greater than those who will show up in an A/B test, because unlike A/B testing, you can survey anyone who is a decent fit for your product, not necessarily those who are coming to your site. What’s more, running these studies works for all types of businesses, even those with large customer bases. Granted, you will then of course come under fire of things like survey bias, but these things can be controlled for, simply because you have control over the length of the study and the makeup of the respondents, rather than hoping and praying the right people see the right things throughout the A/B test.

Sourcing respondents also won’t anchor your customers or upset them. You’re asking for your customers’ willingness to pay, so they won’t be psychologically locked into a preconceived price when they’re ready to buy. Of course, this means you need to survey them properly (more on this in our value based pricing post). Yet, if and when done properly, you’ll minimize anchoring and reduce the risk of customer backlash, while making sure you avoid the ethical grey line of dynamic pricing.

Finally, and most importantly, surveying actually finds your customers’ willingness to pay, which is the ultimate goal of value-based pricing. A/B testing your pricing, on the other hand, only gives you relative data, which means it is practically useless for value-based pricing.

The bottom line:

A/B testing is never the best option for determining an optimal pricing strategy for your company. The practice works great for a lot of things, but its relative nature does more damage than good when coming to the most important aspect of your business. You’re much better off just speaking and sourcing information from your prospects, leads, and current customers.

To learn more about pricing specifics, check out our Pricing Strategy ebook, or speak to one of our experts. We're here to help.