A/B Testing: Don't Build Blind

Have you ever felt that burst of excitement when you ship a brilliant idea for your startup – a killer landing page, checkout experience, or irresistible deal– only to watch it flop? The good news is you’re not alone. As creators and tech-savvy entrepreneurs, we often think we know exactly what our users want, but reality has a way of humbling us.

If you’re wondering how to get more conversions from your marketing and product pushes, we’ve got several tips for you, put together in collaboration with Shmoody – a high growth startup in our portfolio that went from near zero to $160K+ MRR in less than 6 months.

But before we turn those flops upside down, we have to start with the concept of the brilliant idea and your relationship with your customer.

Which are you?

The Omnipotent Being

The Scientist

The Omnipotent Being is all-knowing about the customer; the Scientist asks questions and hypothesizes about the customer. The Omnipotent Being is all-powerful over the customer; the Scientist uncovers customer behavior by testing and observing.

If you can bend your customers to your will with perfect conversion and zero churn, you can stop here! You’re the Omnipotent Being.

But if your conversion rates are less than ideal, you have to realize you are but a mere observer who is curious and eager to help the customer become the hero of this story.

The only way to help the customer achieve greatness is to A/B test their journey to the finish line.

A/B Testing Defined: A experimentation process where two or more versions of a variable (A and B) are shown to different, randomized segments of visitors at the same time to determine which version leaves the maximum impact.

A couple things to keep in mind when A/B testing:

  1. How customers arrive at a call-to-action matters.

Founders often set out to A/B test small details like imagery and copywriting across marketing, sales processes and product but forget about testing the order of steps the user takes in her journey.

For example, if you’re running ads about your product with the goal of Conversion when the target audience isn’t even aware they have a problem yet, you might need to do more work in the Trigger stage to educate them on the problem first.

  1. Using the right tools is key.

Shmoody used Mixpanel to get to their $160K+ MRR. There are other tools out there like Amplify and Pendo. If you have an e-commerce website, Google Analytics could be exactly what you need to understand your marketing funnel, but if you have a tech product, GA alone will not be enough to A/B test effectively (learn more here).

 

3 experiments that Shmoody ran to better understand, convert and retain customers:

PRICING TEST

Here are some variables to test:

  • Dollar Amounts
  • Subscription Durations
  • Trial Periods and Promotional Offers
  • For instance, Shmoody conducted this A/B Test:

On the left (A), they offered a subscription at $69.99 for 24 months.

On the right (B), the subscription was available for $6.99 for 6 months.

The test results revealed that the lower subscription price with a shorter duration was 18% more profitable. Remember that this is just the starting point! You can begin with broader ranges and gradually narrow them down to pinpoint your optimal pricing.

 
 

PAYWALL TEST

Here are some variables to test:

  • Layout
  • Messaging
  • Call to Action

Shmoody, while keeping their product, call to action, and price constant, experimented with two distinct layouts and messaging for their free trial. The outcome favored option A, leading to a 19% increase in users starting a free trial and a remarkable 33% boost in conversion rates. This strategic A/B test catapulted Shmoody to a 10x monthly revenue surge, setting an all-time high record!

 

PRODUCT TEST

Here are some variables to test:

  • Onboarding
  • New Features
  • Colors and Copy
  • Design
  • Notifications and Emails
  • Personalization

Shmoody introduced a new feature on users' homepages, asking them, "How are you feeling today?" This seemingly small change resulted in a 13% increase in user retention, as users kept coming back to engage with the app.

 

BONUS TIPS!

Embrace Failure: Remember that most A/B tests – about 80% – will not move the needle on your desired metrics. Be patient and ensure you accumulate enough data to conduct effective tests. Here, having the right number of users and a noticeable effect size is crucial to determining useful results (tools like MixPanel will tell you when you’ve reached statistical significance on a given test – P.S. no, they’re not paying us to mention them a lot here, we just like how much their tool has helped our startups)

Learn from Winners: Don't hesitate to draw inspiration from successful A/B tests conducted by others. Resources like GoodUI can be a valuable asset.

Build on Success: Once you identify winning patterns, allocate more time and resources to amplify those successes. Minimize, or even eliminate, efforts on the underperforming tests. A/B testing is your beacon of clarity in the uncertainty of startup life. If you aren’t the Omniscient Being, A/B testing is your precision tool which gives you more control over your startup’s destiny with data-driven decisions. Seasoned entrepreneurs know this: A/B testing isn't optional. It’s the key to unlocking the true potential of your startup.

Sales and MarketingKalyn Dickson