Running A/B Tests During the Holiday Season?

Are you considering whether to run website A/B tests during the holiday season? During Q4 seasonality, most ecommerce sites see a considerable increase in site traffic. You may feel apprehensive about launching new experiments due to the risk of negatively affecting the user experience for your shoppers and decreasing your brand’s holiday sales.

However, this does not mean that you need to completely stop testing! In fact, you can take advantage of the seasonal lift in traffic by aligning your testing strategies with the expected purchasing behavior of your users. You may just discover an untapped opportunity to drive more revenue!

When it comes to holiday shopping online, users often change their shopping behavior because of a higher sense of urgency. To encourage users to commit to a purchase, it is important that your site displays stock availability, estimated delivery times, and limited time offers. This fall at Think Retail on Air, Google predicted an increase in comparison shoppers who will be more diligent with their research, comparing products on multiple sites to find the best deal. To win your users over and alleviate their questions and concerns it is critical that your site has clear product messaging and strong social proof.

For more insights from Google’s Think Retail on Air 2022, watch our video (or read the transcript) where ROI’s paid search experts uncover the biggest takeaways!

To understand how to properly target your audience during seasonality, you need to understand the two types of purchasing behaviors.

Early season: Early season customers are less time-sensitive and are more focused on finding the right gift. Testing different verbiage for gifting options or adding a carousel with highly rated products or seasonal merchandise will increase engagement and help users find a product of interest quickly.
Later season: Later season users are last-minute shoppers. They typically rely on stock availability and delivery times. To cater to this audience, you can offer expedited shipping and gift options – and test ways to make these options more prominent on your site.

5 A/B Tests for the Holiday Season

To impact and improve your site’s conversion rate, your A/B testing strategies during the holiday season need to alleviate user fears and uncertainties. Here is a list of five areas we recommend organizing your tests around.

1. Promotions and Discounts

Banners, free shipping, and discount codes are all great ways to entice shoppers to buy from your brand – and make for great places to run A/B tests during the holiday season.

Banners: Banners are great to promote a sale and link users directly to a page or product that you want to receive high engagement. Test the placement of a banner and play with different colors, imagery, and headlines. For instance, you could test “Spend $50 to get 10% off” vs. “Get 10% off on purchases over $50” vs. “Get $5 off a $50 purchase.”
Free shipping: If you offer free shipping, experiment with increasing the prominence of your free shipping threshold by adding messaging to different parts of your site such as the post add-to-cart confirmation modal or the actual cart page.
Discount codes: If you are running a large sale with a discount code, consider increasing the prominence of your promo code field in the cart and/or at checkout for high visibility.

2. Gift Guides

Many users feel pressure to buy the right gift during the holidays, which can result in indecisiveness. One way you can overcome this hurdle is to create gift guides to help users quickly discover products that are recommended.

Test highlighting your gift guide within a carousel, on a grid on your homepage, or on other high-traffic pages. Link to category pages with products that would make great gifts. Some examples of gift guide themes are:

Budget-conscious
Inclusive for all customers
Segmented by age group or gender
Most popular items
Best deals

3. Countdowns

You can capitalize on shoppers’ higher sense of urgency during Q4 due to the time constraints by triggering a mental countdown to motivate users to buy now.

There are different ways you can test this. A countdown clock could tick off the days or hours until:

The end of a sale or promotion
The start of a sale
A product’s availability
Shipping cut-offs for later season shoppers

4. Urgency Messaging

It is important to understand what motivates your users to purchase, and whether adding urgency messaging creates a fear of missing out. You can test around this concept even during non-holiday seasons, but the Q4 shopping period is a time of heightened urgency that provides an excellent opportunity to test copy.

For example, try including messaging that states “only 3 left” next to products that are low in stock, or “25 other people have this item in their cart.”

5. Shipping and Return Policies

Before customers commit to purchase, they want to know if they will be able to return or exchange items – especially if they are browsing for gifts for others.

Experiment with making shipping and return policies more prominent, or test different pages and placements of that messaging to gain the user’s trust and boost their confidence in the purchase they’re considering making.

To get more ideas for holiday A/B tests, read our blog on 8 Holiday A/B Tests to Boost Your Ecommerce Growth This Holiday Shopping Season.

Tips for Q4 A/B Test Durations and Scheduling

Things to keep in mind as you plan for Q4 testing are:

Plan ahead. Start planning your test proposals two to three months ahead of a holiday sale so that you are prepared to execute your tests flawlessly.

Monitor closely. Check your live tests daily to monitor for any dramatic shifts in conversions and avoid any harmful impacts on your brand’s revenue during a critical time.

Consider your scheduling. You may want to limit your test timeframe to the seasonality period or sale. You can run a test across both pre- and post-seasonality, but keep in mind that mixing sale and non-sale period data could skew your test results. If it’s not feasible to run a test during the dates of a promotion, run your tests before or after a sale period.

Test it to set it. Use real-time data to determine the right messaging or treatment for your promotion. We often suggest running a test for the first third of the time a sale is live, then allocating 100% of traffic to the winning variation to maximize the impact. Unless your site receives hundreds of thousands of visitors per day, you may not reach statistical significance during the initial test period, but you can look at trends in the data to determine the likely winner.

If you choose not to run A/B tests during the holiday season or you have a code freeze for this critical time of year, you can still gain valuable insights during this time of year! Use the holiday shopping period to learn about your users by focusing on collecting qualitative data through user testing and surveys.

User testing. This is a great way to understand how your customers are using your site. Having 5-10 users in a test provides enough information to discover common friction points. Design test scripts to discover answers to questions like What stops users from completing their purchase during the holidays? and What motivates them to buy?
Internal surveys. Interview your sales and customer support teams to reveal what questions and feedback they typically receive from your users.
External surveys. Add an exit intent survey to your site that appears when a user attempts to leave. The survey prompts them to provide feedback on how to improve their user experience.

Tying It All Together: A/B Tests During the Holiday Season

The holiday season offers opportunities for you to learn more about your users. This can be done through A/B testing, user testing, external surveys, and internal surveys. Gathering quantitative and qualitative data can elevate your website and enhance the user experience for not only future holidays but also during any high urgency periods. Make sure you follow these best practices to harvest the advantages that testing during the holidays can bring!

ROI has an entire team dedicated to conversion rate optimization. In fact, our experts have an A/B test win rate of 42% compared to the industry average of just 12-15%! That means that 42% of the A/B tests we run end with an actionable result. To explore how our A/B testing and UX experts can help your brand find untapped opportunities for profitable growth, send a message to our team today.

Looking for more on 2022 holiday ecommerce advice? Peruse our resources below!

Sources

Google, Think Retail on Air 2022.