What Is A/B Testing? [Beginner’s Guide]

This article is part of Virtue Media’s Web Pros Series. In this series, we feature articles from our team of experts here at Virtue Media. Our Product Managers, Linux Administrators, Marketers, and Tech Support engineers share their best tips for getting the most out of your website.

Ever wonder what makes some websites and web pages work better than others? Same here. And as a UX designer, it’s important for me to use real data to make design decisions, rather than my own often-biased opinion. A/B testing allows me to get real-world validation of an idea that I or someone else may have. 

Anybody can use A/B testing, including site owners. So, here’s your intro to testing with no study sessions, no grading curve and no wrong answers—only information you can use to make your website easier to use and more appealing to visitors.

What exactly is A/B testing? 

It’s a method to compare two things and, in a nutshell, find out which one is preferred by people who are using your website or app. To make the comparison, you’ll need two different designs, or word choices or images. 

1. The control. This is the one that’s actually on your website, aka in production, in the wild or live. 
2. The experiment. This is the design, copy or image you want to test to see if it works better than what you already have on your website. 

For example, you might wonder if more visitors would add items to their carts if the “buy now” button on your product pages was bigger. Your control for this A/B test is the Buy Now buttons already on your product pages. The experiment is a larger Buy Now button on some product pages. 

How do you know what you should A/B test?

You can A/B test any element on your website that you suspect could be better, but the most efficient way to A/B test is to look at your site analytics first. 

For example, my design team and I look at our site data for areas that have a high bounce rate or low conversions—even or especially if those pages are getting a lot of traffic. In fact, if the page gets lots of visitors, that traffic volume can help you get valid A/B results faster. 

You can also flip the concept around and look for pages that have low bounce rates and high conversions. Then you could A/B test to see if you get better results on lower-performing pages by applying elements from those strong pages.

If you’re still not sure where to start, you can focus on elements on your checkout or signup pages, because those are typically the most important pages on a website, the ones where you want the best possible performance. However, if your checkout and signup rates are already high, you may want to leave things as they are and focus on other elements of your site, like landing page bounce rates or buy now buttons.

How long does an A/B test take?

The short answer is: it depends. The long answer is you need to run your A/B test long enough to get enough data for useful results—and that will depend on your volume of site traffic. 

Typically, at Virtue Media we run our A/B tests for about two weeks to get data with statistical significance. That amount of time gives us confidence that we’ve had enough people going through the test to give us valuable data.

Think of it this way: If you want to know whether your customers prefer chocolate chip or peanut butter cookies, you’re not going to base your cookie-ingredient buying budget on the response from just one customer or even a dozen. 

Before you spend hundreds or thousands of dollars on semisweet morsels or extra-crunchy, you’re going to want input from hundreds or thousands of your customers so you can make more of them happy and sell the most cookies you can.

Still not sure how long to run your test? Distracted by all this cookie discussion? Use Optimizely’s A/B Sample Size Calculator to see how many visitors you’ll need and get an explanation of all the mathy terms the calculator uses.

What if you have a small or new site without much traffic? 

A/B testing is easier with lots of traffic, but you can run informal tests with smaller groups of users. When I was starting out in design, I had a group of 10 or 15 friends and family who were my biggest resource for quick and dirty A/B testing. 

Even though that wasn’t enough users to give me statistically significant results, they were able to give me enough feedback and data to help me figure out how to make my designs more user-friendly. If you go this route, be sure ahead of time that you have a clear goal of what you want to learn. That way you won’t get distracted when your cousin tells you that your checkout page is fine but the whole site needs more pictures of capybaras.

OK, so how do you run A/B tests on a DIY budget?

A/B testing services for enterprise customers typically carry enterprise price tags, but there are other options that fit SMB budgets.

WordPress A/B testing plugins can help you optimize your headlines, calls to action, SEO and other site elements. For example, Thrive Headline Optimizer, SplitHero and SEOScout are all less than $500 a year for a basic plan. 

If free is more in line with your A/B testing budget, Google Optimize is probably your best option. Optimize is free and designed for SMB users. It integrates with your website’s Google Analytics data to help you set up, run and evaluate your A/B tests.

You can also hire someone to do your A/B testing for you, because sometimes it makes more sense to pay for expert guidance than to try to figure out how to set up, run and interpret your tests while you’re also running the rest of your business. Freelance A/B testing specialists usually charge between $50 and $150 an hour and they can save you a lot of time getting your program up and running.

What do you do with your A/B test results?

If your test shows a significant preference for your experimental version, you can make it live on your site. Otherwise, you can leave your existing version in place and look for other elements to test or other variations of that same element to try. 

If you make changes based on your test results, track their performance in your analytics dashboard to make sure those test results are steering your site where you want it to go. 

A/B testing takes a little time to figure out when you’re first getting started, but it can help you and your visitors get more value from your website by finding weak spots and making them better.