The Ultimate Guide To A/B Testing

If you’re reading this blog, I’m going to assume you’re serious about improving your marketing performance

Because let’s be honest, there’s nothing worse than putting time or money into marketing, if visitors are just leaking out the other end without taking action…

In this blog, I’m going to cover everything you need to know about a/b testing, including the common mistakes to avoid that will undo all of your efforts if not approached carefully, so be sure to read the full blog (bookmark this page for later if you don’t have time to read right now!)

So let’s get straight into it!

What is a/b testing & how does it work?

In a nutshell, A/B testing (also sometimes referred to as split testing) is a technique that allows us to test 2 variants of any given marketing asset, against another.

That ‘marketing asset’ could be anything from:

• Landing pages
• Marketing emails
• Video thumbnails
• Ad creatives

The list goes on…

In any A/B test, you’ll have you current version (known as the ‘control’) and then the variant you want to test it against.

However, before starting any A/B test, it’s important to understand the objective behind the test and subsequently which element makes sense to test.

For example, if you wanted to get more people to open your email. There wouldn’t be much point testing, the call to action or even the body copy in your email, as people won’t even see that until they open it.

Instead, you’ll want to test the subject line or opening paragraph here as that’s typically the first thing people will see in their inbox and will often be the deciding factor to whether they open, or skip straight past.

Another thing you could also test here is the sender name. We often see this making a big impact in open rates when doing email a/b testing as people often get ’email blindness’ when they see the same name constantly appearing in their inbox for too long…

Now let’s say on the other hand, we want to get more people to take action from our email. We’d naturally want to be testing the body copy or call-to-actions within the email itself and measuring the CTR (click-through-rate) as our primary metric for the test.

This principle applies to whatever piece of marketing you’re testing…

Make sure you have a clear goal, and you’re testing the RIGHT variable in order to impact that goal

What are the different types of A/B testing?

Before you get started on your A/B testing journey. It’s important to understand the different types of A/B tests available.

1. Regular split test (or A/B test)

Regular split tests involve testing 2 variations of a page, email or ad against each other to see which one performs best.

It’s important to determine what success looks like for you in each particular test.

For example, maybe your goal is to simply get more clicks, or maybe you want more conversions; whatever your goal, be sure to establish this before launching so you can accurately compare the data.

When doing split testing, it’s also important to form a hypothesis first and decide which element to test.

For example, if you look at some heatmaps of your website and find 80% of visitors are leaving upon landing on the page, without even scrolling down – you might decide to change the headline of that page.

Once you have that hypothesis and the element you want to test, it’s important to only test one element at a time, otherwise you introduce more variables which may skew the data.

After running the initial tests you can then test a different element or variable.

2. Multivariate testing (sometimes referred to as MVT)

Multivariate testing is a form of testing that generally involves test 3 or different variations of a landing page, email, or ad.

This can be good for testing different layouts, but it can also be difficult to ascertain exactly which specific element has led to the increase or decrease in results.

It’s also worth bearing in mind that multivariate testing will generally require a much higher amount of traffic since you’re no longer spreading that traffic across just 2 variations! As such, it’ll  typically take longer to gather the same amount of data!

3. Split URL testing

Split URL testing is a specific form of landing page/website testing whereby you use 2 seperate links to test different versions of a page.

This is typically used when you want to test a significantly different version of a page that would otherwise be slowed down if using a traditional a/b test (which rotates the variation from one link only).

4. Multipage testing

Multi page testing is another form of landing page/website testing whereby you’re testing one element across multiple different pages.

For example, this might be the CTA button colour or a certain section which is consistent across several pages on your website, to determine it’s overall impact on your conversion event (whatever that may be)

This blog now converts around 5% of all visitors into leads and even if they aren’t in the market for our services right now – They all go into a lead nurture campaign, so we can keep giving value and building the relationship until we become the go-to choice when they need marketing help.

This allows us to have a consistent pipeline of warm leads/prospects coming into the business.

How long should you run your a/b test  for?

The short answer is you should run your test until it’s statistically significant.

However, this can get slightly complicated as there are different models of statistical significance.

If you’re new to A/B testing we recommend using a simple statistical significance calculator like the one by Qualtrics.

This will give you a confidence score on how likely your data is to be reliable. Sometimes this just comes down to common sense.

For example if you’ve got 2 versions of a landing page and one has had 120 conversions and the other only had 12; yet both had the same amount of visits from the same traffic source or keyword, you can be pretty confident in the results!

Whereas if you have a page with 7 conversions vs 4 conversions, you probably don’t have enough data.

The length of time it’ll take to come to these conclusions will depend on one, the variance between the two pages, but also the amount of traffic you’re currently getting.

For example, if you’re relying on paid traffic and have an ad budget of £1,000 and an average cost per click of £0.10, you’ll have a pretty sizeable sample size of 10,000 visitors in a month.

However, if you have the same budget, but are an in industry with a higher cost per click, say £10, you’ll only be getting 100 visitors for the same budget. As such it could take you years before you can get the same sample size!

What are the best tools for a/b testing?

There’s a number of different tools you can use for a/b testing.

For example if you’re wanting to test marketing emails, most email marketing software and CRM’s, like ConverKit, Mailchimp, HubSpot, ActiveCampaign, etc…will have a/b testing functionality built into the paid plans.

However, if you’re looking to test your website, this will depend on the CMS you’re using.

For example, both Shopify and WordPress websites have a number of free/paid apps and plugins that will allow you to setup and run a/b tests.

You can also use external landing page builders like Clickfunnels or Convertri. 

One of the most popular free tools for a/b testing was Google Optimize but by the end of 2023, the platform will no longer be available.

VWO is a good free alternative to Google Optimize (albeit with some restrictions on traffic and tests on the free plan), however, this does often require you to edit some of the HTML on the variations of your page so it can be quite a technical platform to use.

Alternatively, you can use link shorteners like Linkly or Linksplit which allow you to created a shortened link of your website and automatically rotate the page people will land on, effectively allowing you to do the equivalent of a split URL test as mentioned above!

Best practice for A/B testing and mistakes to avoid at all costs…

One of the most common mistakes we typically see when it comes to a/b testing is people testing too many elements in one test.

For example, if you’re testing 2 different ads, but each ad has a different image, different headline AND different body copy. You then have no idea which of those elements have led to the increase or decrease in conversions

Another trap we see people falling into is thinking they are testing ads but just putting 2 ads up on an ad platform.

The problem with this is that most platforms by default (we’re looking at you Meta! 👀) will automatically rotate ads based on what they think will get the best engagement.

Supposedly the platforms are doing this to help you increase conversions but the reality is, they just choose the ads they believe will keep people on the platform for longer (which is often the opposite to getting you conversions!)

As such we recommend doing manual tests where possible, either by splitting creatives into multiple ad groups with their own dedicated budget, or adjusting the settings accordingly.

LinkedIn for example, give you the option to evenly rotate ads instead of having automatically allocate the budget/impressions, meaning you can make the decisions for yourself based on the metrics that are important to YOU as a business, not the platform!

The next problem we see is when businesses try to run tests based on guesswork instead of being led by the data.

For example, if you were to use heatmapping software on your website and find that the majority of your visits are leaving without even scrolling or that only 5% of visitors reach the bottom of your page, there’s no point testing things at the bottom of your page!

This usually indicates there is a discrepancy in what people have seen in the initial ad (or whatever brought them to your website) to whatever they have seen upon landing on your website (ie your headline or primary CTA)

As such you’ll want to go through the process of testing each of these elements individually and making sure there is congruence at each stage of the user journey, from click to conversion.

Another element that is extremely overlooked in a/b testing is your signup forms or checkouts.

These are typically the areas that will have the highest degree of drop-off as they are the most scary steps for people to take.

Whether I’m giving over my payment details or my email address/phone number as part of a signup process, it’s very likely a lot of doubt will come into the prospects mind.

As such, the more frictionless and easy you can make this process the better.

Sometimes, just changing the order in which you ask questions can lead to a significantly higher conversion rate!

This piece is important to have regardless of which B2B marketing strategy you’re using.

Because here’s the reality, unless you have a truly unique product/service that NOBODY else out there is offering…

How to use a/b testing for messaging or brand positioning

A component people often don’t think about (especially for start-ups) is how you can use a/b testing to determine your messaging and brand positioning.

For example, you could run multiple versions of an ad campaign with the reach/engagement objective; each with a different primary hook, to see which version provides you website the best engagement rate (or even conversion rate if you’re at that stage)

This will give you a good idea of what kind of messaging resonates the most with your target audience before you launch a conversion-based campaign (which will typically have a much higher CPM/CPC) at full budget.

So there you have it guys!

Everything you need to know about A/B testing. We’ll be updating this blog regularly with the latest information around a/b testing so you can make the best decision for your business.

The question is…What will you be testing first?