How to run SEO split tests

Time based SEO Tests are great for quickly seeing whether a change to a page or the site has a positive or negative effect on rankings and clicks. They compare the pre change period against the post change period to see how clicks, impressions, click through rates, and ranking positions have changed.

As you are only testing a single page, but over two different time periods, the results do leave you open to being impacted by algorithm updates and season variances. You can remove this risk by checking for overall site trends during the control and test period to notice any big changes, but it still leaves the results open to interpretation.

One effective way to remove the risk of algorithm updates and seasonal changes is to run an SEO split test.

People are familiar with split testing from a conversion rate optimization perspective. This is where you use a tool such as Visual Website Optimizer or Optimizely to show 2 variations of the same page to different groups of people to see which version converts better.

It’s impossible to do that from an SEO perspective. You can’t create two variations of the same url and submit them to Google to see which ranks the best.

But there is a way to run SEO split tests, by comparing pages of the same type.

Back in 2015 the Pinterest Engineering Team wrote an article explaining how they ran SEO split tests on their site.

Screenshot of the pinterest SEO split test results

From this article, we built out the SEO split testing functionality contained within SEOTesting.com to give you an automated way of collecting results.

But first let’s go into more details on how to run an SEO split test...

How to run an SEO split test

To run an SEO split test you take a number of pages that are of the same type and split them into two groups - a control group and a test group. Each of these pages should be getting impressions and clicks from Google’s organic search listings.

These pages could be blog posts, product pages, category pages, or any other types of pages that are of the same type.

The first group of pages is going to be the control group. No change will be made to these urls.

The second group of pages is the test group. These are the pages you are going to make the change to that you want to test.

Once your pages have been grouped, and your changes to the test group have been made and submitted to Google Search Console for recrawling, you want to start recording results of clicks for a period before the change date. This pre-test period should be at least 2 weeks.

For each day you then calculate:

Daily Difference = Average[test group clicks] - Average[control group clicks]

This will give you a daily difference.

You can then average out the daily difference to create a pre-test baseline number. This gives you something to compare once the changes to the test group of pages are recognised by Google and ranked accordingly.

Once the results start to flow in for and beyond the ‘change date’ you continue to track the daily:

Daily Difference = Average[test group clicks] - Average[control group clicks]

Ideally if your changes are having a positive effect, the total clicks from the test group will increase, and so the daily difference number will get larger and the difference graph will trend up.

If your changes are having a negative effect, the total clicks from the test group will decrease, and therefore the daily difference will decrease and the difference graph will trend down.

Screenshot showing SEO split test results in SEOTesting.com

As daily differences are collected for the test period, a new average of daily differences can be calculated just for the period since the changes were made to the test group.

This test average can then be compared to the baseline calculated previously. If the test average is above the baseline, you can consider the test to have brought more clicks to the test group and been successful. You would then want to consider rolling out that change to all pages of that type.

If the test average is below the baseline, you can say the test changes have not brought about a positive result. I would roll back those changes from the test group.

Why do SEO split tests work?

As you are comparing two groups of pages that are the same type, any seasonal changes or algorithm updates should affect both groups.

You check the average click difference between the two groups, before and after the change, so it is with more certainty you can say the changes to the test group would be the reason for any increase or decrease in click difference.

The fact we average out the clicks for the control and test group before working out the click difference also increases the trustworthiness of the results. It won’t be just a single url that rockets in clicks and affects the results, because the average will bring that down if other pages in the test group haven’t risen accordingly.

Seasonal or algorithm updates would not affect the click difference, as they would apply to both groups. On page changes would therefore be the thing that affects any movement up or down of daily and average click difference.

Can this data be collected manually?

All the data you need for running split tests is available through Google Search Console.

The problem however is the amount of data that needs to be collected on an individual daily basis. If you have 10 pages in your control group, and 10 pages in your test group, you end up looking up the data for 20 pages each day. If you spread that out over an 8 week period - that means looking up clicks 1,120 times. That is a lot of mouse clicks and a lot of time.

Thankfully SEOTesting.com can run SEO split tests for you…

How to setup an SEO split test in SEOTesting.com

SEOTesting.com has the ability to track the results of your SEO split tests each day to calculate the daily differences, the bench mark, and the average of the test period.

You need to make the changes to your test pages, but once those are done you can setup your split test in SEOTesting.com as the screenshot below

Screenshot showing how to create a new SEO split test in SEOTesting.com

Giving the test a meaningful name and description that includes the hypothesis of your test will be really helpful when you are looking back on the results when the test finishes.

You can set the ‘date change’ to be in the past if you want to run a split test on historical changes, but by default this value will set to today.

The control urls text area is where you can paste in the pages from your control group. Each url should be on a new line.

The same goes for the test urls.

You can decide how long you want to run your SEO split test for with the test period drop down. If you want to run your split test for longer than 6 weeks you can tick the box for it to run as an ‘On Going Test’.

Finally you can enter a 3 character ISO country code if you want to filter the data and results to a specific country.

Upon creating the test, SEOTesting.com will do some work in the background to collect data for any periods that are already available. Each day from then on, as Google Search Console updates, your split test will update with new results so you can judge on a daily basis if your test is indicating a positive or negative result.

Summary

I hope this guide has given you a good overview of SEO split testing and you are excited to try it out. If you have any questions about this article please feel free to drop me an email : nick.swan@seotesting.com

SEOTesting.com has a free 14 day trial, so you can set up an SEO split test and start to see the result come in. You can register for free here. Want to learn SEO? Check out our free guide on learning SEO.