At Appius we've done a lot of work to get conversion improvements using Sitecore split testing for some really powerful results. If you have Sitecore then this feature is available to you but if you haven't used it, then here are a few tips to consider. Even if you have used the feature then this might serve as a little checklist for you, and you might just be nodding along in a few places!
Sitecore Split Testing - 10 Top Tips from Appius, a UK based Sitecore Agency
1. Start simple with a single variable test, and pick the right goal
If you have multiple options in mind you can always test the winner of the first test against the other option. A sequence of tests can get you to the same outcome - but for each test will provide cleaner stats from greater volume involved and division of users between just two variants. A Sitecore split test will fundamentally judge a winner on a single goal so think hard about this and make sure it's an important goal, but still feels relevant in the immediate context of the user journey. An overall site conversion can be great, but if volumes are low and there are a lot of steps in the journey, be careful.
2. Try a component test first rather than a page test
Try something like a different 'call to action', button or set of bullet points. Not only is this easier than a whole page change, but if you change too many factors in a split test you don't really know which change is making the difference, and some can cancel each other out. Users may be influenced by smaller things than you think, so start small and build up from there.
3. Think about visitor volumes and a 'test cell' vs. a test applied to all visitors
If your site is converting and you don't want to disrupt this too much think about how many visitors you would need to expose the test to, in order to see a clear difference in the goal conversion. This doesn't have to be 100% of site visitors. If you have a high volume site you might be able to test with as little as 20% of visits, and gain the insight you need without the risk of 50% of visitors experiencing something new.
4. Get your head around Sitecore's % splits - they may not be what you think they are
Sitecore takes the number of variants you are testing and you choose what % to apply the test to - NOT what % of users visiting the site will see a particular variant. So if you want to expose 10% of your site visitors to a particular new variant you need to do a single variable test and expose the test to 20% of your site visitors, with 1 variant being the regular site content. Got it? good - here endeth the maths lesson for the day.
5. Go manual for the conclusion of the Sitecore split test - automating the selection of the winner is only worth it when you have a large volume of tests running
Sitecore configuration for split testing is tailored towards automatic selection of a winner after a period of time, because that's clever and Sitecore is very clever. However we find that strategy is only needed when you have a LOT of split tests running and don't have the time or availability to check in on them - which you probably do have.
6. Work out how you will see your split test variants and test in your site - it's not easy!
Sitecore makes sure that users only see one variant using a cookie, so that if they return to the site or the page while the test is running they will still see that variant. This means that the results have greater integrity. You can see your variants before you publish the test, of course. However if you want to be 100% sure that you have seen variants and tested them fully you'll need to be fully clearing cookies or use multiple browsers a little bit of luck. As with Sitecore personalisation this functionality is very cookie based. Keeping a second browser back for testing these features can avoid you having to delete all your cookies just to fully test your Sitecore website - Firefox can be a good option for this.
7. Sitecore split testing and personalisation don't always play nicely together
While Sitecore has the ability to make a personalised version a variant for a split test, you can't actually run general personalisation rules on a page at the same time as a split test. So if you have an important piece of personalisation in play you are not going to be able to split test in the same way. Best to plan your split testing around your personalisation strategy rather than on top of it.
8. Capture snapshots of your variants and detailed results as you go or you won't be able to see them again
Once a split test is concluded Sitecore throws away this information and the loser of the split test like dumping a friend who has let it down and it really doesn't want to see again. If you want to keep a good reference of user behaviour outside of the main goal, or even to be able to see that losing variant again so you can report internally, then get cropping and keep the conclusion of the test manual and within your control.
9. Be ready for a non-definitive answer
Sitecore has a very big brain so it likes a statistically definitive answer. If you don't get the volume and difference between outcomes of the variants for it to be certain, it's going to tell you so. So be ready to make your own judgement on a smaller set of stats or finer margin for the winner. You may also need to accept that what the split testing may be telling you is that the variation is not really that important to users. Perhaps something else is a more influential factor on the page or you need to try a more radical variant at the component or page level to make a positive change.
10. Move forward one split test at a time
You may be tempted to test multiple variants and not just a single one. If you take this option you might need to run the test for much longer, when you could have tested one variant against another and moved on. With this approach all you need to decide is what your next test is, because you can only run one at a time and you dont know what the outcome is going to be. Good split testing is about constantly testing your next idea against the previous winner, and this can form a consistent strategy.
SUBMIT YOUR COMMENT