A/B testing on CDN improvement

In web development and marketing, A/B testing is an experimental approach to web design which aims to identify changes to web pages that increase or maximize an outcome of interest. There is a great A/B testing article in Smashing Magazine.

In order to demonstrate how CDN can improve page load time, I would like to use an A/B testing with faster load time as the success metric. The 110KB Tokyo Skytree photo in version B is cached on EdgeCast CDN while the one in version A is not.

I need your engagement to collect more RUM (Real User Monitoring) data to make this exercise successful! Would you please help clicking the following links?

• A/B test – version A
• A/B test – version B

Which version do you find is faster?

A/B Testing Results

Thank you very much for your help clicking the above-mentioned links between 27th Feb and 5th Mar 2013! Let’s share the test results.

I used free RUM services from Google Analytics, StatusCake and Torbit to measure the page load times. All of them required to add javascript codes in the web pages to collect RUM data. Below were the average load times from Google and StatusCake between 28th Feb and 5th Mar.

Google Analytics:
A/B test – A: 1.89 seconds (20130320 update: 2.34sec)
A/B test – B: 1.79 seconds (20130320 update: 1.88sec)

StatusCake:
A/B test – A: 2.12 seconds (20130320 update: 2.38sec)
A/B test – B: 2.03 seconds (20130320 update: 2.05sec)

It showed the average page load time of version B – the one using EdgeCast CDN to cache a 100KB photo – was less than that of version A. This simple A/B testing showed EdgeCast CDN improved page load time.

I found both Google and StatusCake could not measure page load time from iOS devices. Possibly because iOS rendering engine did not support W3C navigation timing.

FYI, besides free RUM service, StatusCake also provides free synthetic performance monitoring.

Torbit free service did not provide RUM data of individual URL. That said, Torbit did provide great free Realtime RUM tool. It seemed that Torbit could collect data from iOS devices.

---