In web development and marketing, A/B testing is an experimental approach to web design which aims to identify changes to web pages that increase or maximize an outcome of interest. There is a great A/B testing article in Smashing Magazine.
In order to demonstrate how CDN can improve page load time, I would like to use an A/B testing with faster load time as the success metric. The 110KB Tokyo Skytree photo in version B is cached on EdgeCast CDN while the one in version A is not.
I need your engagement to collect more RUM (Real User Monitoring) data to make this exercise successful! Would you please help clicking the following links?
Which version do you find is faster?
A/B Testing Results
Thank you very much for your help clicking the above-mentioned links between 27th Feb and 5th Mar 2013! Let’s share the test results.
A/B test – A: 1.89 seconds (20130320 update: 2.34sec)
A/B test – B: 1.79 seconds (20130320 update: 1.88sec)
A/B test – A: 2.12 seconds (20130320 update: 2.38sec)
A/B test – B: 2.03 seconds (20130320 update: 2.05sec)
It showed the average page load time of version B – the one using EdgeCast CDN to cache a 100KB photo – was less than that of version A. This simple A/B testing showed EdgeCast CDN improved page load time.
I found both Google and StatusCake could not measure page load time from iOS devices. Possibly because iOS rendering engine did not support W3C navigation timing.
FYI, besides free RUM service, StatusCake also provides free synthetic performance monitoring.
Torbit free service did not provide RUM data of individual URL. That said, Torbit did provide great free Realtime RUM tool. It seemed that Torbit could collect data from iOS devices.
The last two days I found and listened to a great podcast series by Joshua Bixby, VP Application Acceleration at Radware. One podcast was a discussion between Joshua and Torbit CEO Josh Fraser about how performance has changed from a technical metric to a business metric, as well as RUM (Real User Measurement) vs Google Analytics.
As Josh pointed out in the podcast, “accurate measurement is the prerequisite of web site optimization.”
I think one of the challenges of CDN and site optimization adoption in Asia is that many companies do not have good visibilities of their websites. It is not due to lack of budget to buy tools, but the lack of understanding of how to properly manage web sites.
“If you can’t measure it, you can’t manage it!”
More customer education is needed to move the market crossing the chasm!
There are a lot of tools to measure different kind of site performances. Google Analytics is a great FREE tool to gain visibilities of how and by whom your website is used. Together with some iOS apps, you have your site usage info anytime and anyplace! Here is the screen capture of Analytiks about my following demo pages:
With site usage, demographic and performance info, now you know IF you should better serve your users. The next question is HOW.