A/B testing on CDN improvement

In web development and marketing, A/B testing is an experimental approach to web design which aims to identify changes to web pages that increase or maximize an outcome of interest. There is a great A/B testing article in Smashing Magazine.

20130227-124216.jpg

In order to demonstrate how CDN can improve page load time, I would like to use an A/B testing with faster load time as the success metric. The 110KB Tokyo Skytree photo in version B is cached on EdgeCast CDN while the one in version A is not.

I need your engagement to collect more RUM (Real User Monitoring) data to make this exercise successful! Would you please help clicking the following links?

Which version do you find is faster?

A/B Testing Results

Thank you very much for your help clicking the above-mentioned links between 27th Feb and 5th Mar 2013! Let’s share the test results.

I used free RUM services from Google Analytics, StatusCake and Torbit to measure the page load times. All of them required to add javascript codes in the web pages to collect RUM data. Below were the average load times from Google and StatusCake between 28th Feb and 5th Mar.

Google Analytics:
A/B test – A: 1.89 seconds (20130320 update: 2.34sec)
A/B test – B: 1.79 seconds (20130320 update: 1.88sec)

Google Analytics

StatusCake:
A/B test – A: 2.12 seconds (20130320 update: 2.38sec)
A/B test – B: 2.03 seconds (20130320 update: 2.05sec)

StatusCake

It showed the average page load time of version B – the one using EdgeCast CDN to cache a 100KB photo – was less than that of version A. This simple A/B testing showed EdgeCast CDN improved page load time.

I found both Google and StatusCake could not measure page load time from iOS devices. Possibly because iOS rendering engine did not support W3C navigation timing.

FYI, besides free RUM service, StatusCake also provides free synthetic performance monitoring.

StatusCake synthetic monitoring

Torbit free service did not provide RUM data of individual URL. That said, Torbit did provide great free Realtime RUM tool. It seemed that Torbit could collect data from iOS devices.

Torbit Realtime Monitoring

Advertisements

4 Comments on “A/B testing on CDN improvement”

  1. Anonymous says:

    I’ve been surfing on-line more than three hours as of late, yet I by no means discovered any attention-grabbing article like yours. It’s lovely worth enough for me. In my view, if all site owners and bloggers made excellent content as you probably did, the net might be a lot more useful than ever before.

    • Andre says:

      Are you from Italy? It is great you enjoy my blog. Writing helps me to organize the things I learn, as well as to share and contribute back to the great Internet community!

      I am working on another article which I hope will be published late this week:)

  2. deniar says:

    Great Andre.. thx

  3. […] Webpagetest, StatusCake and Load Impact for simple synthetic performance measurement. I will A/B test the revamp site before turning it into production […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s