A/B testing on CDN improvement

In web development and marketing, A/B testing is an experimental approach to web design which aims to identify changes to web pages that increase or maximize an outcome of interest. There is a great A/B testing article in Smashing Magazine.

20130227-124216.jpg

In order to demonstrate how CDN can improve page load time, I would like to use an A/B testing with faster load time as the success metric. The 110KB Tokyo Skytree photo in version B is cached on EdgeCast CDN while the one in version A is not.

I need your engagement to collect more RUM (Real User Monitoring) data to make this exercise successful! Would you please help clicking the following links?

Which version do you find is faster?

A/B Testing Results

Thank you very much for your help clicking the above-mentioned links between 27th Feb and 5th Mar 2013! Let’s share the test results.

I used free RUM services from Google Analytics, StatusCake and Torbit to measure the page load times. All of them required to add javascript codes in the web pages to collect RUM data. Below were the average load times from Google and StatusCake between 28th Feb and 5th Mar.

Google Analytics:
A/B test – A: 1.89 seconds (20130320 update: 2.34sec)
A/B test – B: 1.79 seconds (20130320 update: 1.88sec)

Google Analytics

StatusCake:
A/B test – A: 2.12 seconds (20130320 update: 2.38sec)
A/B test – B: 2.03 seconds (20130320 update: 2.05sec)

StatusCake

It showed the average page load time of version B – the one using EdgeCast CDN to cache a 100KB photo – was less than that of version A. This simple A/B testing showed EdgeCast CDN improved page load time.

I found both Google and StatusCake could not measure page load time from iOS devices. Possibly because iOS rendering engine did not support W3C navigation timing.

FYI, besides free RUM service, StatusCake also provides free synthetic performance monitoring.

StatusCake synthetic monitoring

Torbit free service did not provide RUM data of individual URL. That said, Torbit did provide great free Realtime RUM tool. It seemed that Torbit could collect data from iOS devices.

Torbit Realtime Monitoring

Advertisements

Joshua Bixby’s podcast; RUM vs Google Analytics

The last two days I found and listened to a great podcast series by Joshua Bixby, VP Application Acceleration at Radware. One podcast was a discussion between Joshua and Torbit CEO Josh Fraser about how performance has changed from a technical metric to a business metric, as well as RUM (Real User Measurement) vs Google Analytics.

As Josh pointed out in the podcast, “accurate measurement is the prerequisite of web site optimization.”

I think one of the challenges of CDN and site optimization adoption in Asia is that many companies do not have good visibilities of their websites. It is not due to lack of budget to buy tools, but the lack of understanding of how to properly manage web sites.

“If you can’t measure it, you can’t manage it!”

More customer education is needed to move the market crossing the chasm!

There are a lot of tools to measure different kind of site performances. Google Analytics is a great FREE tool to gain visibilities of how and by whom your website is used. Together with some iOS apps, you have your site usage info anytime and anyplace! Here is the screen capture of Analytiks about my following demo pages:

1.What is CDN – showing how CDN improve large objects delivery

2.CDN video: progressive download and HLS

3.CDN video: access control

4.EdgeCast can turn your H.264 videos into HLS on-demand streams

5.Control your CDN by Rules Engine

6.SSD storage-based cache server to further improve Small object delivery

7.EdgeCast supports flash-based video using RTMP delivery

8.EdgeCast HTTP Large platform speeds up software download

20130224-234504.jpg

Cedexis Radar is a FREE RUM tool which helps you to understand how good is your website performance from the perspectives of real users around the world! Similar to Google Analytics, what you have to do is to add a few JavaScript lines in those webpages you what to measures. Here is the Cedexis Radar data of my above-mentioned demo pages:

20130224-234711.jpg

With site usage, demographic and performance info, now you know IF you should better serve your users. The next question is HOW.


Test file download using iPhone

Compuware Gomez is a popular tool to test website and CDN performance.

For quick demonstration purpose, Firefox and Firebug can be used to show how a webpage is loaded to browser as well as how CDN improves web site performance, as show is this demo:
http://demo.cheungwaikin.com/accelerate-small-objects.html

20130208-022840.jpg

What if you want to do a quick demo of how CDN improves file download when you are having coffee with your customer? No laptop? No big deal. What you need are three little things:

  • the URL of a file at customer origin

  • the URL of the same file cached on CDN

  • an iOS device
  • Gage: Network Speed Test Utility is a FREE iOS app which tests file download speeds on any server. Add the two URLs in Gage and then you can test and compare how fast (in Kbps) and how long (in sec) to download the files from customer origin and CDN.

    20130208-024638.jpg

    Here is a quick demo:
    http://demo.cheungwaikin.com/accelerate-sw-download.html

    20130208-025728.jpg