Ivan Nikitin Ivan Nikitin home facebook linkedin rss

CloudFront vs S3 for hosting software downloads

I always made two assumptions:

The first assumption is based on my personal behavior. However, I never know how important this factor is. The second is solely a result of advertising. If some product is sold as it reduces latency and increases download speed and is sold by a respected company, I tend to believe them.

After switching my ISP this summer, I got a reason to doubt about my assumption #2. I use SummaTelecom provider and it reliably shows better download speeds from an Ireland-based S3 bucket than from a CloudFront account. On average, it is about 1400 vs 600 kbytes/second. These results were stable for a week.

If I pay for a service that promises to improve download speed and it doesn't do the job for my ISP customers, I doubt if it really helps other people in other countries. So, I made an experiment and compared how serving downloads from CloudFront affects sales.

Experiment results. If you don't want to read further, here are results. * Software download speed is indeed important factor for customers. * CloudFront is better than S3 for an average customer from US, UK and Russia.

Here is what I did. I made three download pages. Two of them contained links to software downloads on CloudFront. This was my originals. I needed two of them to surely understand Google Analytics shows me reliable data. In other words, you can see experiment results where variation outperforms original by a margin, but never see it in the real life after switching page variations. This happens because your experiment results were unreliable. To avoid this, I always include two originals into experiment and run it until their conversions match.

For the variation page I created three S3 buckets in every major region: Ireland, US Standard and Sydney. One for every major market: Europe, US and Australia/East Asia.

I uploaded exactly the same software downloads into every bucket, plus a little text file that I use to choose download region. Then I requested that small file from a JavaScript function on page load. Request that completes first is from the closes bucket region. Then my function replaces download link href attribute. Here is how:

try {
    var endpoints = ["https://s3.amazonaws.com/bucket-us/","https://s3-eu-west-1.amazonaws.com/bucket-eu/",
            "https://s3-ap-southeast-2.amazonaws.com/bucket-au/"];
    var $winlink = jQuery("#win-link"), $maclink = jQuery("#mac-link");
    endpoints.forEach(function(endpoint) {
         var handler = function () {
          if ($winlink.attr("href").charAt(0) != "/")   // By default contains relative link like '/download' that points to CloudFront distribution
            return;
          $winlink.attr("href", endpoint + "vmark-installer.exe");
          $maclink.attr("href", endpoint + "vmark-installer.dmg");
         };
         jQuery.ajax(endpoint + "latency.txt", {
           dataType: "jsonp",
           success: handler,
           error: handler
         });
    });
} catch(e) {
  console.log(e);
}

Then I configured an experiment in Google Analytics and sent a portion of traffic into it. Traffic was evenly spread between 3 variations. Experiment goal completes when user opens order page from the app itself.

Here are experiment results for two weeks:

While daily results are unstable because of a small sample count, both pages with CloudFront links show very close results (22.57% vs 22.72%). This makes me believe experiment results are statistically significant and can be implemented in production website.

"Download-from-S3" results are worse by 12% that shows how faster downloads are important for consumer software sales.

comments powered by Disqus