What's a good way to compare Fastly and my old CDN in a speed test?
If you’re doing a test with Fastly, you’ll first need to set your service up so that it’s caching properly. This is a prerequisite to doing any meaningful benchmarking.
Once you have Fastly set up as you’d like to a production, you can test your whole page using WebPageTest. The application supports side by site testing
You can also test single objects with a quick script we wrote that use apache benchmark and MTR: https://gist.github.com/aspires/8189975
The code:
#!/bin/bash
# Dependencies: ApacheBench, MTR
REQUESTS=100
CONCURRENCY=10
FASTLY='www.example.com.global.prod.fastly.net'
CURRENT='www.example.com'
for url in 'path/to/test' 'path/to/test2' 'path/to/test3'; do
for host in $CURRENT $FASTLY; do
ab -n $REQUESTS -c $CONCURRENCY "http://${host}/${url}" >> fastly.ab.log
#echo "http://${host}/${url}"
done
done
for host in $CURRENT $FASTLY; do
mtr -c $REQUESTS -w -r $host >> fastly.mtr.log
done
There are actually a lot of valid testing options available, and each has its tradeoffs.
-
If you're doing a test with Fastly, you'll first need to set your service up so that it's caching properly. This is a prerequisite to doing any meaningful benchmarking.
Once you have Fastly set up as you'd like to a production, you can test your whole page using WebPageTest. The application supports side by site testing
You can also test single objects with a quick script we wrote that use apache benchmark and MTR: https://gist.github.com/aspires/8189975
The code:
```bash
!/bin/bash
Dependencies: ApacheBench, MTR
REQUESTS=100 CONCURRENCY=10 FASTLY='www.example.com.global.prod.fastly.net' CURRENT='www.example.com'
for url in 'path/to/test' 'path/to/test2' 'path/to/test3'; do for host in $CURRENT $FASTLY; do ab -n $REQUESTS -c $CONCURRENCY "http://${host}/${url}" >> fastly.ab.log #echo "http://${host}/${url}" done done
for host in $CURRENT $FASTLY; do mtr -c $REQUESTS -w -r $host >> fastly.mtr.log done ```
-
Also, @vvuksan has open sourced some great tools for inspecting whole requests: https://github.com/vvuksan/fantomTest
-
A few lessons we've learned from doing CDN testing that could help during evaluation:
- using "backbone" tests is often an inaccurate measure of how well a CDN performs. This is because backbone tests are done by testing nodes sitting in large data centers and not subjected to the normal bandwidth and latency constraints that real users are.
- if a testing provider has a "last mile" option (as opposed to backbone), it's generally a much better way of testing CDN performance since it's closer to what real users experience. Last mile tests should either have testing nodes behind actual home routers, or should be emulating real bandwidth/latency conditions.
- when drawing conclusions, make sure you have ample data. This may vary based on what is being tested, and how, but usually the more data the better
- With large amounts of data, looking at medians is generally better than averages since they eliminate egregious outliers.
- often, CDN testing ignores the performance of cacheable, but long-tail content. Long-tail content is stuff that's cacheable, but doesn't get requested as often, so it's not as "hot." This could be highly personalized content, things that were popular once but are less so now, or any other number of objects. When testing, try to figure out some way to look at the performance of long-tail content. Just because something is cacheable, it doesn't mean the CDN has it at the very edge all the time. Testing long-tail content would expose this.
There's lots more; we're learning lessons about this stuff all the time, but these should be some good things to keep in mind.
Please sign in to leave a comment.
Comments
3 comments