Intro to using curl with Fastly

Follow
Chris Kruse

Fastly recognizes the power of being able to manage your configurations via API. The platform was built API first, and most everything that is done in the web GUI, can be done via an API. This document will lay out some of the more useful API calls.

Purging:
Fastly provides types of cache purges, making it easy to purge a single URL, a group of URLs (utilizing a surrogate key) or an entire service. Additionally, there is the soft purge which allows purged (stale) content to remain until the newest version can be retrieved from origin. You can read more about soft purge at [here] (https://docs.fastly.com/guides/purging/soft-purges).

Note: Best practice recommends adding a whitelist for purging to ensure it objects are only purged from authorized sources. This can be found here and another resource here.

Starting simple… Purge by URL:

curl -XPURGE <url>

Or over a TLS connection:

curl -XPOST https://api.fastly.com/purge/<url>

Purge groups of content with surrogate keys:

curl -XPOST -H "Fastly-Key:<Fastly API Key>" https://api.fastly.com/service/<serviceID>/purge/<surrogatekey>

Purge All:

curl -XPOST -H "Fastly-Key:<Fastly API Key>" https://api.fastly.com/service/<serviceID>/purge_all

SOFT Purge URL:

curl -Ssi -XPURGE -H "Fastly-Soft-Purge:1" http://<url>

Managing Configurations:
Fastly archives all of your configurations and locks old configs, so they cannot be overwritten. It is important to understand how to check if the config is locked, how to clone, and how to activate a configuration.

Get versions (notice the “active” key for finding current version):

curl -XGET -H "Fastly-Key:<fastly_api_key>" https://api.fastly.com/service/<service_id>/version

Clone the configuration to a new version:

curl -XPUT -H "Fastly-Key:<fastly_api_key>" https://api.fastly.com/service/<service_id>/version/<current_version_number>/clone

Activate the new version:

curl -XPUT -H "Fastly-Key:<fastly_api_key>" https://api.fastly.com/service/<service_id>/version/<cloned_version_number>/activate

Custom VCL:
Being able to build custom VCLs with Fastly gives the power to the user to programmatically make changes to their configuration. Fastly utilizes Vanish’s VCL language with custom enhancements.
Note: The ability to upload custom VCL is disabled when you sign up by default. It can be enabled by requesting it through support. support@fastly.com

Get the boilerplate VCL to start building your custom VCL:

curl -XGET -H "Fastly-Key:<fastly_api_key>" https://api.fastly.com/service/<service_id>/version/<version>/boilerplate

Uploading a custom VCL:

curl -XPOST -H "Fastly-Key: <fastly_api_key>" https://api.fastly.com/service/<service_id>/version/<version>/vcl -d ‘name=<vcl_name>’ --data-urlencode “(content=$(cat <filename.vcl>)”

Setting the VCL to main (it is set as include by default):

curl -XPUT -H "Fastly-Key: <fastly_api_key>" https://api.fastly.com/service/<service_id>/version/<version>/vcl/<vcl_name>/main

Getting a vcl file from a specific version:

curl -XGET -H "Fastly-Key: <fastly_api_key>" https://api.fastly.com/service/<service_id>/version/<version>/vcl

Debugging:
Visibility into how your request is interacting with a cache node is important when trying to troubleshoot caching behavior. Fastly provides some tools through the html headers to help customers diagnose problems with connection behavior and configuration problems.

Turning on the debug flag for a page request:

curl -svo /dev/null [url] -H "Fastly-Debug: true"

The Details: Adds additional headers into the response to understand how the object is being handled by Fastly caches.

  • Fastly-Debug-Path: which cache servers serve the request
  • Fastly-Debug-TTL: the TTL of the object in the queried server(s)
  • Fastly-Debug-Digest: the hash digest of the object.

Run a test with shielding disabled:

curl -svo /dev/null [url] -H "Fastly-No-Shield: true"

The Details: Forces the cache node to interact directly with origin on a MISS. Shielding can introduce problems to VCL code that works perfectly when the request goes through a single node. Enabling this header helps troubleshoot VCL when shielding is involved.

Determine total time for a request to be completed:

curl -s -w "%{time_total}\n" -o /dev/null <url>

The Details: Gather the time it takes for a request to complete to gather some rough benchmark numbers. Put it in a for loop, and you can get look for performance patterns. Add the sleep command in the loop to pause between tests.

One last Tip:
When troubleshooting your configuration by hand, you will often find yourself trying to parse through a JSON response that is all on one line. This makes it difficult for a user to visually find the data they are looking for in a timely manner. There are two tools that come standard on OSX, json_pp or python –m json.tool. By piping your curl command to one of these tools, the output will be formatted for a human rather than a program. Happy curling.

0

Comments

2 comments

Sort by Date Votes
  • Peter Wohlers

    You can actually expand the -w "%{time_total}\n" timing switch to include dns lookup time, connect time, tls negotiation, etc.. to help pin down where bottlenecks are. I tend to just create an alias or call a bash script to bundle all of the juicy curl bits into a single command.

    e.g.: curl -w "\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n" -svo /dev/null -HFastly-Debug:1 https://www.example.com/ --compress Will give you all your headers but the timings will put things like the following at the bottom:

    ``` Lookup time: 0.526 Connect time: 0.542 PreXfer time: 0.690 StartXfer time: 0.729

    Total time: 0.837 ```

    0
    Comment actions Permalink
  • Rogier Mulhuijzen

    Even better, you can store such statistics format strings in files, and just reference the files.

    Example commandline:

    curl -w "@curl-format.txt" -so /dev/null --compressed http://example.com/

    My curl-format.txt looks like this: \n time_namelookup: %{time_namelookup}\n time_connect: %{time_connect}\n time_appconnect: %{time_appconnect}\n time_pretransfer: %{time_pretransfer}\n time_redirect: %{time_redirect}\n time_starttransfer: %{time_starttransfer}\n ----------\n time_total: %{time_total}\n \n

    But normally I use curl-format-compact.txt which looks like: DNS: %{time_namelookup} Connect: %{time_connect} AppConn: %{time_appconnect} preTX: %{time_pretransfer} Redir: %{time_redirect} TXstart: %{time_starttransfer} Total: %{time_total}\n

    Credit goes to this blog post: https://josephscott.org/archives/2011/10/timing-details-with-curl/

    0
    Comment actions Permalink

Please sign in to leave a comment.

New post