Brotli Compression Support
Chrome is starting to support Brotli compression via HTTPS. See https://groups.google.com/a/chromium.org/forum/#!searchin/blink-dev/brotli/blink-dev/JufzX024oy0/WEOGbN43AwAJ
Will fastly use Brotli compression when it is better than gzip and the browser supports it?
Thanks!
-
Official comment
Hi Community Forum Folks,
Brotli compression at the edge (by Fastly) in beta currently but for now we're limiting participation to accounts with TAMs as the customer VCL needed is not trivial.
We have always and continue to supportss caching/serving of origin content that has already been compressed with Brotli. For more information on this see this comment.
If you are interested in compression at edge, feel free to contact submit a request to our support team for your company to be added to the list of interested participants.
Cheers,
Dara
Comment actions -
Will fastly use Brotli compression when it is better than gzip and the browser supports it?
I expect we'll move in that direction. Rolling a new compression format across our network is something that will take some nontrivial R&D, as with similar changes, but we're always trying to move things forward.
-
Just an update, I have brotli enabled on https://meta.discourse.org it works, I can see
Vary:Accept-Encodingin the response headers. Is there any reason this is not coming through the CDN? -
Good news, everyone! (Did you hear Professor Farnsworth's voice in your head? I did.)
For those wanting to support Brotli right now, with Fastly, there's a way to work around our normalization of the
Accept-Encodingheader. Now first a warning, do not skip the normalization step below, and don't use this workaround if your origin doesn't actually support Brotli. In either case you would be hurting your hit ratio, and that would actually hurt performance!The trick is to use the
Fastly-Orig-Accept-Encodingheader, and normalize that to one of 3 values:gzip,br, or not set at all. (The latter being a value of sorts in this case.)To achieve this, you can use this VCL in
vcl_recv:if (req.http.Fastly-Orig-Accept-Encoding) {
if (req.http.User-Agent ~ "MSIE 6") {
# For that 0.3% of stubborn users out there unset req.http.Accept-Encoding;
} elsif (req.http.Fastly-Orig-Accept-Encoding ~ "br") {
set req.http.Accept-Encoding = "br";
} elsif (req.http.Fastly-Orig-Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} else {
unset req.http.Accept-Encoding;
}
}A couple of notes: 1. While this code is fine, if any new compression scheme shows up that has
brin its identifier, and a browser would support that, but not Brotli, things would break. Now this is pretty unlikely to happen, but if you're worried about it, you can use(^|[, ])br($|[, ])as the regular expression instead. 2. Those of you who have read my Vary blog post might wonder "what aboutdeflate?". Well, if you're really worried about the less than 0.02% of user agents out there that supportdeflatebut notgzip, you can add that back in, but since sending them an uncompressed version of the object doesn't actually cause errors, I wouldn't worry about it personally, and would prefer the slight increase in hit ratio that you gain from leavingdeflateout of things.[quote="sam, post:5, topic:578"] If I request say:
https://cdn.fastly.com/asset.js?brotli=allow
The presence of brotli=allow can flag allowing of brotli in normalization routine. [/quote]
To get this effect, you would expand the code to:
if (req.url ~ "brotli=allow") {
if (req.http.Fastly-Orig-Accept-Encoding) {
if (req.http.User-Agent ~ "MSIE 6") {
# For that 0.3% of stubborn users out there
unset req.http.Accept-Encoding;
} elsif (req.http.Fastly-Orig-Accept-Encoding ~ "br") {
set req.http.Accept-Encoding = "br";
} elsif (req.http.Fastly-Orig-Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} else {
unset req.http.Accept-Encoding;
}
}
} -
So just to clarify,
req.http.Fastly-Orig-Accept-Encodingis something you add from your side? not something requesters are expected to add to the HTTP requests?How do I go about inserting the rule using the Fastly UI? I don't appear to have the ability just to simply edit the generated VCL?
-
[quote="sam, post:7, topic:578"] So just to clarify,
req.http.Fastly-Orig-Accept-Encodingis something you add from your side? not something requesters are expected to add to the HTTP requests? [/quote]Yes. Before we normalize
Accept-Encodingin our master VCL, we copy its value toFastly-Orig-Accept-Encoding.[quote="sam, post:7, topic:578"] How do I go about inserting the rule using the Fastly UI? I don't appear to have the ability just to simply edit the generated VCL? [/quote]
You could add a Header object with the following settings:
Name:
Normalize Accept-Encoding for BrotliType:RequestAction:SetDestination:http.Accept-EncodingSource:if(req.http.User-Agent ~ "MSIE 6", "identity", if(req.http.Fastly-Orig-Accept-Encoding ~ "br", "br", if(req.http.Fastly-Orig-Accept-Encoding ~ "gzip", "gzip", "identity")))Notes:
- Instead of unsetting the header if there's to be no compression, it uses the value
identity. This value means "no compression" as well. See https://tools.ietf.org/html/rfc7231#section-5.3.4 -
Because of the use of
"identity"instead of an empty value, clients unable to do gzip will cause cache misses.If this is a concern, instead of using
"identity"you can use a non-existent header, likereq.http.My-Non-Existent-Header. That will basically unsetAccept-Encoding.For security purposes, you should add a Header object (type:
Request, action:Delete) with a lower priority number, to ensure that that header is indeed non-existent. By setting the header to a value, someone can basically set Accept-Encoding to whatever they want, and when using a unique value for each request, bypass caching.
Alternatively, you can email support to ask for Custom VCL to be enabled for your account. See also https://docs.fastly.com/guides/vcl/uploading-custom-vcl and https://docs.fastly.com/guides/vcl/mixing-and-matching-fastly-vcl-with-custom-vcl.
Let me know how that works, or if you have any further questions!
- Instead of unsetting the header if there's to be no compression, it uses the value
-
Mea culpa. I posted that code without testing it, and it turns out there was something I missed in our Master VCL that made this not work.
Good news is that that has been fixed, and the above code now works.
If you've had this code in your VCL, you'll have to purge things that were requested in the timeframe between deploying that code and roughly an hour ago.
Also, thanks to Donald Stufft of PyPI fame for pointing out the code didn't have the desired effect. (He is running this in production now.)
-
Chrome and Firefox have both shipped brotli and Microsoft Edge is in the process of shipping it. Getting mainstream support for this as @codinghorror suggested would be really nice!
-
Safari 11 on iOS 11 and High Sierra will support brotli as well. Therefore all Mayor Browser will support Brotli in October 2017.
Has there been any progress in a fastly implementation in the last 1 3/4 Years on this matter? And I do not mean the current working hack, with accept-encoding 😉
-
also wanted to ask that... is there any progress up until now? i want to know more... i'm currently sitting home doing nothing but searching for drug coupons because of my health issues and my lack of money so instead of not doing anything i decided to learn. that's why i also would like to talk with more experienced people about some things. here... can i ask questions even if they might seem stupid to you? i'm a newbie... just registered!
-
this vcl is little bad I think.
Client is request to the fastly.
Accept-Encoding:gzip, deflate, brWhen origin server does not support the brotli, fastly got the un-compressible files even if supported the gzip both origin server and clients.
In order to avoid this problem, req.http. Fastly-Orig-Accept-Encoding value should be append (not overwrite)but I can not find this option.
when fastly backend request include gzip, deflate, br, the origin server can compress contents using gzip.
Is my understanding of this correct?
-
I had adjust VCL. but I don't know FASTLY could used vcl backend fetch.
vcl_recv if (req.restarts == 0) { # normalize Accept-Encoding to reduce vary if (req.http.Accept-Encoding) { #brotli handling if (req.http.Accept-Encoding ~ "br") { set req.http.TMP-Accept-Encoding = "br"; set req.http.X-brotli = "true"; } if (req.http.User-Agent ~ "MSIE 6") { unset req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.TMP-Accept-Encoding = regsub(req.http.TMP-Accept-Encoding, "$", ",gzip"); set req.http.TMP-Accept-Encoding = regsub(req.http.TMP-Accept-Encoding, "^,", " "); } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.TMP-Accept-Encoding = regsub(req.http.TMP-Accept-Encoding, "$", ",deflate"); set req.http.TMP-Accept-Encoding = regsub(req.http.TMP-Accept-Encoding, "^,", " "); } else { unset req.http.Accept-Encoding; } } }vcl hash
##brotli if(req.http.X-brotli == "true") { hash_data("brotli"); } vcl_backend_fetch unset bereq.http.Accept-Encoding; unset bereq.http.X-brotli; set bereq.http.Accept-Encoding = bereq.http.TMP-Accept-Encoding; -
Hey,
We looked at this, and none of these solutions were great for us. We went a different route.
We used VCL Snippets like so:
Name: Normalize Accept-Encoding for Brotli Type (placement of the snippet): within subroutine, recv VCL: declare local var.normalized STRING; set var.normalized = ""; if (req.http.Fastly-Orig-Accept-Encoding) { # Supported encodings, in order of preference. if (req.http.Fastly-Orig-Accept-Encoding ~ "([ ]*,[ ]*)?br[ ]*(,[ ]*|$)") { set var.normalized = var.normalized + "br,"; } if (req.http.Fastly-Orig-Accept-Encoding ~ "([ ]*,[ ]*)?gzip[ ]*(,[ ]*|$)") { set var.normalized = var.normalized + "gzip,"; } if (req.http.Fastly-Orig-Accept-Encoding ~ "([ ]*,[ ]*)?deflate[ ]*(,[ ]*|$)") { set var.normalized = var.normalized + "deflate,"; } } if (var.normalized == "" ) { unset req.http.Accept-Encoding; } else { set req.http.Accept-Encoding = regsub(var.normalized, ",$",""); }This allowed us to: - not have to enable custom VCL and carry that burden forward - support many different accept encodings, and have our server pick which it supports - have a simple place to understand where to change the accept-encoding behaviour
Hope this helps someone else.
-
Guided by the philosophy that "origin knows best". Put differently, if we want to disable deflate support, it should be up to the origin, not the edge.
In our case, AFAIK, a cache miss on Accept-Encoding isn't a huge deal (n! == 6), but it's easy to see how this can be adapted if it is.
It would be interesting in practice to have a metric on how many permutations are actually cached, and what their hit/miss ratio is. I wonder if there's a way to check?
-
Regarding Fastly-Orig-Accept-Encoding: I believe it's not currently working as intended.
If you have the origin shield service activated, Fastly-Orig-Accept-Encoding will be rewritten at the shield, losing the original value from the client.
Say your original Accept-Encoding header from the browser is "gzip,deflate,br" and your custom VCL sets it to "br." Once the request reaches the origin shield, Fastly-Orig-Accept-Encoding will be set to "br," not to the original value of "gzip,deflate,br."
It seems to me that Fastly-Orig-Accept-Encoding should reflect the original Accept-Encoding value as sent by the browser/client. Writing it a second time at the shield can cause issues with testing and troubleshooting.
-
Well, in my vcl_recv, I typically run most functionality only once, then mark the request as processed using an X- header, to prevent it from being re-processed at the origin shield. There's no need to scan through a bunch of conditionals all over again at the shield and duplicate the work. But apparently Fastly is normalizing/sanitizing the Accept-Encoding header both at the receiving server and then again at the origin. And further, the Fastly-Orig-Accept-Encoding header is losing the original value that represented what the browser/client had sent over. There may be cases where some customers would like to access their client's actual, original Accept-Encoding headers for analytics or other reasons. Regardless, it does seem to make sense to me to only normalize the header once.
-
Now that it's been a few years, and Brolti is very widely supported, how's that R&D going?
I expect we'll move in that direction.
Rolling a new compression format across our network is something that will take some nontrivial R&D,
as with similar changes, but we're always trying to move things forward.That was 4 years ago. I'd like a progress report.
-
Hi, the snippets provided don't work for me. I tried debugging by adding these headers:
set resp.http.X-Fastly-Accept-Encoding = req.http.Fastly-Orig-Accept-Encoding;
set resp.http.X-Fastly-Accept-Encoding-2 = req.http.Accept-Encoding;My request headers contain:
accept-encoding: gzip, deflate, br
However, the response headers are:
x-fastly-accept-encoding: gzip
x-fastly-accept-encoding-2: gzipMy recv snippet contains:
if (req.http.Fastly-Orig-Accept-Encoding && req.url ~ "(.+\.js)(\?|$)") { set var.path = re.group.1; if (req.http.Fastly-Orig-Accept-Encoding ~ "br") { set req.http.Accept-Encoding = "br"; set var.path = var.path ".br"; } set req.url = var.environment var.path; }This never loads the Brotli files. Gzip is also disabled on Fastly.
Edit: I realized this might be because my company runs Cloudflare in front of Fastly for some reason.
Please sign in to leave a comment.
Comments
35 comments