Changing the URL that gets logged?

I'd posted this in the VCL subcategory, but maybe it would be better here (Fastly staff, please feel free to delete whichever one you feel is less appropriate).

Short version: Can you change req.url in vcl_log before actually doing any of the logging? I thought you could, and I don't get an error when I try, but it doesn't seem to have any effect either.

Background: For complicated historical reasons, our logic flow for 404s is like this:

Original request comes in
Request gets sent to origin.
404 is returned from origin.
req.url is rewritten to be like req.url = "/my404page"
restart; is called.
The request, now with req.url = "/my404page", goes to the origin, appropriate content is returned, and the content is served to the user with status 404.
vcl_log is called and logging is done.

But because we rewrote req.url in step 4, what gets logged is a request for "/my404page", which is not useful because now every single request that 404s looks exactly the same. I would like to see the original URL.

I thought I could do something like "3.5 set req.http.x-original-url = req.url", and then "6.5 set req.url = req.http.x-original-url", but it does not seem to have any effect at all, as though req.url were immutable in vcl_log. Any suggestions?

I know that it's possible to upload content to Fastly to be served as a synthetic 404, but that isn't an option right now, although it's something we're looking at for the future.

Dara Roberts

February 21, 2020 19:14

Hi JD,

Thanks for posting this!

Have you tried storing the URL as a value?

Doing something like this to capture the original url:

if (req.restarts == 0) {
set req.http.orig-url = req.url;
}

Then using that value in the flow.

The flow would look something like this:

Original request comes in
URL for request is stored in a value. (see above)
Request gets sent to origin.
404 is returned from origin.
req.url is rewritten to be like req.url = "/my404page"
restart; is called.
The request, now with req.url = "/my404page", goes to the origin, appropriate content is returned, and the content is served to the user with status 404.
Previously set value is called
vcl_log is called and req.http.orig-url is logged.

Let us know if you have any questions!

Dara

JD Weiner

February 21, 2020 20:51

Thank you, Dara, this looks really useful! I've tested it in one of our services, and it looks like it does exactly what I need; the only other change that has to be made is to explicitly log "req.http.orig-url" instead of the default req.url.

Comments