Fastly's Real-Time Log Streaming feature can send log files to Amazon Kinesis Data Streams. Amazon Kinesis Data Streams (KDS) is a real-time data streaming service that can continuously capture data from a variety of sources.
IMPORTANT: This information is part of a limited availability release. For more information, see our product and feature lifecycle descriptions.
NOTE: This logging endpoint is disabled by default. To enable this endpoint for your account, contact support@fastly.com and request it.
NOTE: Fastly does not provide direct support for third-party services. See Fastly's Terms of Service for more information.
How Amazon Kinesis Data Streams work with Fastly log streaming
Amazon KDS sends data records to a stream. Each stream comprises one or more shards. A shard represents a fixed amount of processing capacity and the total processing capacity of a stream is determined by the number of shards. The number of shards may be increased or decreased over the lifetime of a stream. This is important because the Fastly Kinesis logging endpoint monitors the number of shards and attempts to uniformly distribute the log data records across the available shards. When the number of shards for a stream changes, the Fastly Kinesis logging endpoint automatically adjusts in response. The goal is to make the best use of the throughput capability of the stream while minimizing the configuration overhead required for our customers.
If the log volume exceeds the throughput capacity of the stream, Amazon KDS will return errors and these will be visible in the Fastly logging endpoint logs as output that begins with Failed to put record onto stream. The Fastly logging endpoint will attempt a limited number of retries when these errors occur, but if they occur on a regular basis it is likely an indication that the total stream throughput is insufficient for the log volume and the number of shards should be increased.
TIP: For more information about working with Amazon KDS and understanding the capacity limits, refer to the Kinesis Developer Guide.
Prerequisites
Before adding Amazon KDS as a logging endpoint for Fastly services, we recommend creating an Identity and Access Management (IAM) user in your AWS account specifically for Fastly. Grant the user kinesis:PutRecords and kinesis:ListShards permissions for the logging stream. For more information, see Amazon's guidance on understanding and getting your AWS credentials.
Adding Amazon Kinesis as a logging endpoint
After you've registered for an AWS account and created an IAM user in Amazon Kinesis, follow these instructions to add Amazon KDS as a logging endpoint:
- Review the information in our Setting Up Remote Log Streaming guide.
-
Click the Amazon Kinesis Data Streams Create endpoint button. The Create an Amazon Kinesis Data Streams endpoint page appears.
- Fill out the Create an Amazon Kinesis Data Streams endpoint fields as follows:
- In the Name field, enter a human-readable name for the endpoint.
- In the Log format field, optionally enter an Apache-style string or VCL variables to use for log formatting. The Apache Common Log format string appears in this field by default. Our discussion of format strings provides more information.
- In the Access key field, enter the access key associated with the IAM user you created in your AWS account specifically for Fastly. See Amazon's guidance on understanding and getting your AWS credentials for more information.
-
In the Secret key field, enter the secret key associated with the IAM user you created in your AWS account specifically for Fastly. See Amazon's guidance on understanding and getting your AWS credentials for more information.
NOTE: Password management software may mistakenly treat the Secret Key field as a password field because of the way your web browser works. As such, that software may try to auto-fill this field with your Fastly account password. If this happens to you, the Kinesis integration with Fastly services won't work and you will need to enter Secret Key manually instead.
- In the Stream name field, enter the name of the Kinesis stream to which log data will be sent.
- From the Region menu, select the region to stream logs to. This must match the region where you created your Kinesis stream.
-
Click the Advanced options link of the Create an Amazon Kinesis Data Streams endpoint page and decide which of the optional fields to change, if any.
- In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, None, and waf_debug (waf_debug_log). Selecting None creates a logging object that can only be used in custom VCL. See our guide on WAF logging for more information about
waf_debug_log. - Click the Create button to create the new logging endpoint.
- Click the Activate button to deploy your configuration changes.
NOTE: Although Fastly continuously streams logs into Amazon KDS, the Amazon website and API do not make files available for access until after their upload is complete.