Considerations for implementing IP blocking using Access Control Lists (ACLs) at scale

We provide ways to creating an IP block list using the UI, as described here: https://docs.fastly.com/guides/access-control-lists/using-the-ip-block-list, but a number of people have inquired wondering how to implement IP blocking at scale when hundreds or thousands of IPs are attacking at once. Through a smart combination of real-time log analysis, subnet blocking, and geo-IP blocking, a number of customers have been able to create their own automated systems that will perform effective hands-off attack mitigation. Here is a generic diagram showing what kind of architecture we recommend:

(Diagram by Travis Sanders, Principal Sales Engineer)

You will first want to familiarize yourself with our Access Control List (https://docs.fastly.com/guides/access-control-lists/about-acls) feature, which allows you to build a container of referenceable entries that is attached to a specific service. While you can create and manage an IP-blocking Access Control List (ACL) using our web interface (as described here: https://docs.fastly.com/guides/access-control-lists/working-with-acls-using-the-web-interface), managing this list at scale will necessitate the use of automation via our API since there may be thousands of updates to this list every few minutes depending on your needs. There are ways to make single dictionary updates via our API, but since our API carries a default per-user rate limit of 1000 requests per hour, we highly recommend using an ACL bulk update call (https://docs.fastly.com/api/config#acl_entry_c352ca5aee49b7898535cce488e3ba82), instead of an individual call for each update and deletion.

How are you going to know when to make these calls? This is where the power of our real-time logging (https://docs.fastly.com/guides/streaming-logs/setting-up-remote-log-streaming) comes into play. By funneling the logs into your own analyzing service, you can analyze which traffic might be coming from illegitimate IPs and then use calls to the ACL API to add those IPs to your block list. The needs of different customers will determine how to evaluate what constitutes an illegitimate IP. The most common variables to analyze are:

User-Agent (req.http.user-agent): If a known User-Agent is incompatible with a website (or set of pages) that are being constantly requested, then that User-Agent can probably be safely flagged. For example, Supreme’s websites require JavaScript to run, and will therefore block browsers that are known to not support JavaScript (such as headless browsers that might run bot traffic).

IP (client.ip): It is easy enough to flag a single IP requesting the same page too often, but more sophisticated attacks will originate from various different IPs. However, some attacks will come from different IPs under the same subnet, and luckily, our ACLs allow for subnet designation in a single entry.

URL (req.url): Some attacks will hit the same URL over and over, and many times they are generic paths that may not even exist on a site in question.

Location variables (client.geo.*): If a large number of requests are coming from say, Russia or China, when the majority of a site’s traffic comes from within the US, that may be a flag for questionable behavior.

Through a heuristic approach to logging tailored toward your own needs, data can be fed into an analyzing service in order to make decisions on what needs to be blocked.

As for what that analyzing service is, there is not currently a standard, so you will need to build it on your own. At least one customer has previously built an Elasticsearch, Kibana, and Logstash stack fed by syslog. Based on the dashboards and thresholds that are set up, once can determine what constitutes a mutable action. You can then use lookup tables that concatenate IP addresses and look into a specific timeframe (say, 5-10 minutes) to see who is hitting more than a couple hundred times. One could also look across a subnet to find a correlation, and then normalize in a single subnet and block as needed.

Once attackers have been identified, you will then need to make the API calls to update your ACL block list. The most important part of making the calls here is to use our bulk update API in order to not be rate limited. 

This leads to the question, how often should ACLs be updated during an attack? And how long should an attacker be blocked before they can be unblocked? The answer here again is customer-dependent. Some customers block IPs for an average of 12 minutes before removing them from their block list, and find that is long enough to deter most attackers.

As a final consideration, some attacks may be traced back by geolocation logging to specific countries where the customer’s site has very little normal traffic (such as Russia or China for an e-commerce company that only operates within the U.S.). When this is the case, an entire geographic location can be blocked in VCL fairly quickly by conditionally validating against our geolocation variables (https://docs.fastly.com/vcl/geolocation/) and then blocking that traffic.