On Sun, Feb 23, 2025 at 05:55:43PM -0500, Cecylia Bocovich via anti-censorship-team wrote:
On 2025-02-23 15:40, David Fifield via anti-censorship-team wrote:
The Snowflake broker's request rate limiting, what does it key on? The source IP address, X-Forwarded-For, or something else?
This is where the rate limiting was introduced: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...
location ~ ((proxy)|(client)|(answer)|(metrics)|(prometheus)|(amp/client/.*)|(robots.txt)) { limit_req zone=snowflake burst=3; proxy_pass http://127.0.0.1:8080; proxy_http_version 1.1; }
limit_req_zone $binary_remote_addr zone=snowflake:10m rate=1r/s;
Here's another, more recent snapshot of the configuration: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...
location ~ ((proxy)|(client)|(answer)|(metrics)|(prometheus)|(amp/client/.*)|(robots.txt)) { limit_req zone=snowflake burst=3; proxy_pass http://127.0.0.1:8080; proxy_http_version 1.1; proxy_read_timeout 300; proxy_connect_timeout 300; proxy_send_timeout 300;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;}
limit_req_zone $proxy_protocol_addr zone=snowflake:10m rate=1r/s;
If the limited is using the source IP address, then different clients could be causing each other to be rate-limited, because many requests come from the same CDN IP address (or whatever). If the limiter is using X-Forwarded-For or similar, then it's possible to evade the limiter by putting random or incrementing IP addresses in the header.
There is some discussion about this in the issue you linked. The broker was rate limiting using the source IP address[0], which was likely causing at least some clients to rate limit both themselves and each other. I removed the rate limit entirely at Feb 19 18:41:39 [1] for this reason. I haven't noticed a drop in 504 timeout errors though, so I'm still unsure to what extent this was causing problems. Here's the current configuration:
location ~ ((proxy)|(client)|(answer)|(metrics)|(prometheus)|(amp/client/.*)|(robots.txt)) { # limit_req zone=snowflake burst=3; proxy_pass http://127.0.0.1:8080; proxy_http_version 1.1; proxy_read_timeout 300; proxy_connect_timeout 300; proxy_send_timeout 300;
# map $http_x_forwarded_for $http_x_forwarded_for_new_value { # default "$http_x_forwarded_for, $proxy_protocol_addr"; # "" $proxy_protocol_addr; # }
proxy_set_header X-Forwarded-For $http_x_forwarded_for_new_value; }
I see. I apologize; I didn't see all the updates in #40451 when I asked. LImiting on source IP address is probably the better option (versus limiting on X-Forwarded-For), since X-Forwarded-For can be spoofed, as shelikhoo observes.
You wrote: We're not using this to prevent enumeration attacks, it is only to prevent DoS attacks on the proxy pool or overwhelming the broker's resources. Using the source IP address makes sense in that case.