SQS rendezvous exceeded free tier usage
I got a notification from AWS that our SQS rendezvous service exceeded the free-tier usage this month with over 1,000,000 SQS API requests. This is in some sense exciting news, because it shows that the rendezvous channel is effective and getting some use. It does, however mean that we will have to start paying for the service. The current billing period falls on month boundaries, from January 1st to January 31st. The budget action fired on January 9th, which is pretty early in the month. Looking at our broker metrics[0,1], there were approximately 38,608 client polls using SQS. That's approximately 2.5 requests per poll, which is about what I'd expect. It's reassuring to know that the budget actions work. I'm going to set them a little higher, to something I can reasonably afford. I don't yet know how the cost will scale with the number of polls, and how that will compare with the cost of domain fronted requests. [0] https://snowflake-broker.torproject.net/metrics [1] https://metrics.torproject.org/collector.html#type-snowflake-stats
On 2025-01-16 10:48, Cecylia Bocovich via anti-censorship-team wrote:
It does, however mean that we will have to start paying for the service. The current billing period falls on month boundaries, from January 1st to January 31st. The budget action fired on January 9th, which is pretty early in the month. Looking at our broker metrics[0,1], there were approximately 38,608 client polls using SQS. That's approximately 2.5 requests per poll, which is about what I'd expect. There was an error in my calculation. That should be 25 requests/poll. This might be a little high, but I forgot about the periodic clean up tasks that the broker does. This is probably still accurate for faithful usage of the rendezvous channel. It is possible we could make this more efficient.
On 2025-01-16 11:02, Cecylia via anti-censorship-team wrote:
On 2025-01-16 10:48, Cecylia Bocovich via anti-censorship-team wrote:
It does, however mean that we will have to start paying for the service. The current billing period falls on month boundaries, from January 1st to January 31st. The budget action fired on January 9th, which is pretty early in the month. Looking at our broker metrics[0,1], there were approximately 38,608 client polls using SQS. That's approximately 2.5 requests per poll, which is about what I'd expect. There was an error in my calculation. That should be 25 requests/poll. This might be a little high, but I forgot about the periodic clean up tasks that the broker does. This is probably still accurate for faithful usage of the rendezvous channel. It is possible we could make this more efficient.
Looking at the source code, the client[0] makes a minimum of 3 requests per poll to send a message to the broker queue, get their single-use queue URL, and then receive a message from their single-use queue. But, if the queue hasn't been created or the answer hasn't been sent it will keep retrying those requests for a max of 11 requests total per poll attempt (only poll successes are counted in the metrics). The broker[1] makes 3 requests at poll time, to receive a message from its queue, create the client queue, and send the response. During cleanup, to broker makes a delete request once per queue, but not until the queue has timed out. Since the timeout on this is 2 minutes and cleanup happens once every 30s, we are calling the GetQueueAttributes request approximately 4 times per poll. So the broker is making approximately 7 requests / successful poll. The broker periodic clean up tasks fire once every 30 seconds and make some number of requests to get the list of all queues (there's a max of 1,000 queues per call, which we probably weren't exceeding). Over the course of 9 days, this totals 25,920 requests, which is still pretty low. If I did this accounting right, we should be making a maximum of 18 requests / per client poll *attempt*. The higher than expected number could indicate that some polls are not successful and a client has to try the rendezvous method multiple times. We do have an open issue regarding SQS[2]. In any case, 17 is not so far off 25 as to indicate an attack. [0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [2] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...
Quoting Cecylia Bocovich via anti-censorship-team (2025-01-16 18:05:50)
If I did this accounting right, we should be making a maximum of 18 requests / per client poll *attempt*. The higher than expected number could indicate that some polls are not successful and a client has to try the rendezvous method multiple times. We do have an open issue regarding SQS[2]. In any case, 17 is not so far off 25 as to indicate an attack.
Nice investigation. I guess is worth it working a bit on reducing that number of requests before SQS becomes more popular so we don't hit a very big bill on it. We could argue a way to fit this as part of P146 S3.1, as a censor enumerating will basically DoS our SQS queue. -- meskio | https://meskio.net/ -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- My contact info: https://meskio.net/crypto.txt -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Nos vamos a Croatan.
On 2025-01-16 10:48, Cecylia Bocovich wrote:
I got a notification from AWS that our SQS rendezvous service exceeded the free-tier usage this month with over 1,000,000 SQS API requests. This is in some sense exciting news, because it shows that the rendezvous channel is effective and getting some use.
It does, however mean that we will have to start paying for the service. The current billing period falls on month boundaries, from January 1st to January 31st. The budget action fired on January 9th, which is pretty early in the month. Looking at our broker metrics[0,1], there were approximately 38,608 client polls using SQS.
I raised the budget limit to something that is still very low and here's the accounting for February: - First 1,000,000 Amazon SQS Requests per month are free 1,000,000 Requests - $0.40 per million Amazon SQS standard requests in Tier1 in US East (N Virginia) 5,102,813 Requests Total charges: USD 2.04 I deployed a change to have the broker read from the SQS queue more frequently[0] on Thursday, February 20th, which means that this will likely increase next month. [0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...
On 2025-03-05 09:34, Cecylia Bocovich wrote:
I deployed a change to have the broker read from the SQS queue more frequently[0] on Thursday, February 20th, which means that this will likely increase next month.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable- transports/snowflake/-/merge_requests/514
We did see a rise in API calls during March, I think it's mostly due to the bug fixes allowing us to poll received messages more quickly. - First 1,000,000 Amazon SQS Requests per month are free 1,000,000 Requests - $0.40 per million Amazon SQS standard requests in Tier1 in US East (N Virginia) 10,123,385 Requests Total charges: USD 4.05
Here's a breakdown of SQS costs in April and May. We've seen a steady increase in the usage of SQS in the last few months. I've attached a plot of the number of client poll requests using SQS. It's still a very low fraction of rendezvous polls. # April - First 1,000,000 Amazon SQS Requests per month are free 1,000,000 Requests - $0.40 per million Amazon SQS standard requests in Tier1 in US East (N Virginia) 13,349,269 Requests Total charges: USD 5.34 # May - First 1,000,000 Amazon SQS Requests per month are free 1,000,000 Requests - $0.40 per million Amazon SQS standard requests in Tier1 in US East (N Virginia) 20,056,380 Requests Total charges: USD 8.02
A 50% increase in cost in one month is concerning. I understand you have cost limits set on the account. But you shouldn't be on the hook for it if it keeps going up like this. In the absence of any more complete plan, you could set a low billing limit per month and just let it fail after that. On Wed, Jun 04, 2025 at 02:45:09PM -0400, Cecylia Bocovich via anti-censorship-team wrote:
Here's a breakdown of SQS costs in April and May. We've seen a steady increase in the usage of SQS in the last few months. I've attached a plot of the number of client poll requests using SQS. It's still a very low fraction of rendezvous polls.
# April
- First 1,000,000 Amazon SQS Requests per month are free
1,000,000 Requests
- $0.40 per million Amazon SQS standard requests in Tier1 in US East (N Virginia)
13,349,269 Requests
Total charges: USD 5.34
# May
- First 1,000,000 Amazon SQS Requests per month are free
1,000,000 Requests
- $0.40 per million Amazon SQS standard requests in Tier1 in US East (N Virginia)
20,056,380 Requests
Total charges: USD 8.02
participants (4)
-
Cecylia -
Cecylia Bocovich -
David Fifield -
meskio