[tor-bugs] #32117 [Circumvention/BridgeDB]: Understand and document BridgeDB bot scraping attempts

[Tor Bug Tracker & Wiki]

#32117: Understand and document BridgeDB bot scraping attempts
----------------------------------------+--------------------
     Reporter:  cohosh                  |      Owner:  (none)
         Type:  project                 |     Status:  new
     Priority:  Medium                  |  Milestone:
    Component:  Circumvention/BridgeDB  |    Version:
     Severity:  Normal                  |   Keywords:
Actual Points:                          |  Parent ID:
       Points:                          |   Reviewer:
      Sponsor:                          |
----------------------------------------+--------------------
 We are aware of automated attempts to enumerate bridges in BridgeDB, but
 lack a more rigorous understanding of the problem.

 We have detected bot requests from bridgeDB's web interface and deployed
 some defences by forbidding requests with headers that are commonly
 associated with bots, and handing out fake bridges to suspected bot
 requests (#31252), and

 We also suspect that these bots are solving our CAPTCHAs more accurately
 than users (#24607).

 After a recent campaign to get more volunteer bridges, we set up an
 experiment to test the reachability of a subset of these new bridges from
 a probe site in Beijing and found all new bridges in our sample to be
 blocked (most were blocked from the very start of the experiment): #31701

 This ticket is for documenting bot behaviour and brainstorming ways to
 detect and analyze the automatic scraping of BridgeDB from censor-owned
 bots.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32117>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online