[tor-bugs] #24607 [Circumvention/BridgeDB]: CAPTCHAs on BridgeDB seem to be getting more difficult

[Tor Bug Tracker & Wiki]

#24607: CAPTCHAs on BridgeDB seem to be getting more difficult
-------------------------------------------------+-------------------------
 Reporter:  alison                               |          Owner:  (none)
     Type:  defect                               |         Status:
                                                 |  assigned
 Priority:  Medium                               |      Milestone:
Component:  Circumvention/BridgeDB               |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:  anti-censorship-roadmap-november,    |  Actual Points:
  s30-o22a2                                      |
Parent ID:  #31279                               |         Points:  5
 Reviewer:                                       |        Sponsor:
                                                 |  Sponsor30-must
-------------------------------------------------+-------------------------

Comment (by phw):

 I took a look at our recent BridgeDB metrics to get an idea of how our new
 CAPTCHAs affected users and bots. Here's what we believe are bot requests
 for vanilla bridges over HTTPS (i.e., `https.vanilla.zz`):

 ||= Date =||= # success =||= # failed =||= # total =||= Success rate =||
 || 2020-01-28 || 4,060|| 640|| 4,700|| 86%||
 || 2020-01-29 || 3,700|| 1,120|| 4,820|| 77%||
 || 2020-01-30 || 510|| 4,550|| 5,060|| 10%||

 And here's what we believe are user requests from the U.S. for vanilla
 bridges over HTTPS (i.e., `https.vanilla.us`):

 ||= Date =||= # success =||= # failed =||= # total =||= Success rate =||
 || 2020-01-28 || 170|| 160|| 330|| 52%||
 || 2020-01-29 || 280|| 290|| 570|| 49%||
 || 2020-01-30 || 300|| 70|| 370|| 81%||

 Recall that we deployed new CAPTCHAs on January 29. This is when the
 success rate of bots began to decline, and the success rate of users began
 to increase. I expect the bots to improve over time but we managed to
 increase the success rate of users, which is what matters most. (Granted,
 I'm only looking at requests from the U.S. because we see most requests
 from this region. Other regions that don't have native English speakers
 may not be doing as well.)

 Here's the number of moat requests over time (i.e., `moat.obfs4.??`). Note
 that we don't know the proportion of user and bot requests comprising all
 moat requests:

 ||= Date =||= # success =||= # failed =||= # total =||= Success rate =||
 || 2020-01-28 || 3,780|| 2,780|| 6,560|| 58%||
 || 2020-01-29 || 3,800|| 2,830|| 6,630|| 57%||
 || 2020-01-30 || 3,910|| 590|| 4,500|| 87%||

 I find it surprising that the number of failed requests decreased on Jan
 30 but the number of successful requests didn't increase. This may be a
 bug in the metrics collection. Another possibility is that bots requested
 CAPTCHAs, were not sufficiently confident in their classification, and
 subsequently didn't send a POST request with their solution to BridgeDB.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/24607#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online