tor-dev

tor-dev@lists.torproject.org

3581 discussions

Re: [tor-dev] CAPTCHA Monitoring Project's Dashboard
by Barkin Simsek 11 Aug '20

11 Aug '20

I messed up with replying to my own thread since I was receiving emails in a digest. I'm sorry for creating a duplicate thread. This email intends to reply to Georg's reply from the original thread. > Thanks for that wiki page. That's been really helpful. One thing I kept > wondering while reading through the different graph descriptions with > the default graph style in mind: [..snip..] > So, what's the tick shown per day then? One consensus somehow picked or > data for all consensuses for that day? Or...? I initially wanted to have per-day data points on the graphs, but later I realized that it complicates the calculations a lot (and increases the probability of making a mistake) since some relays show up & disappear (and their exit probabilities change) throughout the day. So, I decided to calculate the CAPTCHA rates per each hourly consensus to simplify calculations. There will be 24 data points (representing each consensus) per day and 30 days of history in a single graph. In total, there will be 24*30=720 data points in a single graph. Since having 720 labels similar to "2020-07-18 14:00" in the x-axis would be too cluttered, I decided to put ticks for each whole day and label them only rather than labeling every single data point. I hope this explanation makes it more clear. I will update the wiki and example graph on the wiki to reflect what I just explained. > How do you plan to show those graphs on the dashboard? Are they all > shown in some order? Grouped by sections (maybe along those on the > wiki)? Grouped by "importance"? I plan to have separate pages for each section mentioned in the wiki. In addition to that, I plan to have a "highlights" page to show a quick summary of all graphs. People visiting the dashboard will first see the highlights and later check more detailed graphs if they are interested in seeing them. I agree that there is a lot of information for a visitor to digest, and making this process as effortless as possible is one of my top priorities. > Somewhat related to that, I think we can try reducing the number of > graphs, at least those which are greatly related. [..snip..] > One thing that could be useful here > is having a switch/buttons next to the graph giving the user the option > to see "by exit relay age" for all CAPTCHAs, or for Cloudflare ones, or > for Akamai ones or for... depending on the switch/button the users > clicked on. The graph would update accordingly once clicked, showing the > user choice. The default could be for all CAPTCHAs or essentially > whatever we think is more important for us. That way you have related > things grouped together AND you have less graphs shown per default on > the dashboard, too, which might help not getting confused. A few other people suggested using this approach as well, and I like it! I will place buttons to achieve what you have suggested. On top of this, there will be checkboxes next to graphs for users to make decisions on what to include in the graphs, and the graphs will be updated accordingly. The default values will be configured in a way to reflect what we care about most. For example, both "Tor Browser" and "Firefox over Tor" measurements could be used to calculate the CAPTCHA rates, but only "Tor Browser" checkbox will be checked by default, and the user will have the ability to include "Firefox over Tor" and others into calculations. This gives the user the ability to compare different options easily. That said, I wonder if having too many options is another problem for users. It might lead to confusion, and determining the right balance is still an open issue. Thank you for your feedback! Best, Barkin

1 0

CAPTCHA Monitoring Project's Dashboard
by Barkin Simsek 11 Aug '20

11 Aug '20

Hi everyone, I created a wiki page [1] for "describing" the graphs that will be used to visualize the CAPTCHA Monitor dataset [2]. There are already a few graphs on the dashboard [3] and they will be replaced with the new ones described on the wiki page. I'm interested in hearing your feedback on the graph descriptions [1] and suggestions for adding new graphs. I want to understand if the graphs I proposed make sense and useful for anyone. The descriptions have a similar style to the one used in the metrics website's "Reproducible Metrics" page [4]. Thank you in advance! Best, Barkin (woswos) [1] https://gitlab.torproject.org/woswos/CAPTCHA-Monitor/-/wikis/Dashboard-Grap… [2] https://gitlab.torproject.org/woswos/CAPTCHA-Monitor/-/wikis/home [3] https://dashboard.captcha.wtf [4] https://metrics.torproject.org/reproducible-metrics.html

2 1

CentOS 8 TOR Repo
by Ladar Levison 07 Aug '20

07 Aug '20

It appears that installing TOR via the CentOS 8 DNF repo isn't working. The pre-flight configuration check fails because the repo package was built using zstd 1.4.4, and CentOS 8 is currently providing zstd 1.4.2. For any that cares to try, the repo is at: https://rpm.torproject.org/centos/8/ L~

2 2

Connection padding set to 1 vs auto
by procmem＠riseup.net 06 Aug '20

06 Aug '20

Hi. I was wondering if setting the connection padding setting in torrc to 1 instead of auto has any benefit in protecting against a passive adversary outside the Tor network.

1 0

CAPTCHA Monitoring Project Updates #3
by Barkin Simsek 25 Jul '20

25 Jul '20

Hi everyone, I have a few more updates since my last email (you might want to check the project's wiki page [0] first if you didn't see my previous emails): -- I have the onion service [1] of the dashboard [2] up and running. -- I was using an SQLite database and migrated it to PostgreSQL. This migration was needed to scale up the whole system. At the moment, there are 50000+ measurements in the database, and access time was very long with SQLite. -- Now there is a properly documented API [3] available for everyone's use. You can perform queries through the API if you prefer to access the raw data, instead of the visualizations on the dashboard. -- The Dockerfile [4] is updated and working again. -- I decided to give priority to the API and Dockerfile for the "reproducibility" of the results and graphs in the dashboard. Before the API, the graphs were using hardcoded SQL queries, which required people to dig in the source code to find out possible mistakes. It was also impossible for people to execute these SQL queries and see the results themselves. API allows me to put a link to the exact API calls to anything I place on the dashboard so that people can double-check with the raw data if they want. -- I'm working on adding Brave Browser's "Private Window with Tor" to the CAPTCHA Monitor. I'm very close to finishing it and should be ready at the end of this weekend. I expect to see a high amount of CAPTCHA since it is using Tor without Tor Browser's user-agent (and thus fingerprint). -- I discussed a possible integration with the metrics website on the metrics team meeting. I initially thought about placing CAPTCHA rate graphs under each relay in the "Relay Search" section. If you have any suggestions, you are welcomed to mention them in ticket tpo/metrics/website/40002 [5]. -- Finally, you can take a look at my blog posts on DIAL's blog [6] to see more detailed explanations, but I already summarized the blog posts in this email. Best, Barkin (woswos) [0] https://gitlab.torproject.org/woswos/CAPTCHA-Monitor/-/wikis/home [1] http://captchaufjq5m2i73up537pldaxnbp6rzcbdrzc7y5rlwtx3mwigznad.onion/ [2] https://dashboard.captcha.wtf/ [3] https://api.captcha.wtf/ [4] https://gitlab.torproject.org/woswos/CAPTCHA-Monitor/-/tree/master/docker [5] https://gitlab.torproject.org/tpo/metrics/website/-/issues/40002 [6] https://hub.osc.dial.community/t/tor-project-cloudflare-captcha-monitoring/…

1 0

Distributing Tor developer keys via Fedora packages
by Andrew Clausen 20 Jul '20

20 Jul '20

Hi everyone, I propose distributing the Tor developer keys inside the Fedora package distribution-gpg-keys.[1] This would give most Linux users a trustworthy chain of signatures from their own distributor (e.g. CentOS or Fedora) to Tor project downloads. I am happy to take care of this, although I am also happy if somebody who is more involved with Tor than me takes this on. I wrote a shell script (attached) to acquire and organise the keys based on https://2019.www.torproject.org/include/keys.txt. My script would install the following keys under /usr/share/distribution-gpg-keys/tor: Arm_releases/Damian_Johnson.gpg Tails_live_system_releases/The_Tails_team.gpg TorBirdy_releases/Sukhbir_Singh.gpg Tor_Browser_releases/Arthur_Edelstein.gpg Tor_Browser_releases/Georg_Koppen.gpg Tor_Browser_releases/Mike_Perry.gpg Tor_Browser_releases/Nicolas_Vigier.gpg Tor_Browser_releases/The_Tor_Browser_Developers.gpg Tor_source_tarballs/Nick_Mathewson.gpg Tor_source_tarballs/Roger_Dingledine.gpg Torsocks_releases/David_Goulet.gpg deb.torproject.org_repositories_and_archives/Tor_Project_Archive.gpg older_Tor_tarballs/Nick_Mathewson.gpg other/Peter_Palfrader.gpg Unless someone else volunteers (please do!), I will set up a weekly job to run the script and alert me to any changes. Can anyone see any potential problems with this plan? The most obvious question is: how do I know that I am distributing unadulterated keys? I think the answer is that I don't! But any attack would have to affect a large group of people, and would be detected quickly as long as many people are looking at the distribution-gpg-keys package. If this solution is unsatisfactory, then perhaps someone who is more involved with the Tor developers -- and hence able to directly check the keys -- ought to take this on. [1] See https://github.com/xsuchy/distribution-gpg-keys and https://rpmfind.net/linux/RPM/fedora/updates/32/x86_64/Packages/d/distribut…

2 2

Proposal 325: Packed relay cells: saving space on small commands
by Nick Mathewson 10 Jul '20

10 Jul '20

``` Filename: 325-packed-relay-cells.md Title: Packed relay cells: saving space on small commands Author: Nick Mathewson Created: 10 July 2020 Status: Draft ``` # Introduction In proposal 319 I suggested a way to fragment long commands across multiple RELAY cells. In this proposal, I suggest a new format for RELAY cells that can be used to pack multiple relay commands into a single cell. Why would we want to do this? As we move towards improved congestion-control and flow-control algorithms, we might not want to use an entire 498-byte relay payload just to send a one-byte flow-control message. We already have some cases where we'd benefit from this feature. For example, when we send SENDME messages, END cells, or BEGIN_DIR cells, most of the cell body is wasted with padding. As a side benefit, packing cells in this way may make the job of the traffic analyst a little more tricky, as cell contents become less predictable. # The basic design Let's use the term "Relay Message" to mean the kind of thing that a relay cell used to hold. Thus, this proposal is about packing multiple "Relay Messages" in to a cell. I'll use "Packed relay cell" to mean a relay cell in this new format, that supports multiple messages. I'll use "client" to mean the initiator of a circuit, and "relay" to refer to the parties through who a circuit is created. Note that each "relay" (as used here) may be the "client" on circuits of its own. When a relay supports relay message packing, it advertises the fact using a new Relay protocol version. Clients must opt-in to using this protocol version (see XXX below) before they can send any packed relay cells, and before the relay will send them any packed relay cells. When packed cells are in use, multiple cell messages can be concatenated in a single relay cell. Only some relay commands are supported for relay cell packing, listed here: - SENDME - DROP - DATA - BEGIN - BEGIN_DIR - END - CONNECTED - PADDING_NEGOTIATE - PADDING_NEGOTIATED If any relay message with a relay command _not_ listed above appears in a packed relay cell with another relay message, then the receiving party MUST tear down the circuit. (Note that relay cell fragments (proposal 319) are not supported for packing.) The command byte "0" is now used to explicitly indicate "end of cell". If the byte "0" appears after a relay message, the rest of the cell MUST be ignored. When generating RELAY cells, implementations SHOULD (as they do today) fill in the unused bytes with four 0-valued bytes, followed by a sequence of random bytes up to the end of the cell. If there are fewer than 4 unused bytes at the end of the cell, those unused bytes should all be filled with 0-valued bytes. # Negotiation and migration After receiving a packed relay cell, the relay know that the client supports this proposal: Relays SHOULD send packed relay cells on any circuit on which they have received a packed relay cell. Relays MUST NOT send packed relay cells otherwise. Clients, in turn, MAY send packed relay cells to any relay whose "Relay" subprotocol version indicates that it supports this protocol. To avoid fingerprinting, this client behavior should controlled with a tristate (1/0/auto) torrc configuration value, with the default set to use a consensus parameter. The parameter is: "relay-cell-packing" Boolean: if 1, clients should send packed relay cells. (Min: 0, Max 1, Default: 0) To handle migration, first the parameter should be set to 0 and the configuration setting should be "auto". To test the feature, individual clients can set the tristate to "1". Once enough clients have support for the parameter, the parameter can be set to 1. # A new relay message format (This section is optional and should be considered separately; we may decide it is too complex.) Currently, every relay message uses 5 bytes of header to hold a relay command, a length field, and a stream ID. This is wasteful: the stream ID is often redundant, and the top 7 bits of the length field are always zero. I propose a new relay message format, described here (with `ux` denoting an x-bit bitfield). This format is 2 bytes or 4 bytes, depending on its first bit. struct relay_header { u1 stream_id_included; // Is the stream_id included? u6 relay_command; // as before u9 relay_data_len; // as before u8 optional_stream_id[]; // 0 bytes or two bytes. } Alternatively, you can view the first three fields as a 16-bit value, computed as: (stream_id_included<<15) | (relay_command << 9) | (relay_data_len). If the optional_stream_id field is not present, then the default value for the stream_id is computed as follows. We use stream_id 0 for any command that doesn't take a stream ID. For commands that _do_ take a steam_id, we use whichever nonzero stream_id appeared last in this cell. This format limits the space of possible relay commands. That's probably okay: after 20 years of Tor development, we have defined 25 relay command values. But in case 2^6==64 commands will not be enough, we reserve command values 48 through 63 for future formats that need more command bits.

3 2

CAPTCHA Monitoring Project Updates & Findings
by Barkin Simsek 10 Jul '20

10 Jul '20

Hi everyone, I made progress on the Cloudflare CAPTCHA Monitoring project since my last email, and I wanted to share some of the updates & findings. This year Tor Project is participating in GSoC under the DIAL umbrella, and I have already been posting updates to the DIAL blog [1] weekly. I started mirroring these updates [2] to my project's wiki page, and I will be posting more frequent updates here. a) Updates: Firstly, I moved the wiki page that explains the project, the code base, and the issue tracker to Tor Project's GitLab. They are all in the same GitLab project. You can find detailed information about the project on the wiki page [3] and leave comments & suggestions within that repository by creating issues. Secondly, I got a fully functioning system up and running. The system fetches various URLs with Tor Browser & Firefox over Tor and checks for CAPTCHAs. The system also checks if any third-party code was injected by comparing the hash of the received page with an expected hash value. It repeats these experiments using different exit relays and records results. You can view the results on the dashboard [4] I created. I'm looking for more URLs to track for CAPTCHAs. Feel free to share the websites you frequently visit and get CAPTCHAs, so that I can track these websites with this tool as well. I want to experiment with all types of CAPTCHAs, and these URLs don't have to be fronted by Cloudflare. b) Findings: So far, I have observed that using the Tor Browser Bundle out of the box without changing its configurations doesn't lead to a high CAPTCHA rate on Cloudflare fronted websites (assuming the website owners don't explicitly block exit relays [5]). That said, modifying the user-agent or any other modifications that deviate your browser's fingerprint from a typical Tor Browser user, significantly increases the chance of getting CAPTCHAs. For example, using the regular Firefox over Tor resulted in getting CAPTCHAs in ~90% of the measurements. I believe Cloudflare is very aggressive against the "Firefox over Tor" users because many people, unfortunately, use Chromium/Firefox + Selenium + Tor to scrape web pages and bypass IP-based rate limits. That's why I'm interested in hearing about your specific browser/Tor configurations to test them with the CAPTCHA Monitor. Not everyone is affected in the same way because of these differences in the way we use Tor, but we can understand which differences affect the CAPTCHA rate more than others by experimenting. Additionally, I observed that the TLS fingerprint has a significant role in whether someone gets a CAPTCHA or not. As a part of the project, I decided to capture the HTTP headers during measurements to understand how they affect the CAPTCHA rates. Initially, I was using a Python library called seleniumwire to capture the HTTP headers by intercepting the traffic between the Tor Browser and Tor. By doing this, I got a very high CAPTCHA rate, like 98% of the time. seleniumwire forwards the traffic transparently, but it has a different TLS fingerprint than Tor Browser. I figured out that the difference in the TLS fingerprints was triggering the MITM detection on the Cloudflare side, thus, resulting in very high CAPTCHA rates. Interestingly, I tried using the exact same Tor Browser & seleniumwire setup, but without Tor and, practically, I didn't get any CAPTCHAs. I believe the MITM detection is more aggressive if the traffic is coming through an exit relay. So, I stopped using seleniumwire to capture headers because it didn't reflect what a real human Tor Browser user is usually experiencing. Please feel free to use the sample code [6] that I used to combine seleniumwire and Tor, if you are interested in doing further experimenting on this. c) Next: I will work on collecting more metrics by testing more configurations and websites. I will create a "Relay Search" section on the dashboard, where CAPTCHA statistics for the relays (exit relays for now) will be available. I will also work on using the collected data to predict the probability of getting CAPTCHAs with a given exit relay and configuration/setup. Best, Barkin [1] https://hub.osc.dial.community/t/tor-project-cloudflare-captcha-monitoring/… [2] https://gitlab.torproject.org/woswos/CAPTCHA-Monitor/-/wikis/Updates [3] https://gitlab.torproject.org/woswos/CAPTCHA-Monitor/-/wikis/home [4] https://dashboard.captcha.wtf/ [5] Cloudflare has a setting to block all traffic originating from the Tor network, but that setting is not "turned on" by default [6] https://gist.github.com/woswos/38b921f0b82de009c12c6494db3f50c5

1 0

Gitolite to Gitlab Sync Change
by Alexander Færøy 09 Jul '20

09 Jul '20

Hello folks! In an attempt to solve Gitlab#41[1] the hooks that is executed when you push to git.torproject.org to synchronize to Gitlab was modified to avoid pruning references in Gitlab that was missing in the Gitolite repository. This /should/ have the following implications: - We no longer delete the "refs/merge-requests/*" namespace each time someone pushes to your repository on git.torproject.org. This should allow people to use the `git mr` alias that can be found online which should make local code-reviews easier and also allow you to handle manual CLI merging more easily. - We no longer delete branches automatically on Gitlab: if a branch is deleted on git.torproject.org. This have to be deleted manually now on Gitlab. Please report any issues you that you might discover on the ticket. Thanks to Hiro for getting this running! All the best, Alex. [1]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/41 -- Alexander Færøy

1 0

Gitlab CI runners available for experimentation on gitlab.torproject.org
by Alexander Færøy 09 Jul '20

09 Jul '20

Hello folks! Hans from The Guardian Project added his CI runners to our Gitlab instance. It looks like some pretty fast machines that allows each team to experiment with Gitlab CI on our Gitlab instance. Hans says that the runners have no uptime promises or anything like that, so if they are down they are down :-) Here's some documentation for getting started: https://docs.gitlab.com/ee/ci/ Thanks to Hans for this! All the best, Alex. -- Alexander Færøy

2 1

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-dev