- tor-project - lists.torproject.org

Anti-censorship team meeting notes, 2024-05-02
by meskio 02 May '24

02 May '24

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2024/tor-meeting.2024-05-02-16.15.html And our meeting pad: Anti-censorship -------------------------------- Next meeting: Thursday, May 9 16:00 UTC Facilitator: shelikhoo ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: meskio == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from sponsors, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Sponsor 96 <-- meskio, shell, onyinyang, cohosh * https://gitlab.torproject.org/groups/tpo/-/milestones/24 * Sponsor 150 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == == Discussion == * Snowfalke API has insufficent support for passing additional settings in constructor * trying to avoid making breaking changes to the API * one option is to use builders: https://docs.rs/http/0.2.9/http/request/struct.Builder.html * another is to use WithXXXX methods: https://pkg.go.dev/google.golang.org/grpc#WithAuthority * shelikhoo will create an issue to start the discussion on a possible version 3 API for snowflake * Snowflake broker Debian version EOL * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * we'll do a fresh install in a new VM * dcf1 will provision a VM for it * shelikhoo will do the setup * ed25519 keys in bridgelines * https://gitlab.torproject.org/tpo/core/torspec/-/issues/257 * there is a conversation on how to integrate ed25519 keys into bridgelines == Actions == == Interesting links == * https://github.com/ooni/utls-light by arturo, an alternative approach to modifying TLS fingerprints that requires less invasive crypto/tls library changes. It works by parsing the serialized native crypto/tls handshake messages before they are sent, and re-serializing them according to a desired schema. == Reading group == * We will discuss "" on * * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2024-05-02 Last week: - opened an issue in core tor about MyFamily for bridges - https://gitlab.torproject.org/tpo/core/tor/-/issues/40935 - snowflake webextension work This week: - make changes to Lox encrypted bridge table - https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/147 - follow up on reported SQS errors - update wasm-bindgen fork to fix some bugs and hopefully upstream changes - create a Lox test environment and instructions for the browser team - https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42503 Needs help with: dcf: 2024-05-02 Last week: Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - open issue to disable /debug endpoint on snowflake broker - move snowflake-02 to new VM Help with: meskio: 2023-05-02 Last week: - implement email distributor in rdsys (rdsys#186) Next week: - test email distributor in rdsys (rdsys#186) - improve privacy on rdsys metrics reusing snowflake techniques (snowflake#40354) Shelikhoo: 2024-05-02 Last Week: - [Merge Request WIP] Add Container Image Mirroring from Tor Gitlab to Docker Hub(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/… - [Dev] Snowflake Performance Improvement - Fix ipv6 connectivity with restricted address support https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webt… - Merge request reviews Next Week/TODO: - Revise Snowflake Performance Improvement MR(cont.) - Merge request reviews onyinyang: 2023-05-02 Last week(s): - Responding to pre-existing MR reviews and other notifications/questions in gitlab This week: - attempt hyper upgrade and tokio upgrade Probably next week and beyond: - preparing for upcoming panel (maybe) - implement some preliminary user feedback mechanism to identify bridge blocking based on Vecna's work - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): https://pad.riseup.net/p/tor-ac-community-azaleas-room-keep - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2023-04-25 Last weeks: - Extended Pion dtls library to support hooking of client hellos, server hellos and certificate request (PR: https://github.com/pion/dtls/pull/631) Fingerprint generation of firefox and chrome + mimicking using the client hello hook have been moved to its own repo: https://github.com/theodorsm/covertDTLS - Analyzed the snowflake captures done in "Snowflake Anonymous Network Traffic Identification" paper: no handshakes found, so not relevant for me - Verified the consistency of the fingerprint generation Next weeks: - Add hook to pion/webrtc so it can be used in snowflake. - Test and validate mimicked client hello with snowflake. - Add randomized fingerprint Help with: Facilitator Queue: shelikhoo onyinyang meskio 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue -- meskio | https://meskio.net/ -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- My contact info: https://meskio.net/crypto.txt -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Nos vamos a Croatan.

1 0

gitolite to gitlab migration completed (TPA-RFC-36)
by Antoine Beaupré 01 May '24

01 May '24

Hi again everyone! This is the last update of the Gitolite migration. It's a little more detailed than previous updates, so I made it in a blog post: https://blog.torproject.org/gitolite-gitlab-migration/ ... which I attach a copy here if you're the kind of people who prefer to read email than web. :) Enjoy! ---- Tor has finally completed a long migration from legacy Git infrastructure ([Gitolite and GitWeb][]) to our self-hosted [GitLab][] server. [GitLab]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab [Gitolite and GitWeb]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git Git repository addresses have therefore changed. Many of you probably have made the switch already, but if not, you will need to change: https://git.torproject.org/ to: https://gitlab.torproject.org/ In your Git configuration. The [GitWeb front page][] is now an archived listing of all the repositories before the migration. Inactive git repositories were archived in GitLab [legacy/gitolite namespace][] and the `gitweb.torproject.org` and `git.torproject.org` web sites now redirect to GitLab. [legacy/gitolite namespace]: https://gitlab.torproject.org/legacy/gitolite/ [GitWeb front page]: https://gitweb.torproject.org/ Best effort was made to reproduce the original gitolite repositories faithfully and also avoid duplicating too much data in the migration. But it's *possible* that some data present in Gitolite has not migrated to GitLab. User repositories are particularly at risk, because they were massively migrated, and they were "re-forked" from their upstreams, to avoid wasting disk space. If a user had a project with a matching name it was *assumed* to have the right data, which might be inaccurate. The two virtual machines responsible for the legacy service (`cupani` for `git-rw.torproject.org` and `vineale` for `git.torproject.org` and `gitweb.torproject.org`) have been shutdown. Their disks will remain for 3 months (until the end of July 2024) and their backups for another year after that (until the end of July 2025), after which point all the data from those hosts will be destroyed, with only the GitLab archives remaining. The rest of this article expands on how this was done and what kind of problems we faced during the migration. # Where is the code? Normally, nothing should be lost. All repositories in gitolite have been either explicitly migrated by their owners, forcibly migrated by the sysadmin team ([TPA][]), or explicitly destroyed at their owner's request. [TPA]: https://gitlab.torproject.org/tpo/tpa/team/ An exhaustive [rewrite map][] translates gitolite projects to GitLab projects. Some of those projects actually redirect to their *parent* in cases of empty repositories that were obvious forks. Destroyed repositories redirect to the GitLab front page. [rewrite map]: https://archive.torproject.org/websites/gitolite2gitlab.txt Because the migration happened progressively, it's technically possible that commits pushed to gitolite were lost after the migration. We took great care to avoid that scenario. First, we adopted a proposal ([TPA-RFC-36][]) in June 2023 to announce the transition. Then, in [March 2024][], we locked down all repositories from any further changes. Around that time, only a [handful of repositories][] had changes made after the adoption date, and we examined each repository carefully to make sure nothing was lost. [handful of repositories]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41214#note_2983302 "handful of repositories" [March 2024]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41213 [TPA-RFC-36]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-36-gitoli… Still, we built a [diff of all the changes in the git references][] that archivists can peruse to check for data loss. It's large (6MiB+) because a lot of repositories were migrated before the mass migration and then kept evolving in GitLab. Many other repositories were rebuilt in GitLab from parent to rebuild a fork relationship which added extra references to those clones. [diff of all the changes in the git references]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41215#note_3023924 A note to amateur archivists out there, it's probably too late for one last crawl now. The Git repositories now all redirect to GitLab and are effectively unavailable in their original form. That said, the GitWeb site was crawled into the [Internet Archive][] [in February 2024][], so at least some copy of it is available in the [Wayback Machine][]. At that point, however, many developers had already migrated their projects to GitLab, so the copies there were already possibly out of date compared with the repositories in GitLab. [Wayback Machine]: https://web.archive.org/web/20240204162238/https://gitweb.torproject.org/ [in February 2024]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41218#note_2992296 [Internet Archive]: https://archive.org/ [Software Heritage][] also has a copy of all repositories hosted on Gitolite [since June 2023][] and have continuously kept mirroring the repositories, where they will be kept hopefully in eternity. There's an [issue][] where the main website can't find the repositories when you search for `gitweb.torproject.org`, instead [search for `git.torproject.org`][]. [search for `git.torproject.org`]: https://archive.softwareheritage.org/browse/search/?q=git.torproject.org&vi… [issue]: https://gitlab.softwareheritage.org/swh/devel/swh-web/-/issues/4787 [since June 2023]: https://gitlab.softwareheritage.org/swh/infra/sysadm-environment/-/issues/4… [Software Heritage]: https://www.softwareheritage.org/ In any case, if you believe data is missing, please do let us know by [opening an issue with TPA][]. [opening an issue with TPA]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new # Why? This is an old project in the making. The first [discussion about migrating from gitolite to GitLab][] started in 2020 (almost 4 years ago). But [going further back][], the first GitLab experiment was in 2016, almost a decade ago. [going further back]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/trac#history [discussion about migrating from gitolite to GitLab]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40472 The current GitLab server dates from 2019, [replacing Trac for issue tracking in 2020][]. It was originally supposed to host only mirrors for merge requests and issue trackers but, naturally, one thing led to another and eventually, GitLab had grown a container registry, continuous integration (CI) runners, GitLab Pages, and, of course, hosted most Git repositories. [replacing Trac for issue tracking in 2020]: https://blog.torproject.org/from-trac-into-gitlab-for-tor/ There were hesitations at moving to GitLab for code hosting. We had [discussions about the increased attack surface][] and [ways to mitigate that][], but, ultimately, it seems the issues were not that serious and the community embraced GitLab. [ways to mitigate that]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/98 [discussions about the increased attack surface]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81 TPA actually migrated its most critical repositories out of shared hosting entirely, into specific servers (e.g. the Puppet Git repository is just on the Puppet server now), leveraging Git's decentralized nature and removing an entire attack surface from our infrastructure. Some of those repositories are *mirrored* back into GitLab, but the authoritative copy is not on GitLab. In any case, the proposal to migrate from Gitolite to GitLab was effectively just formalizing a *fait accompli*. # How to migrate from Gitolite / cgit to GitLab The progressive migration was a challenge. If you intend to migrate between hosting platforms, we strongly recommend to make a "flag day" during which you migrate *all* repositories *at once*. This ensures a smoother transition and avoids elaborate rewrite rules. When Gitolite access was shutdown, we had repositories on both GitLab and Gitolite, without a clear relationship between the two. A priori, the plan then was to import all the remaining Gitolite repositories into the `legacy/gitolite` namespace, but that seemed wasteful, particularly for large repositories like [Tor Browser][] which uses nearly a gigabyte of disk space. So we took special care to avoid duplicating repositories. [Tor Browser]: https://gitlab.torproject.org/tpo/applications/tor-browser When the [mass migration][] started, only 71 of the 538 Gitolite repositories were `Migrated to GitLab` in the `gitolite.conf` file. So, given that we had *hundreds* of repositories to migrate:, we developed some automation to "[save time][]". We already automate similar ad-hoc tasks with [Fabric][], so we used that framework here as well. (Our normal configuration management tool is [Puppet][], which is a poor fit here.) [Puppet]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/puppet [Fabric]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/fabric/ [save time]: https://xkcd.com/1205/ [mass migration]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41215 So a relatively [large amount of Python code][] was produced to basically do the following: [large amount of Python code]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/-/blob/85121b4a8a293cebb… 1. check if all on-disk repositories are listed in `gitolite.conf` (and vice versa) and either add missing repositories or delete them from disk if garbage 2. for each repository in `gitolite.conf`, if its category is marked `Migrated to GitLab`, skip, otherwise; 3. find a matching GitLab project by name, prompt the user for multiple matches 4. if a match is found, redirect if the repository is non-empty * we have GitLab projects that *look* like the real thing, but are only present to host migrated Trac issues * in such cases we cloned the Gitolite project locally and pushed to the existing repository instead 5. otherwise, a new repository is created in the `legacy/gitolite` namespace, using the "import" mechanism in GitLab to automatically import the repository from Gitolite, creating redirections and updating `gitolite.conf` to document the change User repositories (those under the `user/` directory in Gitolite) were handled specially. First, the existing redirection map was checked to see if a similarly named project was migrated (so that, e.g. `user/dgoulet/tor` is properly treated as a fork of `tpo/core/tor`). Then the parent project was forked in GitLab and the Gitolite project force-pushed to the fork. This allows us to show the fork relationship in GitLab and, more importantly, benefit from the "pool" feature in GitLab which deduplicates disk usage between forks. Sometimes, we found no such relationships. Then we simply imported multiple repositories with similar names in the `legacy/gitolite` namespace, sometimes creating forks between user repositories, on a first-come-first-served basis from the `gitolite.conf` order. The code used in this migration is now available publicly. We encourage other groups planning to migrate from Gitolite/GitWeb to GitLab to use (and contribute to) our [fabric-tasks][] repository, even though it does have its fair share of hard-coded assertions. [fabric-tasks]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/ The main entry point is the `gitolite.mass-repos-migration` task. A typical migration job looked like: ``` anarcat@angela:fabric-tasks$ fab -H cupani.torproject.org gitolite.mass-repos-migration [...] INFO: skipping project project/help/infra in category Migrated to GitLab INFO: skipping project project/help/wiki in category Migrated to GitLab INFO: skipping project project/jenkins/jobs in category Migrated to GitLab INFO: skipping project project/jenkins/tools in category Migrated to GitLab INFO: searching for projects matching fastlane INFO: Successfully connected to https://gitlab.torproject.org import gitolite project project/tor-browser/fastlane into gitlab legacy/gitolite/project/tor-browser/fastlane with desc 'Tor Browser app store and deployment configuration for Fastlane'? [Y/n] INFO: importing gitolite project project/tor-browser/fastlane into gitlab legacy/gitolite/project/tor-browser/fastlane with desc 'Tor Browser app store and deployment configuration for Fastlane' INFO: building a new connect to cupani INFO: defaulting name to fastlane INFO: importing project into GitLab INFO: Successfully connected to https://gitlab.torproject.org INFO: loading group legacy/gitolite/project/tor-browser INFO: archiving project INFO: creating repository fastlane (fastlane) in namespace legacy/gitolite/project/tor-browser from https://git.torproject.org/project/tor-browser/fastlane into https://gitlab.torproject.org/legacy/gitolite/project/tor-browser/fastlane INFO: migrating Gitolite repository project/tor-browser/fastlane to GitLab project legacy/gitolite/project/tor-browser/fastlane INFO: uploading 399 bytes to /srv/git.torproject.org/repositories/project/tor-browser/fastlane.git/hooks/pre-receive INFO: making /srv/git.torproject.org/repositories/project/tor-browser/fastlane.git/hooks/pre-receive executable INFO: adding entry to rewrite_map /home/anarcat/src/tor/tor-puppet/modules/profile/files/git/gitolite2gitlab.txt INFO: modifying gitolite.conf to add: "config gitweb.category = Migrated to GitLab" INFO: rewriting gitolite config /home/anarcat/src/tor/gitolite-admin/conf/gitolite.conf to change project project/tor-browser/fastlane to category Migrated to GitLab INFO: skipping project project/bridges/bridgedb-admin in category Migrated to GitLab [...] ``` In the above, you can see migrated repositories skipped then the [fastlane project][] being archived into GitLab. Another example with a later version of the script, processing only user repositories and showing the interactive prompt and a force-push into a fork: [fastlane project]: https://gitlab.torproject.org/legacy/gitolite/project/tor-browser/fastlane ``` $ fab -H cupani.torproject.org gitolite.mass-repos-migration --include 'user/.*' --exclude '.*tor-?browser.*' INFO: skipping project user/aagbsn/bridgedb in category Migrated to GitLab [...] INFO: skipping project user/phw/atlas in category Migrated to GitLab INFO: processing project user/phw/obfsproxy (Philipp's obfsproxy repository) in category Users' development repositories (Attic) INFO: Successfully connected to https://gitlab.torproject.org INFO: user repository detected, trying to find fork phw/obfsproxy WARNING: no existing fork found, entering user fork subroutine INFO: found 6 GitLab projects matching 'obfsproxy' (https://gitweb.torproject.org/user/phw/obfsproxy.git) 0 legacy/gitolite/debian/obfsproxy 1 legacy/gitolite/debian/obfsproxy-legacy 2 legacy/gitolite/user/asn/obfsproxy 3 legacy/gitolite/user/ioerror/obfsproxy 4 tpo/anti-censorship/pluggable-transports/obfsproxy 5 tpo/anti-censorship/pluggable-transports/obfsproxy-legacy select parent to fork from, or enter to abort: ^G4 INFO: repository is not empty: in-pack: 2104, packs: 1, size-pack: 414 fork project tpo/anti-censorship/pluggable-transports/obfsproxy into legacy/gitolite/user/phw/obfsproxy^G [Y/n] INFO: loading project tpo/anti-censorship/pluggable-transports/obfsproxy INFO: forking project user/phw/obfsproxy into namespace legacy/gitolite/user/phw INFO: waiting for fork to complete... INFO: fork status: started, sleeping... INFO: fork finished INFO: cloning and force pushing from user/phw/obfsproxy to legacy/gitolite/user/phw/obfsproxy INFO: deleting branch protection: <class 'gitlab.v4.objects.branches.ProjectProtectedBranch'> => {'id': 2723, 'name': 'master', 'push_access_levels': [{'id': 2864, 'access_level': 40, 'access_level_description': 'Maintainers', 'deploy_key_id': None}], 'merge_access_levels': [{'id': 2753, 'access_level': 40, 'access_level_description': 'Maintainers'}], 'allow_force_push': False} INFO: cloning repository git-rw.torproject.org:/srv/git.torproject.org/repositories/user/phw/obfspro… in /tmp/tmp6orvjggy/user/phw/obfsproxy Cloning into bare repository '/tmp/tmp6orvjggy/user/phw/obfsproxy'... INFO: pushing to GitLab: https://gitlab.torproject.org/legacy/gitolite/user/phw/obfsproxy remote: remote: To create a merge request for bug_10887, visit: remote: https://gitlab.torproject.org/legacy/gitolite/user/phw/obfsproxy/-/merge_re… remote: [...] To ssh://gitlab.torproject.org/legacy/gitolite/user/phw/obfsproxy + 2bf9d09...a8e54d5 master -> master (forced update) * [new branch] bug_10887 -> bug_10887 [...] INFO: migrating repo INFO: migrating Gitolite repository https://gitweb.torproject.org/user/phw/obfsproxy.git to GitLab project https://gitlab.torproject.org/legacy/gitolite/user/phw/obfsproxy INFO: adding entry to rewrite_map /home/anarcat/src/tor/tor-puppet/modules/profile/files/git/gitolite2gitlab.txt INFO: modifying gitolite.conf to add: "config gitweb.category = Migrated to GitLab" INFO: rewriting gitolite config /home/anarcat/src/tor/gitolite-admin/conf/gitolite.conf to change project user/phw/obfsproxy to category Migrated to GitLab INFO: processing project user/phw/scramblesuit (Philipp's ScrambleSuit repository) in category Users' development repositories (Attic) INFO: user repository detected, trying to find fork phw/scramblesuit WARNING: no existing fork found, entering user fork subroutine WARNING: no matching gitlab project found for user/phw/scramblesuit INFO: user fork subroutine failed, resuming normal procedure INFO: searching for projects matching scramblesuit import gitolite project user/phw/scramblesuit into gitlab legacy/gitolite/user/phw/scramblesuit with desc 'Philipp's ScrambleSuit repository'?^G [Y/n] INFO: checking if remote repo https://git.torproject.org/user/phw/scramblesuit exists INFO: importing gitolite project user/phw/scramblesuit into gitlab legacy/gitolite/user/phw/scramblesuit with desc 'Philipp's ScrambleSuit repository' INFO: importing project into GitLab INFO: Successfully connected to https://gitlab.torproject.org INFO: loading group legacy/gitolite/user/phw INFO: creating repository scramblesuit (scramblesuit) in namespace legacy/gitolite/user/phw from https://git.torproject.org/user/phw/scramblesuit into https://gitlab.torproject.org/legacy/gitolite/user/phw/scramblesuit INFO: archiving project INFO: migrating Gitolite repository https://gitweb.torproject.org/user/phw/scramblesuit.git to GitLab project https://gitlab.torproject.org/legacy/gitolite/user/phw/scramblesuit INFO: adding entry to rewrite_map /home/anarcat/src/tor/tor-puppet/modules/profile/files/git/gitolite2gitlab.txt INFO: modifying gitolite.conf to add: "config gitweb.category = Migrated to GitLab" INFO: rewriting gitolite config /home/anarcat/src/tor/gitolite-admin/conf/gitolite.conf to change project user/phw/scramblesuit to category Migrated to GitLab [...] ``` Acute eyes will notice the [bell used as a notification mechanism][] as well in this transcript. [bell used as a notification mechanism]: https://anarc.at/blog/2022-11-08-modern-bell-urgency/ A lot of the code is now useless for us, but some, like "commit and push" or [`is-repo-empty`][] live on in the [git module][] and, of course, the [gitlab module][] has grown some legs along the way. We've also found fun bugs, like a [file descriptor exhaustion in bash][], among other oddities. The [retirement milestone][] and [issue 41215][] has a detailed log of the migration, for those curious. [issue 41215]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41215 [retirement milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/11#tab-issues [file descriptor exhaustion in bash]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=642504 [gitlab module]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/-/blob/85121b4a8a293cebb… [git module]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/-/blob/85121b4a8a293cebb… [`is-repo-empty`]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/-/blob/85121b4a8a293cebb… This was a challenging project, but it feels nice to have this behind us. This gets rid of 2 of the 4 remaining machines running Debian "old-old-stable", which moves a bit further ahead in our late [bullseye upgrades milestone][]. [bullseye upgrades milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/5#tab-issues Full transparency: we tested GPT-3.5, GPT-4, and other large language models to see if they could answer the question "write a set of rewrite rules to redirect GitWeb to GitLab". This has become a standard LLM test for your faithful writer to figure out how good a LLM is at technical responses. None of them gave an accurate, complete, and functional response, for the record. The actual rewrite rules as of this writing follow, for humans that actually like working answers provided by expert humans instead of artificial intelligence which currently seem to be, glorified, mansplaining interns. ## git.torproject.org rewrite rules Those rules are relatively simple in that they rewrite a single URL to its equivalent GitLab counterpart in a 1:1 fashion. It relies on the [rewrite map][] mentioned above, of course. [rewrite map]: https://archive.torproject.org/websites/gitolite2gitlab.txt ``` RewriteEngine on # this RewriteMap connects the gitweb projects to their GitLab # equivalent RewriteMap gitolite2gitlab "txt:/etc/apache2/gitolite2gitlab.txt" # if this becomes a performance bottleneck, convert to a DBM map with: # # $ httxt2dbm -i mapfile.txt -o mapfile.map # # and: # # RewriteMap mapname "dbm:/etc/apache/mapfile.map" # # according to reports lavamind found online, we hit such a # performance bottleneck only around millions of entries, which is not our case # those two rules can go away once all the projects are # migrated to GitLab # # this matches the request URI so we can check the RewriteMap # for a match next # # WARNING: this won't match URLs without .git in them, which # *do* work now. one possibility would be to match the request # URI (without query string!) with: # # /git/(.*)(.git)?/(((branches|hooks|info|objects/).*)|git-.*|upload-pack|receive-pack|HEAD|config|description)?. # # I haven't been able to figure out the actual structure of # those URLs, so it's really hard to figure out the boundaries # of the project name here. I stopped after pouring around the # http-backend.c code in git # itself. https://www.git-scm.com/docs/http-protocol is also # kind of incomplete and unsatisfying. RewriteCond %{REQUEST_URI} ^/(git/)?(.*).git/.*$ # this makes the RewriteRule match only if there's a match in # the rewrite map RewriteCond ${gitolite2gitlab:%2|NOT_FOUND} !NOT_FOUND RewriteRule ^/(git/)?(.*).git/(.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$2}.git/$3 [R=302,L] # Fallback everything else to GitLab RewriteRule (.*) https://gitlab.torproject.org [R=302,L] ``` ## gitweb.torproject.org rewrite rules Those are the vastly more complicated GitWeb to GitLab rewrite rules. Note that we say "GitWeb" but we were actually *not* running [GitWeb][] but [cgit][], as the former didn't actually scale for us. [cgit]: https://git.zx2c4.com/cgit/ [GitWeb]: https://git-scm.com/docs/gitweb ``` RewriteEngine on # this RewriteMap connects the gitweb projects to their GitLab # equivalent RewriteMap gitolite2gitlab "txt:/etc/apache2/gitolite2gitlab.txt" # special rule to process targets of the old spec.tpo site and # bring them to the right redirect on the new spec.tpo site. that should turn, for example: # # https://gitweb.torproject.org/torspec.git/tree/address-spec.txt # # into: # # https://spec.torproject.org/address-spec RewriteRule ^/torspec.git/tree/(.*).txt$ https://spec.torproject.org/$1 [R=302] # list of endpoints taken from cgit's cmd.c # those two RewriteCond are necessary because we don't move # all repositories at once. once the migration is completed, # they can be removed. # # and yes, they are copied all over the place below # # create a match for the project name to check if the project # has been moved to GitLab RewriteCond %{REQUEST_URI} ^/(.*).git(/.*)?$ # this makes the RewriteRule match only if there's a match in # the rewrite map RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND # main project page, like summary below RewriteRule ^/(.*).git/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/ [R=302,L] # summary RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/summary/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/ [R=302,L] # about RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/about/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/ [R=302,L] # commit RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond "%{QUERY_STRING}" "(.*(?:^|&))id=([^&]*)(&.*)?$" RewriteRule ^/(.*).git/commit/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commit/%2 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/commit/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/HEAD [R=302,L] # diff, incomplete because can diff arbitrary refs and files in cgit but not in GitLab, hard to parse RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} id=([^&]*) RewriteRule ^/(.*).git/diff/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commit/%1 [R=302,L,QSD] # patch RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} id=([^&]*) RewriteRule ^/(.*).git/patch/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commit/%1.patch [R=302,L,QSD] # rawdiff, incomplete because can show only one file diff, which GitLab cannot RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} id=([^&]*) RewriteRule ^/(.*).git/rawdiff/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commit/%1.diff [R=302,L,QSD] # log RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} h=([^&]*) RewriteRule ^/(.*).git/log/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/%1 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/log/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/HEAD [R=302,L] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/log(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/HEAD$2 [R=302,L] # atom RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} h=([^&]*) RewriteRule ^/(.*).git/atom/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/%1 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/atom/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/HEAD [R=302,L,QSD] # refs, incomplete because two pages in GitLab, defaulting to "tags" RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/refs/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tags [R=302,L] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} h=([^&]*) RewriteRule ^/(.*).git/tag/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tags/%1 [R=302,L,QSD] # tree RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} id=([^&]*) RewriteRule ^/(.*).git/tree(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tree/%1$2 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/tree(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tree/HEAD$2 [R=302,L] # /-/tree has no good default in GitLab, revert to HEAD which is a good # approximation (we can't assume "master" here anymore) RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/tree/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tree/HEAD [R=302,L] # plain RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} h=([^&]*) RewriteRule ^/(.*).git/plain(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/raw/%1$2 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/plain(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/raw/HEAD$2 [R=302,L] # blame: disabled #RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ #RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND #RewriteCond %{QUERY_STRING} h=([^&]*) #RewriteRule ^/(.*).git/blame(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/blame/%1$2 [R=302,L,QSD] # same default as tree above #RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ #RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND #RewriteRule ^/(.*).git/blame(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/blame/HEAD/$2 [R=302,L] # stats RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/stats/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/graphs/HEAD [R=302,L] # still TODO: # repolist: once migration is complete # # cannot be done: # atom: needs a feed token, user must be logged in # blob: no direct equivalent # info: not working on main cgit website? # ls_cache: not working, irrelevant? # objects: undocumented? # snapshot: pattern too hard to match on cgit's side # special case, we keep a copy of the main index on the archive RewriteRule ^/?$ https://archive.torproject.org/websites/gitweb.torproject.org.html [R=302,L] # Fallback: everything else to GitLab RewriteRule .* https://gitlab.torproject.org [R=302,L] ``` The reference copy of those is available in our (currently private) Puppet git repository. -- Antoine Beaupré torproject.org system administration

2 1

PieroV's Monthly Status Report, April 2024
by Pier Angelo Vendrame 01 May '24

01 May '24

Hi everyone! Here is my status report for April 2024. This month, I explored the automation possibilities I mentioned last month. I wrote a script [0] that automatically rebases a few branches on top of gecko-dev/esr115 [1], cherry-picks commits from our default+maintenance branches, and moves them to the appropriate position of the patchset. In this way, when Mozilla tags a new release, our new branches will be ready without us doing anything. I'm still testing that everything works as expected, so I'm running the script manually. But the following step would be to run it automatically and regularly. Also, we could modify it to send us a notification when a new Firefox tag is published. I also wrote a script to automate release preparations [2], but it's still under review. This script will check for new releases of our dependencies and update our build configuration and changelogs automatically. We already had a script to create changelogs from linked GitLab issues, but we had to add updates manually. This, and a few other details, will be automated by the script, too. Then, I addressed the various comments on the MR to improve error handling in the TorConnect module [3] I mentioned in my previous report. In addition, I started working on Mullvad Browser's Windows installer [4] again. sajolida has prepared some wireframes on how to structure it, and I tried to check what's possible with NSIS. I helped with the releases for this month. I rebased both the release and the alpha channel and backported some fixes from alpha to stable. I'm also continuing my efforts to keep a branch on top of Firefox's rapid release and to (slowly but steadily) address fingerprinting issues. And since the next Firefox ESR (128) is going to be nightly in a couple of weeks [5], I restarted my uplift effort [6] as well. Finally, I made some quick fixes: a fallback mechanism to show the circuit display also when the bridge lines don't have a fingerprint (e.g., with conjure) [7], a patch for our Japanese localization that only partially worked on macOS [8], and more. Best, Pier [0] https://gitlab.torproject.org/pierov/lazy-scripts/-/blob/main/rebase-tool/r… [1] https://github.com/mozilla/gecko-dev/tree/esr115 [2] https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_re… [3] https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… [4] https://gitlab.torproject.org/tpo/applications/mullvad-browser/-/issues/200 [5] https://whattrainisitnow.com/release/?version=128 [6] https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40789#n… [7] https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… [8] https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42535

1 0

Job opening at Tor - Network Health Engineer
by Tyler Corn 29 Apr '24

29 Apr '24

Hi everyone! We have a new job opening for a Network Health Engineer! https://www.torproject.org/about/jobs/network-health/ If you are or know someone who would be a good fit and wants to join our team, please apply/share. Have a great week and thanks for helping us spread the word! 🙂 Best, Tyler

1 0

Anti-censorship team meeting notes, 2024-04-25
by onyinyang 25 Apr '24

25 Apr '24

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2024/tor-meeting.2024-04-25-16.15.html And our meeting pad: Anti-censorship work meeting pad -------------------------------- Anti-censorship -------------------------------- Next meeting: Thursday, May 2 16:00 UTC Facilitator: meskio ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: onyinyang == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from sponsors, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Sponsor 96 <-- meskio, shell, onyinyang, cohosh * https://gitlab.torproject.org/groups/tpo/-/milestones/24 * Sponsor 150 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == == Discussion == * What is the current repo for the snowflake website? * https://gitlab.torproject.org/tpo/web/snowflake * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * is in the process to be moved to tpo/web but not sure if it has already being moved * Snowflake Performance Improvement: * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * There is possible alternative design that would reduce complexity, should we do it? https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… == Actions == == Interesting links == * Maybe already known, Brave Browser has a snowflake proxy feature (since 2023) https://github.com/brave/brave-browser/issues/25315 == Reading group == * We will discuss "" on * * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2024-04-25 Last week: - worked on Lox integration testing - released a new version of the snowflake addon - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… This week: - follow up on reported SQS errors - update wasm-bindgen fork to fix some bugs and hopefully upstream changes - create a Lox test environment and instructions for the browser team - https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42503 Needs help with: dcf: 2024-04-25 Last week: - posted and briefly summarized the paper "Snowflake Anonymous Network Traffic Identification" https://lists.torproject.org/pipermail/anti-censorship-team/2024-April/0003… https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - open issue to disable /debug endpoint on snowflake broker - move snowflake-02 to new VM Help with: meskio: 2023-04-25 Last week: - bridgestrap not publishing collector metrics (bridgestrap#199) - WIP of the email distributor in rdsys (rdsys#186) - document the bridge distribution mechanisms in rdsys to replace bridges.tpo/info (rdsys#137) Next week: - email distributor in rdsys (rdsys#186) Shelikhoo: 2024-04-18 Last Week: - [Merge Request WIP] Add Container Image Mirroring from Tor Gitlab to Docker Hub(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/… - [Merge Request Done] Rename Stable Container Tags to Latest https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - [Fixed!] Investigate broken debian-testing pipeline for snowflake(Remove apt install lbzip2 to avoid broken dependencies: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) - Merge request reviews Next Week/TODO: - Revise Snowflake Performance Improvement MR - Merge request reviews onyinyang: 2023-04-25 Last week(s): - Responding to pre-existing MR reviews and other notifications/questions in gitlab This week: - Responding to pre-existing MR reviews and other notifications/questions in gitlab Probably next week and beyond: - preparing for upcoming panel (maybe) - implement some preliminary user feedback mechanism to identify bridge blocking based on Vecna's work - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum - attempt hyper upgrade again and tokio upgrade (long term things were discussed at the meeting!): https://pad.riseup.net/p/tor-ac-community-azaleas-room-keep - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2023-04-18 Last weeks: - Extended Pion dtls library to support hooking of client hellos (PR: https://github.com/pion/dtls/pull/631) Fingerprint generation of firefox and chrome + mimicking using the client hello hook have been moved to its own repo: https://github.com/theodorsm/covertDTLS - Gave some comments on the recent"Snowflake Anonymous Network Traffic Identification" paper: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…. Next weeks: - Add hook to pion/webrtc so it can be used in snowflake. - Test and validate mimicked client hello with snowflake. - Add randomized fingerprint - Analyze the snowflake captures done in "Snowflake Anonymous Network Traffic Identification" paper. Help with: Facilitator Queue: meskio shelikhoo onyinyang 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue -- --- onyinyang GPG Fingerprint 3CC3 F8CC E9D0 A92F A108 38EF 156A 6435 430C 2036

1 0

Anti-censorship team meeting notes, 2024-04-18
by Shelikhoo 18 Apr '24

18 Apr '24

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2024/tor-meeting.2024-04-18-16.15.html And our meeting pad: Anti-censorship work meeting pad -------------------------------- Anti-censorship -------------------------------- Next meeting: Thursday, Apr 25 16:00 UTC Facilitator: onyingyang ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: shelikhoo == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from sponsors, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Sponsor 96 <-- meskio, shell, onyinyang, cohosh * https://gitlab.torproject.org/groups/tpo/-/milestones/24 * Sponsor 150 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == == Discussion == * Latest snowflake addon reviewer feedback requires a consent prompt for the collection of personal data * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * is already being implemented and the addon is in the store == Actions == == Interesting links == * Snowflake support in Greatfire Envoy (using their own proxies and bridge as I understand) * https://github.com/greatfire/envoy/pull/63 * "Snowflake Anonymous Network Traffic Identification" January 2024 * https://link.springer.com/chapter/10.1007/978-981-99-9247-8_40 * Research from China, too recent to have been referenced in the Snowflake paper * Corresponding author Xu Dawei also has a paper on secure rendezvous using a blockchain https://link.springer.com/chapter/10.1007/978-3-031-15777-6_14 == Reading group == * We will discuss "" on * * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2024-04-18 Last week: - added a consent prompt for the snowflake addon - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - deployed snowflake addon v0.8.0 (got approved) - updated npm script for preparing addon - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - added a button to reopen consent prompt for addon - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - Added a fingerprint field to Lox bridge table after feedback from vecna - https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/147 - reviewed and merged dependency updates - added a new meek bridge to the censorship settings - https://gitlab.torproject.org/tpo/anti-censorship/rdsys-admin/-/merge_reque… This week: - release a new version of snowflake addon once !72 and !73 are merged - follow up on reported SQS errors - update wasm-bindgen fork to fix some bugs and hopefully upstream changes - create a Lox test environment and instructions for the browser team - https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42503 Needs help with: dcf: 2024-04-18 (since 2024-03-21) Last week: - reviewed draft MR for unreliable data channels https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - investigated a momentary drop in users of the snowflake-01 bridge https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - azure CDN bookkeeping https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Snowflake-co… - archived snowflake-webext 0.8.0 https://archive.org/details/snowflake-webextension-0.8.0 Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - open issue to disable /debug endpoint on snowflake broker - move snowflake-02 to new VM Help with: meskio: 2023-04-18 Last week: - persistency for resources in rdsys (rdsys#56) - use the same distributor for bridges sharing IP or fingerprint (rdsys!293) - integration tests for rdsys (rdsys#180) Next week: - email distributor in rdsys (rdsys#186) Shelikhoo: 2024-04-18 Last Week: - [Merge Request WIP] Add Container Image Mirroring from Tor Gitlab to Docker Hub(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/… - [Merge Request Done] Update lyrebird version to v0.2.0 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyre… - [Merge Request] Rename Stable Container Tags to Latest https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - Merge request reviews Next Week/TODO: - Merge request reviews - Investigate broken debian-testing pipeline for snowflake onyinyang: 2023-04-18 Last week(s): - HACS - DRL meeting - AFK for 1 week This week: - Responding to pre-existing MR reviews and other notifications/questions in gitlab Probably next week and beyond: - preparing for upcoming panel - implement some preliminary user feedback mechanism to identify bridge blocking based on Vecna's work - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum - attempt hyper upgrade again (long term things were discussed at the meeting!): https://pad.riseup.net/p/tor-ac-community-azaleas-room-keep - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2023-04-18 Last weeks: - WIP: extended Pion dtls library to support mimicking of client hellos. A github action generate fresh dtls handshakes of the newest firefox and chromium versions. The bytes of the client hello are replayed to mimick common dlts fingerprints. Next weeks: - Test and validate mimicked client hello with snowflake. - PR to pion github, talk with sean. Help with: Facilitator Queue: onyinyang meskio shelikhoo 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue

1 0

Job Opportunity - Grants Coordinator
by Erin Wyatt 16 Apr '24

16 Apr '24

Hello everyone, We have a new job opening for a Grants Coordinator! https://www.torproject.org/about/jobs/grants-coordinator/ If you are or know someone who would be a good fit and wants to join our team, please apply/share. Thanks! :) Cheers, Erin

1 0

OONI Monthly Report: March 2024
by Maria Xynou 15 Apr '24

15 Apr '24

Hello, This email shares OONI's monthly report for March 2024. *# OONI Monthly Report: March 2024* Throughout March 2024, the OONI team worked on the following sprints: * Sprint 112 (1st-10th March 2024) * Sprint 113 (11th-24th March 2024) Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni Highlights are shared in this report below. *## OONI Run* As part of our work on creating the next generation version of OONI Run (“OONI Run v2”), there were several key activities we undertook this month. We continued QA testing for the revamped OONI Probe Android application, and we fixed several issues that were found as a result ( https://github.com/ooni/run/issues/152, https://github.com/ooni/run/issues/149, https://github.com/ooni/run/issues/142, https://github.com/ooni/run/issues/143, https://github.com/ooni/run/issues/141, https://github.com/ooni/run/issues/140) We also continued QA testing for the new OONI Run v2 dashboard, which will allow users to create and maintain their OONI Run links. We fixed several issues that were found as a result of testing ( https://github.com/ooni/run/issues/150, https://github.com/ooni/run/issues/145, https://github.com/ooni/ooni.org/issues/1547, https://github.com/ooni/run/issues/144, https://github.com/ooni/ooni.org/issues/1540) We also finished refactoring the API (https://github.com/ooni/backend/issues/788) *## OONI Probe CLI* In March 2024, we released OONI Probe CLI 3.21.0: https://github.com/ooni/probe-cli/releases/tag/v3.21.0 Moreover, we: * Added support for deploying the Web Connectivity Test Helper using a Docker container on cloud platforms ( https://github.com/ooni/probe-cli/pull/1520) * Merged a community contribution to fix `tlsmiddleboxes` with IPv6 ( https://github.com/ooni/probe/issues/2689) * Fixed a bug in our `./pkg/gobash` tool for automatically using the correct version of Go where we were not correctly validating zip files ( https://github.com/ooni/probe-cli/pull/1521) * Reworked the `./pkg/oonimkall` package to facilitate using OONI Run v2 from mobile applications (https://github.com/ooni/probe-cli/pull/1526) * Merged a community contribution fixing the Tor Snowflake experiment ( https://github.com/ooni/probe-cli/pull/1531) *## OONI Explorer* To identify which censorship findings would be most useful (for the internet freedom community) to present on OONI Explorer ( https://explorer.ooni.org/) we conducted extensive user research in March 2024. Specifically, we created a survey (https://forms.gle/tYv3pecJiUFfUfEh9) which we shared widely with members of our community and with the broader internet freedom community. The goal of this survey was to improve our understanding of how community members use OONI Explorer, the types of information that they find most useful, the challenges they encounter in finding information, and how the platform could be improved to enable access to censorship findings. We analyzed the survey results to inform design and information architecture decisions that will guide the implementation of pages that present thematic censorship findings on OONI Explorer. In addition to the survey, we also carried out interviews with members of our community who use OONI Explorer as part of research and advocacy. The goal of these interviews was to supplement the qualitative data collected through the surveys, and to better understand how they use OONI Explorer and the challenges they encounter in discovering censorship findings through the platform. Based on the findings from the survey and interviews, we created a manually-curated list of domains that could be featured in each of the thematic areas, and we started evaluating various approaches for presenting relevant censorship findings. On the development side of things, we made a few improvements to OONI Explorer. Specifically, we improved the performance of the calendar component on the MAT and Search pages ( https://github.com/ooni/explorer/commit/a437373e15f08ba935ec9772b1382535a78…) and we improved the handling of errors and validations on the OONI Explorer country pages ( https://github.com/ooni/explorer/commit/edb33e2b4e89430d28ceee6530d602616e0… ). Notably, the OONI dataset reached 2 billion measurements in March 2024 ( https://twitter.com/OpenObservatory/status/1764644397700841669) We thank our community for contributing measurements over the past decade, shedding light on internet censorship worldwide. *## Automating censorship detection and characterization based on OONI measurements* In March 2024, we made progress towards shipping the new OONI Data Pipeline into production. Specifically, we integrated the pull request that implements the new Experiment Result model into the main branch ( https://github.com/ooni/data/pull/43) More importantly, we worked on researching orchestration systems for running the periodic tasks necessary for analyzing and processing OONI measurements (https://github.com/ooni/data/issues/46) We opened a pull request that adds support for using Temporal as an orchestration system ( https://github.com/ooni/data/pull/58) We also made a piece of very important refactoring, where we separate the OONI Data Pipeline (that’s to be run on servers) from the command line interface and library component that can be installed by end users ( https://github.com/ooni/data/pull/60) *## Research collaborations with partners on upcoming reports* We continued to coordinate with our partners on research efforts required for upcoming research reports. Specifically, we coordinated with our partners on extensive updates to the Citizen Lab test lists for Russia, Kazakhstan, Bangladesh, and Iran. In March 2024, the updates for the test lists of Russia and Kazakhstan were finalized. Specifically: * Roskomsvoboda updated the Russian test list: https://github.com/citizenlab/test-lists/pull/1691 * Internet Freedom Kazakhstan(IFKZ) updated the test list for Kazakhstan: https://github.com/citizenlab/test-lists/pull/1695 (the pull request was opened on 1st April 2024, but the work was conducted throughout March 2024) *## Research report on internet censorship in Tanzania* Beyond the aforementioned (upcoming) research reports, we have been working on a research report that documents internet censorship in Tanzania based on the analysis of OONI data collected between January 2023 to January 2024. In March 2024, we completed all of the data analysis, research, and writing required for our research report documenting internet censorship in Tanzania. We scheduled the publication of this report in April 2024 to enable more thorough review (both by the OONI team and local partners) prior to publication. *## Planning the OONI Partner Gathering 2024* In preparation for the upcoming OONI Partner Gathering in Malaysia in May 2024, we continued to coordinate on numerous logistics (flights, shuttle service, hotel, catering, etc.). We also assisted participants with visa requirements as needed (though very few participants require e-visas, while most participants don’t need visas to travel to Malaysia), and we coordinated with designers on event-related supplies and materials. Notably, we created and shared a survey with all OONI Partner Gathering participants to collect their feedback, which will help shape and inform the final agenda of the event. As our goal is to ensure that we create an agenda that is valuable to all participants, we requested their feedback on the types of sessions that they would find most useful, the types of skills and knowledge that they would like to learn, and the outcomes that would make their participation feel well spent. As participants started to fill out the survey, we started to analyze the results and create a draft agenda based on participant feedback. In addition to the pre-Partner Gathering survey (which was designed to collect feedback for the agenda), we also prepared a post-Partner Gathering survey to collect participant feedback after the event. These two surveys include a few mutual questions, which will enable us to compare and evaluate the effectiveness of the event in sharing OONI-related skills and knowledge. *## Community use of OONI data### Publication by Oxford on how digital technologies are used to wield authoritarian power during Russia’s 2024 presidential election* Ahead of Russia’s March 2024 presidential election, researchers from the Oxford Internet Institute and the University of Bremen published an article which shares their insights on how digital technologies can be used to wield authoritarian power in the context of the Russian election. Their article makes use of OONI data, and is available here: https://www.oii.ox.ac.uk/news-events/2024-russian-presidential-elections-ho… *## Community activities### OONI demo at the OTF Internet Freedom Expos* On 18th and 19th March 2024, OONI’s Arturo traveled to Washington D.C to participate in the Open Technology Fund’s (OTF) Internet Freedom Expo at the U.S Congress ( https://twitter.com/OpenTechFund/status/1773726871965802906) As part of his participation, Arturo hosted an OONI demo and presented OONI’s work. *### OONI participation in Roskomsvoboda’s DEMHACK 8* Between 29th-31st March 2024, Roskmosvoboda hosted the 8th hackathon in its series of DEMHACK hackathons (https://8.demhack.org/) OONI participated in this hackathon as a partner and as part of the jury board. As part of DEMHACK 8, two teams used OONI tools and methodologies for their hackathon projects. One team developed a prototype of the messengers' accessibility tests for DPI Detector ( https://github.com/nomah4/ubiquitous-rotary-phone) using OONI Probe tests as a reference. This team acquired third place in the hackathon. Another team used OONI data and the OONI API to compare the Roskomnadzor blocklist with the domains identified as blocked by OONI Probe, with the goal of potentially creating an alternative blocklist based on OONI measurements (https://github.com/1andrevich/ooni-zapret-list) We hope to stay in touch with both teams to further collaborate on investigating censorship in Russia. *### OONI Community Meeting* On 26th March 2024, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/) The topic of this meeting was “Network measurements and censorship circumvention”, and we invited the following speakers: * Ain Ghazal, Information Controls OTF Fellow with OONI * Cecylia Bocovich, Tor Project * Keith McManamen, Psiphon * Diwen Xue, PhD Student, University of Michigan, Censored Planet As part of this meeting, our invited guests shared their experiences and some of the questions discussed ( https://pad.riseup.net/p/ooni-community-meeting-keep) include: * Why is it important for circumvention projects to measure networks? How do existing network measurement projects contribute to circumvention technologies? * What could both network measurement projects and circumvention development projects do better in terms of measurement or in terms of data sharing? What are the challenges to fill the existing gaps? * How can network measurement data help inform circumvention tool development and deployment efforts? What type of data would be most useful, and what questions would you want such data to answer? * What are the challenges associated with measuring and evaluating whether a circumvention tool works? * What is needed to help ensure that circumvention technologies are more resilient to censorship? *## Measurement coverage* In March 2024, 58,901,099 OONI Probe measurements were collected from 2,904 networks in 173 countries around the world. This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/ ~ OONI team.

1 0

OONI Monthly Report: February 2024
by Maria Xynou 15 Apr '24

15 Apr '24

Hello, This email shares OONI's monthly report for February 2024. *# OONI Monthly Report: February 2024* Throughout February 2024, the OONI team worked on the following sprints: * Sprint 110 (1st-11th February 2024) * Sprint 111 (12th - 25th February 2024) Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni Highlights are shared in this report below. *## New partnership with Digital Rights Foundation Pakistan* In February 2024, we had the opportunity to formally establish a partnership with Digital Rights Foundation ( https://digitalrightsfoundation.pk/) a leading digital rights organization in Pakistan. As part of this partnership, we aim to collaborate on the study of internet censorship. *## Published report on blocks in Pakistan ahead of 2024 elections* In February 2024, we published a report (based on OONI data) on the blocking of an investigative news platform (Fact Focus), as well as on the blocking of PTI political party websites in Pakistan leading up to the country’s 2024 general election. The report is available here: https://explorer.ooni.org/findings/108298926901 *## Published FOCI paper on EU internet sanctions on Russian media* In collaboration with researchers from the University of Illinois Chicago, the University of Twente and the University of Amsterdam, we co-authored a paper which analyzes how different ISPs in EU member states implement sanctions on Russian media. This paper (which made extensive use of OONI data) was published by FOCI in February 2024: https://petsymposium.org/foci/2024/foci-2024-0001.pdf SIDN Labs published a blog post, providing a summary of the paper: https://www.sidnlabs.nl/en/news-and-blogs/internet-sanctions-on-russian-med… This blog post was cross-posted on the OONI blog: https://ooni.org/post/2024-eu-sanctions/ *## OONI Probe Mobile* We released OONI Probe Mobile 3.8.6 on iOS: https://github.com/ooni/probe-ios/releases/tag/v3.8.6 The latest version includes an important fix for the data quality of the Signal Private Messenger App experiment results. The Signal experiment fix is also available in the latest versions of OONI Probe Android (3.8.6), OONI Probe Desktop (3.9.3), and OONI Probe CLI. As part of the OONI Probe iOS 3.8.6 release, we also fixed an issue with RTL language support on the iOS app (https://github.com/ooni/probe/issues/2578) and we made several other bug fixes and improvements. To enable the localization of Deutsche Welle’s News Media Scan app ( https://play.google.com/store/apps/details?id=com.dw.ooniprobe) we added support for Transifex in the app ( https://github.com/ooni/translations/pull/32) *## OONI Run* As part of our work on creating the next generation version of OONI Run (“OONI Run v2”), there were several key activities we undertook this month. The main highlight is that we officially started QA testing for the Android version of the new OONI Probe app that has both an improved UI and support for new OONI Run links. This is a big next step in the progress for this project! Thus far, we completed a major initial testing pass to test the new user-interface. We also started QA testing for the new OONI Run v2 dashboard, which will allow users to create and maintain their OONI Run links. Some other highlights worth noting: * We ensured alignment between the spec and backend ( https://github.com/ooni/backend/issues/787) and we refactored the API as a result (https://github.com/ooni/backend/issues/788) * We started implementing support for OONI Run v2 in the iOS app ( https://github.com/ooni/ooni.org/issues/1518) *## OONI Probe CLI* Most of the OONI Probe CLI development work in February 2024 was focused on continuing to bring Web Connectivity v0.5 to production (as documented further in the next section). We also: * Added Wikimedia DNS as a DNS-over-HTTPS session resolver ( https://github.com/ooni/probe/issues/2532) * Reworked the code to generate summaries and made it more robust and safe ( https://github.com/ooni/probe/issues/2667) * Improved building tor as a library to make it work under ArchLinux ( https://github.com/ooni/probe/issues/2651) * Reworked test case generation for our QA suite to reduce churn and simplify the process of spotting bugs ( https://github.com/ooni/probe/issues/2677) * Fixed a bug in the test lists Gardener script to avoid trusting a URL database that has become stale (https://github.com/ooni/probe/issues/2684) *## Creating a methodology for measuring throttling* To measure throttling more effectively, we need to ship Web Connectivity v0.5, which is based on an underlying measurement library specifically designed to collect additional throttling-related information and, more generally, easier-to-process measurements. To this end, we continued our efforts to ensure a high standard of data quality for Web Connectivity v0.5 and made progress on our plans for rolling it out to production. Specifically, we: * Fixed a regression introduced in a previous release where Web Connectivity v0.5 was missing network events ( https://github.com/ooni/probe/issues/2674) * Refined the functionality to read partial response bodies when interrupted by a timeout (https://github.com/ooni/probe/issues/2654) which is relevant for throttling. * Ensured Web Connectivity v0.5 counted the bytes sent and received ( https://github.com/ooni/probe/issues/2655) * Made some updates to the Web Connectivity v0.5 specification ( https://github.com/ooni/probe/issues/2666) * Added the `tcptls_experiment` tag that was missing in Web Connectivity v0.5 to be backward compatible with v0.4 ( https://github.com/ooni/probe/issues/2673) * Addressed an issue that left the `client_resolver` field empty to ensure backward compatibility with v0.4 (https://github.com/ooni/probe/issues/2676 ). * Addressed a list of to-do items to clean up, refactor and add tests to the codebase (https://github.com/ooni/probe/issues/2669) which included: * Implementing DNS cache expiration for the DNS whoami code used by Web Connectivity v0.5 so that we can periodically update our understanding of the DNS we’re using (https://github.com/ooni/probe-cli/pull/1499) * Using a random DNS-over-UDP resolver drawn from a list of well known resolvers (rather than using 8.8.4.4) in response to community feedback ( https://github.com/ooni/probe-cli/pull/1500) *## OONI Design System* As part of our work on the OONI design system ( https://github.com/ooni/design-system/) we upgraded dependencies and configured storybook to use Vite instead of Webpack ( https://github.com/ooni/design-system/pull/172) which makes Storybook faster and requires less configuration. We also replaced Eslint and Prettier with Biome for code linting and formatting ( https://github.com/ooni/design-system/pull/173) which is all part of maintenance work to simplify the repository. We explored different frameworks for building and maintaining frontend components. To this end, we created two different examples of Button component implementation, using two of the most popular approaches in 2024: * TailwindCSS as an example of utility-first CSS library, using Class Variance Authority (https://github.com/ooni/design-system/pull/170) * Vanilla-Extract as an example of zero-runtime CSS-in-TS ( https://github.com/ooni/design-system/pull/171) *## OONI Explorer* To identify which censorship findings would be most useful (for the internet freedom community) to present on OONI Explorer ( https://explorer.ooni.org/) we organized a user research study ( https://github.com/ooni/ooni.org/issues/1543) so that we can better understand the habits and needs of our community (and determine how best to present this information on OONI Explorer). In February 2024, we drafted and finalized the user research survey, and we prepared for the user research interviews. We disseminated the survey and conducted user research interviews in March 2024. *## Test list updates* Ahead of Pakistan’s 2024 general elections, we updated the Citizen Lab test list for Pakistan: https://github.com/citizenlab/test-lists/pull/1651 *## Research collaborations with partners on upcoming reports* In February 2024, we continued to coordinate with our partners on research efforts required for upcoming research reports. Specifically, we coordinated with our partners on extensive updates to the Citizen Lab test lists for Russia, Kazakhstan, Bangladesh, and Iran. Each of these test list updates has a thematic focus based on the specific research questions of each (upcoming) research report. *## Planning the OONI Partner Gathering 2024* In preparation for the upcoming OONI Partner Gathering in Malaysia in May 2024, we continued to coordinate with the travel agency on booking flights for participants and other travel logistics. We also continued to coordinate with participants and the hotel on numerous other logistics. *## Updated the OONI Data Policy* On 23rd February 2024, we updated the OONI Data Policy: https://ooni.org/about/data-policy As part of this update, we replaced the use of Matomo analytics with Umami analytics. The Data Policy changes are visible here: https://github.com/ooni/ooni.org/pull/1550/files *## Rapid response efforts### Blocking of TikTok in Senegal* Starting from (at least) 25th January 2024, OONI data suggests that Senegal had been blocking access to TikTok. OONI data shows that the block was primarily visible on Sonatel (AS8346), and appears to have been implemented by means of TLS interference. In response, we posted about the block on Twitter/X ( https://twitter.com/OpenObservatory/status/1754944166130327716) and we shared relevant OONI data with local partners and advocacy groups. *### Unblocking of social media in Guinea* On 23rd February 2024, we reported the unblocking of social media platforms in Guinea on Twitter/X ( https://twitter.com/OpenObservatory/status/1761049064752177282) and shared relevant OONI data and information with advocacy groups. We had previously published a report on the blocking of social media platforms in Guinea, sharing OONI data on the blocking of WhatsApp, Telegram, Facebook, Twitter, Instagram, and YouTube ( https://explorer.ooni.org/findings/296303006301) OONI data shows that the blocks lasted between 24th November 2023 to 22nd February 2024. *## OONI citations### Tor Project blog post about defending internet freedom during elections in 2024* In February 2024, the Tor Project published a blog post about defending internet freedom with Tor during elections in 2024. As part of this blog post, they cite OONI research reports documenting censorship events that emerged during previous elections in countries around the world, and encourage the use of OONI tools and data. Their blog post is available here: https://blog.torproject.org/2024-defend-internet-freedom-during-elections/ *### CPJ statement on West African lawsuit against Senegal internet shutdowns* In February 2024, the Committee to Protect Journalists (CPJ) published a statement on the lawsuit filed against Senegal at the Economic Community of West African States (ECOWAS) Court of Justice challenging Senegal’s internet shutdowns in 2023. This statement cites OONI’s research report, which documented social media blocks and network outages in Senegal during political unrest in 2023 ( https://ooni.org/post/2023-senegal-social-media-blocks/) CPJ’s statement is available here: https://cpj.org/2024/02/cpj-welcomes-west-african-lawsuit-against-senegal-i… *## OONI Community Meeting* On 27th February 2024, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/) The topic of this meeting was “Litigation against internet censorship and shutdowns”, and we invited the following speakers: * Natalia Krapiva, Tech-Legal Counsel, Access Now * Toby Mendel, Executive Director of the Centre for Law and Democracy * Yelzhan Kabyshev, Director, Internet Freedom Kazakhstan (IFKZ) * Unggul Sagena, Head of Internet Access Division, Southeast Asia Freedom of Expression Network (SAFEnet) As part of this meeting, our invited guests shared their experiences from litigating against cases of internet censorship and internet shutdowns, while other community members who participated in the meeting shared questions (https://pad.riseup.net/p/ooni-community-meeting-keep) *## Measurement coverage* In February 2024, 57,616,784 OONI Probe measurements were collected from 2,981 networks in 173 countries around the world. This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/ ~ OONI team.

1 0

OONI Monthly Report: January 2024
by Maria Xynou 15 Apr '24

15 Apr '24

Hello, This email shares OONI's monthly report for January 2024. *# OONI Monthly Report: January 2024* Throughout January 2024, the OONI team worked on the following sprints: * Sprint 108 (1st - 14th January 2024) * Sprint 109 (15th - 28th January 2024) Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni Highlights are shared in this report below. *## Published report on news media censorship in Bangladesh during 2024 elections* In January 2024, we published a short report documenting the blocking of news media websites in Bangladesh (based on OONI data) amid the country’s 2024 general elections. The report is available here: https://explorer.ooni.org/findings/11686385001 *## OONI Probe Mobile* We released OONI Probe Mobile 3.8.6 on Android: https://github.com/ooni/probe-android/releases/tag/v3.8.6 This release includes an important fix for the data quality of the Signal Private Messenger App experiment (https://ooni.org/nettest/signal) results. It also ensures that the measurement engine is synced with OONI Probe CLI v3.20.0, and it includes several bug fixes and improvements. *## OONI Run* As part of our work on creating the next generation version of OONI Run (“OONI Run v2”), there were several key activities we undertook this month. Specifically, we continued to work towards achieving a ‘feature complete’ status for the Android application so that we can begin end-to-end testing. This included focusing on finishing the initial implementation of the add link flow, including addressing some feedback from early testing ( https://github.com/ooni/probe/issues/2595) We worked on implementing the dashboard link loading and we reviewed the updates ( https://github.com/ooni/probe/issues/2594) and we continued to work on the dashboard (https://github.com/ooni/run/pull/131) which is now ready for testing. Additionally, we enumerated the remaining backend work we need to complete as part of this project ( https://github.com/ooni/ooni.org/issues/1519) *## OONI Probe CLI* Most of the development work in January 2024 was focused on bringing Web Connectivity v0.5 to production (as documented in the next section below). In addition to this work, we reworked the output of the DNS Ping experiment to make it more actionable (https://github.com/ooni/probe-cli/pull/1444) by taking into account specific feedback provided by community members. We also increased the robustness of the internal libtor package to prevent concurrent tor instances, which would lead to a crash because libtor uses several shared, unlocked, global data structures ( https://github.com/ooni/probe-cli/pull/1445) This issue emerged while debugging Tor Snowflake experiment crashes ( https://github.com/ooni/probe/issues/2406) While addressing this issue is not enough to prevent crashes (given that we were not running libtor concurrently anyway), we tried to structurally prevent this from happening directly inside the libtor codebase. *## Creating a methodology for measuring throttling* To measure throttling more effectively, we need to ship Web Connectivity v0.5, which is based on an underlying measurement library specifically designed to collect additional throttling-related information and, more generally, easier-to-process measurements. To this end, in the previous month we ensured that Web Connectivity v0.5 produced the same results as version v0.4 (the one currently running in production) for all test cases in our QA suite. In January 2024, we continued this work by fixing additional data quality issues affecting Web Connectivity v0.4 and v0.5 so that when we release Web Connectivity v0.5, we are sure there are hopefully no regressions and only improvements. Specifically, we: * Ensured that the data analysis engine used by Web Connectivity v0.5 behaves correctly and detects multiple cases of blocking ( https://github.com/ooni/probe/issues/2640) While the overall result of the experiment uses a logic that is still compatible with v0.4, the result also contains flags indicating all the anomalies observed. For example, as part of the same measurement, OONI could detect both DNS and TLS interference. * Enumerated and fixed a number of edge cases where otherwise the measurement result would have been inaccurate, including cases where (a) a domain does not exist anymore but there’s still DNS censorship returning the IP address where a blockpage is hosted and (b) domains for which there are no A or AAAA records, which are now correctly recognized as not accessible and not blocked (https://github.com/ooni/probe/issues/2652) * Handled more cases where a website is not TCP or TLS reachable, which are now marked as not accessible, not blocked ( https://github.com/ooni/probe/issues/2299) * Ensured that IDNA (Internationalized Domain Names) are handled correctly by adding test cases and fixes (https://github.com/ooni/probe/issues/1925) * Added test cases to avoid flagging TCP blocking when IPv6 is available but there are no IPv6 routes (https://github.com/ooni/probe/issues/2456) * Ensured that cases where a website address is a loopback address (due to censorship or misconfiguration) are handled correctly ( https://github.com/ooni/probe/issues/1517) * Ensured that v0.5 does not include cached DNS responses to avoid confusing pipelines processing OONI data ( https://github.com/ooni/probe/issues/1530) * Fixed a bug where v0.5 was not able to correctly process URLs containing IPv4 or IPv6 addresses (https://github.com/ooni/probe/issues/1511) In addition to the code and test case changes described above, we also verified ~40 data quality issues and we started sketching out fixes for other outstanding issues. All the data quality issues that we vetted and closed in January 2024 can be found here: https://github.com/ooni/probe/issues?q=is%3Aissue+label%3A2024-01-data-qual… *## OONI Explorer* In January 2024, we worked on updating dependencies for OONI Explorer ( https://github.com/ooni/explorer/issues/901) and we started working towards building design system components ( https://github.com/ooni/design-system/issues/168) We also started working towards presenting thematic censorship findings on OONI Explorer through brainstorming sessions to discuss the potential scope of work and agree on next steps. *## OONI Backend* We made significant progress on coming up with a plan about the future of OONI infrastructure. Specifically, we started exploring several solutions for having a better pattern for deploying frontend and server-side components. As part of this work, we carried out an internal survey with our development team. After experimenting with several different solutions, we concluded that it’s probably ideal to go for an approach which can be adopted iteratively and that connects well with the existing patterns. We will therefore have a system for defining our infrastructure as code and start using (where it makes sense) more cloud infrastructure ( https://github.com/ooni/backend/issues/785) We also started making a big refactor of backend components to make it easier to deploy our OONI data analysis tool (https://github.com/ooni/data) as part of the OONI backend (https://github.com/ooni/backend/pull/791) *## Test list updates* Throughout January 2024, we coordinated with community members on updating the Citizen Lab test lists and we reviewed many pull requests. These include: * Updates to the Bangladesh test list ahead of the country’s January 2024 general elections (https://github.com/citizenlab/test-lists/pull/1544, https://github.com/citizenlab/test-lists/pull/1543, https://github.com/citizenlab/test-lists/pull/1531) * Updates to the Indonesian test list ( https://github.com/citizenlab/test-lists/pull/1527) * Updates to the Japanese and Global test lists ( https://github.com/citizenlab/test-lists/pull/1504, https://github.com/citizenlab/test-lists/pull/1511) * Updates to the Philippines test list ( https://github.com/citizenlab/test-lists/pull/1461, https://github.com/citizenlab/test-lists/pull/1559) * Updates to the Senegalese test list ( https://github.com/citizenlab/test-lists/pull/1600, https://github.com/citizenlab/test-lists/pull/1602) * Updates to the Thai test list ( https://github.com/citizenlab/test-lists/pull/1521) *## Research collaborations with partners on upcoming reports* In January 2024, we coordinated with the Tor Project on signing Fixed Award Agreements (FAAs) with 4 of our local partners to financially support their contributions to upcoming research reports (as part of a DRL grant). This officially initiated the research collaborations, and we started collaborating with our partners on several research reports. Specifically, we started this research process by finalizing the research scope for each of the upcoming reports, and collaborating with our partners on updating the Citizen Lab test lists for Russia, Kazakhstan, Bangladesh, and Iran. Each of these test list updates has a thematic focus based on the specific research questions of each (upcoming) research report. *## Planning the OONI Partner Gathering 2024* We are excited to share that on 8th and 9th May 2024, we will host an in-person OONI Partner Gathering in Kuala Lumpur, Malaysia. As part of this 2-day event, we will bring our partners (primarily from Asia and the Middle East) together to share skills and knowledge on internet censorship research. The goal of the event is to strengthen regional and global collaboration on censorship measurement research and advocacy. To host this event, we had previously (in 2023) carried out all the necessary fundraising and logistical planning work. As we were not able to reach the target overall budget that would have allowed us to bring all our partners from around the world, we decided to limit the scope of the event primarily to our partners from Asia and the Middle East (while also inviting some of our international partners, whose work is relevant/important to our regional partners). We decided this because countries in Asia and the Middle East experience the most pervasive forms of internet censorship (and we therefore prioritized these regions over other regions which experience less internet censorship). In an effort to ensure that the OONI Partner Gathering is hosted in a location that is as visa-friendly as possible for our participants, we developed a script which identifies such locations based on various parameters, such as visa requirements, safety index, and flight travel time (https://github.com/hellais/global-gather) Based on this data-driven approach, Malaysia was identified as one of the top visa-free countries in the world for our specific list of participants. This was another reason why we decided to limit the event primarily to our partners from Asia. That said, we hope to host additional OONI Partner Gathering events for our partners in Africa and Latin America over the next few years. In January 2024, we sent official invitations (along with a Concept Note) to our partners from Asia (Southeast Asia, South Asia, East Asia, Central Asia), the Middle East, and to some of our international partners. We also started coordinating with a travel agency on the booking of flights, with the hotel on the booking of rooms and meeting spaces, and on a number of other logistics. We expect to welcome 46 participants at the OONI Partner Gathering in May 2024. *## OONI workshops and presentations### OONI training for Digital Rights Foundation Pakistan* On 12th January 2024, OONI’s Elizaveta hosted an online OONI workshop for Pakistan’s Digital Rights Foundation (https://digitalrightsfoundation.pk/) training their team members on how to use OONI tools in preparation for Pakistan’s upcoming 2024 elections. The goal of this training was to enable Digital Rights Foundation to train local journalists on how to use OONI tools to measure and monitor censorship events. *### OONI Q&A for BebasID Indonesia* On 12th January 2024, OONI’s Elizaveta and Simone hosted an online OONI Q&A session with the BebasID community in Indonesia (https://bebasid.com/) As part of this session, we answered questions regarding the use of OONI tools, we collected community feedback, and we discussed the current state of internet censorship in Indonesia with the BebasID community. *### OONI training for human rights advocates in Senegal* Between 16th-18th January 2024, OONI’s Elizaveta co-hosted a 3-day hybrid training for human rights advocates in Dakar, Senegal. We organized this training in collaboration with our local partners, Computech ( https://computechinstitute.com/) and Jonction ( https://jonction.e-monsite.com/) and the event was possible thanks to support from Access Now. The goal of the training was to share OONI skills and knowledge that would enable Senegalese human rights advocates and trainers to run OONI workshops for their communities in 5 different regions of Senegal. As part of this training, Elizaveta facilitated the following sessions: * Introduction to Internet Censorship * Using OONI Probe and OONI Run: Hands-on workshop * Updating the Senegalese test list: Hands-on workshop * Using OONI Explorer: Hands-on workshop As part of this training, the participants created the programmes for their own training and scheduled them for January and February 2024. Overall, the goal was to enable local Senegalese trainers and human rights defenders to lead and facilitate OONI censorship measurement workshops in their communities in preparation for the country’s 2024 elections. *### OONI presentation at IAB Workshop on Barriers to Internet Access of Services (BIAS)* On 17th January 2024, OONI’s Simone presented our research report on how internet censorship in Russia changed during the first year of military conflict in Ukraine ( https://ooni.org/post/2023-russia-a-year-after-the-conflict/) as part of the IAB Workshop on Barriers to Internet Access of Services (BIAS), 2024 ( https://datatracker.ietf.org/group/biasws/about/) The presentation is available on YouTube: https://youtu.be/rz2qkRfaNVE?si=i33_45UTX96Vs21z&t=3404 *## Activities by the OONI community### OONI workshop by Digital Rights Foundation for journalists in Pakistan* In January 2024, our partner, Digital Rights Foundation ( https://digitalrightsfoundation.pk/) facilitated two workshops for 60 journalists in Pakistan in preparation for the country’s 2024 elections. As part of this workshop, Digital Rights Foundation introduced participants to OONI tools and encouraged them to run OONI Probe tests leading up to the election. As part of their 2024 election monitoring efforts, Pakistan’s Digital Rights Foundation encouraged the use of OONI Probe as part of their ElectionDesk resource: https://election2024.digitalrightsfoundation.pk/internet-shutdowns/ *### OONI training by Digital Rights Lab for journalists in Sudan* On 13th January 2024, our partner, Digital Rights Lab Sudan ( https://ooni.org/partners/drlab/) facilitated online OONI training sessions in Arabic for journalists in Sudan. As part of the training sessions, they introduced participants to OONI tools and internet measurement. *### OONI training by Computech and Jonction in Senegal* On 27th January 2024, our partners, Computech ( https://computechinstitute.com/) and Jonction ( https://jonction.e-monsite.com/) hosted an OONI training session in Saint Louis, Senegal. This training (run by local trainers) included hands-on workshops on using OONI Probe and OONI Run. *### OONI training by Advocacy Assembly fellow for human rights defenders in Sierra Leone* On 29th January 2024, the Center for Advocacy and Sustainable Empowerment hosted an in-person workshop in Bo, Sierra Leone, as part of the Advocacy Assembly Internet Shutdown Mentored Training Program ( https://advocacyassembly.org/en/news/236) As part of this workshop, they introduced participants to OONI tools. OONI’s Elizaveta joined this workshop for the online Q&A session to address participants’ questions in relation to OONI’s work and tools. *## OONI Community Meeting* On 30th January 2024, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/) during which we discussed the following topics: Preparing for elections in 2024: How to monitor and respond to censorship events Improving OONI Explorer: Request for community feedback on needs and challenges *## Measurement coverage* In January 2024, 64,404,108 OONI Probe measurements were collected from 3,028 networks in 167 countries around the world. This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/ ~ OONI team.

1 0