- tor-dev - lists.torproject.org

PT bridge reporting significant numbers of <OR> IP addresses after upgrade to 0.4.8.6; Conflux related?
by David Fifield 10 Oct '23

10 Oct '23

The Snowflake bridges do not expose their plain ORPort (they have `ORPort 127.0.0.1:auto` in torrc), and consequently they have always reported ≈0 <OR> IP addresses in the bridge-ip-transports line of bridge-extra-info descriptors, with virtually all connecting IP addresses being instead attributed to the snowflake transport. (The <OR> count is low but nonzero, which I have rationalized as tor making some small number of internal connections, or something.) I upgraded one of the two Snowflake bridges from 0.4.7.13 to 0.4.8.6 on 2023-09-24. Since then, the number of <OR> IP addresses has been roughly equal to the number of snowflake IP addresses. The ORPort is still not exposed; these are not external vanilla bridge users. Did something change between these versions that might cause PT connections to be double-counted, once for the transport and once for <OR>? Here are excerpted bridge-extra-info descriptors from before and after the version upgrade. Note the bridge-ip-transports lines. ``` @type bridge-extra-info 1.3 extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0 master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks published 2023-09-19 10:46:08 transport snowflake bridge-ip-versions v4=13528,v6=1384 bridge-ip-transports <OR>=8,snowflake=14912 ``` ``` @type bridge-extra-info 1.3 extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0 master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks published 2023-09-29 17:33:20 transport snowflake bridge-ip-versions v4=2880,v6=336 bridge-ip-transports <OR>=1632,snowflake=1592 ``` Below is a time series for a whole month. The thing to notice is not the overall decrease in users on 2023-09-20; that's due to https://forum.torproject.org/t/problems-with-snowflake-since-2023-09-20-bro…. Rather, the thing to notice is that before 2023-09-24, <OR> ≈ 0; and after, <OR> ≈ snowflake. date <OR> snowflake 2023-08-30 4 45240 ← running 0.4.7.13 here 2023-08-31 21 193412 2023-09-01 30 195647 2023-09-02 40 200405 2023-09-03 40 201730 2023-09-04 42 195097 2023-09-05 35 190842 2023-09-06 30 189694 2023-09-07 26 191524 2023-09-08 35 195833 2023-09-09 32 196244 2023-09-10 27 146972 2023-09-11 6 55491 2023-09-12 18 182519 2023-09-13 17 173536 2023-09-14 21 161396 2023-09-15 21 166379 2023-09-16 16 174350 2023-09-17 17 165063 2023-09-18 23 164734 2023-09-19 32 164075 2023-09-20 23 121623 2023-09-21 2 25268 2023-09-22 0 4314 2023-09-23 0 746 2023-09-24 239 606 ← upgrade to 0.4.8.6 on this day 2023-09-25 1919 1900 2023-09-26 3533 3470 2023-09-27 11919 11687 2023-09-28 20242 19890 2023-09-29 26583 26100 2023-09-30 25929 25400

2 2

Proposal 345: Migrating the tor specifications to mdbook
by Nick Mathewson 03 Oct '23

03 Oct '23

``` Filename: 345-specs-in-mdbook.md Title: Migrating the tor specifications to mdbook Author: Nick Mathewson Created: 2023-10-03 Status: Open ``` # Introduction I'm going to propose that we migrate our specifications to a set of markdown files, specifically using the [`mdbook`] tool. This proposal does _not_ propose a bulk rewrite of our specs; it is meant to be a low-cost step forward that will produce better output, and make it easier to continue working on our specs going forward. That said, I think that this change will enable rewrites in the future. I'll explain more below. # What is mdbook? Mdbook is a tool developed by members of the Rust community to create books with [Markdown]. Each chapter is a single markdown file; the files are organized into a book using a [`SUMMARY.md`] file. Have a look at the [`mdbook` documentation][`mdbook`]; this is what the output looks like. Have a look at this [source tree]: that's the input that produces the output above. Markdown is extensible: it can use numerous [plugins] to enhance the semantics of the the markdown input, add diagrams, output in more formats, and so on. # What would using mdbook get us _immediately_? There are a bunch of changes that we could get immediately via even the simplest migration to mdbook. These immediate benefits aren't colossal, but they are things we've wanted for quite a while. * We'll have a document that's easier to navigate (via the sidebars). * We'll finally have good HTML output. * We'll have all our specifications organized into a single "document", able to link to one another and cross reference one another. * We'll have perma-links to sections. * We'll have a built-in text search function. (Go to the [`mdbook`] documentation and hit "`s`" to try it out.) ## How will mdbook help us _later on_ as we reorganize? Many of the benefits of mdbook will come later down the line as we improve our documentation. * Reorganizing will become much easier. * Our links will no longer be based on section number, so we won't have to worry about renumbering when we add new sections. * We'll be able to create redirects from old section filenames to new ones if we need to rename a file completely. * It will be far easier to break up our files into smaller files when we find that we need to reorganize material. * We will be able make our documents even _easier_ to navigate. * As we improve our documentation, we'll be able to use links to cross-reference our sections. * We'll be able to include real diagrams and tables. * We're already using the [`mermaid`] tool in Arti in generate [protocol diagrams] and [architecture diagrams]; we can use this in our specs too, instead of hand-drawn ASCII art. * We'll be able to integrate proposals more easily. * New proposals can become new chapters in our specification simply by copying them into a new 'md' file or files; we won't have to decide between integrating them into existing files or creating a new spec. * Implemented but unmerged proposals can become additional chapters in an appendix to the spec. We can refer to them with permalinks that will still work when they move to another place in the specs. # How should we do this? ## Strategy My priorities here are: * no loss of information, * decent-looking output, * a quick automated conversion process that won't lose a bunch of time. * a process that we can run experimentally until we are satisfied with the results With that in mind, I'm writing a simple set of [`torspec-converter`] scripts to convert our old torspec.git repository into its new format. We can tweak the scripts until we like the that they produce. After running a recent `torspec-converter` on a fairly recent torspec.git, here is how the branch looks: https://gitlab.torproject.org/nickm/torspec/-/tree/spec_conversion?ref_type… And here's the example output when running mdbook on that branch: https://people.torproject.org/~nickm/volatile/mdbook-specs/index.html > Note: these is not a permanent URL; we won't keep the example output > forever. When we actually merge the changes, they will move into > whatever final location we provide. The conversion script isn't perfect. It only recognizes three kinds of content: headings, text, and "other". Content marked "other" is marked with \`\`\` to reneder it verbatim. The choice of which sections to split up and which to keep as a single page is up to us; I made some initial decisions in the file above, but we can change it around as we please. See the [configuration section] at the end of the `grinder.py` script for details on how it's set up. ## Additional work that will be needed Assuming that we make this change, we'll want to build an automated CI process to build it as a website, and update the website whenever there is a commit to the specifications. (This automated CI process might be as simple as `git clone && mdbook build && rsync -avz book/ $TARGET`.) We'll want to go through our other documentation and update links, especially the permalinks in spec.torproject.org. It might be a good idea to use spec.torproject.org as the new location of this book, assuming weasel (who maintains spec.tpo) also thinks it's reasonable. If we do that, we need to decide on what we want the landing page to look like, and we need _very much_ to get our permalink story correct. Right now I'm generating a .htaccess file as part of the conversion. ## Stuff we shouldn't do. I think we should continue to use the existing torspec.git repository for the new material, and just move the old text specs into a new archival location in torspec. (We *could* make a new repository entirely, but I don't think that's the best idea. In either case, we shouldn't change the text specifications after the initial conversion.) We'll want to figure out our practices for keeping links working as we reorganize these documents. Mdbook has decent redirect support, but it's up to us to actually create the redicrets as necessary. # The transition, in detail * Before the transition: - Work on the script until it produces output we like. - Finalize this proposal and determine where we are hosting everything. - Develop the CI process as needed to keep the site up to date. - Get approval and comment from necessary stakeholders. - Write documentation as needed to support the new way of doing things. - Decide on the new layout we want for torspec.git. * Staging the transition: - Make a branch to try out the transition; explicitly allow force-pushing that branch. (Possibly nickm/torspec.git in a branch called mdbook-demo, or torspec.git in a branch called mdbook-demo assuming it is not protected.) - Make a temporary URL to target with the transition (possibly spec-demo.tpo) - Once we want to do the transition, shift the scripts to tpo/torspec.git:main and spec.tpo, possibly? * The transition: - Move existing specs to a new subdirectory in torspec.git. - Run the script to produce an mdbook instance in torspec.git with the right layout. - Install the CI process to keep the site up to date. * Post-transition - Update links elsewhere. - Continue to improve the specs. # Integrating proposals We could make all of our proposals into a separate book, like rust does at https://rust-lang.github.io/rfcs/ . We could also leave them as they are for now. (I don't currently think we should make all proposals part of the spec automatically.) # Timing I think the right time to do this, if we decide to move ahead, is before November. That way we have this issue as something people can work on during the docs hackathon. # Alternatives I've tried experimenting with Docusaurus here, which is even more full-featured and generates pretty react sites [like this](https://docusaurus.io/). (We're likely to use it for managing the Arti documentation and website.) For the purposes we have here, it seems slightly overkill, but I do think a migration is feasible down the road if we decide we _do_ want to move to docusaurus. The important thing is the ability to keep our URLs working, and I'm confident we could do that The main differences for our purposes here seem to be: * The markdown implementation in Docusaurus is extremely picky about stuff that looks like HTML but isn't; it rejects it, rather than passing it on as text. Thus, using it would require a more painstaking conversion process before we could include text like `"<state:on>"` or `"A <-> B"` as our specs do in a few places. * Instead of organizing our documents in a `SUMMARY.md` with an MD outline format, we'd have to organize them in a `sidebar.js` with a javascript syntax. * Docusaurus seems to be far more flexible and have a lot more features, but also seems trickier to configure. <-- References --> [`mdbook`]: https://rust-lang.github.io/mdBook/ [source tree]: https://github.com/rust-lang/mdBook/tree/master/guide/src/ [Markdown]: https://en.wikipedia.org/wiki/Markdown [`SUMMARY.md`]: https://rust-lang.github.io/mdBook/format/summary.html [plugins]: https://github.com/rust-lang/mdBook/wiki/Third-party-plugins [`mermaid`]: https://mermaid.js.org/ [architecture diagrams]: https://gitlab.torproject.org/tpo/core/arti/-/blob/main/doc/dev/Architectur… [protocol diagrams]: https://gitlab.torproject.org/tpo/core/arti/-/blob/main/doc/dev/hs-overview… [`torspec-converter`]: https://gitlab.torproject.org/nickm/torspec-converter [configuration section]: https://gitlab.torproject.org/nickm/torspec-converter/-/blob/main/grinder.p…

1 0

[Update] Network Status APIs
by Mattia Righetti 03 Oct '23

03 Oct '23

Greetings Tor devs, I wanted to give a quick update on the progress of the Network Status APIs[0]. First of all, a couple of weeks ago we have deployed an initial version of the service internally so that we can benchmark and test it and see how it holds. Plus, we now have developed three more APIs: `/details`, `/clients`, `/weights`. In total, we now have covered almost every endpoint that is currently provided by onionoo. Just to give you a little bit more insights, `/summary` and `/details` fetch data from the Network Health team database and return a response identical to the one that onionoo clients expect right now. On the other hand, `/clients`, `/weights` and `bandwidth` will proxy the request to VictoriaMetrics. In the upcoming weeks we plan to stabilize and test these endpoints further and do some perf improvements. Meanwhile, we are also working on documenting the APIs so that potential clients know how to use them, when they'll be available. Those can be found in the project's repo Wiki[1]. Please do reach out to me, hiro or GeKo if you have any ideas or feedback you'd like to share with us. Cheers, Matt [0]: https://gitlab.torproject.org/tpo/network-health/metrics/networkstatusapi/ [1]: https://gitlab.torproject.org/tpo/network-health/metrics/networkstatusapi/-…

1 0

funion: A Tor Client in Elixir
by Emil Engler 21 Sep '23

21 Sep '23

Hello Tor hackers, I just want to announce "funion", a Tor client implementation in Elixir which has been my primary side-project from March 2023 to August 2023. I began this project as an exercise in order to understand both, the Tor protocol and Elixir, a little bit better. Choosing Elixir came to my mind after watching a Computerphile video[1] introducing Erlang. After watching it, I concluded that Tor's hierarchy of connections -> circuits -> streams fits in very well -- which it actually does. Currently, the implementation is capable of creating streams across several hops with the ability of sending and receiving data. The implementation has a size of roughly 3500 LOC. A very neat feature is that each connection is a dedicated process having the circuits as dedicated child processes and the streams as dedicated grandchild processes respectively. This design leads to some very nice benefits, including the fact that only the circuit processes hold the cryptographic keys used in communication (except the TLS keys). Keep in mind however that it still lacks many features, most notably support for the directory protocol, resulting in the necessity of hard-coding each OR with the keys extracted manually from the consensus. I am not sure if I will find the time to continue hacking on this though, but it has been a very fun and refreshing experience so far. The Elixir code might not be the most beautiful out there, as this was my first project in this language. I do not consider this to be anything near production ready. Beside this, I purposely decided to ignore certain practices common in cryptographic applications (such as overwriting sensitive keys with zeros) for the sake of simplicity, as my primary intention was to understand the Tor protocol, not to write a tool that allows secure and anonymous communication across the internet. If you seriously depend on anonymity: DO NOT USE THIS! The repository can be found here: https://github.com/emilengler/funion A talk given at BornHack 2023 talking about the internals and pitfalls can be found here: https://media.ccc.de/v/bornhack2023-56123-funion-a-tor-client-i My dedications go to the following people who have helped me dealing with understanding various internals of Tor, be it direct or indirect: - Roger Dingledine - Alexander Færøy - Ian Jackson - Nick Mathewson - The Talla Authors - trinity [1]: https://www.youtube.com/watch?v=SOqQVoVai6s

1 0

Automating dependency updates in Tor
by micah anderson 17 Sep '23

17 Sep '23

Tor has been undertaking security audits of code that we've been changing. Security audits are a good thing! They uncover blind spots, peel back assumptions, and present us with ways to improve our overall security posture. We intend to publish the results of the two that we've done recently, and commit to publishing every one we undertake - stay tuned. The first audit we did recently was a great success! The auditors remarked that although the scope was large, the number of issues uncovered was low, and that Tor in general adopts an admirably robust and hardened security posture and sound design decisions: "[Tor's] code was written to a first-rate standard and conformed to secure coding practices ... adopt[ing] highly-advanced and deliberately security focused building processes ... all which contribute towards considerable defense-in-depth security posture". One of the issues that came up was the overall lack of automated resiliency in our software supply chain. What does this mean? That means several dependencies in our software were outdated. Why were these outdated? Because we lack the automation. Tracking dependencies manually is difficult, you need to manually search for those updates individually (although some package managers offer automated functionality), it can be difficult to handle So now we have a solution, Renovate: a highly configurable system for dependency update automation. It scans your software, discovers dependencies, automatically checks to see if an updated version exists, and helps you by submitting automated pull requests. It is an open source project that we are self-hosting on our gitlab (its like 'dependabot' if you know that). A number of Tor projects are using it already, please consider using it for your project! Its very simple to use, and there is no harm in giving it a try. We are still trying this out, so your feedback[2] is important for how to move forwards. Ideally, we will have this problem solved automatically for all of our projects, but lets make sure things work well for everyone first. How do I use it? ----------------- To have renovate work on your gitlab project, you simply have to invite the 'renovate-bot' user (its a bot!) to your project (with the `Developer` access level), and then wait for it to do its work. Next time it runs, it will open an "Onboarding" issue[0], to get you started. The first time it runs, there may be a number of dependencies that need updating, which will result in a MR for each[1]. That could be overwhelming, but after the initial wave, things will calm down. Simply review the MR and merge it if it makes sense (making any code adjustments necessary). If you don't want that MR to happen, simply close it, and Renovate will stop bugging you about it. How does it work? ----------------- There is a project in our gitlab[2], which has a scheduled CI that runs every 30 minutes. When it runs, it looks to see what projects have the gitlab bot user 'renovate-bot' as a member, with 'developer' level access. For each of those projects, it then scans the project for any dependencies that need updating, and will open MRs to update those out-of-date dependencies (triggering CI builds). Your project must also have a CI that is being tended to, so that it runs and succeeds. I want to change its behavior ----------------------------- Renovate is highly configurable. You can decide what you do, and do not, want from Renovate. There are knobs for practically everything[3]. Renovate has a default[4] set of configurations that we've set organization-wide, you can override those in your project, and set any other configuration options[5] you might want. How to give feedback, ask questions, etc. ----------------------------------------- If you are looking for help, have questions, or want to give some feedback on global defaults or other aspects that could be improved, please file an issue[1]! 0. eg. https://gitlab.torproject.org/tpo/core/onionmasq/-/merge_requests/101 1. eg. https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/merge_requests/151 2. https://gitlab.torproject.org/tpo/tpa/renovate-cron/ 3. https://docs.renovatebot.com 4. https://github.com/renovatebot/renovate/blob/main/docs/development/configur… 5. https://docs.renovatebot.com/getting-started/use-cases/

1 0

Prop#344: Prioritizing Protocol Information Leaks in Tor
by Mike Perry 23 Aug '23

23 Aug '23

As part of Sponsor 112, Tor has an objective to address covert channels that malicious relays can use to deanonymize users. This proposal lays the groundwork by categorizing these issues, and prioritizes them according to their severity. Subsequent proposals will deal with specific solutions. I am posting just the intro section of this proposal here. The full proposal is at: https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/344-pr… Filename: 344-protocol-info-leaks.txt Title: Prioritizing Protocol Information Leaks in Tor Author: Mike Perry Created: 2023-07-17 Purpose: Normative Status: Open 0. Introduction Tor's protocol has numerous forms of information leaks, ranging from highly severe covert channels, to behavioral issues that have been useful in performing other attacks, to traffic analysis concerns. Historically, we have had difficulty determining the severity of these information leaks when they are considered in isolation. At a high level, many information leaks look similar, and all seem to be forms of traffic analysis, which is regarded as a difficult attack to perform due to Tor's distributed trust properties. However, some information leaks are indeed more severe than others: some can be used to remove Tor's distributed trust properties by providing a covert channel and using it to ensure that only colluding and communicating relays are present in a path, thus deanonymizing users. Some do not provide this capability, but can be combined with other info leak vectors to quickly yield Guard Discovery, and some only become dangerous once Guard Discovery or other anonymity set reduction is already achieved. By prioritizing information leak vectors by their co-factors, impact, and resulting consequences, we can see that these attack vectors are not all equivalent. Each vector of information leak also has a common solution, and some categories even share the same solution as other categories. This framework is essential for understanding the context in which we will be addressing information leaks, so that decisions and fixes can be understood properly. This framework is also essential for recognizing when new protocol changes might introduce information leaks or not, for gauging the severity of such information leaks, and for knowing what to do about them. Hence, we are including it in tor-spec, as a living, normative document to be updated with experience, and as external research progresses. It is essential reading material for any developers working on new Tor implementations, be they Arti, Arti-relay, or a third party implementation. This document is likely also useful to developers of Tor-like anonymity systems, of which there are now several, such as I2P, MASQUE, and Oxen. They definitely share at least some, and possibly even many of these issues. Readers who are relatively new to anonymity literature may wish to first consult the Glossary in Section 3, especially if terms such as Covert Channel, Path Bias, Guard Discovery, and False Positive/False Negative are unfamiliar or hazy. There is also a catalog of historical real-world attacks that are known to have been performed against Tor in Section 2, to help illustrate how information leaks have been used adversarially, in practice. We are interested in hearing from journalists and legal organizations who learn about court proceedings involving Tor. We became aware of three instances of real-world attacks covered in Section 2 in this way. Parallel construction (hiding the true source of evidence by inventing an alternate story for the court -- also known as lying) is a possibility in the US and elsewhere, but (so far) we are not aware of any direct evidence of this occurring with respect to Tor cases. Still, keep your eyes peeled... 0.1. Table of Contents 1. Info Leak Vectors 1.1. Highly Severe Covert Channel Vectors 1.1.1. Cryptographic Tagging 1.1.2. End-to-end cell header manipulation 1.1.3. Dropped cells 1.2. Info Leaks that enable other attacks 1.2.1. Handshakes with unique traffic patterns 1.2.2. Adversary-Induced Circuit Creation 1.2.3. Relay Bandwidth Lying 1.2.4. Metrics Leakage 1.2.5. Protocol Oracles 1.3. Info Leaks of Research Concern 1.3.1. Netflow Activity 1.3.2. Active Traffic Manipulation Covert Channels 1.3.3. Passive Application-Layer Traffic Patterns 1.3.4. Protocol or Application Linkability 1.3.5. Latency Measurement 2. Attack Examples 2.1. CMU Tagging Attack 2.2. Guard Discovery Attacks with Netflow Deanonymization 2.3. Netflow Anonymity Set Reduction 2.4. Application Layer Confirmation 3. Glossary The remainder of this proposal can be read at: https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/344-pr… -- Mike Perry

1 0

Calling Tor Volunteers and Developers! Help Shape the Future of Tor Relay Updates!
by Ravi Kumar 25 Jul '23

25 Jul '23

Dear Tor Volunteers and Developers, We are reaching out to you today to seek your invaluable expertise and insights in shaping the future of Tor relay updates. As part of an MSc Dissertation, a comprehensive study is underway to gain a deeper understanding of the significance of Tor relay updates from the unique perspective of developers like yourself. Your participation is pivotal to the success of this research, as we seek to explore your thoughts and opinions on a proposed automatic update mechanism for the relay. By lending your expertise, you will play a key role in advancing the health of the Tor network and enhancing the overall experience for millions of users around the globe. Rest assured that all your responses will be treated with the utmost care and confidentiality. To safeguard your data, the survey has been thoughtfully designed on an end-to-end encrypted platform, aligning with our commitment to ethical principles. We genuinely value your privacy and will handle your contributions with utmost respect and protection. We acknowledge the value of your time and effort, and we assure you that your input, which requires only 10 minutes, will wield a significant influence on the advancement of the Tor network. Don't miss out on this chance to contribute; the survey will remain open until August 5th. We invite you to join us today in shaping transformative research for Tor's bright future. Should you have any questions or require further information, please don't hesitate to contact us at S2450888(a)ed.ac.uk. We wholeheartedly welcome your feedback and queries, and your cooperation is genuinely appreciated. Together, we can make a meaningful difference and construct a stronger, more secure online world for everyone. Participate now and be an integral part of shaping the future of Tor relay updates through this link: https://cryptpad.fr/form/#/2/form/view/pb7wKm8zl0D62PsW93fz4wiW73i9nzDM-Jxc… [https://cryptpad.fr/customize/images/opengraph_preview/og-form.png]<https://cryptpad.fr/form/#/2/form/view/pb7wKm8zl0D62PsW93fz4wiW73i9nzDM-Jxc…> Encrypted Form<https://cryptpad.fr/form/#/2/form/view/pb7wKm8zl0D62PsW93fz4wiW73i9nzDM-Jxc…> CryptPad: end-to-end encrypted collaboration suite cryptpad.fr Thank you for your essential role in this important journey. Best regards, Ravi Kumar S2450888(a)ed.ac.uk The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

1 0

Network Health team and analysis work
by Georg Koppen 06 Jul '23

06 Jul '23

Hello everyone! The Network Health team is best known for its work in the bad-relays area and being concerned with providing metrics + keeping an eye on the health of the Tor network. While that involves doing analyses to answer our own questions it was not clear so far what we should do with questions about the network or available data which got raised by other teams (i.e. you). Should we do those analyses as well or should we just provide the necessary data and each team then tries to answer their questions by themselves? This got raised a bunch of times in the past[1][2] and I am happy to announce that the Network Health team is stepping up and helping you with your analyses, if desired. The idea would be that we'd do the analysis work in those cases, alone or together, and accumulate/improve a set of scripts and tools over time at a single place, so that both we and other interested parties can do those analyses faster and better in the long(er) run. The tools should be open so that other folks can help improving and shaping them according to their needs. We have a dedicated analysis project[3] where we track all analysis related issues and which contains tools and documentation about how to use them for our work. If there is any analysis work you would like us to get done, please file a ticket in that project. If you did a similar analysis in the past by yourself we'd love to hear about it as it might speed up our work and we could build documentation out of it and add available code to the analysis project. Additionally, please assume we are no experts in your area of work, but don't hesitate to file tickets because of that. We'll ask if things are unclear. :) To get your analysis questions answered in a timely manner it would greatly help if they were attached to some sponsored work as that makes it usually easier to justify picking them up among all the other things on our plate. Additionally, emergencies (like new blocking events in country X) should be easily justifiable as well, in particular as we have a "surprise 'anomaly analysis' on the network as needed"-item on our roadmap throughout the year. I plan to make sure that we have sufficient spare cycles available for that work should it be needed. If your desired analysis does not fall into either of those categories don't despair, we'll try to find a way to get it done nonetheless. Thanks, Georg P.S.: If you are not part of any of our teams but want to do an analysis by yourself or just note an analysis idea somewhere, feel free to start by filing a ticket in the analysis project as well. Others might just jump on it and get it done because they might be curious about the answers themselves. :) [1] https://gitlab.torproject.org/tpo/network-health/metrics/relay-search/-/iss… [2] https://gitlab.torproject.org/tpo/network-health/team/-/issues/250#note_286… [3] https://gitlab.torproject.org/tpo/network-health/analysis

1 0

[Update] Network Status APIs
by Mattia Righetti 06 Jul '23

06 Jul '23

Greetings Tor devs, During the last two weeks there has been an improvement in the Network Status APIs[0]. Network Status APIs is going to replace Onionoo web protocol as the Network Health Team moves most of the current bridges and relays data from files stored on disk to Postgres and VictoriaMetrics. We now have an initial working version of the `/summary` and `/bandwidth` endpoint. The first endpoint already has some working filters, while the latter does not yet. In the upcoming weeks we plan to stabilize and test these endpoints while we continue the development of others. The project initially started and was setup to look like a Onionoo copy to prevent most clients from breaking and for a smoother transition to the new protocol. We are now in the process of redefining some responses as the Onionoo implementation introduces some limitations. In particular, changes will affect responses that contain timeseries data stored on VictoriaMetrics as we thought it made more sense to just return a direct ref to VictoriaMetrics[1] for those requests, this way a client can decide to fetch referenced data or not based on its needs. Please do reach out to me, hiro or GeKo if you have any ideas or feedback you'd like to share with us. Cheers, Matt [0]: https://gitlab.torproject.org/tpo/network-health/metrics/networkstatusapi/ [1]: https://gitlab.torproject.org/tpo/network-health/metrics/networkstatusapi/-…

1 0

Network Status APIs
by Mattia Righetti 22 Jun '23

22 Jun '23

Greetings Tor devs, In an effort to improve current resource utilisation, the Network Health Status team is developing a new version of their pipeline [1]. Previously, much of the data was stored on files, this made many operations slow due to their I/O bound nature. The new pipeline will transfer much of the data related to Tor nodes and bridges from files stored on a single server's disk to two separate databases: Postgres and Victoria Metrics. With this new approach, there's the need to develop a new service that is going to replace the current onionoo web protocol [2]. The main objective of this project is to design a RESTful API service, using the actix_web framework, that is going to be integrated in the new pipeline v2.0 to support data retrieval of bridges and relays from the two databases and additionally provide new features such as historic data search. The project will be developed over the next few weeks as part of the GSoC sponsored program. Please do reach out to me, hiro or GeKo if you have any ideas or feedback you'd like to share with us. Cheers, Matt [1] https://gitlab.torproject.org/tpo/network-health/team/-/wikis/metrics/colle… [2] https://gitlab.torproject.org/tpo/network-health/metrics/networkstatusapi/

1 0