Dear Tor relay operators, Inspired by the Berkeley Protocol on Digital Open Source Investigations, Network Health Team and Community Team has developed a framework for conducting Network Health investigations. This proposal provides clear guidelines for conducting ethically sensitive investigations transparently, responsibly, and in a way that upholds the trust of the Tor user base and wider internet freedom community. You can read and comment on this proposal here: https://gitlab.torproject.org/tpo/community/policies/-/issues/25 Alternatively, you can read the full text below. Gus ``` * Filename: 101-running-network-health-investigations.md * Title: Framework for running Tor Network Health investigations * Author: Gus - gus at torproject.org, GeKo - gk at torproject.org * Created: 2024-10-15 * Status: Draft ``` ## Overview This proposal outlines a framework for conducting open source investigations of Tor Network Health related events and actors that potentially harm Tor users or the Tor network, such as denial-of-service attacks or malicious relay operators. Investigations of other types of network incidents, like censorship events, are out of scope of this proposal, even though they might want to follow the framework outlined here as well. Inspired by the Berkeley Protocol on Digital Open Source Investigations [1] (hereafter: Berkeley Protocol), but adapted to the needs of the Tor Community, this proposal provides clear guidelines for conducting ethically sensitive investigations transparently, responsibly, and in a way that upholds the trust of the Tor user base and wider internet freedom community. ## Motivation It is easy to contribute to the Tor network by setting up and running relays. While this low threshold is good for growing the network it, unfortunately, makes it easy as well for malicious actors trying to harm Tor users or the network. Using publicly accessible data and open-source resources empowers the Network Health team and open-source investigators to actively protect the Tor network against such malicious actors. However, thus far there are no guidelines on how to do such investigations in an ethical, reliable and reproducible manner which might prevent volunteers from looking closer at the Tor network out of fear they might do something wrong or harm users or the Tor Project. This proposal is meant to resolve that issue, encouraging interested volunteers to stay vigilant and help protecting the Tor network and its users. At the same time, it aims at providing transparency into how we at the Tor Project use publicly available data during network health investigations ourselves, as we follow the framework developed below as well. ## Framework The framework outlined in this proposal is following the open source investigation cycle as developed in Chapter 6 of the Berkeley Protocol, concluding with a section about guidelines for reporting the findings safely to the Tor Project, so we can take action if needed. The individual stages of the investigation process will be adapted to the Tor context while following the methodological, professional, and ethical principles collected in Chapter 2 of the Berkeley Protocol. Those principles are in particular important as advancing human rights is at the core of Tor's mission. The framework below does not address any security or legal concerns that could arise from doing open source investigations into network incidents or actors. In particular, if the planned investigation might result in danger or harm (be it physical, legal, or otherwise) to yourself or a third party, we urge you to additionally consult the legal and security sections of the Berkeley Protocol first for advice and proper planning of the investigation before it starts. Finally, keep in mind that investigators must adhere to the Tor Project's [Community Policies](https://community.torproject.org/policies). For instance, investigators must respect relay operators' human rights and privacy rights conventions as described in the [evalution criteria for solution to combat malicious relays](https://community.torproject.org/policies/relays/evaluation-criteria-for-sol...). ## Table of Contents 1. Searching and Monitoring 2. Preliminary assessment 3. Collection 4. Preservation 5. Verification 6. Analysis 7. Reporting ### 1. Searching and Monitoring 1.1. When searching for information and monitoring sources, investigators need to be aware of biases, both their own and those inherent in information online, which might taint the results. Developing and deploying several working hypotheses, which are guiding the search and monitoring efforts, is helping with biases but is not a panacea. Ultimately, it's crucial to be aware of potential biases involved in investigative work and countering them e.g. by broadening and diversifying search and monitoring efforts. 1.2. Besides useful external tools, like regular search engines or specialized ones (e.g., https://shodan.io/), investigators can deploy a collection of tools developed and maintained by the Tor Project to look into network anomalies and bad relay behavior. These tools can be found on the [Network Health GitLab repository](https://gitlab.torproject.org/tpo/network-health/). It is recommended to avoid rewriting tools that already exist, for example, a Tor exit scanner to spot machine-in-the-middle (MITM) attacks. Instead please contribute to the code that's actively developed. In case you're working on innovative open-source methods to detect malicious actors and publishing them could tip attackers, please contact bad-relays@lists.torproject.org so we can think about ways to use those methods while not giving attackers an even larger advantage in the bad relay arms race.[2] 1.3. Running scanners is important to keep the network healthy, but be mindful that this is putting strain on the network and the data you are looking for might already be available by other means. 1.4. As a public network, the Tor network already provides relay information such as relays IP addresses, ISPs, operating systems, fingerprints, Tor versions, and more. Investigators can query or consult this data as long as it does not disrupt relay operations or compromise the privacy of Tor users. Such publicly accessible network information is available via Tor Metrics and Onionoo. To learn more, refer to the Onionoo[3] and Tor Metrics sources[4] documentation. Moreover, there are tools available as well, like OrNetStats[5], that build upon those sources providing further insight into the network and its relays. 1.5. Moreover, as a transparent and open source project we provide public analyses and investigations of a lot of previous and current incidents in our [bug tracker](https://gitlab.torproject.org/tpo). Please consult that issue tracker as well to avoid duplicated work or false positives. For instance, sometimes there are some DNS resolution issues at exit nodes preventing users from reaching a website. We are aware of that as we are continuously scanning for that problem and dealing with affected exit nodes on a regular basis. 1.6. It must be acknowledged that legit investigations into malicious activities and bad relays may extend beyond using open-source data. Other methodologies and strategies for gathering information, like whistleblowing or using a relay's MetricsPort[6], are out of scope of this framework. 1.7. Finally, it is important to highlight that running modified Tor software or deploying offensive hacking techniques against relays or the Tor network which could harm users, relay operators and the Tor network itself won't be considered part of an open-source investigation under this framework. Rather, those actions will be considered a malicious activity. ### 2. Preliminary assessment 2.1. As searching and monitoring efforts lead to more and more results it is important to make a preliminary assessment of them early on to avoid over-collection. Over-collection is not just problematic because it will make the analysis process slower and more difficult: there are more results to analyze, more storage used and needed, etc. The risks of leaking information or infringing on rights to privacy of individuals are rising as well. 2.2. Similarly to the search and monitoring efforts, assessing the results through the lens of hypotheses is important as they help to focus on the evidence and avoid speculation. 2.3. During the assessment of results make sure they are relevant, reliable and safe to collect. If there is no risk of results being removed from the internet then their collection is not needed either. ### 3. Collection 3.1. Data collected close to the event under investigation may produce better results. For example, tracking IP addresses involved in a DDoS attacks might not be useful after a long period of time as the attackers' infrastructure might have been reassigned to a new customer. 3.2. As the investigation progresses and new evidence shows up, older, previously collected data might not be needed anymore. Please consider deleting it for the same reasons as stated in section 2.1. above. 3.3. Apart from collecting relevant content itself, it is important to save associated metadata as well in order for other investigators to more easily reproduce the investigation's findings and double-check used source material: URLs of the content, the time when it was accessed, source of a website in question, and anything else that helps to explain the context of the capture/collection should be stored, too. 3.4. In cases where malicious content, such as injected JavaScript, is identified during the scanning of exit nodes, it is crucial to capture and archive the website’s source code for both analysis and as evidence of a bad exit relay. ### 4. Preservation 4.1. Preserving the collected content is essential in network health investigations because the information can change, disappear, or be altered over time, making it unreliable and unusable for the investigation. To ensure the integrity of the data, network health investigators must follow preservation principles, based on archivist standards, as outlined in the Berkeley Protocol (section 6D). These principles include authenticity, availability, identity (keeping track of the origin and nature of the evidence), persistence, renderability, and understandability. 4.2. The decentralized and dynamic nature of the Tor network makes collecting and preserving public relay information a significant challenge. The network changes as relays join or leave, making historical data collection and preservation essential for investigations. For example, relay information is only available on the Tor Metrics website for one week before it is archived and removed from the relay search portal. To ensure access to historical relay information, investigators need to download relay descriptors from [CollecTor](https://metrics.torproject.org/collector.html). 4.3. Tools like archive.today and the Internet Archive (web.archive.org) are useful for capturing and publicly preserving snapshots of digital content, such as relay information, websites, social media content, and other online information that needs to be collected during an investigation. For a private archive collection, see the [ArchiveBox project](https://github.com/ArchiveBox/ArchiveBox). 4.4. When using open source tools to monitor the network, it is important to preserve the results. This includes maintaining logs with metadata such as timestamps, relay fingerprints, targeted URLs and other information like errors. Trusted services like archive.today or the Internet Archive should be used to preserve the original version of the website. 4.5. Investigators must maintain both evidentiary and working copies of collected digital content. An evidentiary copy is a preserved, unaltered version of the original evidence, used to maintain the authenticity of the original content. Working copies can be created for analysis, allowing changes in format or structure without compromising the integrity of the original data. 4.6. Investigators should regularly back up both evidentiary and working copies to secure storage systems, ensuring that data is not lost or corrupted during the investigation process. Please note that while relay descriptors are important for verification and analysis, they can be re-downloaded from the Tor Metrics website, so backing up this data is not needed. ### 5. Verification 5.1. Verification is a process to ensure the accuracy and validity of information collected during network health investigations. Verification can be done using closed and confidential sources or based exclusively on open sources. Without using a verification method, the information collected may be invalid or unreliable for detecting and removing bad relays. (Verification can be split into three key methods: source analysis, technical and investigative analysis.) 5.2. Investigators must evaluate the credibility and impartiality of the source from which the evidence originates. This includes verifying whether the output from open source tools, social media or websites are reliable. 5.3. Information should be cross-verified by consulting other sources to avoid relying on non validated, manipulated, biased or outdated data. Please be aware that some relay information displayed on the Tor Metrics website are not validated, for example, operator's contact information. 5.4. When verifying collected information, network health investigators should begin by looking for unique or distinguishing features. Such features might include groups of relays joining or leaving the network from a particular Autonomous System (AS), new relay operating in an obscure or popular AS, similarities and other patterns. These identification methods rely on searching and monitoring the Tor network activity. 5.5. While geolocation can be a useful tool in network health investigations and analysis, it is important to remember that the geolocation of Tor relays is often inaccurate. This data must be cross-verified with other IP geolocation databases to ensure reliability. 5.6. Investigators should evaluate the information collected for internal consistency during the verification process. This helps determine whether the information is consistent and coherent, and aligns with itself, ensuring its credibility on its own terms. ### 6. Analysis 6.1. Be mindful that a network health analysis might find false-positives. There are a number of possible reasons for that. Among other things: - the data from Tor Metrics isn't always reliable, and what could initially appear as an action of a malicious actor could actually be the result of a glitch in the Metrics database or other network events, like a denial-of-service attack or a major Tor version upgrade. Please double check if Tor Metrics services (Metrics website, relay-search, Onionoo, etc) status are/were healthy and accurate at the time of the incident. - properly analyzing scanning results can be tricky. During an anomaly investigation, you may discover new and intriguing censorship facts, where exit nodes seem to mess with user traffic resulting in blocked websites. Yet, the underlying problem is that they are just using [the DNS resolvers of their ISP](https://gitlab.torproject.org/tpo/network-health/team/-/issues/92) or the blocking of websites is done even further upstream. - Tor has location-biases that might impact the analysis. For instance, when investigating bandwidth related anomalies in the network or outright attacks it might just look like some relays are involved based on bandwidth scanner data. However, that might in fact be related to the still existing location-dependency of our bandwidth scanning efforts instead. 6.2. When starting the analysis keep in mind that it might be useful to translate, aggregate or reformat collected material so that it's easier to process with tools planned to be deployed. For instance, it is not uncommon to change the format of collected information to make it more easily searchable. 6.3. If any of your collected data is at risk of getting altered, lost or destroyed during an analysis it is advisable to use a working copy instead. That way it is possible to re-do the analysis with the same data later on, for instance with a different set of tools that might produce better results. See items 4.5. and 4.6. 6.4. There are many open source tools and services available to assist with your data analysis. As the Network Health Team needs to verify findings without the need to buy or purchase any product, it is important to prioritize platforms and/or products that do not require the purchase of a license to access the results. 6.5. Shifting the focus from 'who' to 'how' an anomaly or attack is happening may yield more valuable insights. In general, attackers have a _modus operandi_. Making a profile of a potential attacker, including their resources and techniques, can significantly help and support your analysis. 6.6. During the analysis investigators may be able to identify attacks such as Sybil, MITM, (distributed) denial-of-service, impersonation of relay operators, DNS poisoning, and social engineering (for example, Tor relay MyFamily evasion) attacks. A list of harmful behaviors can be found in [Criteria for rejecting bad relays](https://gitlab.torproject.org/tpo/network-health/team/-/wikis/Criteria-for-r...) and also on [Network Health blog posts](https://blog.torproject.org/malicious-relays-health-tor-network/). ### 7. Reporting 7.1. As said in 6.6, the canonical definition and criteria for bad relays in the Tor network are established and followed by the [Criteria for rejecting bad relays](https://gitlab.torproject.org/tpo/network-health/team/-/wikis/Criteria-for-r...) document. However, if you've observed a behavior that you may consider malicious and can cause harm to Tor users or to the network, but is not covered by the criteria, please file a ticket on [GitLab](https://gitlab.torproject.org/tpo/network-health/team). Make sure the ticket is confidential[7] if there is a risk of tipping off an attacker (see 7.7.). 7.2. If investigators detect relays with one of the described issues or behaviors from the Criteria for rejecting bad relays, please report them with a detailed description to our bad relays mailing list (bad-relays@lists.torproject.org) including: - The relay's IP address or fingerprint. The fingerprint is a forty-character hex string such as 203933ED4E55EF8A3C3518427D1A1ED6A4CC285E. - What kind of behavior did you see and when? - Any additional information we'll need to reproduce the issue. 7.3. It is better to report attacks earlier than later, as attackers might turn them on and off to avoid the risk of detection. Waiting too long before getting back to us might make it therefore harder or outright impossible to replicate your findings. 7.4. Without replicable findings, the Network Health Team is unable to reject relays. However, we'd like to hear your results nevertheless, as we might have additional information or did our own investigation already that could help reproducing them. 7.5. When reporting to us, investigators don't need to identify themselves by their legal names. Pseudonyms and anonymous tips are welcome. 7.6. Writing a blog post or a research paper about your findings is very much welcomed. However, please coordinate the disclosure part with us first, so that the risk and potential harm to our users is minimized. 7.7. Don't tip off the attacker. While much of the data on suspicious relay activity is public, putting together all the pieces of an investigation should be done discretely. Revealing publicly an investigation result could inadvertently inform the attacker, helping them to evade future detection. 7.8. Additionally in the case of bridges, investigators must be careful on not publicly revealing the bridges' IP addresses. Revealing publicly the IP addresses of bridges without consulting the Network Health Team might be considered as malicious intent. ## References [1] Berkeley Protocol on Digital Open Source Investigations (2022) - https://www.ohchr.org/en/publications/policy-and-methodological-publications... [2] [tor-talk] Why make bad-relays a closed mailing list? (2014) - https://lists.torproject.org/pipermail/tor-talk/2014-July/034219.html [3] Onionoo documentation - https://metrics.torproject.org/onionoo.html [4] Metrics sources documentation - https://metrics.torproject.org/sources.html [5] OrNetStats - https://nusenu.github.io/OrNetStats/ [6] MetricsPort - https://support.torproject.org/glossary/metrics-port/ [7] Confidential issues in GitLab - https://docs.gitlab.com/ee/user/project/issues/confidential_issues.html -- The Tor Project Community Team Lead