- tor-dev - lists.torproject.org

Proposal draft: Better hidden service stats from Tor relays
by George Kadianakis 14 Dec '14

14 Dec '14

Hello there, I inline a copy of a proposal we've been working on lately. Discussion can be found in the "Feedback on obfuscating hidden-service statistics" thread. The proposal suggests that Tor relays add some stats about hidden service usage. We believe that these stats are not dangerous and can be useful to Tor developers and to people who want to understand hidden services and the onionspace better. Any feedback is appreciated :) ====== Filename: 238-hs-relay-stats.txt Title: Better hidden service stats from Tor relays Author: George Kadianakis, David Goulet, Karsten Loesing, Aaron Johnson Created: 2014-11-17 Status: Draft 0. Motivation Hidden Services is one of the least understood parts of the Tor network. We don't really know how many hidden services there are and how much they are used. This proposal suggests that Tor relays include some hidden service related stats to their extra info descriptors. No stats are collected from Tor hidden services or clients. While uncertainty might be a good thing in a hidden network, learning more information about the usage of hidden services can be helpful. For example, learning how many cells are sent for hidden service purposes tells us whether hidden service traffic is 2% of the Tor network traffic or 90% of the Tor network traffic. This info can also help us during load balancing, for example if we change the path building of hidden services to mitigate guard discovery attacks [0]. Also, learning the number of hidden services, can give us an understanding of how widespread hidden services are. It will also help us understand approximately how much load is put in the network by hidden service logistics, like introduction point circuits etc. 1. Design Tor relays will add some fields related to hidden service statistics in their extra-info descriptors. Tor relays collect these statistics by keeping track of their hidden service directory or rendezvous point activities, slightly obfuscating the numbers and posting them to the directory authorities. Extra-info descriptors are posted to directory authorities every 24 hours. 2. Implementation 2.1. Hidden service statistics interval We want relays to report hidden-service statistics over a long-enough time period to not put users at risk. Similar to other statistics, we suggest a 24-hour statistics interval. All related statistics are collected at the end of that interval and included in the next extra-info descriptors published by the relay. Tor relays will add the following line to their extra-info descriptor: "hidserv-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL [At most once.] YYYY-MM-DD HH:MM:SS defines the end of the included measurement interval of length NSEC seconds (86400 seconds by default). A "hidserv-stats-end" line, as well as any other "hidserv-*" line, is first added after the relay has been running for at least 24 hours. 2.2. Hidden service traffic statistics We want to learn how much of the total Tor network traffic is caused by hidden service usage. There are three phases in the rendezvous protocol where traffic is generated: (1) when hidden services make themselves available in the network, (2) when clients open connections to hidden services, and (3) when clients exchange application data with hidden services. We expect (3) to consume most bytes here, so we're focusing on this only. More precisely, we measure hidden service traffic by counting RELAY cells seen on a rendezvous point after receiving a RENDEZVOUS1 cell. These RELAY cells include commands to open or close application streams, and they include application data. Tor relays will add the following line to their extra-info descriptor: "hidserv-rend-relayed-cells" SP num NL [At most once.] Approximate number of RELAY cells seen in either direction on a circuit after receiving and successfully processing a RENDEZVOUS1 cell. The actual number observed by the directory is multiplied with a random number in [0.9, 1.1] and then gets floored before being reported. The keyword indicates that this line is part of hidden-service statistics ("hidserv") and contains aggregate data from the relay acting as rendezvous point ("rend"). 2.3. HSDir hidden service counting We also want to learn how many hidden services exist in the network. The best place to learn this is at hidden service directories where hidden services publish their descriptors. Tor relays will add the following line to their extra-info descriptor: "hidserv-dir-published-ids" SP num NL [At most once.] Approximate number of unique hidden-service identities seen in descriptors published to and accepted by this hidden-service directory. The actual number observed by the directory is multiplied with a random number in [0.9, 1.1] and then gets floored before being reported. This statistic requires keeping a separate data structure with unique identities seen during the current statistics interval. We could, in theory, have relays iterate over their descriptor caches when producing the daily hidden-service statistics blurb. But it's unclear how caching would affect results from such an approach, because descriptors published at the start of the current statistics interval could already have been removed, and descriptors published in the last statistics interval could still be present. Keeping a separate data structure, possibly even a probabilistic one, seems like the more accurate approach. 3. Security The main security considerations that need discussion are what an adversary could do with reported statistics that they couldn't do without them. In the following, we're going through things the adversary could learn, how plausible that is, and how much we care. (All these things refer to hidden-service traffic, not to hidden-service counting. We should think about the latter, too.) 3.1. Identify rendezvous point of high-volume and long-lived connection The adversary could identify the rendezvous point of a very large and very long-lived HS connection by observing a relay with unexpectedly large relay cell count. 3.2. Identify number of users of a hidden service The adversary may be able to identify the number of users of an HS if he knows the amount of traffic on a connection to that HS (which he potentially can determine himself) and knows when that service goes up or down. He can look at the change in the total reported RP traffic to determine about how many fewer HS users there are when that HS is down. 4. Discussion 4.1. Why count only RP cells? Why not also count IP cells? As discussed on IRC, counting only RP cells should be fine for now. Everything else is protocol overhead, which includes HSDir traffic, introduction point traffic, or rendezvous point traffic before the first RELAY cell, etc. Furthermore, introduction points correspond to specific HSes, so publishing IP cell stats could reveal the popularity of specific HSes. 4.2. How to use these stats? 4.2.1. How to use RP Cell statistics We plan to extrapolate reported values to network totals by dividing values by the probability of clients picking relays as rendezvous point. This approach should become more precise on faster relays and the more relays report these statistics. We also plan to compare reported values with "cell-*" statistics to learn what fraction of traffic can be attributed to hidden services. Ideally, we'd be able to compare values to "write-history" and "read-history" lines to compute similar fractions of traffic used for hidden services. The goal would be to avoid enabling "cell-*" statistics by default. In order for this to work we'll have to multiply reported cell numbers with the default cell size of 512 bytes (we cannot infer the actual number of bytes, because cells are end-to-end encrypted between client and service). 4.2.2. How to use HSDir HS statistics We plan to extrapolate this value to network totals by calculating what fraction of hidden-service identities this relay was supposed to see. This extrapolation will be very rough, because each hidden-service directory is only responsible for a tiny share of hidden-service descriptors, and there is no way to increase that share significantly. Here are some numbers: there are about 3000 directories, and each descriptor is stored on three directories. So, each directory is responsible for roughly 1/1000 of descriptor identifiers. There are two replicas for each descriptor (that is, each descriptor is stored under two descriptor identifiers), and descriptor identifiers change once per day (which means that, during a 24-hour period, there are two opportunities for each directory to see a descriptor). Hence, each descriptor is stored to four places in identifier space throughout a 24-hour period. The probability of any given directory to see a given hidden-service identity is 1-(1-1/1000)^4 = 0.00399 = 1/250. This approximation constitutes an upper threshold, because it assumes that services are running all day. An extrapolation based on this formula will lead to undercounting the total number of hidden services. A possible inaccuracy in the estimation algorithm comes from the fact that a relay may not be acting as hidden-service directory during the full statistics interval. We'll have to look at consensuses to determine when the relay first received the "HSDir" flag, and only consider the part of the statistics interval following the valid-after time of that consensus. 4.3. Multiplicative or additive noise? A possible alternative to multiplying the number of cells with a random factor is to introduce additive noise. Let's suppose that we would like to obscure any individual connection that contains C cells or fewer (obscuring extremely and unusually large connections seems hopeless but unnecessary). That is, we don't want the (distribution of) the cell count from any relay to change by much whether or not C cells are removed. The standard differential privacy approach would be to *add* noise from the Laplace distribution Lap(\epsilon/C), where \epsilon controls how much the statistics *distribution* can multiplicatively differ. This is not to say that we need to add noise exactly from that distribution (maybe we weaken the guarantee slightly to get better accuracy), but the same idea applies. This would apply the same to both large and small relays. We *want* to learn roughly how much hidden-service traffic each relay has - we just want to obscure the exact number within some tolerance. We'll probably want to include the algorithm and parameters used for adding noise in the "hidserv-rend-relayed-cells" line, as in, "lap=x" with x being \epsilon/C. [0]: guard discovery: https://lists.torproject.org/pipermail/tor-dev/2014-September/007474.html

3 12

Call for Papers: Switzerland - International Conference on Semantic Web Business and Innovation (SWBI2015)
by ANNOUNCEMENT 12 Dec '14

12 Dec '14

The International Conference on Semantic Web Business and Innovation (SWBI2015) The University of Applied Sciences and Arts Western Switzerland (HES-SO Valais-Wallis) October 7-9, 2015 http://sdiwc.net/conferences/swbi2015/ All registered papers will be included in SDIWC Digital Library ================================================================ The proposed conference on the above theme will be held at he University of Applied Sciences and Arts Western Switzerland (HES-SO Valais-Wallis) on October 7-9, 2015 which aims to enable researchers build connections between different digital applications. The conference welcomes papers on the following (but not limited to) research topics: *Semantic Web and Linked Data - Database, IR, NLP and AI technologies for the Semantic Web - Geospatial Semantic Web - Information Extraction from unstructured data - Information visualization of Semantic Web data and Linked Data - Internet of things - Languages, tools, and methodologies for representing and managing Semantic Web data - Linked open data - Management of Semantic Web data and Linked Data - Ontology engineering and ontology patterns for the Semantic Web - Ontology modularity, mapping, merging, and alignment - Ontology-based data access - Privacy and Security - Query and inference over data streams - Search, query, integration, and analysis on the Semantic Web - Semantic business process management - Semantic Sensor networks - Semantic technologies for mobile platforms - Semantic Web and Linked Data for Cloud Environments - Semantic Web, Ontologies - Social networks and processes on the Semantic Web - Supporting multi-linguality in the Semantic Web - User Interfaces and interacting with Semantic Web data and Linked Data *Business and Innovation in Semantic Web - Business Model Innovation - Business Models and E-Commerce - Business Technology Intelligence - Challenges for change in semantic services - Collaborative improvement and innovation - E-Business Applications and Software - E-commerce Technology Adoption - E-commerce, E-Business Strategies - E-tailing and Multi-Channel selling - Evolution of Business model for Semantic Web Applications - High-tech marketing - Implementation strategies for responsible innovation - Innovation for E-Business - Innovative methods and tools for products and services - Practices and Cases in E-Commerce - Production of Knowledge Economy - Technology and Business Transformation - Technology strategies - The Latest Trends in Linked Data - Web Advertising and Web Publishing - Web and Mobile Applications Researchers are encouraged to submit their work electronically. All papers will be fully refereed by a minimum of two specialized referees. Before final acceptance, all referees comments must be considered. Important Dates ============== Submission Date: Open from now until Sept. 1, 2015 Notification of Acceptance: 6 weeks from the submission date Camera Ready Submission: September 14, 2015 Registration Deadline: September 14, 2015 Conference Dates: October 7-9, 2015

1 0

use tor in my app
by Terry Spearman 11 Dec '14

11 Dec '14

I've written an Windows 7 Visual Basic application that downloads info from a website, that I'd like to use tor with. I've downloaded tor browser, and if I open the browser it establishes a connection to the tor network and my application runs just fine. However, if I instead run tor via tor.exe, my app gives the exception " No connection could be made because the target machine actively refused it 127.0.0.1:9150" which is the same message I get if I run the app without first starting tor. I confirmed that tor.exe is running as a process in Task Manager. I've tried multiple versions of torrc. My current one is: # This file was generated by Tor; if you edit it, comments will not be preserved # The old torrc file was renamed to torrc.orig.1 or similar, and Tor will ignore it AvoidDiskWrites 1 Log notice stdout #Log debug file c:/program files/tor/debug.log DataDirectory C:\Users\Terry\Desktop\Tor Browser\Browser\TorBrowser\Data\Tor DirReqStatistics 0 GeoIPFile C:\Users\Terry\Desktop\Tor Browser\Browser\TorBrowser\Data\Tor\geoip GeoIPv6File C:\Users\Terry\Desktop\Tor Browser\Browser\TorBrowser\Data\Tor\geoip6 SocksPort 9150 ControlPort 9151 CookieAuthentication 1 ## fteproxy configuration ClientTransportPlugin fte exec TorBrowser\Tor\PluggableTransports\fteproxy --managed ClientTransportPlugin flashproxy exec TorBrowser\Tor\PluggableTransports\flashproxy-client --register :0 :9000 ## meek configuration ClientTransportPlugin meek exec TorBrowser\Tor\PluggableTransports\terminateprocess-buffer TorBrowser\Tor\PluggableTransports\meek-client-torbrowser --exit-on-stdin-eof -- TorBrowser\Tor\PluggableTransports\meek-client I can't figure out what configuration settings the tor browser is using that are different from what's in the torrc. I've searched the documentation, and requested help from help(a)rt.torproject.org, who tell me that my problem is outside the scope of the help desk, and suggested I try here. Any help/suggestions greatly appreciated.

1 0

Distributing TBB and Tails via Torrents
by Chuck Peters 11 Dec '14

11 Dec '14

I have been trying to download the new Tails the past week via torrent and the tracker seems to be broken. I know it isn't a good idea to use torrents over tor, but why doesn't torproject.org distribute the TBB, Tails and other tor related software with torrents? And if torproject.org doesn't want to host a tracker, what would you recommend for the trackers system requirements? If I were to do it, a few requirements I have are: DNSSEC enabled domain. Not an open tracker, only serve Tails, TBB and other tor related packages. The torrent files are available through https with a valid certificate. Thanks, Chuck

5 5

contribute to org
by Ayush Jhunjhunwala 11 Dec '14

11 Dec '14

Hello, I am ayush jhunjhunwala. I am interested in the contributing to the project - "Rewrite Tor Weather". I have read through the ideas page given in the website. As I am new to this it will be grateful if someone can guide me. Thanks in advance. Yours Sincerely, Ayush Jhunjhunwala IRC nick:ayushjjwala email:ayushjjwala94@gmail.com

2 1

Understanding bwauth data in Stem?
by Anna Kornfeld Simpson 09 Dec '14

09 Dec '14

I'm trying to understand how the bandwidth authorities work, and reading the spec [0] got me only partway to an understanding, so I'm trying to use Stem to see what the bwauth measurements look like in practice. I'm working off the "Votes by Bandwidth Authorities" example on the Stem webpage [1]. The query there returns a RouterStatusEntryV3, so I've been looking at the "bandwidth" and "measured" fields to try to understand what's going on [2]. "Bandwidth" maxes out at 10,000 (kb/s??) but "measured" doesn't seem to have the same ceiling, which made me realize that they aren't in the same units. Damian mentioned on IRC that "measured" might be returning the bwauth weight rather than a bandwidth, but what is the meaning of that weight? Does a higher "measured" value mean a higher bandwidth, or a higher bandwidth relative to what the relay advertises? In other words, if I sorted the descriptors by "measured" value, what would that order mean? Separately, is there a way (using Stem or some other tool) to see the raw bwauth measurements rather than the weights? Is that a calculation I can reverse? I haven't looked into the historical data on CollecTor yet, but ideally, I would like to use the historical data to figure out how effective the bwauth measurements seem to be in different situations (for example, the misconfigured to very high bandwidth relay this past February seems to have produced confusing bwauth measurements [3]). If I'm looking for interesting events in the historical bwauth data, would I be looking for high "measured" values, rapid changes, or ...? Thanks! Anna PhD Student, University of Washington [0] https://gitweb.torproject.org/torflow.git/blob/HEAD:/NetworkScanners/BwAuth… [1] https://stem.torproject.org/tutorials/examples/votes_by_bandwidth_authoriti… [2] https://stem.torproject.org/api/descriptor/router_status_entry.html [3] https://lists.torproject.org/pipermail/tor-talk/2014-February/032094.html

4 6

Time change for Tor patch workshop meetings: 1800 UTC on Tuesdays
by Nick Mathewson 09 Dec '14

09 Dec '14

Hi! By request, I'm changing the time for the patch workshops from 1700 UTC to one hour later at 1800 UTC on Tuesdays. Bring your patches for tor, and let's chat about them! (Because of the late notice, I will try to be around for the whole time window, but I can't promise too much. I've got meetings all afternoon.) -- Nick

1 0

Git hosting changes, git:// support discontinued
by Sebastian Hahn 08 Dec '14

08 Dec '14

Hi there, git users! Today, weasel changed our gitweb setup to use cgit from now on in the wake of a migration to wheezy. This move hopefully improves speed without taking away any needed functionality. At the same time, we've discontinued supporting clones via the git:// Protocol. It is unauthenticated and you probably shouldn't use it if at all possible. Access via https:// has been provided for years, and should continue to work without any hiccups. If there are questions or concerns, let's here them. Cheers Sebastian

4 9

Re: [tor-dev] [tor-reports] Help desk report for November 2014
by David Fifield 08 Dec '14

08 Dec '14

On Mon, Dec 01, 2014 at 09:41:58AM -0600, Colin C. wrote: > Top answer templates used by the support team for this month: > > usage | article > -------+----------------------------------------------------------------- > 67 | Get your own bridges. > 36 | Get TB's Tor Log. > 33 | Exclude Tor Browser from an Antivirus/Firewall/AntiMalware > 30 | Tor Browser doesn't work (Windows) > 13 | Redirect to Guardian Project's support > 11 | How to get bridges and use Tor in mainland China > 11 | Please try meek > 10 | [debug] Tor Browser 4.0 is crashing on Windows > 9 | Can't connect to my favorite onion site! > 8 | Unsupported language > 7 | Redirect to Tails support > 7 | Suggest the tor-talk mailing list > 6 | Tor-related error window from browser. > 5 | How to use bridges > 5 | Set Country > 4 | TBB no Flash/Java other plugins (Replied to from https://lists.torproject.org/pipermail/tor-reports/2014-December/000711.html.) I'm interested to know if we can use help desk requests to detect new censorship events. When you answer with "Get your own bridges" or "Please try meek" or "How to use bridges," do you have an idea of what country or what language the request comes from? Do you keep track of any such numbers, even just a gut feeling? What about cases like the recent https://trac.torproject.org/projects/tor/ticket/12727 blocking in Iran last August, did you have an increase in Farsi requests or requests for bridges then? I don't mean to impose a lot of bookkeeping on you. I just want to see if this is a worthwhile idea. David Fifield

1 0

botnet detection
by Enkidu Mo Shiri 07 Dec '14

07 Dec '14

is there any new method for botnet detection through tor?anyway can find the hidden service which been used by botmaster to c&c botnets? as communication all are encrypted and the decryption happens only node by node, how can i detect which HS been used bt botmaster? help me plz

1 0