July 2012 - tor-dev - lists.torproject.org

[GSoC] Stem - Status update
by Ravi Chandra Padmala 25 Jul '12

25 Jul '12

Hello The network status descriptor parsing has taken a lot longer than expected. I should be putting it up for review soon. Followed by the microdescriptor parsers. Given the current status of the project, I have prepared a revised schedule for the remaining GSoC period. The final evaluation is about a month away. I have spoken to Damian and we decided to reschedule the Arm port deliverable to after the GSoC timeline (most likely the week immediately following the end of the GSoC coding period). July 25 - July 28 Descriptor parsing July 29 - August 4 | Event handling methods for the controller class. August 5 - August 11 | I will be working on them as alloted to week 4, 5 and August 11 - August 20 | 6 from my original proposal[1]. 1. https://www.torproject.org/about/gsocProposal/gsoc12-proposal-stemImproveme… -- neena

1 0

ApachePayloadServer pushed
by vmonmoonshine＠gmail.com 24 Jul '12

24 Jul '12

Disc.:In the spirit of our discussion in Italy about unifying mailing list and that we should submit our technical discussions on tor-dev instead of tor-assistants, I'm moving my conversations with Zack here. If there's a strong opposition I'll go back to the old model (but you should have opposed it in Italy). Hey Zack, I pushed the ApachePayloadServer on your git on f--payload-servers branch. It compiles but I have to test it yet. However, it would be much easier to be tested when I change the current payload-based-on-trace to be inherited from the same parent. so I'm postponing testing for few days after I write that part (so you don't need to look at it till then). On another note, your pgen_pcap.cc and pgen_fake.cc don't compile on g++ 4.7.0 on linux, because it doesn't like the fact that you don't care about the return value of fwrite. That's it for now. Cheers, vmon I

2 1

GSoC Status Update - Testing and Transports
by Damian Johnson 24 Jul '12

24 Jul '12

Sending this on behalf of Brandon since tor-dev@ is bouncing his emails. ======================================== The last two weeks were spent getting the code in better shape, fixing bugs and streamlining so that I could get to the point that I could actually write some transports. Now it’s transport implementation time. I’ve written a Dust transport, using the most barebones version of Dust, which is basically just an ECDH handshake and Skein encryption. I’ll be adding the rest of Dust over time. This barebones Dust implementation was really just to lay the foundation to start working on obfs3 and then obfs2. It was easier to start with Dust because I already have the code and it works. Next I’m bringing in an AES implementation for obfs3 and obfs2 to use. The schedule has changed somewhat in that I an now going to be focusing on obfs2 and obfs3 as the main protocols. This is going to be easier because they both follow the pattern of a handshake followed by encrypted bytes. Both full Dust support and flash proxy were going to require a more complicated API that should probably wait until the basic API is working.

1 0

hide my site in clearweb
by Salva . 22 Jul '12

22 Jul '12

Hello, I have launched a website in TOR and I can access it through the .onion address and also typing the ip of my server in my brower.I want my site was only accessible from TOR and not through clearweb, how can I solve it ? Thank you guys.

2 2

Automating Bridge Reachability Testing
by Isis 21 Jul '12

21 Jul '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello Everyone. I've already gotten a bit of good feedback on the trac ticket[0], but I'm going to post my research/writeup/proposal/whateveryouwanttocallit here so more people can discuss it. <(A)3 isis agora lovecruft [0] https://trac.torproject.org/projects/tor/ticket/6414 ________________________________________________________________________________ Automating Bridge Reachability Testing - -------------------------------------- An effort was made earlier this year to create a discovery system for current bridge reachability status #5028. This resulted in the development and deployment of [https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/plugins/bridge… OONI's bridgeT], which uses txtorcon to attempt a connection, speaking the full Tor protocol, to the set of bridges being tested. Some bridges were scanned, and results were gathered. We would like to go back and automate this process, and possibly revise it if a better methodology is proposed. Anyone with ideas or interest should feel free to join the discussion here. While this automation is intended to be geolocationally agnostic, it is trivial to test a bridge's reachability from a country which does not block Tor, and therefore automation methodology should be developed according to the worst-case scenarios. Countries which block Tor, or have blocked Tor, include China, Iran, Lebanon, Qatar, United Arab Emirates, and Ethiopia. In order to ensure that the fewest amount of Tor bridges are blocked during reachability testing, it seems wise to assume that the test is being conducted from one of these countries. Also, any test methodology which produces accurate results from inside China or Iran would likely work just as well from any non-Tor-blocking country. I. Brief Overview of Dynamic Tor Bridge Blocking - From my understanding so far (please correct me if I have misunderstood something, or if there is more information), China's mechanism for blocking Tor bridges takes the following steps (unconfirmed data is prefaced by a question mark): 1. OP --> OR/Bridge Connection a. Alice (OP/client in China) connects to Bob (OR/bridge), completes the TLS handshake, and sets up circuits. b. This works for roughly fifteen minutes. 2. Protocol Identification & Fingerprinting a. The GFC identifies Tor via fingerprinting the cipher list in the TLS Server Helo. b. Tests for the precise trigger in the fingerprint were conducted (I'll leave said tester(s) anonymous unless they would like to speak up) by fuzzing the TLS handshake ServerHello, and the precise fingerprint for triggering the GFC's nascent probes was determined to be a specific 5 bytes. (?) It was also found that the GFC blocks packets <= 79 bits. c. Philip Winter's research showed that fragmentation of the ciphersuite list would not trigger a probe [5]. 3. Network Enumeration a. The GFC adds Bob's IP and port to a queue of addresses to be checked. These queues are processed every fifteen minutes (hence why Alice's connection functions normally at first). b. A probe is sent to Bob during queue processing. The GFC probes are not yet fully understood, and unverified data in this section is prefaced by a '?'. Thus far, the following is believed to occur: * (?) Reportedly (speak up if you wish), there are eight "edge routers" in China. The reporter stated that there was "one for each province", however there are twenty-two Provinces in PRC -- twenty-three if you count Taiwan. There is one "core router" which controls/routes to the eight "edge routers". Because all traffic into and out of China passes through these eight routers, all netblocks within China are essentially a private network behind the "edge routers". (See question #2 below.) * (?) Because these "edge routers" are intercepting all traffic, they are able to temporarily hijack any IP from the contained netblocks. * A hijacked IP and a random port (the range appears to be ~35000-60000) are used as the source to send a probe to the queued IP:port of the suspected bridge. (See question #3 below.) * The probe does a TCP connect. * Then it sends a TLS ClientHello and waits for the cipher list in the ServerHello message. * If the cipher list matches that used by Tor, the IP:port gets blacklisted. Previous research has shown that this blacklisting is not permanent, but lasts for 12 hours after the last successful connection by a probe [1]. (See question #4) II. Testing Bridge Reachability As Roger has stated on the Tor Blog, we can either do active or passive scans to check if a bridge has been blocked [4]. Passive scans, wherein either the bridge or the client report connections, are unreliable without results from active scans in the former case [5], and could potentially reduce privacy and anonymity in the later case. II. A. Active Scans ==== Direct Methods ==== - From most innocuous (least Tor-like) to most conspicuous (most Tor-like): ICMP type-8 ping / echo Tells us if the host running the Tor bridge is online, but not necessarily if the ORPort is open. TCP ping / ACK If TCP ACKs are timed to be sent infrequently (probably no more than one every five minutes or so), they can appear to be random network noise rather than a scan. If we get a RST back, we know that we can at least communicate with the bridge's ORPort though the GFC. This might look odd, if it gets noticed, especially since the GFC is stateful and might realize the ACKs are unsolicited. TCP SYN This still doesn't tell us if Tor is running, but, again, a SYN/ACK would let us know if the ORPort is reachable and accepting connections, a RST that it is reachable and not accepting connections (or the GFC is sending false TCP RSTs), and no response would mean that the GFC, or some other hop is dropping packets. Philip Winter's research showed that the client's SYN is transmitted through the GFC, which instead drops the SYN/ACK response of known Tor relays/bridges [2]. TCP connect() We could try a normal full TCP connect (SYN & ACK). This would be the most genuine-to-the-Tor-protocol test available for regions where SSL is being blocked. It could be useful here to test different types of fragmentation, for example, the old trick involving overlapping fragments to rewrite the TCP headers in the first fragment [25]. SSL Handshake We could try doing a normal SSL handshake, as if contacting, for example, an Apache webserver over HTTPS. Another interesting idea would be to run an SSLObservatory from inside China, and simply pretend that the bridges are HTTPS webservers, which would look just like the normal SSLObservatory for bridges whose ORPort is set to :443 [14, 15]. As of this morning, a quick check on Tor relays shows that 27% of relays are run on :443 : {{{ isis@acab:/var/lib/tor$ cat cached-microdesc-consensus | grep -e \ >"^r\ [a-zA-Z0-9]*\ /*" | grep " 443 " -c 779 isis@acab:/var/lib/tor$ cat cached-microdesc-consensus | grep -e \ >"^r\ [a-zA-Z0-9]*\ /*" -c 2912 isis@acab:/var/lib/tor$ python -c 'from __future__ import \ >division;a=799/2912;print a' 0.274381868132 }}} with the most common ports being: {{{ isis@acab:/var/lib/tor$ cat cached-microdesc-consensus | grep -e \ >"^r\ [a-zA-Z0-9]*\ /*" | cut -d " " -f 7 | sort | uniq -ic | sort -gr 1592 9001 762 443 217 80 34 9090 33 8080 21 9002 20 444 11 9031 11 110 9 22 7 21 [...] }}} I would assume that the percentage of bridges running on :443 is higher than that of relays (question #5). We could safely automate the testing of those relays without actually speaking Tor to them, by appearing to be an SSLObservatory (question #6). This would provide us with an extensive set of canaries to help mitigate the zig-zag enumeration attack [9] (see question #7). However, in regions which block Tor based on the ciphersuite list in the ServerHello, such as in Iran in June 2011, it doesn't matter what ciphersuite we send as the client [16]. For those bridge not running on :443, we could have the bridge scanner mimic another protocol and service which uses TLS/SSL, such as IMAPS, SFTP, for instance it could pretend to be a client connecting to a Dovecot or vsftp server. '''Tor TLS/SSLv3 Handshake''' We can drive a Tor Client, or a script pretending to be Tor (which should know about the different handshake versions, specifically their command and CERT cells [10]), to handle the TLS negotiation. Interestingly, for the v2 and v3 protocols, we can use any ciphersuite list we like, as long as we include TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA in addition to at least one extra that is not any of those four. Tor clients before 0.2.3.11-alpha send a fixed ciphersuite list, and the GFC sends a probe based on this fixed ciphersuite list [12]. It is apparently also the case that the GFC will ''not'' send a probe if the standard fixed ciphersuite is altered by at least two ciphers [12]. To assist with this, hellais wrote a handy Python script for grabbing the default ciphersuite list from the source code of Firefox [13]. Also, as mentioned previously, we can fragment the sending of the ciphersuite list to avoid triggering a probe [5]. ==== Indirect Methods ==== As Roger also mentions, we could use some variant of the idle scan. [4, 8, 17] There are a few: 1. Use nmap / hping. a. For nmap, there is an NSE script for zombie discovery, which can be combined with blockfinder to collect lists of hosts (probably printers or other archaic networked devices) with globally sequential IPIDs [7, 18]. 2. Use idlescanner, a Python script which uses the "content upload" feature of popular sites, e.g. Reddit, Imgur, Facebook, Digg, Tinypic, Tineye, etc., to attempt a connection to the bridge [19, 20]. This may not be entirely accurate, because it is based purely on the waiting for the upload site to timeout. 3. Use FTP PROXY or some other obscure bounce mechanism [21]. These need to be further researched. 4. Now we start to get into some crazier ideas. If we set up a bridge purposefully to act as a canary, then we could send from an box inside China a bunch of TCP SYNs with spoofed IP headers to the canary bridge to trigger a bunch of probes. Then we trigger the probes with something (Winter wrote a program to do this called tcis [22, 23], and hellais ported it to Python in OONI [24]) forcing the probes to go after the canary bridge, during the two minutes that the probes have hijacked IP addresses, we use the probes' hijacked IP addresses as zombies for idle scan of bridge. This would require some preliminary mucking with the probes to see if they have any mechanism we could leverage to "see" if the bridge's packets made it to the probe. Basically we force the probe to hijack an IP, which we then zombify while it's chasing the canary, and get the zombie probe to scan the the bridge for us, without ''it'' actually scanning it, so it doesn't get blocked, and the traffic doesn't look suspicious to anyone keeping an eye on the probes. 5. A commenter on the Tor blog had the idea to try to "borrow a Chinese botnet" to do the scans for us, since the botnet would probably attract a lot more attention by the Chinese officials than any amount of Tor bridges. Also, with this idea, the scan could be made to look like your standard botnet running around launching PHP exploits at everyone and their mothers. This is a highly entertaining idea, but it's also a bit unethical (though I'm not certain -- do the ends justify the means in this case?), and it might come back to bite us. a. If there were a way to get an in-country botnet to "take notice" of certain bridges, we could do a sort of "Here boy, fetch!" trick. For example, if a botnet appears to be having infected hosts report-back to an IRC channel, or scanning for Windows hosts with port 139 open, we could mimic the responses an infected host would give while spoofing the bridge's IP. I have no idea how feasible or reliable that would be. III. Automation Concerns and Desired Features We should avoid scanning bridges that we suspect are not blocked. Therefore, eventually there should be an easy way to automate feedback loops between Karsten's metrics and the bridge scanner. That way, once connections in a certain country drop significantly, the automated tests initiate in order to discover if those bridges are in fact unreachable. Design Features: 1. Allow for either eventual integration with, or some type of feedback mechanismfor, metrics-db. 2. Should be automatable in a safe manner, i.e. the bridge scanner should know that a a full Tor connection to a specific bridge will likely result in that bridge being blocked, and thereby skip running any test which include a full Tor connection. 3. Should be easily incrementable, meaning it should be simple to tell the test "only try TCP SYNs for this list of bridges", or "try everything up until a Tor-specific TLS/SSL handshake". 4. GeoIP awareness. IV. Implementation I propose the test have all of the Active Direct Methods outlined above, and an easy way to test one at a time. For the actual testing, I want to err on the side of caution, in order to avoid getting bridges blocked. Therefor, during bridge reachability testing, we should test via most innocuous method first, wait a while (probably a day or two), see what we learn, then proceed to the next method. I was planning to use Python, because it's fast (in terms of coding time), we don't need to worry about portability in this instance, and it gives me less headaches than C. And Java makes me want to set things on fire. James Arthur Gosling, take it back. For the indirect scanning methods, I believe these will be difficult to entirely automate, but I plan to implement them so that they require as little human interaction as possible. If any of them prove reliable, they can be used as fallback methods when information concerning specific bridges is needed immediately and there is a human willing to run the tests. IV. A. Project Timeline July 2012 Two weeks of continued research and discussion until end of July. August 2012 Four weeks for initial development phase. Beta tests should be deployed by 31 August, and gathered data saved for evaluation of testing methods. September 2012 Four weeks for evaluation of data previously gathered from beta testing, and continued development of bridge reachability testing tools. Alpha release should be deployed by 30 August. October 2012 Two weeks for final development, with a useable, automated bridge reachability testing tool produced by 14 October. Two weeks for final testing, data collection and report generation, and discussion of further steps for integrating the automation of bridge reachability testing with general Tor metrics. November 2012 The project should be completed by 1 November 2012. - -------------------------------------------- == Active Questions: == 1. Should this automation be considered part of OONI? Or BridgeDB? Or is it part of some other project? 2. If there are only eight "edge routers": a. What are their IP addresses? b. Which protocols return traceroute data for these routers? c. Is the "core router" on this side of the "edge routers", or the other? d. What is the usual TTL of packets from the probes? 3. For how long is an IP hijacked by the GFC probe? 4. Roger mentions that "if the bridge had no other interesting services running (like a webserver), they just blackholed the IP address...but if there was an interesting service, they blocked the bridge by IP and port." Do the probes enumerate all ports, just common ones, or just privileged ports? 5. What percentage of current bridges are running on port 443? 6. Does the GFC automatically flag connections to TLS/SSL services which did not previously complete a DNS resolve? a. If so, (because most browsers cache DNS resolutions) what is the max time interval between the last successful clientside DNS resolution and a client's request for the GFC to remember that DNS was resolved? b. Do connection directly to IP addresses on port 443 stand out due to a lack of DNS resolution? 7. Does the GFC queue all TLS/SSL connections for later enumeration? - ---------------------------------------------- '''References''' - ---------------- [1] "How China Is Blocking Tor". Winter, Philip, and Lindskog, Stefan. Karlstad University, Sweden (2011). p.7, section 5.1. http://www.cs.kau.se/philwint/pdf/torblock2012.pdf [2] Ibid. p.6, section 4.2. [3] Ibid. p.19, section 6.3. [4] "Research problem: Five ways to test bridge reachability". Dingledine, Roger. The Tor Project (2011). https://blog.torproject.org/blog/research-problem-five-ways-test-bridge-rea… [5] "Case study: Learning whether a Tor bridge is blocked by looking at its aggregate usage statistics". Loesing, Karsten. The Tor Project (2011). https://metrics.torproject.org/papers/blocking-2011-09-15.pdf [6] "Level Four Traceroute". http://pwhois.org/lft/ [7] "ipidseq.nse - nmap script for globally sequential IP ID discovery" http://nmap.org/nsedoc/scripts/ipidseq.html [8] "Idle Scan". http://nmap.org/book/idlescan.html [9] "paketto". http://dankaminsky.com/2002/11/18/77/ [10] "Research problems: Ten ways to discover Tor bridges". Dingledine, Roger. The Tor Project (2011). Point #10. https://blog.torproject.org/blog/research-problems-ten-ways-discover-tor-br… [11] "Tor Protocol Specification". Dingledine, Roger, and Mathewson, Nick. The Tor Project (2012). Sections 2-4. https://gitweb.torproject.org/torspec.git/blob_plain/HEAD:/tor-spec.txt [12] "GFW probes based on Tor's SSL cipher list". https://trac.torproject.org/projects/tor/ticket/4744 [13] "get_mozilla_ciphers.py - Get the default ciphers of Mozilla Firefox". https://trac.torproject.org/projects/tor/attachment/ticket/4744/get_mozilla… [14] "EFF's SSL Observatory". https://www.eff.org/observatory [15] "SSLObservatory git repository". https://git.eff.org/public/observatory.git [16] "Iran blocks Tor; Tor releases same-day fix". Dingledine, Roger. The Tor Project (2011). https://blog.torproject.org/blog/iran-blocks-tor-tor-releases-same-day-fix [17] "new tcp scan method". Sanfilippo, Salvatore. (1998). http://seclists.org/bugtraq/1998/Dec/79 [18] "Ioerror's blockfinder git repository". https://github.com/ioerror/blockfinder [19] "Zombie Scans using Unintended Public Services". http://blog.makensi.es/post/3884103946/zombie-scans-using-unintended-public… [20] "idlescanner.py - Use unintentional web services for portscanning". http://makensi.es/tools/idlescanner.txt [21] "FTP Bouncing for Portscanners - FTP PROXY". http://nmap.org/nmap_doc.html#bounce [22] "How the Great Firewall of China is Blocking Tor". Winter, Philipp. Karlstads Universitet (2012). http://www.cs.kau.se/philwint/static/gfc/ [23] "NullHypothesis' tcis git repository". https://github.com/NullHypothesis/tcis [24] "OONI - chinatrigger.py - Python port of tcis". https://github.com/hellais/ooni-probe/blob/master/ooni/plugins/chinatrigger… [25] "An Analysis of Fragmentation Attacks". Anderson, Jason. (2001). http://www.ouah.org/fragma.html -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJQCxmBAAoJEKOttnos24s1hkcQAJ/eoc57f8Uk3vVsN9cpHQ9b /1v5S7+1j/kcIkLVaWRA8oDEPl1c6K1m6YaNsKEYceR180RZEg7MHPquxw/WC2QO DkwwQBQI4QfpvnJ+99egfy5MyErk34lfZzG+jTaUDzrFdtx9v07XbhFR391ly1iw AHGoyFaPcy8kFudByxptn8ecjYBfnyY7jD/eV21sx+/5tDV9a5vHP602PNV7CLfV XEVh38CtmM0+mZBjBFd0lZyO9x+ggeof2HRKjsT5Pzvu5UgOsNVTUWaRjD3GVVVg TND4H26TDgZz2wiWu25VUJO70OO1swfazzl4t0oFmpQHsZS5BIHJttBkgTzKb26j onVMd4Wa7LShJq7XukZUYQSx8Xfwo11dq/lIzRTa9pYmnu45GQUtm/a5jd55NgHp Ene7qtWecOGbQzDzxpCXYynH0iEQtBINdo1Yyl/WfVLIY4a1YKIQX54kl1DLKGos sqENNjsKA3ADN3ry+hKeIQkFUjJlI55RLWK26tcEaZUgOBLw0wiqFEvS6JY/l10e G5W+9FnRWANQJqg2xKuMdEnTLhTdmmu8KTKr0suo7t/Ac1jYdiOx/0AzbDpZeL51 E02mJf4cc149HlcA2LZAa7rzQGpXF6sTqSWrkOyqQMrN1+MWah+B/5rSoBQYZ+nt 9mDxRxhidxYs8nEg3+8p =NCy4 -----END PGP SIGNATURE-----

1 0

email2trac?
by tagnaq 21 Jul '12

21 Jul '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, is there a way to communicate with tpo's trac via mail? (create new tickets, answer to existing ones) If not: Has that already been discussed before? What would you think about it? https://subtrac.sara.nl/oss/email2trac/ I'm not sure how authentication is handled. thanks. -----BEGIN PGP SIGNATURE----- iF4EAREKAAYFAlAKA7UACgkQyM26BSNOM7aVhwD/f8PehnoevFLuhqytMk0T/CMe JrPROSvhNirLhNEvso8BAK0TyFTFL/80htpdsERG6h+TEGi76DW8zg4CNVocWCCU =RkIo -----END PGP SIGNATURE-----

1 0

stegotorus.git has been rebased
by Zack Weinberg 21 Jul '12

21 Jul '12

[ I was told to send this to tor-commits but I seem to be unable to send email there. ] In order to remove some inconveniently large and also potentially copyright-infringing files from the revision history, https://gitweb.torproject.org/stegotorus.git has been rebased. If you pulled from that repository, you're going to need to do a forcible branch update or a re-clone. (I am under the impression that mostly nobody did that because they were waiting for me to do this.) zw

1 0

Reachability testing brainstorming. (OONI, metrics, pluggable-transports)
by Aaron 20 Jul '12

20 Jul '12

BridgeDB was extended to filter bridges that are blocked in a specified country from its response. BridgeDB needs a list of what is blocked, and where, for this to work. We want to answer the question "What is blocked, and where", with the constraint that the list of bridges should not become public. Here are a few questions to kick off the discussion. Any insight, more questions (or answers) are appreciated. 1. Do we want active or passive reachability testing (or both)? 2. For active scanning, OONI comes to mind. Do we trust operators with the list of bridges? A partial list of bridges? 3. Does an open TCP port mean that a bridge is 'reachable' or should the test complete a full handshake? 4. What about pluggable transports (PT)? Will the Bridge Authority do independent reachability testing of PTs? 5. What concerns are there regarding detectability of active scanning? 6. For passive censorship detection, are there other indicators other than relay/bridge usage-by-country over time? 7. Should relays report per-country usage by address/transport? What are the upsides? Downsides? Thanks in advance! --Aaron

1 1

Building a two-hop circuit
by Ray Sweha 20 Jul '12

20 Jul '12

Hi all, I was wondering if there is a way to force Tor to build a two-hop path/circuit through torrc. It seems that both "Pathlen 2" and "PathlenCoinWeight 0.0" are depreciated, is there a substitute for them? Thanks, -Ray

2 1

[OONI] Designing the OONI Backend (OONIB). RESTful API vs rsynch
by Arturo Filastò 19 Jul '12

19 Jul '12

I would like to follow up on the discussion we had in Florence on some design choices behind OONIB. In particular the most controversy was around using HTTP or rsync. Before discussion the pro and contra about one choice over the other it would be useful to frame what are exactly the requirements for OONIB. # What is OONIB OONIB is the backend of OONI. It will run mainly on one centralized machine and may in a later stage run distributed across multiple ones. Currently we have not though of how to make it scale to being distributed, so we will look at it as if it were running only on one central machine. It will be responsible for: * Reporting a) Collecting reports from OONIProbes. Such reports are the results of tests. b) Collecting reports from third party censorship measurement tools (e.x. Bismark, NeuBot, etc.) * Assistance in test running Certain OONI Tests require to have a backend component (e.x. b0wser). On OONIB we will have the serverside component that will assist us in running the test. Note: Certain tests require the server to make connections to the client. This means that the client will need to request the server to probe them. * Control Channel Signaling This is required for making some measurements to verify that the data received by the backend specific to the test matches with the one sent by the client. # What properties we would like it to have note: these are not ordered. * Efficient even over high latency networks. * Ease of integration for third party developers. * Expandable to support requirements of new tests we develop. * Anonymous * Secure # HTTP and rsych comparison note: I will not deal with the security aspects of OONIB. We will suppose to have an encrypted and authenticated transport (this can be TLS, Tor Hidden Services, etc.) ## Rsync: Pro: * It supports good compression algorithms * It's efficient and supports resume * It does integrity checking on the uploaded files Contra: * It's designed only for copying files, this means we can't implement any more advanced API like logic. [*] * It's not supported by many languages (for example in python we only have an implementation of the rsync algorithm, not of the protocol [1]) * It's not as commonly used by other application developers that have similar requirements. * Painful to do sanitization of the data sent by clients. * Does not allow bidirectional communication (Request-Response pattern) [*] I would like to be able to create a session ID for a specific test and be able to reference such test ID when interacting with the Test helpers. rsync is one way, I push data to the server, but the server cannot signal me back with some data. This largely impeeds it's usefulness as an API interface. ## HTTP: note: I am not necessarily talking only about HTTP, we could use any other protocol with similar properties (e.x. SPDY). I will discuss HTTP because it is the one that I am most familiar with, but don't Pro: * Industry standard for exposing APIs * Supported natively in most programming languages * Well understood protocol * Implementation of sanitization of passed data can be done more easily * Allows bidirectional communication * Good support in twisted (what we use as a language for OONIB) Contra: * Compression is not enabled by default (we can use gzip compression with HTTP 1.1), and no compression for headers. * No resume support (this can be implemented on top of HTTP, we could even implement the rsyc algorithm on top of HTTP). * No support for deltas (we can use rsych protocol over HTTP if we really need this). I feel like we are a bit comparing apples and oranges here and I don't see why we could not use rsync algorithm on top of HTTP. Anyways I would like to get some feedback as to what we should use for something that should have the above described properties. Thoughts? - Art. [1] https://github.com/isislovecruft/pyrsync

7 10