- tor-dev - lists.torproject.org

txtorcon 0.11.0
by meejah 16 Aug '14

16 Aug '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 txtorcon 0.11.0 is now released. This adds a few API improvements: * launch_tor() allows access to stdout, stderr in real-time and control whether we kill Tor on and stderr output. See issue #79. * Warning about ``build_circuit`` being called without a guard first is now optional (default is still warn) (from arlolra) * available_tcp_port() now in util (from arlolra) * TorState now has a .routers_by_hash member (from arlolra) You can download the release from PyPI or GitHub (or of course "pip install txtorcon"): https://pypi.python.org/pypi/txtorcon/0.11.0 https://github.com/meejah/txtorcon/releases/tag/v0.11.0 sha256sum reports: 5efb7ba4faf698e68daea3d8711ce0ecfd8b95501cbec00b2093dd99f72dc8cc txtorcon-0.11.0-py2-none-any.whl aa206ab8a31053fd34798659c3d4cd7354a3e6872a86065c55098866df3dcaca txtorcon-0.11.0-py2-none-any.whl.asc ecb7cc8ee002ead5481c17fd62576521862e93081c58d7118f53cf3c8817f857 txtorcon-0.11.0.tar.gz 76ee6983269347d475ff4df9aefb7c43ddd25d24dd67c4e3293f5c8557377652 txtorcon-0.11.0.tar.gz.asc txtorcon 0.10.1 is in Debian, and no doubt this release will migrate there quickly as usual (thanks to Lunar^) thanks, meejah -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJT7vo4AAoJEMJgKAMSgGmnzjsH/36wtSA2Cb1coSuK+5hGNIGa flmGETgEOo4mHRCnSLQdO5JrXAKR9zeEt9tT9IfCTO1kRy4Q5cqzE569RhPsciBn 1g3fMlgXewjJuXx1atefO1sbqQgK74TsIqxkGnK85LEUOcqIcW71JT8lI6CDn3xg LymqLLhbC28uYLot+Y0/KPAME7MAlgugny5Me2sGzey1aKfjZhPXQ6gRJsGVlPjQ qhMD6ck9Xe+QZNkE4udzEzNkvA7Og/HJdwwi3yC/ENKcWKit/9PeKvmttVjyQ9Aj KGeN2CMA5jTHHznvohvAKgeUnkDDR4tqpcoKdJZjCVg9yAaaiNoPw2wj/3O3tTw= =elqw -----END PGP SIGNATURE-----

1 0

Decentralized VOIP over Tor
by terryz＠Safe-mail.net 16 Aug '14

16 Aug '14

Hi, I'm interested an anonymous decentralized VOIP network on Tor. All traffic routed through Tor. Every Tor user being an optional server for PTT or Continuous Speech. How feasible does this sound? --TZ

4 3

Guard nodes and network down events
by George Kadianakis 15 Aug '14

15 Aug '14

Hello friends :) This is a post to discuss how Tor should treat its entry guards when its network goes down. This is part of ticket #12595 [0] which aims to design better interfaces and data structures for entry guards. This thread investigates what should happen when the network goes down and Tor's connection to a guard fails. How should Tor recognize that and connect back to that guard when the network goes up again? I recently sent an email to tor-dev [1] which explains Tor's current behavior and its problems. tl;dr there are edge cases [2] where Tor will not detect the "network back up" event and will connect to low priority guards instead of connecting to its primary guards. The fundamental issue here is that Tor does not have a primitive that detects whether the network is up or down, since any such primitive stands out to a network attacker [3]. This means that when Tor fails to connect to an entry guard, Tor can never be sure whether the guard was actually down or whether the network is down, and that complicates its algorithm significantly. In #12595, I laid down a few different ways that Tor could improve its current "network down" entry guard algorithm [4]. After thinking some more, I've been leaning towards algorithm (a), which is: Everytime we manage to connect to a guard, if it's not the top guard in our list, mark all previous guards as retriable and try again from the top. For me, this seems like the most complete way of ensuring that the guards in the top of the list are always going to get a fair hearing even if the network was down when they were first probed. However, that algorithm is not without its problems. Here are some notes: - This algorithm suffers from an infinite loop. The naive version suffers from an obvious infinite loop if the first guards in our list are actually down. To protect against that, we will probably need to rewrite the algorithm to: Everytime we manage to connect to a guard, if it's not the top guard in our list, mark all previous guards as retriable and try again from the top: this time pick whichever guard we can connect to even if it's not the top one. - This algorithm is not actually robust. If we wanted to design a truly robust algorithm, it would have to be robust even against adversarial "network down" events. That is, the algorithm would need to work well even if you imagine the network as an "on"/"off" switch that the adversary can toggle at will. Here is how that algorithm can fail against such an adversary: 1) Tor starts up. Adversary switches network off. 2) Tor starts going through guards and fails to connect to them. 3) Adversary switches network on. 4) Tor detects "network up" event, marks all guards as up and goes from the top. 5) Adversary switches network off. 6) Tor starts going through guards and fails to connect to them. 7) Adversary switches network on, and Tor establishes a circuit to the guard that it was currently walking over. If the adversary is a LAN adversary, she learns the order of the guards in Tor's list, so she can basically choose whichever guard she wants. FWIW, whether such an adversary is realistic is up for debate. - This algorithm is not very elegant. - This algorithm might make guard fingerprinting worse. Imagine that the first 2 guards in your list are actually down. Everytime Tor detects a "network up" event, it will attempt to connect to those 2 dead guards before connecting to the third guard which is up. . A LAN/WAN adversary will be able to see those failed connections and the third successful connection, which form a nice tight guard fingerprinting vector (similar to #10969). And that's all folks. I'm looking forward to some feedback on the proposed algorithms as well as improvements and suggestions. PS: I think that Tor was doing a trick with UDP to learn its public IP address or something. I need to read up on that trick, and see if it can be used to build a "network up" primitive. [0]: https://trac.torproject.org/projects/tor/ticket/12595 [1]: https://lists.torproject.org/pipermail/tor-dev/2014-June/007042.html [2]: https://trac.torproject.org/projects/tor/ticket/12450 [3]: http://stackoverflow.com/questions/3764291/checking-network-connection https://trac.torproject.org/projects/tor/ticket/12595#comment:6 [4]: https://trac.torproject.org/projects/tor/ticket/12595#comment:5

4 3

[GSoC 2014] Revamp GetTor status update #6
by Israel Leiva 15 Aug '14

15 Aug '14

Hi people. This is my sixth and last report for the 'Revamp GetTor' GSoC project. I'll try to summarize what I've done in the last two weeks. If you have any comments, please tell me! Achievements: [*] Solved pending issues on the XMPP module, and tested it on a couple more providers. [*] Created a simple guide for testing both SMTP and XMPP module. [*] Opened tickets for documenting and discussing what's been done during GSoC. [*] Forked and modified the current code* of BridgeDB Twitter distributor to create a basic Twitter module. I haven't finished this, since there are some issues** I'd like to discuss before doing further development. [*] Started to work on blacklisting of users (not finished). There are still some days left until the hard deadline for GSoC, so I'll try to finish the last point during the weekend. I haven't been able to test the SMTP module on a real environment, I'll try to do that in the weekend too. I guess I'll write another e-mail to this list once GSoC is over, to talk about the entire process and development of the project. One of the things I can tell you is that so far I've written ~1500 lines (I know, quantity doesn't mean quality). I have already communicated to my mentors that I want to keep working on GetTor after GSoC, and given that I've re-written most of the code, I'll be happy to maintain it, too. *: at least what is public on Github. **: the issues are related to Twitter API terms and privacy. I sent an e-mail to this list [0] for discussion. Feel free to comment. As always, you can view the code on [1]. Have a nice weekend! [0] https://lists.torproject.org/pipermail/tor-dev/2014-August/007320.html [1] https://github.com/ileiva/gettor/tree/master/src <https://github.com/ileiva/gettor/tree/master/src> -- israel

1 0

RPM: "hardcoded" config options in /usr/bin/torctl
by Nusenu 15 Aug '14

15 Aug '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Ondrej, I filed a bug regarding the rpm packages [1]: "hardcoding" config options in torctl. Please let me also know what you think about migrating to systemd. thanks! [1] https://trac.torproject.org/projects/tor/ticket/12834 -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJT5kYQAAoJEDcK3SCCSvoefxoP/04Ac6/zFFg0cb8XjeNHdvjW qQAKWjKiWXRY00LWmCzeFsKp8MIyIVMDIDGja2XeaLcQd360iG/zbEKtHUm6RtXa AEH0OAzrG5eK7fUXoqXvpnSoAEkr/0cqdI1dcBJ9wUpUAMT+W6h8awpya3IuOfXv E/4V3g/C1PeQp9ISsaJIJTkn7E6enZ7teyPgW4gXx//+rljhW/mTgwGH94h2UDRE /JpH91VfBmO3zZipT1T/soIiWl8F5lm2FobOlJx5JPm9zfeeYntxVVHN7tbo6I4W xcs5Piq5O8cRvyBHliAH4i7uir/g/Fq8dwNJtRZGvyW+LLOVd98Djw+sdpqQ5Ddn zthNYU2/XUEDT1pHNYD73qrbA4fY1Thk1wU8MuYciiJvDL6uclEUFt2CBx1haAcN KJd3XiupUgY0nxpB53qUEqbX9hRmqWy8P7hbjguD5xk+exYH60VAkDBHT+HVjJS6 eWwMDiZcddPq5lNWyY1kHX0Px5xHiZWHBRUbw7OBZ/ImJAItLMIj47OTz3vD8nT/ CA5Rb5fLsJMGm1mmnqHH0wXX8ZqE60b3iBYbT/7U3fXejkqav4+PqMx3JVvN6xN6 1yb+1swrX0ZPIhKRH9CA5WGJOvrZwau9xlDj0R86U2PoV7KE7y52S9a9ywPMT/WY 2KYM9ebQa0HdPY5NeR5P =Ev1C -----END PGP SIGNATURE-----

2 7

only one signature on TBB 3.6.4
by Leif Ryge 15 Aug '14

15 Aug '14

Looking at https://www.torproject.org/dist/torbrowser/3.6.4/ I see that there is currently only one signature on sha256sums.txt for this release. As far as I can remember, every other stable release in the 3.x series has had signatures from at least 3 people. Is this an aberation or should users not be expecting multiple signatures in the future? I was just advocating for torbrowser-launcher to require multiple signatures when this happened: https://github.com/micahflee/torbrowser-launcher/issues/113#issuecomment-51… ~leif

1 0

Re: [tor-dev] [tor-talk] Tor Research Framework update
by str4d 14 Aug '14

14 Aug '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 08/14/2014 08:14 AM, George Kadianakis wrote: > Tim <t_ebay(a)icloud.com> writes: > >> On 13 Aug 2014, at 22:33 , George Kadianakis >> <desnacked(a)riseup.net> wrote: >> >>> My plan was to make a Peach fuzzer to achieve this [0], but as >>> I mentioned in a previous email I never got past the V3 link >>> handshake since I actually had to implement Tor's crypto to get >>> past. >>> >>> Someone would need to implement all this stuff to be able to >>> fuzz the Tor protocol as I was intending to. >> >> Gareth has implemented Tor's crypto in tor-research-framework[0] >> in Java. >> >> Would this be sufficient for Peach, or does it need to be written >> in Python? >> > > I don't know. I think it needs to be written in Python, or you need > to write bindings or something... But better check this yourself, > I haven't looked into Peach in years. You could also use Jython to run tor-research-framework and use the Java crypto in the Python code. See Jy2P [0] for an example, it uses Jython to start the Java I2P router and run Python code to interact with it. str4d [0] https://github.com/majestrate/jy2p > _______________________________________________ tor-dev mailing > list tor-dev(a)lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev > -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJT7HVTAAoJEIA97kkaNHPnjaQP/A+lWfVpETGUGDEVoDRnGK/X bAS0IXdt1O1G4Pv3aYSwbf/BDlJU2jkp/4OCygyIzEN8JxtJofqV9vMUlaubV31k NZJfLuojHmOICDKqWiiloYqF68BhVXF166ZXQHthCQKgZ1U8Y6QOwa0gknlB3uDn n/fZ+10fMxYoV1WjdCsBpqgsZwgVLWjt2iMmvnoILLqOENH6t4OaCCxtT1aUMDor cMnyZ9z/kaNLWqyFoFGucsMdUv9Z9WIJnLsym3wCnU2pbMTypvrDcMIZPA2U1DoW IBL5pqR7wGICjCoWEtGIvVWLHATybcMItC7I3bHsHj+06E+KDY4c3qZK07qgDTtw /3SkEt2/ZBxdbh6wcFL0oZivmQnUqTHPyOMHsDo8Id0d8qD0D2XTiBZY1w+di4ac MxaM6RH4SgK31aKq5S4Ei0ZWICFL96TO8V/dINB+UKvlgJ9Z81eSrSciaI08ol7v Z5Rx59KnRtbu7vmHpv5vxusjcM3aES1M+XGLGXRfe6G7Wf78f7GlSdsWkAi9MU0Y qMSJp94nT7npEX/dhGh0p8lBpzRsSGurLsvYn63tVI/uw4iz2ba84+cqEZl/pbcv yC1Rxzi7EOYEMNC/umxXDvsWfjPbNv0LW6rXilYT7fqPPHXi/A09N8ns5RqR63IE yoN+HGJQgR7VUYHHEKGu =biUC -----END PGP SIGNATURE-----

1 0

OFTC-i2p Relay Bot for #tor/#tor-dev
by ihave2p 14 Aug '14

14 Aug '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello and greetings, I was referred to this list by velope@oftc and have spoken directly with isis@oftc with regard to administering a relay bot for #tor/#tor-dev between Irc2p (i2p) and OFTC. To my joy, isis@oftc has happily granted permission to start a 'testing' phase of the bot pending the final decision from the Tor devs about the bot's #tor/#tor-dev fate. As it stands, the only hiccups the bot has seen were within the first 15-20 minutes of initial deployment but now, after several days, it has proven to be more helpful than not and it has since ran without incident. The bot is a barebones Limnoria with default plugins (set to private) and LinkRelay from https://github.com/ProgVal/Supybot-plugins. The bot masks hostnames with either 'oftc' or 'i2p' and currently serves no other function than relaying. As it stands, all bot responses are privmsg'd so as not to spam the channels and the bot will not relay non-privmsg relay messages i.e., joins, parts, nicks, quits, modes, etc. to anyone. I'm writing to clarify how you would like to approach the administration of the bot should things go awry. It will only connect to OFTC via tor as I cannot expose myself to clearnet/i2p identity correlation (I'm well aware of OFTC's temporary blacklistings of Tor exit nodes so I've also taken that into consideration with regard to management). Listed are my preferred options for your consideration. All comments/suggestions are welcome. 1) Off-topic chatter from i2p can be redirected to #nottor (a general "hey you, go to #nottor" to an i2p user would suffice. Note: #nottor on i2p is *NOT* relayed to OFTC but instead, redirects i2p users to our non-Tor community channels). 2) Currently, the bot will *not* auto-rejoin when kicked. If it is ever kicked, we can discuss why and then take further action if needed. 3) "ChanOp Exchange Initiative": OFTC ChanOps could have the ability to mute or kick users on i2p's side, and vice versa. This could be very useful to both parties once both have considered the other trustworthy. 4) OFTC ChanOps could have the ability to start/stop relaying and/or start/stop nick-specific relaying (tricky, but I believe possible). 5) Worst-case scenario: the bot would relay *all* OFTC users but only +voiced i2p users. Presently, I'm founder/op (the only op) for #tor/#tor-dev/#tails on Irc2p but if more ops are needed, I'll see who is available. Irc2p has flood protection and I have yet to see any Irc2p abuse outside of the occasional troll or ill-informed user. Let it be known: irc bot management/deployment is not my forte nor do I find it terribly interesting but I'm passionate about facilitating easy/anonymous communications between our two networks - so I'll "take one for the team" and try to make everyone happy with what we're achieving. For the time being, I'll attend to the bot very regularly and I don't foresee ever leaving it alone for more than a day or two (even after the 'testing' phase is complete). If I'm away for longer than that, I'll make the appropriate notifications/arrangements and, if needed, I'll simply take the relay offline. To reach me if I'm not available on either network, the bot's "real name" has my i2pmail.org (mail.i2p) address and PGP key (FYI, in case you were wondering, the bot has its own SSL cert and is registered on both networks). Thank you for your time and consideration. For those who aren't familiar with Irc2p, is not nearly as large as OFTC but I feel that strong minds/talents are on both sides and that all would benefit from the relay. I look forward to a growing bond between our communities. ihave2p 0xF8034C5F <ihave2p [at] mail.i2p> -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT5eR2AAoJEHlp9aX4A0xf/RUQAKCAFSgiXHCI6NPwBL/+TDUp 0dp4R2oqxspcs1x7v2wV4SL2iuwYjWDMAjLtW7qwPFLLlPX+2nS0Jxbye01pLxHy ftJ0Bj+MY2q+4o4mQ5x1iGRPEukWm6zJgK0kI7BU/0yoennD4+TrBUygFIq0eS9g muJReWX3iNsFjhkJnE3ZoRA6CFequ+1Aiy5erInZkjqzG7tqlP6qZ3LM5cY4Ig3M uYWZCLBpSnlEimZX2EdPeFuwbSPGGur0QfIEfw3RHyibDy4XU1E1clAROHv+8bww x3+06CG4Xbn5UxMOicHBr//8nHFngGMWPZgKDKcPpEqiNPskkMX018xVYeLi7VYY syNC2Ioq/fY5w0XvIRZ+tQoeizndqurn2Lq5ZT+Rfwjg6irbPlOnR7Cld/SDAjZ0 7/u0pB8X//Eg/wCq+piFe7KFMGh29/QBCa6f0I8T+vQ5ktoLz8TD8Q+OVWmvk4uZ sGH7OTb6yUjwe3kGZFuHPwQ7tT/0FO4n8IEq0btAfuLU6ve2EAFl5BMie+2W8KhK zQ/SxX2ihiFaGnp7K/pVhR16n7AVLQbWFk4E5tQZgwByQSev5c+PUIdfjLBGM+lo rnU5OwEk+ppSMJeizACX9eU+j97lp8kVCAXzBjN8L5lvGTZPkbGEcL1M+KtWRI3+ ggv0IQw2pB76ulg85Dmw =m6lI -----END PGP SIGNATURE-----

3 2

Role of Bandwidth weight
by saurav dahal 14 Aug '14

14 Aug '14

Hello, I studied about section 3.5.3 computing bandwidth weights in Tor Directory Protocol Version 3. I know now how to compute the values of these weights. 1) I am not able to understand how these values are used in path selection by the client. Anybody please highlight on this. 2) There are guard flagged nodes, Exit flagged nodes and Guard+Exit flagged nodes in Tor network which are used in guard position and exit position while making circuit by the client. I know that for choosing guard flagged node in guard position, weight Wgg is used, for choosing exit flagged node in exit position, Wee is used, and for choosing Guard+Exit flagged node, Wgd or Wed is used. lets say that Wgd < Wgg and Wed < Wee. Then how is it that G+E flagged node be choosen in guard position or exit position? Thanks, -- Saurav

1 0

[Patch] src/test/test.h
by Gisle Vanem 13 Aug '14

13 Aug '14

The 'U64_PRINTF_TYPE' is defined twice for MSVC. An 'I64_PRINTF_TYPE' is needed too: --- orig/src/test/test.h 2014-05-13 23:24:22 +0000 +++ src/test/test.h 2014-05-13 23:26:51 +0000 @@ -61,7 +61,7 @@ #ifdef _MSC_VER #define U64_PRINTF_TYPE uint64_t -#define U64_PRINTF_TYPE int64_t +#define I64_PRINTF_TYPE int64_t #else #define U64_PRINTF_TYPE unsigned long long #define I64_PRINTF_TYPE long long --gv

2 1

Feedback on new OONI test deck format
by Arturo Filastò 13 Aug '14

13 Aug '14

Hi all, We have been discussing lately some changes that we would like to make to the ooni-probe test deck format [1] and would like to have some feedback on what we have come up with. For those of you not familiar with ooni-probe, a test deck is basically a way of telling it "Run this list of OONI tests with these inputs and by the way be sure you also set these options properly when doing so". The previous deck format involved a lot of boilerplate and did not make it possible to bake inputs into the deck itself. This new format is supposed to overcome some of the limitations of the old design and we hope that a major redesign will not be needed in the near future. The specification can be found in git [3], but to facilitate commenting I am also going to paste it into this email. All feedback is greatly appreciated. -- BEGIN SPEC -- # Test deck specification * version: 0.1.0 * date: 2014-08-13 * author: Alejandro López (kudrom), Arturo Filastò # 0. Terminology Analyst: The person who writes a deck. Collector: A machine running the ooni-backend that collects the reports generated by the execution of ooni-probe. Nettest: A test whose execution by ooniprobe creates a report sent later to the collector if provided. Ooni: Open observatory of network interference. Ooni-probe: The client side of ooni used to execute a set of nettests. Tester: The person who executes ooni-probe. YAML: A human-readable data serialization format. # 1. Rationale To ease the execution of nettests, the ooni developers came up with the idea of a container that would allow a tester to easily execute a bunch of configured nettests.That container was called a deck. This way, an analyst interested in a particular behaviour would write a deck; then, she would distribute that deck to every tester interested in the ongoing analysis. Finally, the tester would execute ooni-probe with that deck to properly create and send the nettests' reports. Unfortunately, right now both the deck format and ooni-probe don't allow that level of automation. The way we want to solve this situation is by writing a new spec for the deck format that would allow us to write, with great confidence, the given functionality to ooni-probe. The document that you're reading is that spec. # 2. Goals Allow an analyst to reuse well written and tested nettests in an easy way. Allow a tester to execute easily a battery of nettests with a complex configuration. Grant privacy of the tester. # 3. The data format Every deck is a yaml file composed of two major sections: the header and the body. The header is a dictionary that provides all the shared and global configuration of every nettest included in the deck. Its main purpose is to reduce boilerplate by letting the analyst express common behaviour in one section instead of in every nettest execution. The ooni-probe options allowed in this section are: 1. collector: Address of the collector of test results. 2. bouncer: Address of the bouncer for test helpers. 3. annotations: Annotate the report with a key:value[, key:value] format. 4. no-collector: Disable the collector. (FLAG) 5. no-geoip: Disable the geoip support. (FLAG) Every option of the header section is a dictionary's key except that labeled with a FLAG, which are members of a list called flags. So for example a valid header section is the following: ``` header: collector: 'http://localhost' annotations: key1: value1 key2: value2 flags: - no-collector ``` The deck header can also contain metadata associated to the test deck. The possible fields are: 1. name: The name of the test deck. 2. description: A short description of the test deck. 3. author: The author of the test deck. 4. version: A version number for the test deck. 5. requires-root: A flag to indicate that the test deck requires root to run. (FLAG) 6. requires-tor: A flag to indicate that the test deck requires tor to run. (FLAG) The header may also be omitted. The body is a list composed of one element per nettest execution. Every nettest execution is a dictionary composed of the following three keys: 1. nettest: name of the nettest to execute (MANDATORY) 2. local_options: local_options of the test execution (OPTIONAL) 3. global_options: global options of the test execution (OPTIONAL) In the same way that with the header, every option can be a member of a flag list if that options doesn't have any arguments. A valid body can be: ``` body: - nettest: manipulation/http_request local_options: url: 'http://torproject.org' global_options: collector: 'http://localhost' flags: - no-geoip - nettest: manipulation/captiveportal ``` All file paths must be relative and they must start with "deck/" if they are referring to files contained inside of the test deck. They must start with "http(o|s)://" if they are referring to files to be downloaded via Tor. All other file paths should be ignored and raise an exception. This is because we do not want an analyst creating a deck to be pass as arguments to a test arbitrary files on the testers filesystem. # 3.1 Container format The container proposed is tar+gzip because it's well supported in python. The deck container will be composed of a directory named "deck" containing the deck file and the inputs. The directory layout will be: deck/test.deck deck/input-filename-1.txt deck/input-filename-2.txt This will then be compressed using tar+gzip. # 4. Implementation details ## 4.1 Introduction Each execution of a nettest in ooniprobe needs four main inputs. 1. The global config file 2. The global cmd options 3. The nettest/deck 4. The local options for each nettest The first three ones are mandatory to run a nettest, the last one depends on the nettest. When a single nettest is executed, all options except the first one are passed in the cmd line. The difference between the global config options and the global cmd options is that the second one has some shared options with the config file plus some additional subcommands to ooniprobe. When a deck is involved in the execution, the last three inputs are meant to be passed (at least partially) in the deck. In fact, the deck is no more than the three last inputs, with some subtleties that I expect to explain in the following sections. ## 4.2 Who overwrites who In a single-nettest execution, the global config is parsed and a global object is built. Then the cmd line is parsed and it overwrites the global configs. This is done to allow the setting of the most dynamic options in the console without the need of writing them each time to the config file. All the ooni-probe's code base is allowed to access that global object, included the nettests. So, the global options affect the way ooni-probe behaves but also can affect the way the executing nettest behaves. That's the reason why the deck must provide the specification of some global options, but not all because there are some powerful options that would put in danger the tester. The allowed options are listed in the section [3. The data format]. In a deck-nettest execution, the header section of the deck is parsed first and a global object is built. Then the config file is parsed and the object is overwritten when it applies. Finally the cmd line of the ooni-probe is parsed and the object is overwritten. We read the header section first to avoid the analyst overwrite some sensitive options of the config file, which only should be modified by the tester. ## 4.3 Copy-on-write To reduce the boilerplate in the deck, the file is splitted into a shared section for all nettests and a local one for each nettest. The idea is to allow the writer of the deck to express common and global behaviour of all the nettests in the header of the deck and to put specific and local options in each nettest element, as was already explained in [3. The data format]. What this means for the execution of the nettests is that when a nettest overwrites a global configuration of the header, this change is only visible to this nettest, not to every nettest who may ever execute in the same instance of ooni-probe. So what follows is that every change to the global options of ooni-probe in the local section of a deck should attend a copy-on-write policy regarding the global config object. ## 4.4 The input file Ooni-probe invokes every nettest method with the information saved in a file called the input file. This file is part of the local configuration of every test, and therefore must be provided with the deck. So the deck should be a compressed container which includes both the deck file and every input file necessary to every nettest included in the deck. Otherwise, the analyst would have to send to the tester the input files separately, which is unacceptable. The container proposed is tar+gzip because it's well supported in python. # 5. Example deck The complete.deck provided with each installation of ooni-probe would be: ``` header: name: Complete description: Runs all the existing ooniprobe tests author: 'OONI <ooni-dev(a)lists.torproject.org>' version: 0.1.0 flags: - requires-root - requires-tor body: - nettest: blocking/http_request local_options: input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' - nettest: blocking/dns_consistency local_options: input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' - nettest: manipulation/http_invalid_request_line - nettest: manipulation/http_header_field_manipulation - nettest: manipulation/traceroute - nettest: blocking/http_host local_options: input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' ``` -- END SPEC -- ~ Art. [1] https://trac.torproject.org/projects/tor/ticket/12823 [2] https://gitweb.torproject.org/ooni/spec.git/blob_plain/HEAD:/test-decks/td-…

1 0

UX Idea - A controller inside TBB
by Nima Fatemi 13 Aug '14

13 Aug '14

Hi Devs, [For those of you who have missed Vidalia, and for those of you who hated it dearly but still needed an easy-to-use Tor controller] A long while ago, when I was bored on an airplane, I tried to imagine a nice controller inside TBB, with all the good features of Vidalia and some more stuff. Fast forwarding to couple days ago... I was hitting a weird exit node that wouldn't let me connect to any site except Google. I only had TBB on that machine, and couldn't find a way to find my Tor IP address. That UX nightmare reminded me of this image I complied on that trip. Although it's not finished and is based on imagination only! I thought I'd share it anyways, hoping to inspire people to start hacking on it. https://people.torproject.org/~nima/ux/about-tor.png Here's my question for you: How would you perfect that image? What's missing? Let's talk about your ideal controller (inside browser). Who knows... maybe someone would read this thread and make it happen. Bests, -- Nima 0XC009DB191C92A77B | @nimaaa | mrphs "I disapprove of what you say, but I will defend to the death your right to say it" --Evelyn Beatrice Hall

7 9

Proposal idea: Stop assigning (and eventually supporting) the Named flag
by Sebastian Hahn 13 Aug '14

13 Aug '14

Filename: xxx-kill-named-flag.txt Title: Stop assigning (and eventually supporting) the Named flag Authors: Sebastian Hahnn Created: 10 April 2014 Target: 0.2.5 Status: Draft 1. Intro and motivation Currently, Tor supports the concept of linking a Tor relay's nickname to its identity key. This happens automatically as a new relay joins the network with a unique nickname, and keeps it for a while. To indicate that a nickname is linked to the presented identity, the directory authorities vote on a Named flag for all relays where they have such a link. Not all directory authorities are currently doing this - in fact, there are only two, gabelmoo and tor26. For a long time, we've been telling everyone to not rely on relay nicknames, even if the Named flag is assigned. This has two reasons: First off, it adds another trust requirement on the directory authorities, and secondly naming may change over time as relays go offline for substantial amounts of time. Now that a significant portion of the network is required to rotate their identity keys, few relays will keep their Named flag. We should use this chance to stop assigning Named flags. 2. Design None. Tor clients already support consensuses without Named flags, and testing in private Tor networks has never revealed any issues in this regard. 3. Implementation The gabelmoo and tor26 directory authorities can simply remove the NamingAuthoritativeDirectory configuration option to stop giving out Named flags. This will mean the consensus won't include Named and Unnamed flags any longer. The code collecting naming statistics is independent of Tor, so it can run a while longer to ensure Naming can be switched on if unforeseen issues arise. Once this has been shown to not cause any issues, support for the Named flag can be removed from the Tor client implementation, and support for the NamingAuthoritativeDirectory can be removed from the Tor directory authority implementation. 4. Open questions None.

3 6

Rescheduling of next OONI development meeting
by Arturo Filastò 13 Aug '14

13 Aug '14

The next weekly OONI development meeting is supposed to happen on the 15th of August. Since the this day is holiday in a lot of countries I am going to propose we skip next weeks meeting and move it to the week after: the 22nd of August. The time will be the same 5 PM CET (3 PM UTC). The upcoming weeks the meeting should happen every Friday as usual. Happy holidays, ~ Art.

1 0

Globe Dev
by Mike 13 Aug '14

13 Aug '14

I'd like to help and last I read on this list wpapper was taking on Globe dev https://github.com/wpapper/globe. Their fork is looking a bit quite of late so I'm not sure where I should start. I'm happy to start from where makepanic left off, or continue with wpapper's fork??? Mike.

2 1

Pluggable transports meeting tomorrow (16:00UTC Wednesday 13th of August 2014)
by George Kadianakis 12 Aug '14

12 Aug '14

Hola brothers and sisters, just wanted to remind you that the regular biweekly pluggable transports meeting is going to occur tomorrow at 16:00 UTC. Place is the #tor-dev IRC channel in the OFTC network. As you see, the meeting day was recently changed from Fridays to Wednesdays: https://lists.torproject.org/pipermail/tor-dev/2014-August/007317.html Thanks for your attention!

1 0

Tor API doc?
by CJ 11 Aug '14

11 Aug '14

Hello, I've searched, but didn't find anything regarding how we can interact with Tor API (through the Administration Port). Would be nice if you could provide some link :). Thanks in advance. Cheers, C.

3 4

little-t tor "projects" in need of another keyboard
by tibicen＠riseup.net 11 Aug '14

11 Aug '14

Hi everyone, Is there any project in little-t tor that needs another hacker? I have (under another name) been writing tor unit tests, which I've "finished". What should I work on next? Happy to work on either new code or more unit tests. I'm also reasonably skilled with python, if any pythonic project needs a hand, particularly OONI, which I love the idea of (I won't CC the ooni dev list, as I assume you're all on here). tibicen

2 1

Torflow error
by spriver 11 Aug '14

11 Aug '14

Hi, I am trying to setup Torflow, but when I am about to start metatroller I'm getting following error: sudo ./metatroller.py Traceback (most recent call last): File "./metatroller.py", line 330, in <module> main(sys.argv) File "./metatroller.py", line 327, in main listenloop(*startup()) File "./metatroller.py", line 310, in startup h = PathSupport.PathBuilder(c, __selmgr) # StatsHandler(c, __selmgr) File "/home/dodo/flowtor/TorCtl/PathSupport.py", line 1447, in __init__ TorCtl.ConsensusTracker.__init__(self, c, RouterClass) File "/home/dodo/flowtor/TorCtl/TorCtl.py", line 1582, in __init__ self.update_consensus() File "/home/dodo/flowtor/TorCtl/TorCtl.py", line 1674, in update_consensus self._update_consensus(self.c.get_consensus()) File "/home/dodo/flowtor/TorCtl/TorCtl.py", line 1011, in get_consensus nsData = self.sendAndRecv("GETINFO dir/status-vote/current/consensus\r\n")[0][2] File "/home/dodo/flowtor/TorCtl/TorCtl.py", line 877, in sendAndRecv raise ErrorReply("%s %s"%(tp, msg), status = code, message = msg) TorCtl.TorCtl.ErrorReply: 552 Unrecognized key "dir/status-vote/current/consensus" Does anybody has an idea what's going wrong? Tor is running properly configured (with opened ControlPort). Cheers spriver

1 0

[GSoC] BridgeDB Twitter Distributor report
by Kostas Jakeliunas 11 Aug '14

11 Aug '14

Hi all, preferring existing code over shiny code and being mad late, I * (re)wrote a simple but working churn control mechanism[1], which uses * a general persistable storage system: * in particular, the bot now has a central storage controller which takes care of storage handlers which, in turn, may be of different varieties. Each variety knows how to handle its own kind of storage containers (simple objects with data as attributes). Some of them may be persistable, others necessarily ephemeral (wipe data on close); * right now we only make use of simple pickle-dump-to-file-and-gzip persistable storage; we use it for churn control and for challenge responses; everything is self-contained so to speak; * we hash the user twitter handles (unique usernames / screen names) and round up bridges-last-given-at timestamps; * we handle bot shutdown by catching the appropriate signal (then properly closing down the twitter stream listener and asking the storage controller to close down the handlers); * we use the storage system in the core bot via a general "bot state" object (which is itself oblivious to how storage is actually implemented); * wrote a simple and generic challenge-response system[2] (which makes use of the persistent storage); * instead of doing something very smart, we use a general CR system which takes care of particular challenge-responses; the general CR is usable as-is; the particular CR objects can be easily subclassed (and that's what we do now); * the current mock/bogus CR system that is in place (for testing etc.) is a naive text-based question-answer CR, which asks the users to add the number of characters in their twitter username to a given verbal/English-word number; * I should now finish up with ``BridgeRequest``s, which are the proper way to handle bridge requests in the bot while doing challenge-responses (the current interaction between the core bot and the CR system will lead / has been leading nowhere); * also, there's a question to be had whether the cached (and hashed) answers to CRs should be persisted to storage (if bot gets shutdown while some challenges are pending) in the first place. I've been unable to find[3] or to come up with a concept of a user-friendly *text-based* CR that would stand against any kind of thief who is able to create lots of Twitter users and to write twenty-line scripts solving any text-based challenges/questions presented. Either it will to be a difficult problem that will be easier solved by a computer than by a human (hence unfeasible general-UX-wise), or it will be so "symmetrical" in the sense that one only has to view the source (if even that) to come up with a script trivially solving the challenge presented. Hence I've been slowly moving on with the captcha-over-twitter-direct-messages idea, which is not pretty, but which would at least ensure that we don't give up bridges more easily than in, say, the current IPDistributor. [1]: https://github.com/wfn/twidibot/compare/master...churn_rewrite [2]: https://github.com/wfn/twidibot/compare/churn_rewrite...simple_cr2 [3] it's quite hard to find anything of use in the "chatroom problem" / "text-based challenge response" area. Basically, it would be great to have a "reverse Turing test"[4] that is not about captcha/OCR. I realize this is in itself a very ambitious topic. [4]: some context on early CAPTCHAs / precursors (have been trying to familiarize myself with the general area), http://www2.parc.com/istl/projects/captcha/docs/pessimalprint.pdf -- Kostas. 0x0e5dce45 @ pgp.mit.edu

2 1

A change in the schedule of Pluggable Transport meetings
by George Kadianakis 09 Aug '14

09 Aug '14

tl;dr: From now on, Tor PT meetings happen on Wednesdays. Same time. Next PT meeting: Wednesday 13th of August, at 16:00 UTC. Hello friends, I want to inform you of a schedule change that affects Pluggable Transport meetings. Because of the diverse set of timezones participating in the biweekly PT meetings, we have decided to move the meeting to Wednesdays (instead of Fridays) since for some people the PT meeting occurs during Friday night, which is considered precious social time in some cultures. So the plan is to start having PT meetings on Wednesdays. The time will remain the same for now (16:00 UTC), but it might also change in the future (I will inform you) to make it more convenient for some people. BTW, the changes are effective immediately, which means that our next meeting is on Wednesday, 13th of August 2014, at 16:00. Thanks :)

1 0

New tor relay configuration
by Richard Dennis 08 Aug '14

08 Aug '14

Hi guys, I have recently created a python library that creates a circuit through tor and then can send streams through etc, for the next phase of this project however i require access to a tor relay node to monitor. i have set one up using a raspberry pi, following this tutorial : http://www.instructables.com/id/Raspberry-Pi-Tor-relay/?ALLSTEPS i have three flags already : Running, V2Dir and Valid however when i try to connect to the node i get a destroy cell back after a lengthy wait. the node is Goblin500 and is listed on the consensus, but i am not sure why i am unable to connect to it, Would it be possible for some one to try and create a circuit using this node to see if it works if this does not work does any one know where i have likely gone wrong ? Thanks Rich

1 0

New time for Tor dev meetings: 1330 UTC on Wednesdays
by Nick Mathewson 08 Aug '14

08 Aug '14

Hi, all! Due to time zone changes for a bunch of developers, we're going to have a new time for the weekly core Tor development meetings. Going forward, they will be each Wednesday at 1330 UTC. This is the weekly IRC meeting for people working on the program "tor". (It won't cover all the other programs developed under the Tor umbrella.) As usual, we'll do it on the #tor-dev IRC channel. best wishes, -- Nick Mathewson

1 0

Sybil attack detection (was: Karsten's July 2014)
by Philipp Winter 07 Aug '14

07 Aug '14

On Tue, Aug 05, 2014 at 11:37:45AM +0200, Karsten Loesing wrote: > Started looking into better algorithms to detect Sybil attacks on the > Tor network. Current thinking is that we should define relay similarity > metrics like common IP address prefix length or time between first seen > in a consensus, go throw the consensus archives, and see which relays > look similar but are not defined to be in the same family. Do you already have some code or more details on that? I'm quite interested in this topic and I'm wondering if it wouldn't be best to start with something simple like cosine similarity [0]. We would have to transform a relay descriptor into a numerical vector and then calculate the cosine similarity to other relay vectors. However, this might scale poorly as we would need (n^2)/2 comparisons. We might also want to weigh the vector's elements differently as some of them are easy to control for an attacker (advertised bandwidth, uptime, flags, exit policy) and others require more effort (IP address, ASN, observed bandwidth). Like you mentioned, the key thing to look at might be time, i.e., uptime and derived features such as "total online time in last month" etc. [0] https://en.wikipedia.org/wiki/Cosine_similarity Cheers, Philipp

5 5

Damian's Status Report - July 2014
by Damian Johnson 06 Aug '14

06 Aug '14

For one brief, glorious moment in time I touched code. Really, it happened! House plants can vouch as my witness. Or that is to say, I've coded a lot less than I'd like. Barring a long overdue vacation to Sol Duc this month went toward a lot of random distractions... * Worked with Philipp on overhauling our procedure for flagging bad relays... https://trac.torproject.org/projects/tor/wiki/doc/ReportingBadRelays Philipp's taken the lead here and is doing a great job! While I find this space interesting I'm already spread too thin, so going forward I won't be a co-maintainer for it after all. :( * Looked over Nick Hopper's consensus validation change, adding crypto blob type validation to simplify his work. I now owe him a more thorough code review. (#11045) * Handful of DocTor issues including OOM troubles (#12457), unpleasant amount of noise from consensus parameters (#12708), and a several minor revisions. * Discussed adding a new X- extrainfo descriptor field with Virgil, suggesting revisions and helping him make it conformant with the spec. * System I do development on died a puzzling death. Even after twenty hours investigating I'm still not quite sure what happened. Cleaning out dust, removing the RAID controller, and starting over seems to have done the trick. * GSoC administrative work. We're now less than two weeks from the final evaluations! * Continued overhauling arm but as already mentioned focused on this a lot less on this than I'd like. So little coding makes me grumpy, so like many of you I'm taking a hard look at where my time's evaporating to. Maybe there's something I can change or extricate myself from to better focus. Cheers! -Damian

1 0