- tor-dev - lists.torproject.org

On status of Stegotorus on github vs SRI
by vmonmoonshine＠gmail.com 10 Jul '13

10 Jul '13

Hi Jeoren, Thanks for your comments or tor-dev IRC channel. I thought I may as well take my time and reply to them here so I can also include Zack, Vinod and others into the discussion. >I think the large chunks of commits without specific >goals/descriptions what the commit fixes will be timecostly to > integrate, I started working on Stegotorus as a GSoC student last summer and I have been more or less continuously working on it since then. I always kept the tor-dev list informed on my progress. So a quick search on [Stegotorus] should gives anybody a good idea on stuff I was working on. But I can summarize all my works in four categories: - Debug. - Writing more unit and behavior tests, including tester-proxy, webpage-tester and bad-dropper-proxy. - Integrating Stegotorus with a web server (e.g Apache Httpd) so the covers can be served by the web server. - Debugging the ack/transmit protocol and the packet transmission - reliability in general. In general, along the way, I tried to reconstruct the code into being more C++ and more object oriented (I'm continuously doing so). Obviously Stegotorus was designed to be as modular as possible. That is the main motivation behind "pluggable transport": the DPI got more intelligent? Plug the compromised module out, plug something else in with minimum effort. All of Stegotorus different modules are begging for OOP and inheritance. However, forcing the code into C had just made the steg folder unmanageable and un-expandable. The different steg modules (pdf, swf and js) have something like 70%-90% code in common but instead of being inherited from a common parent the code has been copied multiple times. It seems like an artificially generated example by OOP evangelists to convert people into OOP by showing how terrifying functional programming can be. >especially as the hidden upstream has diverged quite a >bit in some places (which one can nicely attribute to the wrong of >that 'model', but unfortunately not an easy way around that); Until recently that Roger sent an email to Vinod, I did not know that the upstream in SRI is making any progress (I'm not sure if even Roger (or anybody else on the Tor side for that matter) was fully aware of the amount of progress on the other side) . Last winter, Zack told me that I'll be the maintainer of the code (at least from Zack's prospective) in the sense I'd decide what patch to keep and what to discard. > nevertheless, I think the best plan to get those in there would be to > see when that code is out, and have a list of important > ... fixes/commits available and then at one point go over everything > and try to get them merged into that code base, will be big > job though to get that done, and would 'require' > that upstream takes them, which mostly depends on > available time I would say.... As Roger and other have mentioned multiple time, it is hard to fully reconcile the SRI model of development with the open source model. In that sense, we'll probably always have an open source version of Stegotorus and a SRI version of it (At least till SRI development is active). Nevertheless, my goal is to reconstruct the best piece of software possible out of all available code and I'll do my best to integrate all improvement into the open source version. > I think for that matter, trying to really have upstream have a > github version and accept patches would be a good way to avoid such > scenarios; hence also why we'll be pushing the acs code there into > the safdef repo, just the easiest way to make that happen if one > wants it; but first to finish that code and stop restructuring it > and adding new features etc.... then it will be pushed there, > otherwise not much use for people looking at it anyway. I do agree that the best thing is to wait for updated code to be published before going ahead to avoid re-invention of the wheel and waste of time. But I have a contract to add some features to Stegotorus with deadlines and I think Roger has asked Vinod for the code release on March 25 and it is not clear how longer it is going to take. Nonetheless, I'm going to ask Vinod if he can have a quick look at those features and decide which one are more likely to be affected by the recent development. For example when you told me about the ACK, I moved to the other parts. Thanks for your advice and updating me about the other side. Please keep me updated of any detail that you think will help with Stegotorus development. Cheers, Vmon

1 0

atlas.torproject.org question
by me＠rndm.de 08 Jul '13

08 Jul '13

Hi there, I rebuild atlas.torproject.org using Ember.js instead of Backbone. It looks like this right now: search page: http://i.imgur.com/GEM86mk.png details page: http://i.imgur.com/1UwpwlA.png I'd like to share the code and thought about putting it on github but I don't know if it's ok for you with all the Tor assets (tor logo). Would love to hear what you think about it. thank you

7 25

Torsocks development status
by David Goulet 07 Jul '13

07 Jul '13

Hi everyone, For those who don't know, I've been working on a new version of Torsocks in the last three weeks or so. https://lists.torproject.org/pipermail/tor-dev/2013-June/004959.html I just wanted to give a quick status report on the state of the development. The DNS resolution is working for domain name (PTR) and IPv4 address. Currently, Tor does not support IPv6 resolution but the torsocks code support it. Hidden service onion address resolution is also working using a "dead IP range" acting as cookie that is sent back to the user and mapped to the .onion address on the hijacked connect(). I've changed quite a bit the configuration file (torsocks.conf) to fit the style of tor (torrc). At this point, the tor address and port can be configured as well as the "dead IP range" mention above. More is coming but pretty simple for now. Logging is working, connection registry and thread safety as well. There is also a compat layer for mutexes and once I start porting the project to other *nix system (BSD, OS X, ...) probably more subsystem will be added to that compat layer. So, in a nutshell, some libc calls still need to be implemented, *moar* tests and other OS supports. I'm confident to have a beta version to present to the community in a couple of weeks (if nothing goes wrong). Feel free to browse the code, comment on it, contribute!, etc... https://github.com/dgoulet/torsocks/tree/rewrite Cheers! David

2 2

Vidalia resources no longer accepting translations
by Runa A. Sandvik 06 Jul '13

06 Jul '13

Hi everyone, As of today, the Vidalia resources on our Transifex page [1] will no longer be accepting translations. I feel translators should not work on resources which are currently not being maintained by a developer. When/if this does change, I will happily enable translations again. [1]: https://www.transifex.com/projects/p/torproject/ -- Runa A. Sandvik

1 0

Lead Automation Engineer Job Posting
by Mike Perry 06 Jul '13

06 Jul '13

Good news, everyone! Tor is looking for a QA/automation engineer. We want to deploy nightly builds and continuous integration for as many of our key software components and platform combinations as possible. Your job would be build and deploy the initial functional versions of a wide range of testing frameworks and continuous integration systems. This is a contract position. Candidates are expected to be capable of taking the lead in selecting, deploying, and maintaining multiple automation systems in several different programming languages. Candidates should also be capable of reproducing bugs and writing new reproduction test cases for one or more of the testing frameworks. Eventually, we hope to add additional staff to assist in this project, but to start, you will be expected to prioritize your own work such that the most important tasks get attention first, without letting any specific core component starve for attention. For more details, including information on how to apply, see the job posting: https://www.torproject.org/about/jobs-lead-automation.html.en -- Mike Perry

1 0

Re: [tor-dev] Migration of Tor Weather to Stem
by Damian Johnson 05 Jul '13

05 Jul '13

> Hi, > > My name is Sreenatha. I've made an attempt to migrate Tor Weather > from TorCtl to Stem as stated in my gsoc proposal(Module 2). > > Can you please take a look at the top 3 commits that I've made at my > github account? > Please suggest any additional changes that you think are necessary. > > Cheers, > Sreenatha > > P.S : I am lucyd@OFTC on IRC. Hi Sreenatha, glad that you've started in on this! We'd much prefer for development discussions to be on tor-dev@ so looping that in. This looks like a great start. I suspect there is a little confusion though around the socket object that the Controller uses. The Controller doesn't take a socket.socket instance, but rather a stem.socket.ControlSocket... https://stem.torproject.org/api/socket.html This comes up a couple places, most notably... https://github.com/lucyd/weather/commit/ff22e74c247bd61163a974329cd2feb2a12… I doubt that function presently works. Rather, it should be... def listen(): controller = Controller.from_port(port = config.control_port) controller.authenticate(config.authenticator) controller.add_event_listener(newconsensus_listener, EventType.NEWCONSENSUS) I'm not sure what 'config.authenticator' is so that part might be wrong. Also note that this creates a control socket then forgets about it (Weather never cleans it up). This isn't a bug you've introduced but rather what Weather already did. Not the end of the world (interpreter shutdown will clean it up, though possibly with a stacktrace about lingering threads). Have you gotten a Weather instance up and running to test your changes? Cheers! -Damian

2 2

IRC meeting to discuss sponsor F progress on Wed July 3, 16:00 to 17:00 UTC in #tor-dev
by Karsten Loesing 04 Jul '13

04 Jul '13

Hi all, I'd like to schedule an IRC meeting to discuss what progress we made on sponsor F deliverables in June. Suggested time and place are: Wed July 3, 16:00 to 17:00 UTC in #tor-dev That time in other timezones is: 9:00 in San Francisco 12:00 in Boston 18:00 in Berlin 19:00 in Athens 21:30 in New Delhi People who should attend are: Roger, Nick, Andrea, George, David, Sathya, Moritz, Linus, Andrew, Tom, and anyone else who wants to attend. If your name is on that list and the suggested date or time doesn't work for you, please let me know, either here or in private mail. An alternative date would be: Tue July 2, 16:00 to 17:00 UTC in #tor-dev I'm going to send out another reminder by the end of next week that will contain the definite date. Thanks, Karsten

3 5

Metrics Plans
by Damian Johnson 01 Jul '13

01 Jul '13

Hi Kostas. Now that we no longer need to worry about accidentally leaking GSoC selection we can talk more openly about your project. Below is an interchange between me and Karsten - thoughts? ---------- Forwarded message ---------- From: Karsten Loesing <karsten(a)torproject.org> Date: Thu, May 23, 2013 at 11:37 AM Subject: Re: Metrics Plans To: Damian Johnson <atagar(a)torproject.org> Cc: Tor Assistants <tor-assistants(a)lists.torproject.org> On 5/23/13 7:22 PM, Damian Johnson wrote: > Hi Karsten. I just finished reading over Kostas' proposal and while it > looks great, I'm not sure if I fully understand the plan. Few > clarifying questions... > > * What descriptor information will his backend contain? Complete > descriptor attributes (ie, all the attributes from the documents), or > only what we need? His proof of concept importer [1] only contains a > subset but that's, of course, not necessarily where we're going. > > If we're aiming for this to be the 'grand unifying backend' for > Onionoo, Exonerator, Relay Search, etc then it seems like we might as > well aim for it to be complete. But that naturally means more work > with schema updates as descriptors change... This GSoc idea started a year back as a searchable descriptor search application, totally unrelated to Onionoo. It was when I read Kostas' proposal that I started thinking about an integration with Onionoo. That's why the plan is still a bit vague. We should work together with Kostas very soon to clarify the plan. > * The present relay search renders raw router status entries. Does it > actually store the text of the router status entries within the > database? With the new relay search I suppose we'll be retrieving the > attributes rather than raw descriptor text, is that right? The present relay search and ExoneraTor store raw text of router status entries in their databases. But that doesn't mean that the new relay search needs to do that, too. > * Kostas' proposal includes both the backend importing/datastore and > also a Flask frontend for rendering the search results. In terms of > the present tools diagram [2] I suppose that would mean replacing > metrics-web-R and having a python counterpart of metrics-db-R (with > the aim of later deprecating the old metrics-db-R). Is that right? Not quite. We cannot replace metrics-db-R yet, because that's the tool that downloads relay descriptors for all other services. It needs to work really stable. Replacing metrics-db-R would be a different project. The good thing though is that metrics-db-R offers its files via rsync, so that's a very clean interface for services using its data. In terms of the tools diagram, Kostas would write a second tool in the "Process" column above Onionoo that would feed two replacement tools for metrics-web-R and metrics-web-E. His processing tool would use data from metrics-db-R and metrics-db-E. If his tool is supposed to replace more parts of Onionoo and not only replace relay search and ExoneraTor, it would use data from metrics-db-B and metrics-db-P, too. > Maybe we should focus on a 'grand unified backend' rather than > splitting Kostas' summer between both a backend and frontend? If he > could replace the backends of the majority of our metrics services > then that would greatly simplify the metrics ecosystem. I'm mostly interested in the back-end, too. But I think it won't be as much fun for Kostas if he can't also work on something that's visible to users. I don't know what he prefers though. In my imagination, here's how the tools diagram looks like by the end of summer: - Kostas has written an Onionoo-like back-end that allows searches for relays or bridges in our archives since 2007 and provides details for any point in the past. Maybe his tool will implement the existing Onionoo interface, so that Atlas and Compass can switch to using it instead of Onionoo. - We'll still keep using Onionoo for aggregating bandwidth and weights statistics per relay or bridge, but Kostas' tool would give out that data. - Thomas has written Visionion and replacements for metrics-web-N and metrics-web-U. You probably saw the long discussion on this list. This is a totally awesome project on its own, but it's sufficiently separate from Kostas' project (Kostas is only interested in single relays/bridges, whereas Thomas is only interested in aggregates). I'm aware that not all of this may happen in one summer. That's why I'm quite flexible about plans. There are quite a lot of missing puzzle pieces in the overall picture, people can start wherever they want and contribute something useful. > I was very, very tempted to start up a thread on tor-dev@ to discuss > this but couldn't figure out a way of doing so without letting Kostas > know that we're taking him on. If you can think of a graceful way of > including him or tor-dev@ then feel free. Let's wait four more days, if that's okay for you. Starting a new discussion there about this together with Kostas sounds like a fine plan. This will be an exciting summer! :) Best, Karsten > [1] https://github.com/wfn/torsearch/blob/master/tsweb/importer.py#L16 > [2] https://metrics.torproject.org/tools.html >

3 9

Run With Limited Capabilities - GSOC
by Cristian-Matei Toader 01 Jul '13

01 Jul '13

Hello, My name is Cristian Toader, and I feel very excited about designing and implementing a capabilities based sandbox for the central Tor project, as part of the GSOC program. ---- About myself: I have been a Linux enthusiast for almost 6 years and have first started using Tor around 3 years ago. I am currently studying in the UK. In approximately one month I will be graduating the Computer Science programme at the University of Surrey, and plan on pursuing a master's degree in Advanced Computer Science at the University of Cambridge for the following academic year. I have completed a placement year at Qualcomm (UK) LTD which involved implementing and testing security solutions for the Linux Android OS. These were based on cryptography and the TrustZone run-mode of the ARM processors. Most of the development during the placement year was performed in C, with some tests written in Java and C++ for the upper layers. ---- About the project: The project I will be working on as part of GSOC is based on the "Run With Limited Capabilities" proposal [1] mentored by Nick Mathewson (nickm) and Andrea Shepard (athena). The project is still in the planning stage. I will start working on an appropriate design as soon as I finish my last exams, which is the 3rd of June. As part of the project I will need to: - investigate research papers regarding capability based sandboxes - get familiar with the Tor code structure - decide on whether there should be different states starting from which the tor program only has a limited set of capabilities, depending on what syscalls it should be able to perform; or maybe have a more complex approach based on a trusted process representing a root of trust (with no network interactions) which controls the capabilities of the processes it launches - integrate an appropriate solution - develop and run tests for the project A constraint agreed with nickm, would be that once the program capabilities are set they should not be modifiable (otherwise a potential attacker could have the option of first enabling capabilities and then execute privileged code). Some additional details can be found in tickets #7005 [2], #5219 [3], and #5220 [4]. I will try to keep everyone updated. I am looking forward to advice and suggestions. If anyone needs to contact me, this is my primary email, my irc.oftc.net username is ctoader, and I am geographically located in GMT+2. Best regards, Cristian Toader. [1] https://www.torproject.org/getinvolved/volunteer.html.en#limitCapabilities [2] https://trac.torproject.org/projects/tor/ticket/7005 [3] https://trac.torproject.org/projects/tor/ticket/5219 [4] https://trac.torproject.org/projects/tor/ticket/5220

5 6

[GSOC] Status report - Tor capabilities
by Cristian-Matei Toader 01 Jul '13

01 Jul '13

Hello tor-dev, As a small reminder the purpose of the project is to create capabilities based sandboxing for Tor, which may only allow the program to execute a number of predefined syscalls. For the past 2 weeks: - I have consulted with Nick Mathewson (nickm) and agreed upon using seccomp2 [1], and more recently a library built on top of that called libseccomp [2]. - I have set up a public remote branch [3]. - We have agreed on a 3 step plan for the project: 1. General sandbox based on a single (permisive) filter which restricts tor to using a number of syscalls. 2. Add configuration option for step 1, if any parts were broken in phase 1 by adding capabilities, they can be re-enabled at the cost of security. 3. Figure out what functionality should be split into separate processes, based on our experience from step 1 and step 2. - So far I have implemented step 1 using both libseccomp and seccomp2 [3]. Step 1 was developed in such a way that nothing from tor should be broken at the moment; What this means is that sandboxing currently exists in the remote branch, but is fairly coarse and will need some fine tuning at a later stage such as only allowing specific files to be open, or allowing the exec syscall to be called with specific parameters. These days I will be adding command line support, which is basically step 2, which will be followed by a code review and merge in the master branch. [1] http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-precise.git;a=blob;f=Documenta… [2] http://sourceforge.net/projects/libseccomp/ [3] https://github.com/cristiantoader/tor-gsoc-capabilities

2 2