Hi Kostas. Now that we no longer need to worry about accidentally
leaking GSoC selection we can talk more openly about your project.
Below is an interchange between me and Karsten - thoughts?
---------- Forwarded message ----------
From: Karsten Loesing <karsten(a)torproject.org>
Date: Thu, May 23, 2013 at 11:37 AM
Subject: Re: Metrics Plans
To: Damian Johnson <atagar(a)torproject.org>
Cc: Tor Assistants <tor-assistants(a)lists.torproject.org>
On 5/23/13 7:22 PM, Damian Johnson wrote:
> Hi Karsten. I just finished reading over Kostas' proposal and while it
> looks great, I'm not sure if I fully understand the plan. Few
> clarifying questions...
>
> * What descriptor information will his backend contain? Complete
> descriptor attributes (ie, all the attributes from the documents), or
> only what we need? His proof of concept importer [1] only contains a
> subset but that's, of course, not necessarily where we're going.
>
> If we're aiming for this to be the 'grand unifying backend' for
> Onionoo, Exonerator, Relay Search, etc then it seems like we might as
> well aim for it to be complete. But that naturally means more work
> with schema updates as descriptors change...
This GSoc idea started a year back as a searchable descriptor search
application, totally unrelated to Onionoo. It was when I read Kostas'
proposal that I started thinking about an integration with Onionoo.
That's why the plan is still a bit vague. We should work together with
Kostas very soon to clarify the plan.
> * The present relay search renders raw router status entries. Does it
> actually store the text of the router status entries within the
> database? With the new relay search I suppose we'll be retrieving the
> attributes rather than raw descriptor text, is that right?
The present relay search and ExoneraTor store raw text of router status
entries in their databases. But that doesn't mean that the new relay
search needs to do that, too.
> * Kostas' proposal includes both the backend importing/datastore and
> also a Flask frontend for rendering the search results. In terms of
> the present tools diagram [2] I suppose that would mean replacing
> metrics-web-R and having a python counterpart of metrics-db-R (with
> the aim of later deprecating the old metrics-db-R). Is that right?
Not quite. We cannot replace metrics-db-R yet, because that's the tool
that downloads relay descriptors for all other services. It needs to
work really stable. Replacing metrics-db-R would be a different
project. The good thing though is that metrics-db-R offers its files
via rsync, so that's a very clean interface for services using its data.
In terms of the tools diagram, Kostas would write a second tool in the
"Process" column above Onionoo that would feed two replacement tools for
metrics-web-R and metrics-web-E. His processing tool would use data
from metrics-db-R and metrics-db-E.
If his tool is supposed to replace more parts of Onionoo and not only
replace relay search and ExoneraTor, it would use data from metrics-db-B
and metrics-db-P, too.
> Maybe we should focus on a 'grand unified backend' rather than
> splitting Kostas' summer between both a backend and frontend? If he
> could replace the backends of the majority of our metrics services
> then that would greatly simplify the metrics ecosystem.
I'm mostly interested in the back-end, too. But I think it won't be as
much fun for Kostas if he can't also work on something that's visible to
users. I don't know what he prefers though.
In my imagination, here's how the tools diagram looks like by the end of
summer:
- Kostas has written an Onionoo-like back-end that allows searches for
relays or bridges in our archives since 2007 and provides details for
any point in the past. Maybe his tool will implement the existing
Onionoo interface, so that Atlas and Compass can switch to using it
instead of Onionoo.
- We'll still keep using Onionoo for aggregating bandwidth and weights
statistics per relay or bridge, but Kostas' tool would give out that data.
- Thomas has written Visionion and replacements for metrics-web-N and
metrics-web-U. You probably saw the long discussion on this list. This
is a totally awesome project on its own, but it's sufficiently separate
from Kostas' project (Kostas is only interested in single
relays/bridges, whereas Thomas is only interested in aggregates).
I'm aware that not all of this may happen in one summer. That's why I'm
quite flexible about plans. There are quite a lot of missing puzzle
pieces in the overall picture, people can start wherever they want and
contribute something useful.
> I was very, very tempted to start up a thread on tor-dev@ to discuss
> this but couldn't figure out a way of doing so without letting Kostas
> know that we're taking him on. If you can think of a graceful way of
> including him or tor-dev@ then feel free.
Let's wait four more days, if that's okay for you. Starting a new
discussion there about this together with Kostas sounds like a fine plan.
This will be an exciting summer! :)
Best,
Karsten
> [1] https://github.com/wfn/torsearch/blob/master/tsweb/importer.py#L16
> [2] https://metrics.torproject.org/tools.html
>
Hello,
My name is Cristian Toader, and I feel very excited about designing and
implementing a capabilities based sandbox for the central Tor project, as
part of the GSOC program.
----
About myself:
I have been a Linux enthusiast for almost 6 years and have first started
using Tor around 3 years ago.
I am currently studying in the UK. In approximately one month I will be
graduating the Computer Science programme at the University of Surrey, and
plan on pursuing a master's degree in Advanced Computer Science at the
University of Cambridge for the following academic year.
I have completed a placement year at Qualcomm (UK) LTD which involved
implementing and testing security solutions for the Linux Android OS. These
were based on cryptography and the TrustZone run-mode of the ARM
processors. Most of the development during the placement year was performed
in C, with some tests written in Java and C++ for the upper layers.
----
About the project:
The project I will be working on as part of GSOC is based on the "Run With
Limited Capabilities" proposal [1] mentored by Nick Mathewson (nickm) and
Andrea Shepard (athena). The project is still in the planning stage. I will
start working on an appropriate design as soon as I finish my last exams,
which is the 3rd of June.
As part of the project I will need to:
- investigate research papers regarding capability based sandboxes
- get familiar with the Tor code structure
- decide on whether there should be different states starting from which
the tor program only has a limited set of capabilities, depending on what
syscalls it should be able to perform; or maybe have a more complex
approach based on a trusted process representing a root of trust (with no
network interactions) which controls the capabilities of the processes it
launches
- integrate an appropriate solution
- develop and run tests for the project
A constraint agreed with nickm, would be that once the program capabilities
are set they should not be modifiable (otherwise a potential attacker could
have the option of first enabling capabilities and then execute privileged
code).
Some additional details can be found in tickets #7005 [2], #5219 [3], and
#5220 [4].
I will try to keep everyone updated. I am looking forward to advice and
suggestions. If anyone needs to contact me, this is my primary email, my
irc.oftc.net username is ctoader, and I am geographically located in GMT+2.
Best regards,
Cristian Toader.
[1]
https://www.torproject.org/getinvolved/volunteer.html.en#limitCapabilities
[2] https://trac.torproject.org/projects/tor/ticket/7005
[3] https://trac.torproject.org/projects/tor/ticket/5219
[4] https://trac.torproject.org/projects/tor/ticket/5220
Hello tor-dev,
As a small reminder the purpose of the project is to create
capabilities based sandboxing for Tor, which may only allow the
program to execute a number of predefined syscalls.
For the past 2 weeks:
- I have consulted with Nick Mathewson (nickm) and agreed upon using
seccomp2 [1], and more recently a library built on top of that called
libseccomp [2].
- I have set up a public remote branch [3].
- We have agreed on a 3 step plan for the project:
1. General sandbox based on a single (permisive) filter which
restricts tor to using a number of syscalls.
2. Add configuration option for step 1, if any parts were broken
in phase 1 by adding capabilities, they can be re-enabled at the cost
of security.
3. Figure out what functionality should be split into separate
processes, based on our experience from step 1 and step 2.
- So far I have implemented step 1 using both libseccomp and seccomp2
[3]. Step 1 was developed in such a way that nothing from tor should
be broken at the moment; What this means is that sandboxing currently
exists in the remote branch, but is fairly coarse and will need some
fine tuning at a later stage such as only allowing specific files to
be open, or allowing the exec syscall to be called with specific
parameters.
These days I will be adding command line support, which is basically
step 2, which will be followed by a code review and merge in the
master branch.
[1] http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-precise.git;a=blob;f=Documenta…
[2] http://sourceforge.net/projects/libseccomp/
[3] https://github.com/cristiantoader/tor-gsoc-capabilities
Hi there,
During the first two weeks of my GSoC project, I have implemented a HTTP CONNECT-based pluggable transport. In short, I use HTTP CONNECT semantics to establish a secure channel between the client and the bridge. Specifically, this is the scenario:
1. Connection establishment:
+------------------------------+
|CONNECT bridge_ip:443 HTTP/1.0|
|User-agent: blablabla |
+------------------------------+
|
|
+----------+ | +----------+ +----------+
| client |----->| proxy |----->| bridge |
+----------+ +----------+ | +----------+
|
|
+------------------------------+
| (Establish a connection) |
+------------------------------+
2. Data relay
+------------------------------+
| (Encrypted Payload) |
+------------------------------+
|
|
+----------+ | +----------+ +----------+
| client |<----->| proxy |<----->| bridge |
+----------+ +----------+ | +----------+
|
|
+------------------------------+
| (Encrypted Payload) |
+------------------------------+
I hope the diagrams above are self-explanatory. It is only my initial attempt to get familiar with HTTP protocol. Once I make sure it is working correctly under squid proxy I will upload it to the repository.
The use of CONNECT method is restricted in many networks, so it is better to implement the HTTP transport using the usual HTTP methods as POST, GET, etc. In the next stage, I plan to implement a new HTTP transport based on BOSH[1]. There are many issues to care about (in order of priority):
* bi-directional data transfer over HTTP
* proxy cache
* HTTP request/response encoding
* encryption
* scanning resistance
Have a nice weekend!
[1]: http://xmpp.org/extensions/xep-0124.html
Best,
Chang
Sent from my mobile device. Sorry for the brevity.
> This is rather exciting. Do you think that this method can be adapted to
> build the pluggable transports bundles?
Yes.
I originally started working on this so I could submit a patch to include
the tor-fw-helper in the TBB.
But since the helper is only needed in the PT bundles,
reworking their build process seems the more productive route.
I'll see what I can do.
> This is currently done
> semi-manually, with a makefile that builds the pluggable transports,
> unzips the bundle, copies in some files, and re-zips the bundle.
>
> https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/Ma…
>
> We also have instructions for setting up a build VM from scratch--which
> it sounds like is what Vagrant does--but I guess we would instead use
> whatever is the build machine for the vanilla bundle.
>
> https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/Ma…
>
> It would solve a lot of problems for us to have the PT bundles built at
> the same place and time as the vanilla bundles.
Hey friends,
i want to inform you that i am working on a complete new control tool
for Tor. I know some people use Vidalia for it, but i never liked the
program and the people i showed vidalia often got confused.
So i started to rewrite it from scratch and i am close to the release of
the source code. (Currently i need to clean up some classes in there)
I had a few things on my feature list for the new control tool and i
want to share/discuss it with you. Because maybe some of my ideas are
bad or maybe you have some additional ideas for features.
A: "native work flow". Thats one of the main goals of my rewrite. The
current Vidalia does things a little bit different than people are used
to do it. So they get confused, because it does not behave "naturally".
So instead of QT i used GTK and followed as always the GNOME HIG.
(http://developer.gnome.org/hig-book/3.5/)
B: "Multi configuration setup". What do i mean by that? Well, in my case
for example i use Tor on my laptop and i am often in different places
where i need different settings for tor. For example, if i am at home i
can give tor all the bandwith i have, because at home i don't care. In
my company on the other hand i cant do that. I am allowed to run Tor in
the company, but with a bandwith limit of 2000kbit/s. (Yes, my company
allows this ... well, i am the technical directory, so it was my call :) )
But i need a different Tor configuration there. Or if i am giving a talk
and only have mobile internet connections i need every bit of bandwith
for checking my emails so i want no server, only Tor client support.
Currently i have multiple torrc files for that and always move them
arround. That works for command line junkies like me, but even i am
anoyed to do this multiple times a day.
So i added the "multi configuration" support into the tool. This allows
you to define multiple configurations and simply change them with a
click from the gnome2 notification panel.
C: "GUI for config settings". Well, some users are simply not ready to
modify config files on there own. So i provide a simple gui (tab based)
for all config options i found in the normal torrc file. Always with an
explonation (with needs spellchecking, because my spelling sucks in
german and is even worse in english :) ) so the user can deside what an
option would to.
D: "Notifications": My control tool uses the normal desktop
notifications to inform the user about changes in the running tor
application. For example if there is a problem building the circuit or
something you get the notifyd message and can simply react to events
without having the arm log open all the time.
Thats just a quick overview of what i am duing right now. The source
code will be released soon and is of course 100% open source. If you
have some addiional ideas, feedback, wishes, ... i would like to hear them.
Thanks and greetings
Leo
PS: I am Germam and very sorry for every spelling error i did above.
--
Leo Unglaub
Nachreihengasse 39 / Top 7
A-1170 Vienna
Austria
Phone: 0650 / 6575103
Website: http://www.leo-unglaub.net
Twitter: http://twitter.com/LeoUnglaub
Send with Mozilla Thunderbird on Linux (Debian)
Hi all,
In the first 4 weeks of my GSOC project steganography browser add-on,
I have implemented the basic UI parts in first two weeks, and later I
spent more time on web content downloading part. Up to now, I have
finished downloading image and get them as bytes. Now when you right
click on an image and click on test button, you will be getting the
byte format of image which can be used with steganography algorithms
to write messages. Since I'm mainly concentrating on features rather
than algorithms now,
my plan for next 2 week is to extend the file downloading for video
and sound files.
You can find the source code under git repository [1].
[1] https://github.com/rharishan/Steganography-Browser/
--
Hareesan
Ian Goldberg:
> On Wed, Jun 26, 2013 at 03:55:58PM -0400, David Goulet wrote:
>> Hi everyone,
>>
>> For those who don't know, I've been working on a new version of Torsocks
>> in the last three weeks or so.
>>
>> https://lists.torproject.org/pipermail/tor-dev/2013-June/004959.html
>>
>> I just wanted to give a quick status report on the state of the development.
>>
>> The DNS resolution is working for domain name (PTR) and IPv4 address.
>> Currently, Tor does not support IPv6 resolution but the torsocks code
>> support it.
>>
>> Hidden service onion address resolution is also working using a "dead IP
>> range" acting as cookie that is sent back to the user and mapped to the
>> .onion address on the hijacked connect().
>>
>> I've changed quite a bit the configuration file (torsocks.conf) to fit
>> the style of tor (torrc). At this point, the tor address and port can be
>> configured as well as the "dead IP range" mention above. More is coming
>> but pretty simple for now.
>>
>> Logging is working, connection registry and thread safety as well. There
>> is also a compat layer for mutexes and once I start porting the project
>> to other *nix system (BSD, OS X, ...) probably more subsystem will be
>> added to that compat layer.
>>
>> So, in a nutshell, some libc calls still need to be implemented, *moar*
>> tests and other OS supports. I'm confident to have a beta version to
>> present to the community in a couple of weeks (if nothing goes wrong).
>>
>> Feel free to browse the code, comment on it, contribute!, etc...
>>
>> https://github.com/dgoulet/torsocks/tree/rewrite
>
> Are non-blocking sockets, select/poll/etc. (especially at connect()
> time), and optimistic data on the to-do list?
Yes! Good point I should have put the todo list. So yes, non block socket support.
For optimistic data, it is kind of tricky. I can use it for DNS resolution
without a problem because torsocks control the complete flow of data from
opening a SOCKS5 connection to closing it after the DNS response is received
however for actual real data (sendmsg, send, ...) a connect is needed before so
it would means that a connect() call will return "yes OK socket connected" but
where in fact it is not really true. So, when the first data are sent, there is
a possibility that the Tor connections failed or even we block for an unknown
amount of time during the send*/write() call.
Now the question is, is this the kind of behavior that would be acceptable
meaning basically lying to the caller at connect() and possibly blocking I/O
calls and returning something like ECONNRESET or ENOTCONN if the Tor socks5
connection fails.
This is *real* tricky especially with non blocking socket, if torsocks needs to
do some possible blocking call for the SOCKS5 replies during an I/O call from
the caller that is not suppose to block. Furthermore, having pending data that
*might* come at any time on the connection from the SOCKS5 negotiation, the
caller could put the file descriptor in poll() mode, wake up and try to receive
the data but where in fact it's the socks5 reply... it's possible to handle that
but it seems here a VERY intrusive behavior. Does optimistic data worth it here
vis-a-vis the complexity of handling that it and high intrusiveness ?
Cheers!
David
>
> Thanks,
>
> - Ian
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi *,
The first two weeks of working on my GSoC project[0], which has been
dubbed "EvilGenius", are nearly over. and I just wanted to share a bit
of the experience I made during that time.
EvilGenius is a tool that - much like Descartes' idea of an "evil
genius", it "presents a complete illusion of an external world,
including other minds, to Descartes' senses, where in fact there is no
such external world in existence." (Source: Wikipedia), it distorts
the guests' view of the real internet, in short, it creates a virtual
environment that simulates censorship in order to verify OONI's
nettest modules.
So, after getting started with vagrant, the virtualization abstraction
layer used for my project, i started simulating a virtual netwotrk and
configuring the hosts to act as if they were connected to each other
and the rest of the internet through a router that is part of the
simulated environment.
the "precise32" box is used as base image for all nodes, and I created
scripts that configure the network and packed it up in a nice vagrantfile.
Then I began deploying ooniprobe and oonibackend and executed the
first successful tests.
Next week, I will begin working on censoring environments.
Have a nice weekend everyone!
Johannes
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJRzKDWAAoJEAgie5o3Q/JpUhAQAIt2WEpDsNyLVDuSdMBM0ETf
q0sOxdx20Q6o77J93FJ4ZBJRzpBX4mlsE4t22Dl/TPgnHu3b+DG+dvVhfCBrQOqw
wGUuA6/X/khCikAOo+f3T9fNtaHKw54phW1Tw9Yj3Gea7SBN5ZikfXKjIIHOT+lu
n0xqWVzxAxqGOETy0byDznsdgc2OfaR2NkdxzCsLlZTLtE75h/EQ3U96ZCej8lw/
xmZZUD+q+4ZCE6bBKTebufQeh0eI/rUPgo0UX1rNLgUapfCs9oLVIF+8nwBKBU0j
E2myEOUbXMCxtTURbHikRQKzGKlsXMCEqqSatik37Rc3VaeVdkxNiiyXFVRwzTr0
IwRTibzJ8hD+aqh9d9VDD6s7dIdRio42p+XzD8jswFltEyYcbBGeAFFiybdNeeE2
EFb7n7iHz7Gr1csezyma1fdqvvMHZaaeCMTqeRENQtBaazOnqDwrGJn83r/FC7QF
m8i+3eVDm8suM7CzYvuYi8YSIhF7yMVerIBOcBSZ2A929SVKtQFT4by1V2Jc4pz+
0ZJV4teS8xct0wjhpDLANBCXgnV07OO6Z6EgLhcWpJBWoZDONDaxikH7clmTlzdg
w/9ZktL3Ac3PjhSdMjOI/TKqc68+CHQ7ln1mG+YtSimMjf5Abpx1wPgLLILmzFNS
lpZ2vev2pW0DsCnvs946
=abNV
-----END PGP SIGNATURE-----
