[tor-dev] [OONI] Designing the OONI Backend (OONIB). RESTful API vs rsynch

[Arturo Filastò]

On 7/16/12 2:15 AM, Ondrej Mikle wrote:
> On 07/15/2012 02:56 PM, Arturo Filastò wrote:
>> I would like to follow up on the discussion we had in Florence on some
>> design choices behind OONIB.
>>
>> In particular the most controversy was around using HTTP or rsync.
> [...]
>
>> # What properties we would like it to have
>> note: these are not ordered.
>>
>> * Efficient even over high latency networks.
>>
>> * Ease of integration for third party developers.
>>
>> * Expandable to support requirements of new tests we develop.
>>
>> * Anonymous
>>
>> * Secure
> Even though you will probably not end up using this, it may be a good idea to
> know that it exists:
>
> ZeroC Ice - http://www.zeroc.com/ice.html

There are a bunch of very fancy and nice libraries out there to do RPC like
things that support a variety of languages.

Another one I am very fond of is ZeroMQ, but I think we should not start
worrying
about supporting such advanced and scalable libraries at this time.

This is the reason why I stated at the beginning of the requirements
that we should
"so we will look at it as if it were running only on one central machine".

The scalability issues will be dealt with once we have properly defined
the problem
scope. Making the node communication rely on Zero(C|MQ) can be something
that we integrate
without having to require clients to change their behavior.

I think the overall feeling from the responses is that going for
something like an HTTP
RESTful API is what we are looking for.

HTTP is a well understood technology and I have quite some experience in
designing and
building RESTful APIs based on twisted (cyclone) and this is also what
is used in other [1]
projects [2] belonging [3] to the [4] Tor community [5].

Moreover by looking at how the reporting systems of other network
measurements tools
worked [5] we found that almost all of them used an HTTP API to collect
reports [6][7][8][9].
I see this as an indication that such a strategy is the best practice.

For the time being we should go for something simple like this and once
we encounter major
scalability/performance bottlenecks we can quantify them and figure out
what the best
path to a solution may be.

If you were the developer of a censorship detection tool would you like
to have to report
to anything that is not a RESTful HTTPs API?

- Art.

[1] https://github.com/mmaker/APAF
[2] https://github.com/gsathya/pyonionoo
[3] https://github.com/globaleaks/Tor2web-3.0
[4] https://github.com/globaleaks/GLBackend
[5]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools
[6]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/Herdict#Howdoesthereportingsystemwork
[7]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/Netalyzr#Howdoesthereportingsystemwork
[8]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/NeuBot#Reportingsystem
[9]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/ProjectBismark#Reportingsystem