Hi,

so you were right the databases were corrupt, but they shouldn't have been
there in the first place. :-)


I didn't want to include large files in the git repo (~120mb in total), so
there is a generation script in the tools directory (now added).

I added some instructions on the readme file too. Of course, this procedure
is done only once.


I tried it and it should work now, but let me know if there is still a
problem.

There is a plan to optimize the way the pre-computed values are used, and
this could potentially shrink the size of the databases by a lot.

Vasilis

On 17 October 2015 at 15:56, str4d <str4d@i2pmail.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> str4d wrote:
> > Vasilios Mavroudis wrote:
> >> Hello,
> >
> >> I would like to introduce our project "Crux", which enables the
> >> computation of privacy preserving statistics on sensitive data.
> >> The project was developed at University College London (UCL) by
> >> me, in close collaboration with George Danezis.
> >
> > This is fantastic work. Haven't read all of your thesis yet, but
> > it looks similar to the stats collection system that we ran for a
> > while inside I2P, but with the privacy protections that I wanted it
> > to have. Well done!
> >
> >
> >> "Crux" was designed with Tor hidden services in mind (in
> >> accordance with this
> >> <https://research.torproject.org/techreports/hidden-service-stats-201
> 5
> >
> >>
> - -04-28.pdf>
> >
> >
> > Tor
> >> Tech Report), but it can be applied on various kinds of data
> >> (given an appropriate parser).
> >
> > Excellent. I want to use this in I2P :)
> >
> >
> >> It supports three statistics (i.e. mean, median, variance) and
> >> its threat model is compatible Tor's (see technical document
> >> below).
> >
> > I will look into this, but I doubt there will be any
> > insurmountable incompatibilities with I2P's threat model.
> >
> >> Currently, we don't have a parser specific for the data
> >> collected by the Tor relays, but we plan to keep adding
> >> functionality.
> >
> >> The source code can be found here:
> >> https://github.com/mavroudisv/Crux The theoretical background,
> >> threat model, security argument and technical details can be
> >> found here: link <http://mavroudisv.eu/files/thesis.pdf>
> >
> >
> >> Ideas, comments, feedback much appreciated!
> >
> > Firstly: needs more docs. I shouldn't have to read your entire
> > thesis in order to figure out how to run it :P The README is a good
> > start, but the command parameters need clarification/definitions,
> > and it is not at all obvious how the relays work. An XLS file is
> > also mentioned without explanation.
>
> I have now tried running the code, but the "unit test" run by
> authority.py fails because bsbdb.btopen() returns an empty dict. I
> don't know if this is because the table files are broken or because of
> a local issue with my setup (I am using a virtualenv, but I get the
> same empty dict when I try to open the tables manually outside the
> virtualenv). I am not going to try and fix the latter on my own,
> because bsddb is deprecated since Python 2.6, so you shouldn't really
> be using it :) (But it must be something else, because I tried using
> bsddb3 instead and that also returns an empty dict).
>
> str4d
>
> >
> > Secondly, you could probably simplify configuration by managing
> > hidden service connections yourself. I believe the Stem library can
> > do this? I2P has a python library that can be used to listen on and
> > connect to I2P Destinations, with a socket-like interface.
> >
> > It's not clear to me from your thesis whether you plan to merge
> > the relay component into the Tor relay code (and have the Tor
> > relays talk directly to the query processors), or have your relays
> > running separately and interfacing with the Tor relays to fetch
> > stats. This will potentially alter how it would be used with other
> > networks. In I2P's case I'd imagine it would be easiest (in terms
> > of usability by those setting up relays) to write an I2P plugin in
> > Java that performs the relay role, but I wouldn't want to be
> > duplicating a lot of logic that subsequently needs to be kept
> > in-sync with upstream :)
> >
> > str4d
> >
> >
> >> Vasilis
> >
> >
> >
> >> _______________________________________________ tor-dev mailing
> >> list tor-dev@lists.torproject.org
> >> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
> >
> > _______________________________________________ tor-dev mailing
> > list tor-dev@lists.torproject.org
> > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
> >
> -----BEGIN PGP SIGNATURE-----
>
> iQIcBAEBCgAGBQJWItIgAAoJEBO17ljAn7PglsQP/3tOrL9EZfYMb5fxxVI6/wJm
> qDvA8RpbEJDhoMjOc9sLNyMVgvGGPxEdma8I/R8H37yIPOp2GJg82MoXfUk+AGFe
> 5ZIIXwf2EzHH0RFLRI5xaQ4DW3Ru1yIzH+E3vhuGclL87YRKnAM0xkMQcueR5xo5
> /ETe9TI7oPCdETI4J6iFoM+Ob9qsRiEVeuT5HEFFKwQhBvl0NSpfMHV9ftuxQHuu
> I2B9jomnBdzF9OXnSQrYXg3ho2rxX4hO1TZfQHg5DgHvkt8TQYQM3cwdGE7/rsqC
> d4+kBW2RuAz68wq+fLPeeZ2U2ff2QqmDLqVyUJvpP5RqeUcv66J/iYm7md886tTc
> palwVGodKiTJ6DBM0z1K97x6Sl1jJXEAkZQAr2yQ8jrC/BJqQRPu2vHeZILqbMEQ
> UU3JLjIk/Jc2RGK6zitvj4kTrLdfACEQMYy3brQUawZhaM42QNQgb3LQq1bJXzUo
> vISZckX/CVfF+R63oO085LYOO7oTLyY3EhA1PCXiV09Usl/To66Oh4AVYtz/lPLH
> 6uoWR/ic5X9gFIXt1t+617+i/L/ILMfqkqN7GzoSbQ8QfyBJrjIikuj575M7AH3o
> RuhRYHUTmaCRL+9niDFG/mSVGBxBqiSfW1eS5pGE3hIAcLQd0eFT/38HRdonkfLP
> Ug5ihMFSKfKmCIljQ6mm
> =jDmS
> -----END PGP SIGNATURE-----
>