March 2014 - tor-dev - lists.torproject.org

[Orbot patch] Properly skip tests when building libevent
by Daniel Martí 27 Mar '14

27 Mar '14

This fixes the remaining $(top_srcdir) errors and properly generates event2/event-config.h Signed-off-by: Daniel Martí <mvdan(a)mvdan.cc> --- external/Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/external/Makefile b/external/Makefile index bce8d83..7a9f637 100644 --- a/external/Makefile +++ b/external/Makefile @@ -116,6 +116,8 @@ openssl-clean: # libevent libevent/Makefile: + sed -i 's@$SUBDIRS = . include$ sample test@\1@' libevent/Makefile.am + sed -i 's@$AC_OUTPUT(Makefile include/Makefile$ test/Makefile sample/Makefile)@\1)@' libevent/configure.in cd libevent && ./autogen.sh cp config.sub libevent cp config.guess libevent @@ -126,7 +128,7 @@ libevent/Makefile: --disable-shared libevent-build-stamp: libevent/Makefile - $(MAKE) -C libevent all-am + $(MAKE) -C libevent ./include/event2/event-config.h all-am touch libevent-build-stamp libevent: openssl libevent-build-stamp -- Daniel Martí - mvdan(a)mvdan.cc - http://mvdan.cc/ PGP: A9DA 13CD F7A1 4ACD D3DE E530 F4CA FFDB 4348 041C

2 2

Re: [tor-dev] StegoTorus gsoc2014
by vmon＠riseup.net 27 Mar '14

27 Mar '14

Hey Amin, Good to hear from you. As the first step try to download the latest stegotorus code: git clone https://github.com/zackw/stegotorus cd stegotorus git checkout tor-improve and try to compile it: autoreconf ./configure PKG_CONFIG_PATH=`pwd`&& make When you are successfully able to build it then, then I'll ask you to write a minor patch for Stegotorus to test your ability for being a successful GSoC candidate. > I will send the edited proposal to you in order to discuss it. > Again, I am so sorry for the delay. Sure, It is OK to submit a proposal to address the open tasks, however you need to show in your proposal that you understood the tasks and you have some idea about how to resolve them. Best of luck, vmon AMIN SLEIMI <amin.sleimi(a)gmail.com> writes: > Hi vmon, > > I would like to thank you for your comment and I apologize for the > delay. For my personal experience, I joined a CV summarizing my > studies and projects. I am currently hacking StegoTorus's source code > and I would enhance my proposal according to this study along with the > paper "StegoTorus: A Camouﬂage Proxy for the Tor Anonymity System". > > I will send the edited proposal to you in order to discuss it. > Again, I am so sorry for the delay. > > Regards, > > > Amin Sleimi.

1 0

Re: [tor-dev] Stegotorus tickets
by vmon＠riseup.net 27 Mar '14

27 Mar '14

Hey Noah, I've updated the tickets: 8098, 8096. I don't know much about 8090, it is not coding related and hence seems irrelevant to GSoC. I made a ticket for the task we discussed as well: #11337: Reimplement (move relevant functions, delete extra redundant code) of pdfSteg.cc, swfSteg.cc, jsSteg.cc as children of FileStegMod] If you want me to update any other specific ticket to current status of Stegotorus please give me the specific ticket number (I'm a bit short of time in the congestion of end-of-month tasks). If you'd like us to go over the code related to your task together, schedule a meeting time using Google calendar: vmonmoonshine(a)gmail.com And we can discuss it on #tor-dev(a)irc.oftc.net Good luck, vmon Noah Rahman <selimthegrim(a)gmail.com> writes: > In the interim, I'll start looking at 8090, 8098, 8096 for the > challenge period. I presume the security factor still has some > improvements left to do and if the PDF/JS/SWF modules need to be > refactored I'll count it under this. > > On Wed, Mar 26, 2014 at 12:24 AM, Noah Rahman <selimthegrim(a)gmail.com> > wrote: > > Did you get around to updating the trac yet? > > > > > > On Tue, Mar 25, 2014 at 7:40 AM, Noah Rahman > <selimthegrim(a)gmail.com> wrote: > > > do you mean the refactoring and the config files, or the key > handshake? > > > > > > On Tue, Mar 25, 2014 at 3:34 AM, <vmon(a)riseup.net> wrote: > > Hey Noah, > > It seems that you successfully compiled Stegotorus, > congratulation! ;) > > Although, you still have problem with compiling the unit > tests in google > framework, they are not essential in solving your > challenge. I recommend > that you start working on your challenge now. > > Per Google's recommendation, I've included the google unit > test code > inside the code, so that g_unittests_LDFLAGS probably won't > help > (because you are linking to the object file directly). You > might want to > download the google frame work and try to compile it > independently > without the stegotorus headache: > > https://googletest.googlecode.com/files/gtest-1.7.0.zip > > Keep the good work going! > vmon > > Noah Rahman <selimthegrim(a)gmail.com> writes: > > > after fixing the local branch and having to remove - > Werror by hand > > from the makefile yet again...maybe I am missing > something else in the > > unit tests? I checked git branch -v and it now matches > the tor-improve > > on remote/tor-improve > > > > selimthegrim@Rimsky-Korsakoffee:~/stegotorus$ make -j2 > > make all-am > > make[1]: Entering directory > `/home/selimthegrim/stegotorus' > > depbase=`echo src/steg/payload_scraper.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/steg/payload_scraper.o -MD -MP -MF > $depbase.Tpo -c > > -o src/steg/payload_scraper.o > src/steg/payload_scraper.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > /bin/bash ./src/genmodtable.sh protolist.cc > src/protocol/chop.cc > > src/protocol/chop_blk.cc src/protocol/null.cc > > protolist.cc is unchanged > > touch stamp-protolist > > /bin/bash ./src/genmodtable.sh steglist.cc > src/steg/b64cookies.cc > > src/steg/cookies.cc src/steg/embed.cc src/steg/http.cc > > src/steg/http_apache.cc > src/steg/http_steg_mods/file_steg.cc > > src/steg/http_steg_mods/pdfSteg.cc > src/steg/http_steg_mods/swfSteg.cc > > src/steg/http_steg_mods/jsSteg.cc > src/steg/http_steg_mods/jpgSteg.cc > > src/steg/http_steg_mods/pngSteg.cc > src/steg/http_steg_mods/gifSteg.cc > > src/steg/nosteg.cc src/steg/nosteg_rr.cc > src/steg/payload_server.cc > > src/steg/trace_payload_server.cc > src/steg/payload_scraper.cc > > src/steg/apache_payload_server.cc > > steglist.cc is unchanged > > touch stamp-steglist > > depbase=`echo src/pgen_fake.o | sed 's| > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/pgen_fake.o -MD -MP -MF $depbase.Tpo > -c -o > > src/pgen_fake.o src/pgen_fake.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > /bin/bash ./src/test/genunitgrps.sh unitgrplist.cc > > src/test/unittest_base64.cc > src/test/unittest_compression.cc > > src/test/unittest_crypt.cc src/test/unittest_pdfsteg.cc > > src/test/unittest_socks.cc > > touch stamp-unitgrplist > > depbase=`echo src/test/tltester.o | sed 's| > [^/]*$|.deps/&|;s|\.o$||' > > `;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/tltester.o -MD -MP -MF > $depbase.Tpo -c -o > > src/test/tltester.o src/test/tltester.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/webpage_tester.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/webpage_tester.o -MD -MP -MF > $depbase.Tpo -c - > > o src/test/webpage_tester.o src/test/webpage_tester.cc > &&\ > > mv -f $depbase.Tpo $depbase.Po > > src/steg/payload_scraper.cc: In member function ‘int > > PayloadScraper::scrape()’: > > src/steg/payload_scraper.cc:283:49: warning: ignoring > return value of > > ‘int system(const char*)’, declared with attribute > warn_unused_result > > [-Wunused-result] > > system(ftp_unmount_command_string.c_str()); > > ^ > > src/steg/payload_scraper.cc:312:49: warning: ignoring > return value of > > ‘int system(const char*)’, declared with attribute > warn_unused_result > > [-Wunused-result] > > system(ftp_unmount_command_string.c_str()); > > ^ > > depbase=`echo protolist.o | sed 's| > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT protolist.o -MD -MP -MF $depbase.Tpo -c - > o protolist.o > > protolist.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo steglist.o | sed 's| > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT steglist.o -MD -MP -MF $depbase.Tpo -c - > o steglist.o > > steglist.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > g++ -Wall -Wextra -Wno-missing-field-initializers - > Wformat=2 - > > std=c++0x -g -O2 -std=gnu++11 -o pgen_fake > src/pgen_fake.o src/util.o > > src/rng.o src/base64.o -lcrypto > > depbase=`echo src/test/tinytest.o | sed 's| > [^/]*$|.deps/&|;s|\.o$||' > > `;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/tinytest.o -MD -MP -MF > $depbase.Tpo -c -o > > src/test/tinytest.o src/test/tinytest.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/unittest.o | sed 's| > [^/]*$|.deps/&|;s|\.o$||' > > `;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/unittest.o -MD -MP -MF > $depbase.Tpo -c -o > > src/test/unittest.o src/test/unittest.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/unittest_base64.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/unittest_base64.o -MD -MP -MF > $depbase.Tpo -c > > -o src/test/unittest_base64.o > src/test/unittest_base64.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/unittest_compression.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/unittest_compression.o -MD -MP - > MF > > $depbase.Tpo -c -o src/test/unittest_compression.o > > src/test/unittest_compression.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/unittest_crypt.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/unittest_crypt.o -MD -MP -MF > $depbase.Tpo -c - > > o src/test/unittest_crypt.o src/test/unittest_crypt.cc > &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/unittest_pdfsteg.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/unittest_pdfsteg.o -MD -MP -MF > $depbase.Tpo - > > c -o src/test/unittest_pdfsteg.o > src/test/unittest_pdfsteg.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/unittest_socks.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/unittest_socks.o -MD -MP -MF > $depbase.Tpo -c - > > o src/test/unittest_socks.o src/test/unittest_socks.cc > &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo unitgrplist.o | sed 's| > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT unitgrplist.o -MD -MP -MF $depbase.Tpo - > c -o > > unitgrplist.o unitgrplist.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/gtest/gtest_main.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/gtest/gtest_main.o -MD -MP -MF > $depbase.Tpo - > > c -o src/test/gtest/gtest_main.o > src/test/gtest/gtest_main.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/gtest/gtest-all.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/gtest/gtest-all.o -MD -MP -MF > $depbase.Tpo -c > > -o src/test/gtest/gtest-all.o > src/test/gtest/gtest-all.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/test/steg_test/steg_mod_unittest.o | > sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/steg_test/steg_mod_unittest.o - > MD -MP -MF > > $depbase.Tpo -c -o > src/test/steg_test/steg_mod_unittest.o > > src/test/steg_test/steg_mod_unittest.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo > src/test/steg_test/payload_scraper_unittest.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT > src/test/steg_test/payload_scraper_unittest.o -MD -MP - > > MF $depbase.Tpo -c -o > src/test/steg_test/payload_scraper_unittest.o > > src/test/steg_test/payload_scraper_unittest.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > g++ -Wall -Wextra -Wno-missing-field-initializers - > Wformat=2 - > > std=c++0x -g -O2 -std=gnu++11 -o tltester > src/test/tltester.o > > src/util.o src/util-net.o -levent > > depbase=`echo src/test/tester_proxy/tester_proxy.o | sed > 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/test/tester_proxy/tester_proxy.o -MD > -MP -MF > > $depbase.Tpo -c -o src/test/tester_proxy/tester_proxy.o > > src/test/tester_proxy/tester_proxy.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > depbase=`echo src/http_parser/http_parser.o | sed 's| > > [^/]*$|.deps/&|;s|\.o$||'`;\ > > g++ -I. -I./src -I./src/steg -I./src/steg/http_steg_mods > - > > I./src/test/gtest -I./src/test/gtest/include -Wall - > Wextra - > > Wno-missing-field-initializers -Wformat=2 -std=c++0x -g - > O2 - > > std=gnu++11 -MT src/http_parser/http_parser.o -MD -MP - > MF $depbase.Tpo > > -c -o src/http_parser/http_parser.o > src/http_parser/http_parser.cc &&\ > > mv -f $depbase.Tpo $depbase.Po > > rm -f libstegotorus.a > > ar cru libstegotorus.a src/base64.o src/compression.o > > src/connections.o src/crypt.o src/mkem.o src/network.o > src/protocol.o > > src/rng.o src/socks.o src/steg.o src/util.o > src/util-net.o > > src/evbuf_util.o src/curl_util.o src/transparent_proxy.o > > src/protocol/chop.o src/protocol/chop_blk.o > src/protocol/null.o > > src/steg/b64cookies.o src/steg/cookies.o > src/steg/embed.o > > src/steg/http.o src/steg/http_apache.o > src/steg/http_apache.o > > src/steg/http_steg_mods/file_steg.o > src/steg/http_steg_mods/pdfSteg.o > > src/steg/http_steg_mods/swfSteg.o > src/steg/http_steg_mods/jsSteg.o > > src/steg/http_steg_mods/jpgSteg.o > src/steg/http_steg_mods/pngSteg.o > > src/steg/http_steg_mods/gifSteg.o src/steg/nosteg.o > > src/steg/nosteg_rr.o src/steg/payload_server.o > > src/steg/trace_payload_server.o > src/steg/payload_scraper.o > > src/steg/apache_payload_server.o src/subprocess-unix.o > protolist.o > > steglist.o > > : libstegotorus.a > > /bin/bash ./src/audit-globals.sh src/base64.o > src/compression.o > > src/connections.o src/crypt.o src/mkem.o src/network.o > src/protocol.o > > src/rng.o src/socks.o src/steg.o src/util.o > src/util-net.o > > src/evbuf_util.o src/curl_util.o src/transparent_proxy.o > > src/protocol/chop.o src/protocol/chop_blk.o > src/protocol/null.o > > src/steg/b64cookies.o src/steg/cookies.o > src/steg/embed.o > > src/steg/http.o src/steg/http_apache.o > src/steg/http_apache.o > > src/steg/http_steg_mods/file_steg.o > src/steg/http_steg_mods/pdfSteg.o > > src/steg/http_steg_mods/swfSteg.o > src/steg/http_steg_mods/jsSteg.o > > src/steg/http_steg_mods/jpgSteg.o > src/steg/http_steg_mods/pngSteg.o > > src/steg/http_steg_mods/gifSteg.o src/steg/nosteg.o > > src/steg/nosteg_rr.o src/steg/payload_server.o > > src/steg/trace_payload_server.o > src/steg/payload_scraper.o > > src/steg/apache_payload_server.o src/subprocess-unix.o > protolist.o > > steglist.o src/main.o > > sed: -e expression #1, char 113: unknown option to `s' > > touch stamp-audit-globals > > g++ -Wall -Wextra -Wno-missing-field-initializers - > Wformat=2 - > > std=c++0x -g -O2 -std=gnu++11 -o unittests > src/test/tinytest.o > > src/test/unittest.o src/test/unittest_base64.o > > src/test/unittest_compression.o > src/test/unittest_crypt.o > > src/test/unittest_pdfsteg.o src/test/unittest_socks.o > unitgrplist.o > > libstegotorus.a -levent -lcrypto -lz -lcurl - > lboost_system - > > lboost_filesystem -lconfig++ > > g++ -Wall -Wextra -Wno-missing-field-initializers - > Wformat=2 - > > std=c++0x -g -O2 -std=gnu++11 -o tester_proxy > > src/test/tester_proxy/tester_proxy.o -levent_openssl - > levent -lssl - > > lcrypto > > g++ -Wall -Wextra -Wno-missing-field-initializers - > Wformat=2 - > > std=c++0x -g -O2 -std=gnu++11 -o webpage_tester > > src/test/webpage_tester.o src/util.o src/util-net.o > src/curl_util.o > > src/http_parser/http_parser.o -levent -lcrypto -lz - > lcurl - > > lboost_system -lboost_filesystem -lconfig++ > > g++ -Wall -Wextra -Wno-missing-field-initializers - > Wformat=2 - > > std=c++0x -g -O2 -std=gnu++11 -o stegotorus src/main.o > libstegotorus.a > > -levent -lcrypto -lz -lcurl -lboost_system - > lboost_filesystem - > > lconfig++ > > g++ -Wall -Wextra -Wno-missing-field-initializers - > Wformat=2 - > > std=c++0x -g -O2 -std=gnu++11 -o g_unittests > > src/test/gtest/gtest_main.o src/test/gtest/gtest-all.o > > src/test/steg_test/steg_mod_unittest.o > > src/test/steg_test/payload_scraper_unittest.o > libstegotorus.a -levent - > > lcrypto -lz -lcurl -lboost_system -lboost_filesystem - > lconfig++ > > /usr/bin/ld: src/test/gtest/gtest-all.o: undefined > reference to symbol > > 'pthread_key_delete@@GLIBC_2.2.5' > > /lib/x86_64-linux-gnu/libpthread.so.0: error adding > symbols: DSO > > missing from command line > > collect2: error: ld returned 1 exit status > > make[1]: *** [g_unittests] Error 1 > > make[1]: Leaving directory > `/home/selimthegrim/stegotorus' > > make: *** [all] Error 2 > > > >

1 0

Pluggable transports meeting tomorrow (Friday 28th of March 2014)
by George Kadianakis 27 Mar '14

27 Mar '14

Greetings humans, this is an email to remind you that the regular biweekly pluggable transports meeting is going to happen tomorrow. Place is the #tor-dev IRC channel in OFTC. Time is 17:00 UTC. Cheers!

1 0

[Orbot patch] Add all /external build stamps to .gitignore, cleanup
by Daniel Martí 27 Mar '14

27 Mar '14

Signed-off-by: Daniel Martí <mvdan(a)mvdan.cc> --- .gitignore | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/.gitignore b/.gitignore index 4824f92..f3fb750 100644 --- a/.gitignore +++ b/.gitignore @@ -1,16 +1,11 @@ -external/bin/ -external/include/ -external/jtorctl-build-stamp -external/lib/ -external/libevent-build-stamp -external/obfsproxy-build-stamp -external/openssl-static-build-stamp -external/privoxy-build-stamp -external/privoxy/ -external/test.c -external/test.h.gch -external/tor-build-stamp -external/translation/ +/external/bin/ +/external/include/ +/external/*-build-stamp +/external/lib/ +/external/privoxy/ +/external/test.c +/external/test.h.gch +/external/translation/ bin gen native -- Daniel Martí - mvdan(a)mvdan.cc - http://mvdan.cc/ PGP: A9DA 13CD F7A1 4ACD D3DE E530 F4CA FFDB 4348 041C

1 0

Re: [tor-dev] Implications of switching to a single guard node: some conclusions
by Nicholas Hopper 27 Mar '14

27 Mar '14

On Tue, Mar 25, 2014 at 3:41 PM, Mike Perry <mikeperry(a)torproject.org> wrote: > For control, it would be nice to know what we would be giving up if we > used that same T=2000 cutoff with three guards and compare that to > T=2000 with just one guard. Is it easy for you to rerun just that > comparison (since you seem to be suggesting that T=2000 is a good cutoff > anyway)? Takes a few hours to run the scripts again, but the results look like this. For the "unluckiest 10%" with T=1000, having 3 guards is still better than having one, while with T=2000 the difference between 3 guards and 1 is less noticeable: https://www-users.cs.umn.edu/~hopper/unlucky_threeguard_threshold_bandwidth… Interestingly, though, for the median user with the higher thresholds, having one guard results in a higher median circuit bandwidth (because a user with 3 guards is more likely to have selected one guard close to the cutoff, dragging the typical performance down): https://www-users.cs.umn.edu/~hopper/threeguard_threshold_bandwidth.png ...but I don't think these results speak to this concern: > ** The reason I ask is because I suspect there is actually an interplay > between the current circuit build timeout code and the pool of 3 guards. > The CBT cutoff will tend to cause you to avoid whichever of your guards > may be temporarily overloaded at a given time. because TorPS is a python reimplementation of the circuit selection algorithm only and doesn't have any way to simulate the CBT logic: http://torps.github.io/ -- ------------------------------------------------------------------------ Nicholas Hopper Associate Professor, Computer Science & Engineering, University of Minnesota Visiting Research Director, The Tor Project ------------------------------------------------------------------------

1 0

Using the HS protocol for unlinkability only
by Michael Rogers 26 Mar '14

26 Mar '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi all, (Please let me know if this belongs on tor-talk instead of here.) I'm working on a messaging app that uses Tor hidden services to provide unlinkability (from the point of view of a network observer) between users and their contacts. Users know who their contacts are, so we don't need mutual anonymity, just unlinkability. I wonder whether we need everything that the Tor hidden service protocol provides, or whether we might be able to save some bandwidth (for clients and the Tor network) and improve performance by using parts of the hidden service protocol in a different way. First of all, we may not need to publish hidden service descriptors in the HS directory, because we have a way for clients to exchange static information such as HS public keys out-of-band. Second, we may not need to use introduction points to protect services from DoS attacks - we can assume that users trust their contacts not to DoS them. Third, we may be able to reduce the number of hops in the client-service circuits, because we don't need mutual anonymity. This isn't the first app to use hidden services for unlinkability, so I expect this topic's come up before. Are there any discussions I should look at before coming up with hare-brained schemes to misuse the hidden service protocol? Cheers, Michael -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iQEcBAEBCAAGBQJTMwZDAAoJEBEET9GfxSfM330H/2seo/oZgz2K54W5oxkRKa07 Jh+W4swi2utFs728bEdbWDl6EaWiTYvwlUxTBllrXVTGQolPxUsHo4jHLk0Xt5ah Jo3RZxbiKF6rZkcRC66nxF6aGAQ0JZn+xkvVB4xb/2vzMg7jQ9N+GACQ7fRKEvqA GgqqEjKVJdzgtBKQBl0eZYZi/4VXisCEtN7pY1MHiO5k/wFMsg3z1MceN9HSw7EG JedzCLT0r/OVW/f07/1iQU6TWRohcOuE/pBCHi+6ctgp/6a+NehKBw4gIcm1aBur kZiSxea0FIpWdarTcMVctwayLhhzpTuMB6/0YOBO+1/u2rqkLl6njqnDaALkicQ= =AnYI -----END PGP SIGNATURE-----

2 1

[RFC] Proposal draft: The move to a single guard node
by George Kadianakis 26 Mar '14

26 Mar '14

I promised Nick I would have something for him by today's little-t-tor meeting. Here is a draft of the guard node security proposal. Definitely needs more work: Filename: xxx-single-guard-node.txt Title: The move to a single guard node Author: George Kadianakis Created: 2014-03-22 Status: Draft and potentially a bad idea 0. Introduction It has been suggested that reducing the number of guard nodes of each user and increasing the guard node rotation period will make Tor more resistant against certain attacks [0]. For example, an attacker who sets up guard nodes and hopes for a client to eventually choose them as their guard will have much less probability of succeeding in the long term. Currently, every client picks 3 guard nodes and keeps them for 2 to 3 months (since 0.2.4.12-alpha) before rotating them. In this document, we propose the move to a single guard per client and an increase of the rotation period to 9 to 10 months. 1. Proposed changes 1.1. Switch to one guard per client When this proposal becomes effective, clients will switch to using a single guard node. That is, in its first startup, Tor picks one guard and stores its identity persistently to disk. Tor uses that guard node as the first hop of its circuits from thereafter. If that Guard node ever becomes unusable, rather than replacing it, Tor picks a new guard and adds it to the end of the list. When choosing the first hop of a circuit, Tor tries all guard nodes from the top of the list sequentially till it finds a usable guard node. A Guard node is considered unusable according to section "5. Guard nodes" in path-spec.txt. The rest of the rules from that section apply here too. XXX which rules specifically? XXX Do we need to specify how already existing clients migrate? 1.1.1. Alternative behavior to section 1.1 Here is an alternative behavior than the one specified in the previous section. It's unclear which one is better. Instead of picking a new guard when the old guard becomes unusable, we pick a number of guards in the beginning but only use the top usable guard each time. When our guard becomes unusable, we move to the guard below it in the list. This behavior _might_ make some attacks harder; for example, an attacker who shoots down your guard in the hope that you will pick his guard next, is now forced to have evil guards in the network at the time you first picked your guards. However, this behavior might also influence performance, since a guard that was fast enough 7 months ago, might not be this fast today. Should we reevaluate our opinion based on the last consensus, when we have to pick a new guard? Also, a guard that was up 7 months ago might be down today, so we might end up sampling from the current network anyway. 1.2. Increase guard rotation period When this proposal becomes effective, Tor clients will set the lifetime of each guard to a random time between 9 to 10 months. If Tor tries to use a guard whose age is over its lifetime value, the guard gets discarded (also from persistent storage) and a new one is picked in its place. XXX We didn't do any analysis on extending the rotation period. For example, we don't even know the average age of guards, and whether all guards stay around for less than 9 months anyway. Maybe we should do some analysis before proceeding? XXX The guard lifetime should be controlled using the (undocumented?) GuardLifetime consensus option, right? 1.2.1. Alternative behavior to section 1.2 Here is an alternative behavior than the one specified in the previous section. It's unclear which one is better. Similar to section 1.2, but instead of rotating to completely new guard nodes after 9 months, we pick a few extra guard nodes in the beginning, and after 9 months we delete the already used guard nodes and use the one after them. This has approximately the same tradeoffs as section 1.1.1. Also, should we check the age of all of our guards periodically, or only check them when we try to use them? 1.3. Age of guard as a factor on guard probabilities By increasing the guard rotation period we also increase the lack of clients for young guards since clients will rotate guards even more infrequently now (see 'Phase three' of [1]). We can try to mitigate this phenomenon by giving higher priority to young guards to be picked as guards: To do so, everytime an authority needs to vote for a guard, it reads a set of consensus documents spanning the past NNN months, and calculates the age of the guard; that is, in how many consensuses its public key has been included in the past. The authorities include the age of each guard by appending '[SP "Age=" INT]' in the guard's "w" line. When a client picks a guard, it applies the age of each guard as a weight on its guard probability. XXX unspecified how XXX How much should the age of a guard influence its probability? Should we say that a guard that just appeared should have 10% more chance of being selected as a guard node than the oldest guard in town? XXX Should the authorities include the age itself, or just the weight that clients should apply to the probability? XXX Is this risky? Maybe we shouldn't give too much priority to new guards, otherwise an adversary can start up a few new relays every month, enjoy maximum priority when they get the guard flag, leave them running for a bit till the next batch gets the guard flag and then trash them. 1.4. Raise the bandwidth threshold for being a guard From dir-spec.txt: "Guard" -- A router is a possible 'Guard' if its Weighted Fractional Uptime is at least the median for "familiar" active routers, and if its bandwidth is at least median or at least 250KB/s. When this proposal becomes effective, authorities should change the bandwidth threshold for being a guard node to 2000KB/s instead of 250KB/s. Implications of raising the bandwidth threshold are discussed in section 2.3. XXX Is this insane? It's an 8-fold increase. 2. Discussion 2.1. Guard node set fingerprinting With the old behavior of three guard nodes per user, it was extremely unlikely for two users to have the same guard node set. Hence the set of guard nodes acted as a fingerprint to each user. When this proposal becomes effective, each user will have one guard node. We believe that this slightly reduces the effectiveness of this fingerprint since users who pick a popular guard node will now blend in with thousands of other users. However, clients who pick a slow guard will still have a small anonymity set [2]. All in all, this proposal slightly improves the situation of guard node fingerprinting, but does not solve it. See the next section for a suggested scheme that would further fix the guard node set fingerprinting problem 2.1.1. Potential fingerprinting solution: Guard buckets One of the suggested alternatives that moves us closer to solving the guard node fingerprinting problem, would be to split the list of N guard nodes into buckets of K guards, and have each client pick a bucket [3]. This reduces the fingerprint from N-choose-k to N/k guard set choices; it also allows users to have multiple guard nodes which provides reliability and performance. Unfortunately, the implementation of this idea is not as easy and its anonymity effects are not well understood so we had to reject this alternative for now. 2.2. What about 'multipath' schemes like Conflux? By switching to one guard, we rule out the deployment of 'multipath' systems like Conflux [4] which build multiple circuits through the Tor network and attempt to detect and use the most efficient circuits. On the other hand, the 'Guard buckets' idea outlined in section 2.1.1 works well with Conflux-type schemes so it's still worth considering. 2.3. Implications of raising the bandwidth threshold for guards By raising the bandwidth threshold for being a guard we directly affect the performance and anonymity of Tor clients. We performed a brief analysis of the implications of switching to one guard and the results imply that the changes are not tragic [2]. Specifically, it seems that the performance of about half of the clients will degrade slightly, but the performance of the other half will remain the same or even improve. Also, it seems that the powerful guard nodes of the Tor network have enough total bandwidth capacity to handle client traffic even if some slow guard nodes get discarded. On the anonymity side, by increasing the bandwidth threshold to 2MB/s we half our guard nodes; we discard 1000 out of 2000 guards. Even if this seems like a substantial diversity loss, it seems that the 1000 discarded guard nodes had a very small chance of being selected in the first place (7% chance of any of the being selected). However, it's worth noting that the performed analysis was quite brief and the implications of this proposal are complex, so we should be prepared for surprises. 2.4. Should we stop building circuits after a number of guard failures? Inspired by academic papers like the Sniper attack [5], a powerful attacker can choose to shut down guard nodes till a client is forced to pick an attacker controlled guard node. Similarly, a local network attacker can kill all connections towards all guards except the ones she controls. This is a very powerful attack that is hard to defend against. A naive way of defending against it would be for Tor to refuse to build any more circuits after a number of guard node failures have been experienced. Unfortunately, we believe that this is not a sufficiently strong countermeasure since puzzled users will not comprehend the confusing warning message about guard node failures and they will instead just uninstall and reinstall TBB to fix the issue. 2.5. What this proposal does not propose Finally, this proposal does not aim to solve all the problems with guard nodes. This proposal only tries to solve some of the problems whose solution is analyzed sufficiently and seems harmless enough to us. For example, this proposal does not try to solve: - Guard enumeration attacks. We need guard layers or virtual circuits for this [6]. - The guard node set fingerprinting problem [7] - The fact that each isolation profile or virtual identity should have its own guards. XXX It would also be nice to have some way to easily revert back to 3 guards if we later decide that a single guard was a very stupid idea. References: [0]: https://blog.torproject.org/blog/improving-tors-anonymity-changing-guard-pa… http://freehaven.net/anonbib/#wpes12-cogs [1]: https://blog.torproject.org/blog/lifecycle-of-a-new-relay [2]: https://lists.torproject.org/pipermail/tor-dev/2014-March/006458.html [3]: https://trac.torproject.org/projects/tor/ticket/9273#comment:4 [4]: http://freehaven.net/anonbib/#pets13-splitting [5]: https://blog.torproject.org/blog/new-tor-denial-service-attacks-and-defenses [6]: https://trac.torproject.org/projects/tor/ticket/9001 [7]: https://trac.torproject.org/projects/tor/ticket/10969

1 0

Implications of switching to a single guard node: some conclusions
by George Kadianakis 26 Mar '14

26 Mar '14

tl;dr: analysis seems to indicate that switching to one guard node might not be catastrophic to the performance of Tor. To improve performance some increased guard bandwidth thresholds are proposed that seem to help without completely destroying the anonymity of the network. Enjoy the therapeutic qualities of the graphs and please read the whole post. We start this post by assuming that we _should_ switch to one guard for the security/anonymity arguments that were detailed in Tariq's paper and Roger's blog post. === Performance implications of switching to 1 guard === The question now becomes, if we indeed switch to 1 guard, how does that influence the performance of the Tor network? To answer this question we look at the following graph which shows the expected bandwidth for a client circuit: https://people.torproject.org/~asn/guards2/perf_cdf_guard_bw_desc.png (see green and orange lines) (I calculate the bandwidth using the descriptor bandwidth values [0] and in the case of 3 guards we measure the expected bandwidth as the average of the bandwidths of the three guard. [1]) For example, looking at the graph, we see that when three guards are used, 1/5th of the clients will have performance below 5MB/s, whereas with one guard 1/5th of the clients will have performance below 3MB/s. Assuming that our assumptions are logical, this is almost half of the bandwidth for the unlucky 1/5th single guard clients that happened to pick a weak guard: not good. At a later stage of our CDF, we see that in the three guards case, half of the clients will have performance below 8MB/s whereas in the one guard case they will have performance below 7MB/s. This is not terribly bad, and the reason for this is that powerful guards have more chance to be selected, so single-guard clients will tend to pick those. Finally, a crossover happens for the lucky 2/5ths of the single guard clients, where they actually experience better performance than the three guards clients since they picked a powerful guard and they only use that. This is interesting but in real life the results might not be so peachy, because the powerful guards will get more overloaded. === Client performance implications of bumping up the guard bandwidth threshold === So, now that we analyzed the performance implications of using a single guard, let's see if we can improve the performance. One obvious way of doing so is by increasing the bandwidth threshold for the Guard flag. The threshold is currently at 250KB/s (according to dir-spec), but let's see what happens from a performance perspective if we bump it up to 2MB/s. Looking at the same graph as before, now pay attention to the blue line. We can see that for the unlucky 1/5th of the single guard clients who had a bandwidth of 3MB/s, their bandwidth now becomes 4MB/s, which seems like a decent improvement. Furthermore, the crossover happens earlier now, which means that _supposedly_ half of the clients are going to have better performance (modulo guard overload) compared to the three guard case! I also made graphs for a bandwidth threshold of 1MB/s (since 2MB/s sounded too crazy), you can find them here [2]: https://people.torproject.org/~asn/guards2/perf_cdf_guard_bw_desc_1000.png https://people.torproject.org/~asn/guards2/perf_cdf_guard_bw_consensus_1000… === Network performance implications of bumping up the guard bandwidth threshold === Now that we analyzed the performance difference for individual clients, let's see what will happen to the total bandwidth of the Tor network if we bump up the guard bandwidth threshold. This might help us understand how much we will overload the Tor network with this change. Here is a graph that shows the fraction of the total guard bandwidth we discard when we impose various bandwidth thresholds [3]: https://people.torproject.org/~asn/guards2/perf_bw_fraction.png {1} The graph above is not very meaningful on its own, but it combos well with the following metrics graph: https://metrics.torproject.org/network.html#bandwidth-flags {2} (see yellow and orange lines) >From {2}, we see that the Tor network has 6000MiB/s advertised guard bandwidth (orange line), but supposedly is only using the 3500MiB/s (yellow line). This means, that supposedly we are only using 3/5ths of our guard capacity: we have 2500MiB/s spare. Looking back at {1}, we see that if we increase the guard bandwidth threshold to 2MB/s we will discard 1/10th of our total guard bandwidth. This is not a terrible problem if we have 2/5ths of spare guard capacity... .oO(this sounds too good to be true, doesn't it?) === Security implications of bumping up the guard bandwidth threshold === Unfortunately, we can't just simply go about and discard most of our guard nodes. Discarding nodes has definite implications to the anonymity of the Tor network. Let's try to understand them. Here is a graph that shows the number of guard nodes and how that changes over different bandwidth thresholds: https://people.torproject.org/~asn/guards2/diversity_guards_n.png For example, we see that increasing the bandwidth threshold to 2MB/s will cut our guard nodes to half: from 2000 to 1000. This is not really good. Even a smaller threshold of 1MB/s will cut them down to 1400 or so. But before we pull a Filliol, let's try to understand how much discarding 1000 guard nodes influences the diversity of our guard selection. Here is a graph that shows what's the probability of picking any of the guard nodes we discarded for different bandwidth thresholds: https://people.torproject.org/~asn/guards2/diversity_discarded_prob.png So for example, we see that those 1000 nodes that we discarded in the 2MB/s case, only had 0.07 probability of being selected. That's around 1/15 chance of picking one of those 1000 guard nodes, so even though there were many of them they were not providing much diversity to the guard selection process. Of course, there are many possible attacks and threat models involving guards, so this analysis might be valid to some and irrelevant to others. The fact that those guards had only 1/15 chance of being selected also gives us hope that we will not overload the network by discarding them, since only a "small" portion of clients were choosing them anyway. These clients will now spread to the rest of the other 1000 nodes which are much better at handling them (famous last words). === Fingerprinting implications of switching to 1 guard === See https://bugs.torproject.org/10969 for the background of this. Here is a graph with the expected number of clients for the biggest and smallest guard over different bandwidth thresholds: https://people.torproject.org/~asn/guards2/fingerprinting_expected_clients.… The graph considers 500k clients choosing guards simultaneously. Switching to 1 guard will make guard set fingerprinting harder if you are a lucky client that picked a popular guard, since now you are blending in with thousands of other clients who are using that guard. If you were unlucky to chose a small guard, your anonymity set is still shit. For example, without considering bandwidth cutoffs, the smallest guard has an expected number of clients less than 1, which means that it will uniquely represent you. Even with a bandwidth cutoff of 2MB/s, the expected number of clients is 10 which is not much better. Heck, even with a cutoff of 9MB/s, there will only be 100 clients in average for the smallest guard; that's a pretty small number if we consider Tor clients all over the globe. === Conclusions === It seems that the performance implications of switching to 1 guard are not terrible. The performance of some clients will indeed get worse, but we might be able to help that by increasing the bandwidth threshold for being a Guard node. A guard bandwidth threshold of 2MB/s (or 1MB/s if that sounds too crazy) seems like it would considerably improve client performance without screwing terribly with the security or the total performance of the network. The fingerprinting problem will be improved in some cases, but still remains unsolved for many of the users (TODO: calculate the percentage). A proper solution might involve guard node buckets as explained in : https://trac.torproject.org/projects/tor/ticket/9273#comment:4 Also, through the analysis it seems that people who pick slow guards are unlucky (even though they will share those guards with less people). Should we do anything about people who are going to choose new guards till they hit the good ones? Or torrc lines on the Internet that statically pick the best guard nodes? === Closing notes and disclaimers === I would say that our analysis has shown that switching to one guard is probably viable but we should be aware of the drawbacks and be prepared for possible surprises. Furthermore, I would like to disclose that one month ago I didn't even know how guard node selection happens and now I'm partly responsible for choosing whether we switch to one guard node or not. Also, even though this project is a serious research project, I felt that I had to rush it and do it in 3 weeks. This was not ideal, because I don't feel I understand all the variables in the equation. So please read the whole document and make sure that I have not fucked up majorly. I would like to avoid being the man who destroyed the Tor network ;) Also, it's my first time producing graphs with Python, so I wouldn't be surprised if there are errors. Hopefully most of the graphs that I produced seem to agree with the graphs that Nick Hopper or Tariq have produced, which gives me some slight confidence. The code I used can be found in https://gitorious.org/guards/guards [4] You can find all the graphs here: https://people.torproject.org/~asn/guards2/ Don't worry be happy. [0]: Important note: even though I calculate the plotted bandwidth using descriptor bandwidth values, I still calculate the guard probabilities using the consensus bandwidth values. This seemed to me to be the correct way; if it's not I can easily change it. Also see https://people.torproject.org/~asn/guards2/perf_cdf_guard_bw_consensus.png for the same graph but using the bandwidth values from the consensus (measured by the bandwidth authorities) everywhere. [1]: This graph is taking the pretty bold assumption that "higher guard bandwidth' == "better client performance" which is probably not entirely true because of the bandwidth-based load balancing during path selection. However, we need an assumption to work with and this one might not be too bad. It also takes the assumption, that the mean of the bandwidth of three guards represents the actual performance of a client, which is not entirely true. A correct solution in this case should take the circuit-build-times (CBT) logic of tor into account. [2]: Because of technical difficulties I could not put everything in one graph! Graphs are hard! [3]: Nick Hopper made a similar graph earlier in this thread: https://www-users.cs.umn.edu/~hopper/guards/guard_thresholds_bandwidth.png [4]: It's rushed research quality code, which means that I'm probably the only person who can use it atm. If you feel experimental, you can try generating some graphs, for example: $ python guard_probs.py consensus descriptors

2 2

Re: [tor-dev] GSoC - Search Engine for Hidden services
by George Kadianakis 26 Mar '14

26 Mar '14

Juha Nurmi <juha.nurmi(a)ahmia.fi> writes: >> And what would you like to do over the summer so that: a) Something >> useful and concrete comes out of only 3 months of work. b) Your >> work will also be useful after the summer ends. >> >> I would be interested to see some areas that you would like to work >> on over the summer, and how that would change the ahmia.fi user >> experience. > > I have drafted a timetable for the possible new features to ahmia.fi: > > https://docs.google.com/document/d/1XB42HM4uESYBAnoHHRuaqKMP64VFDI91Qa-CtIu… > Hello Juha, here are some comments on your proposal: > Search development > > Full text search development > Popularity tracking (catch users clicks and tell YaCy the popular pages): development of a popularity tracking feature for ahmia.fi and Integration of that feature with YaCy API (providing stats for popular pages and suggestions for relevant results) > 1-3 workdays > Yes, this is definitely useful. I would also like you to check out how backlinks work, and whether your crawler can start counting HS backlinks too. Mainly because popularity tracking is easily gameable, whereas backlinks might be harder to game (still definitely gameable though; SEO is crazy). To make sure that this section is done properly, I would suggest to compile a list of well-known HSes and verify that they all appear on/near the top of the ahmia search results by the end of development of these features. I would suggest using more than 1-3 workdays for this. > Use an another crawler to search .onion pages from the public Internet > Search new .onion domains from different online sources > This is an excellent case to test open source crawlers like Heritrix and Apache Nutch > 1 workweeks > Yes, this is very useful. > Public open YaCy back-end for everyone > letâ€™s make our YaCy network open so anyone can join to it with their YaCy nodes > This way we could get real P2P decentralization > Share installation configuration package that joins a YaCy node to ahmia.fiâ€™s nodes > 1 workweek > I guess this is also useful. > Better edited HS descriptions > Design and development of a more useful and complete UI including more comple and exaustive descriptions and details (e.g., show the whole history of descriptions and let the users edit it better) > 1 workweek > Yes this seems like a good idea. Improving the UX is very important. Because of the security nature of ahmia, the UX should be security conscious too. For example, you shouldn't give your users too much confidence on the ordering of the search results since a motivated adversary can probably influence it. Maybe you could also expose some of your popularity/backlinks information to users, in case that lets them pick results more safely. > Comment and vote about the content (safe/unsafe) > Ahmia.fi needs a commenting and rating systems for hidden services > It is useful to gather a user's knowledge about the sites > 1 workweek > I think that this needs more thinking. The rating idea is trivially gameable. Do we assume that all users are good citizens? Given that there are shitloads of phising websites registered to ahmia, we take it that there are bad people out there who know of ahmia. How will the rating system interact with bad people? What about the comming system? Is this also an argument against popularity tracking? How do we use this technologies usefully in the face of bad people? > Tor browser friendly version of the ahmia.fi > Development of a JavaScript free version of ahmia.fi > 1 workweek > TBB has javascript enabled these days. I would probably spend this one week on other stuff. > Search API > 1 workweek > What do you mean by this? Do other search engines provide this sort of API? This would need more than one week to design and deploy properly, no? > Automated statistics and visualizations about hidden services and their content > Development of an Analytics feature > As the result of the indexing Tor network's content ahmia.fi can produce an authoritative and exact quantitative research data about what is published through the Tor network. > 2 workweeks > > Automated visualizations > It is very practical to visualize the data > 2 workweeks > Both of the above items are statistics and they seem to require 1 month of development. Are there really that many stats that we can/should produce? What kind of stats are you thinking of, other than the "number of HSes added per month", "number of ahmia visitors", etc.? BTW, we should be very careful that stats are privacy preserving. > Show cached text versions of the pages > 1 workweek > Useful. I thought you had this feature in the past though; no? > API development > > In addition, ahmia.fi provides RESTful API to integrate other services to use hidden service description information (see https://ahmia.fi/documentation/) Hidden services can integrate their descriptions directly to the hidden service list (see https://ahmia.fi/documentation/descriptionProposal/) Ahmia.fi knows which hidden services are online and you can use the API to check hidden service's online status. This API should be maintained general and simple. > > Integration with softwares that are using hidden services > Integration with Tor2web > Thanks to our suggestion recently, Tor2web has implemented a feature that provides secure and anonymous statistics within a day. I want to implement to implement an automatic fetch and handling of this data. > Ahmia.fi should fetch these and add each new .onion page > Child pornographic is a plague for the Tor network and a well designed and authoritative entity may be useful for provide some filtering lists. To this aim we are currently handling manually a filter list already integrated with Tor2web and in use on quite all the nodes of the Tor2web network (https://ahmia.fi/policy/, https://github.com/globaleaks/Tor2web-3.0/issues/25) In collaboration with Tor2web i want to develop an efficient and automated system to handle and share a filtering information in a secure manner. > 1 workweek > Hm, this is interesting but potentially controversial. Where is this data? > Development of a Content Abuse Signaling feature in order to allow fast handling of abuse comments; i want to implement a Callback API in order to publish this data to Tor2web nodes in real-time. > 1-3 workdays > Ehm, so you are going to expose all the banned pages to Tor2Web? Is this API going to be public? Will anyone be able to see the banned pages? If it's not public, how are you going to protect it? Is this doable in 1-3 workdays? Is this worth doing? > Globaleaks integration > Currently, GlobaLeaks informs ahmia.fi to index new hidden services > Ahmia.fi could extend the visibility of Globaleaks on the search results > Together with GlobaLeaks: RESTful API according to Globaleaksâ€™ needs > 1 workweek > So you will make an API that allows people to submit HSes to ahmia? Will this be useable by anyone; can it be exploited? If not, how will you protect it? Is this really worth doing? > Estimated amount of work is 13 weeks. All in all, the timetable looks good. I'm quite excited about the changes to your crawler (that will give us a bigger list of HSes), and the changes to your indexing (popularity tracking/backlinks etc.). I think you should devote more time to these so that they are done properly. You currently estimated 1.5 weeks to those tasks, but maybe you could bump it to 3 or 4 weeks. OTOH, I don't know much about search engines so it might be easier than I think. I'm also excited about the UX changes and statistics, but I'm not sure if I would devote one month just for statistics. Maybe steal some time from statistics and give it to the crawler/indexing and UX? Maybe not? The API stuff and the "Integration with *" projects are probably harder/riskier to do than they seem. Are we sure we want to do them? Better to do fewer things properly, than many things sloppily. Or not? I would also like to see the code base cleaned up a bit. For example a README file, some basic description of what each file is doing. Probably also include the YaCy/crawler configs? I would also like Ahmia to have some docs on the website. I would like to see a doc on how ahmia works, including how its components interact with each other. And I would also like to see a doc that explains to users the threat model of Ahmia; that is, what technologies ahmia has in place to defend against phishing, how likely they are to succeed, and how cautious users should be.

6 15