[tor-reports] GSoC: Weekly report for ahmia, week 21

[Juha Nurmi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi All,

I started coding GSoC early before 18.05.2014 so this more than one week
work. Because the GSoC is now officially in the coding phase I will be
sending weekly report from now on.

All the tasks are in the GitHub. There are three milestones for the
tickets: https://github.com/juhanurmi/ahmia/issues/milestones

What I have done:

1) Demo environment for ahmia.fi
- - There I can test most of the features and use real data
- - Read-only to ahmia.fi's database and write to local database
- - Automatically uses DEBUG state on demo server and in the prod server
sets DEBUG = False
- - Useful because it is very hard to test the new features (it still is)

2) The default search page is now JavaScript free
- - The main search does not any more require JavaScript: the XML query
answer is translated to HTML with Python
- - This creates a little more work to the server-side, but I think that
the main search should work for people who scare the client side code

3) Crontab task: Integration with Tor2web: gather data from tor2web nodes
- - This feature is now implemented. Ahmia.fi is downloading statistics
from Tor2web nodes and saving this information to the database, see
https://github.com/juhanurmi/ahmia/issues/3
- - This and a lot of other crontab tasks help to automate tasks as much
as possible

4) Backlink tester for the public WWW backlinks to onions
- - Ahmia.fi gathers backlink data from the public WWW, see
https://github.com/juhanurmi/ahmia/issues/7

5) Every search result click is now detected and the number of clicks
is saved
- - Ahmia.fi catches the user's search result clicks and saves the
number of them to the database

6) API that shares the statistical data
- - A skeleton for the stats API is created
- - https://ahmia.fi/address/k5zq47j6wd3wdvjq/popularity
- - https://ahmia.fi/stats/popularity
- -
https://ahmia.fi/stats/popularity?limit=4&offset=0&order_by=public_backlinks

7) Show TOP 10 sites by different popularity measurements and
visualize stats
- - I haven't officially even started yet with the visualizations. Having
said that, I decided to try to show and visualize some of the stats:
https://ahmia.fi/stats/viewer
- - This is just a start and mainly created now for myself so it is
easier for me to design the algorithm that sorts search results using
this data. This simple page tells at least something from the data I
am gathering now.
- - In the future, more useful stats visualizations to the users

8) HUGE code refactoring (90% code is now refactored); that one ugly
views.py is now splitted to multiple code files
- - This will be very useful in the future

9) ahmia.fi/rule section enables admin-users to login and ban content
- - Easy way to manage the content

10) Report button for child abuse
- - Simple way to push a button that sends the user's abuse notice message
for me using email
- - https://ahmia.fi/address/

Moreover, a lot of bugs fixed and minor features created.

The back-end servers were updated and the Solr search index is now very
fresh. Unfortunately, I messed it up when I removed some CP sites
directly using Solr. As a result, sooner or later, the whole search
database have to be re-created.

I have (poorly) documented some technical information to the GitHub wiki
https://github.com/juhanurmi/ahmia/wiki and I should write more.

Furthermore, I created a simple documentation to the site
https://ahmia.fi/documentation/ how to use different data APIs.

It is time to drink a beer, made it too myself! I am brewing my own
beers because it easy, inexpensive, fun, and the beer is superior to
those cans they sell in the shops.

Have a nice weekend everyone!

- -Juha
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTf3q+AAoJELGTs54GL8vAyLkH/01AbZuO2ACHWeZQeB1XCnjK
CckfkHAycZuaj2mYYw17yEgNzzOOpJUVEr6rxeNZ468B43ft/5YTvSxQP5uwDQ98
N8BAqOqWLRrsHl7b/vN5/TkG+NDzeXAWz+LgquaAcnkbIIweKIoltVDopkiY8Pvz
/ouFDcmBZphlJBeIDkrghGpUN/ecW0YuGNSx/K7ZLL5FLKNHjyuRTZWuIJugB6uO
edRUaJKSJqRriEftVbdeQPIkzRaxEL6fLS4c4VL/PMa/IJjsWDMJkdPnUlXu3hCn
lzh0qb4aB3DWB8pw1uHRG6xoyzQ70UV1um8K58vPP3oFsheS5KGspItE4CPme4Q=
=Mj3Q
-----END PGP SIGNATURE-----