[tor-bugs] #17588 [GetTor]: GetTor Logging

[Tor Bug Tracker & Wiki]

#17588: GetTor Logging
---------------------+---------------------
 Reporter:  sukhbir  |          Owner:  ilv
     Type:  defect   |         Status:  new
 Priority:  Medium   |      Milestone:
Component:  GetTor   |        Version:
 Severity:  Normal   |     Resolution:
 Keywords:           |  Actual Points:
Parent ID:           |         Points:
  Sponsor:           |
---------------------+---------------------

Comment (by ilv):

 Replying to [ticket:17588 sukhbir]:
 > This is the main ticket for GetTor logging. Let's try to discuss
 everything here or you can open new tickets and reference this as the
 parent ticket.
 >
 \\
 Sounds great, thank you.
 \\
 >
 > Here is what we will be storing (counters):
 >
 > - Number of requests for the email bot. (A "request" is considered if we
 reply to an email with the links to the bundles.)
 > - Number of requests for the other distribution channels: Twitter and
 XMPP. (A "request" is considered if we reply to a query with the links to
 the bundles.)
 \\
 What about if a "request" is considered whenever we reply to an email (or
 a message, or a DM), and we make the distinction about the type of request
 i.e. between sending links, help message, mirrors?
 \\
 > - OS: Windows, Linux or OS X.
 > - Locale: Language of the request (en, es, etc.)
 \\
 Sounds good, I agree.
 \\
 > - Requests per day: this is useful if in events of censorship, if there
 was an increase in the number of requests for a given day.
 >
 \\
 Is this related to the number of requests mentioned above (number of
 requests)?
 \\
 > Talking with ilv, he described how we are storing user data.
 >
 > - In the SQLite database, we have a table that stores the sha256 of the
 address so that we can prevent GetTor from being spammed. Let's clear this
 after a day so that we don't keep the hashed email address for long and
 also because since we are not actually sending out the bundles, we
 shouldn't enforce harsh limits on blacklisting addresses.
 >
 \\
 Yes, clearing this after a day sounds good. I established a limit to avoid
 having to process zillions of automated requests. Main reason for this was
 to prevent an overload of the service, although we process the request
 anyways, so having a blacklist might be an overkill. However, a good
 reason to have a short limit for this is that in case someone is making
 automated requests we don't get our stats ruined.
 \\
 > On a related note, after how many requests does an email address get
 blacklisted?
 >
 \\
 Right now after 100 requests. I changed this a while ago in order to make
 some tests, and I didn't change it back. I think 10 or less should be a
 good number.
 \\
 > - In the request table we only store the counter for the requests. This
 is fine. From the log files, we should extract the other information,
 update this request table and then use that to generate the automatic
 reports.
 >
 \\
 I think we have two options here:

   1) extracting this from the log files, and then insert it into the
 database.
   2) storing this on the database directly when we process a request.

 I would say 2), because I think we should use the log files to help
 debugging if/when something goes wrong, not for extracting data.
 \\
 \\
 What do you think?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/17588#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online