[tor-bugs] #11573 [Onionoo]: Ponder using a database for Onionoo rather than keeping indexes in memory and contents on disk

[Tor Bug Tracker & Wiki]

#11573: Ponder using a database for Onionoo rather than keeping indexes in memory
and contents on disk
-----------------------------+-----------------
     Reporter:  karsten      |      Owner:
         Type:  enhancement  |     Status:  new
     Priority:  minor        |  Milestone:
    Component:  Onionoo      |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+-----------------

Comment (by iwakeh):

 * It might be also interesting to look at the access log in terms of how
 many request were there at a given instance in time.
 * Maybe, separate the lookup and prepare measurements from the response
 times. Response times could be longer if the connection of the client is
 slow, or not correctly terminated, or in case of too many concurrent
 session threads (that's why I asked about access logs). Seperate lab-
 measurements for lookup and compile could be easily compared to db
 solutions. Seperate response time measuremnts in-vitro and in-lab might
 help identifying server issues.

 * Pondering the database question should not only be done b/c of
 performance reasons, but also in the light of what Onionoo serves. Many
 requests especially those using the history subtype are database
 concerns.Future requests (like #13137) too. Databases are made for that;
  - why reprogram the functionality? The reprogramming won't scale forever,
 and, what could turn out more difficult, is the code maintenance.
  - A highly optimized "query" on a proprietary in-memory index, might pose
 a problem when it needs to be extended/changed. An sql-query is way more
 readable, b/c the database takes care of optimization (of course one needs
 to define indexes and the like).

 So, maybe just think about using a database anyway.

 In this light I would not opt for sqllite anymore, but start with a
 reasonable gpl-db-system.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/11573#comment:7>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online