[tor-bugs] #15844 [Onionoo]: Develop database schema to support Onionoo's search parameter efficiently

[Tor Bug Tracker & Wiki]

#15844: Develop database schema to support Onionoo's search parameter efficiently
-----------------------------+-----------------
     Reporter:  karsten      |      Owner:
         Type:  enhancement  |     Status:  new
     Priority:  normal       |  Milestone:
    Component:  Onionoo      |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+-----------------

Comment (by leeroy):

 Thank you for all the clarifications. To start with:

  1. Create an updated normal form with tables to ensure document coverage.
  1. Create related SQL for document queries.
  1. Deploy a test database, and a test Onionoo instance, using a
 restricted data-set for comparison.
  1. Benchmark them for similar queries.
  1. Iteratively make adjustments to database to achieve desired quality of
 service.

 With updates and refinements occurring in this ticket along the way.

 ----
 >Slow clients would show up in the next line, "Milliseconds to build
 response", which includes reading documents from disk as well as writing
 them to the response.  (We could split up that line, but then we'd have
 to buffer complete responses in memory, which might be bad.)

 >Happy to add new statistics there, if they help us investigate potential
 bottlenecks.

 It might be useful to look at how much time is spent just building the
 JSON part of the response. In particular, the data is serialized into JSON
 using `GsonBuilder` in `ResponseBuilder` . That class appears to contain
 non-trivial branching and logic. PostgreSQL can serialize the response
 into JSON too. I heard it's shiny.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/15844#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online