[tor-bugs] #14826 [Onionoo]: backend: hourly update without crontab or the like

[Tor Bug Tracker & Wiki]

#14826: backend: hourly update without crontab or the like
-----------------------------+----------------------------
     Reporter:  iwakeh       |      Owner:
         Type:  enhancement  |     Status:  needs_revision
     Priority:  normal       |  Milestone:
    Component:  Onionoo      |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+----------------------------

Comment (by karsten):

 The idea to run around :15 of the hour is that we can be quite confident
 that new data will be available at that time.  The consensus gets
 published at :00, and CollecTor starts fetching the consensus and other
 descriptors at :05.  That process usually takes a few minutes.  If we have
 Onionoo download new data from CollecTor at :15 or :20, it's going to be
 new data, and the earlier we process new data the fresher Onionoo's data
 will be.

 You have a point about not having all mirrors start at the exact same
 time.  What we could do is start between :15 and :20, depending on when
 the program starts.  For example, if the program starts at :01, we start
 the first run at :16 and the next run 60 minutes later.

 Note that we also must avoid running more than once per hour.  There's a
 bug where if we don't parse a fresh consensus, the exit/guard/middle
 probabilities cannot be calculated correctly.  If we plan to run more
 often, we'll have to fix that bug first.  I could imagine running every
 five minutes and stopping a run if we figure out that CollecTor has no new
 data for us.

 Thoughts?  Thanks for starting this!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/14826#comment:4>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online