[tor-bugs] #19018 [Metrics/CollecTor]: run CollecTor modules without crontab

[Tor Bug Tracker & Wiki]

#19018: run CollecTor modules without crontab
-------------------------------+------------------------
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  enhancement        |         Status:  new
 Priority:  Medium             |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:  ctip               |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+------------------------

Comment (by karsten):

 I tried to apply the suggested configuration scheme to match what we're
 currently doing with cron.  But the simplest way I could come up with that
 does what we need requires two values per module:
  - `offsetMinutes`: offset in minutes since the epoch and
  - `periodMinutes`: period in minutes.

 Using these two values we can compute `initialDelay` and `period` for
 `ScheduledExecutorService.scheduleAtFixedRate()` as follows (mostly
 untested):

 {{{
 long ONE_MINUTE = 60L * 1000L;
 long initialDelay = periodMinutes * ONE_MINUTE
     - (System.currentTimeMillis() - offsetMinutes * ONE_MINUTE)
     % (periodMinutes * ONE_MINUTE);
 long period = periodMinutes * ONE_MINUTE;
 }}}

 Here's how I would configure runs on collector.torproject.org using that
 scheme:

 {{{
 bridgedescs 9,60
 bridgepools -1,-1
 exitlists 2,60
 relaydescs 5,30
 torperf 1,360
 updateindex 0,2
 }}}

 Explanation:
  - Bridge descriptors are copied over from the bridge authority at :07
 every hour, so we should start the sanitizing soon after at :09 every hour
 and be ready when Onionoo and others fetch sanitized bridge descriptors at
 :15.
  - Bridge pool assignments are not sanitized anymore, so I picked -1,-1 as
 special value to disable the module, similar to your suggestion.  (Note
 that we could also remove the entire code for sanitizing bridge pool
 assignments.  If you currently don't have any branches based on master and
 if you'd like me to remove that code, I would go ahead and do that today.
 But regardless, we would need a way to disable a module, and -1,-1 should
 work okay for that.)
  - Exit lists should be fetched once per hour at :02.
  - Relay descs can be fetched at :05, but we should try hard not to miss
 an hour there.  If the server happens to be down at :05 we should try
 again half an hour later at :35.  We're currently not doing that, but I
 think it would be smart to do it.
  - Torperf results are only fetched every six hours to reduce load on the
 Torperf hosts.
  - Updating the index should happen every two minutes.

 How does that sound?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/19018#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online