[tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances

[Tor Bug Tracker & Wiki]

#18910: distributing descriptors accross CollecTor instances
-------------------------------+---------------------------------
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_review
 Priority:  High               |      Milestone:  CollecTor 1.1.0
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:  ctip               |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+---------------------------------

Comment (by karsten):

 Here's a fresh bug report: synchronizing relay descriptors takes a few
 ''hours'' on average on my laptop, and I think I know why.  Whenever we
 append to an existing file in `recent/`, we copy the entire file content
 to a temporary file, append the new content there, and then rename.  And
 we do that for every single server descriptor or extra-info descriptor we
 receive.  That is bad for two reasons.  First reason is that it obviously
 doesn't scale, with some sync runs taking 4 hours or longer.  Second
 reason is that files in `recent/` shouldn't change after they're included
 in `index.json` (with the inglorious exception of Torperf measurements),
 which is why we should append all descriptors to a temporary file and
 rename when we're sure we're done.  See also
 `ArchiveWriter.cleanUpRsyncDirectory()` where we're renaming temporary
 files to destination files at the very end of the update run.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/18910#comment:88>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online