[tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances

[Tor Bug Tracker & Wiki]

#18910: distributing descriptors accross CollecTor instances
-------------------------------+-----------------------------------
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_information
 Priority:  High               |      Milestone:  CollecTor 1.1.0
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:  ctip               |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+-----------------------------------

Comment (by karsten):

 Alright, I read the wiki page and the comment above.  Just two
 clarifications of what I meant above:

  - The only code with permission to write to (and delete from)
 `SyncFolder` should be `DescriptorCollector`, which would also delete
 files as soon as they disappear remotely.  We shouldn't move away or
 delete files while going through that directory and looking at
 descriptors, because that would mean that `DescriptorCollector` would have
 to download them again next time.  Every time it runs.

  - There will now be two cases where we want to write a descriptor and
 need to check if we already have it: 1) when downloading it locally and 2)
 when syncing from another CollecTor instance.  In the first case, if a
 file already exists and has different contents, we now overwrite it.  It
 could be a descriptor we synced from another instance in a previous run or
 obtained earlier by downloading it from somewhere.  Note that we're
 looking at our `out/` directory to decide whether we already have a
 descriptor, not at our `recent/` directory.  In the second case, we simply
 don't store the descriptor.  I think that's fine as initial strategy.

 If this all makes sense, feel free to work on the code, and I'll take a
 look once there's something to review.  If it doesn't make sense yet, feel
 free to ask more questions.  Thanks!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/18910#comment:16>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online