[tor-bugs] #23421 [Metrics/CollecTor]: Use persistence functionality throughout all modules

[Tor Bug Tracker & Wiki]

#23421: Use persistence functionality throughout all modules
-------------------------------+-----------------------------------
 Reporter:  iwakeh             |          Owner:  metrics-team
     Type:  enhancement        |         Status:  needs_information
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:  metrics-2017       |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+-----------------------------------

Old description:

> As discussed in
> [https://trac.torproject.org/projects/tor/ticket/21272#comment:18 #21272
> comment:18] and before.
>
> All storing of files should be done by the persist-mechanism, i.e.,
> a o.t.c.persist.*Persistence classes should also be used for storing
> descriptors initially (i.e., when fetched from authorities).

New description:

 As discussed in
 [https://trac.torproject.org/projects/tor/ticket/21272#comment:18 #21272
 comment:18] and before.

 All storing of files should be done by the persist-mechanism, i.e.,
 a o.t.c.persist.*Persistence classes should also be used for storing
 descriptors initially (i.e., when fetched from authorities).

 Goal:
 We want to identify a common approach for storing descriptors during sync-
 runs and imports. The various modules should diverge from it as little as
 possible.  The resulting changes (once the approach is defined) will be
 implemented in the various modules and the implementation of these changes
 will make use of the persistence classes in order to avoid code
 multiplication for file storing/writing/renaming/etc.
 Currently, there is quite a variety of handling storage that is in many
 cases historically grown but not due necessity (cf. comment:1).

--

Comment (by iwakeh):

 I added some text to the description of this ticket, because it was a
 little short.  Please, refer to the description before continuing.

 (I tried to take your comment:6 into account in the following.)

 What about the following general rules for all modules:

 During descriptor imports:
 After parsing (and possibly sanitation) the descriptor is written; if one
 descriptor cannot be parsed (and possibly sanitized), it is skipped with a
 debug log message, unless it contains at least enough information to sort
 it into out&recent (for relays and bridges and onionperf and ?).

 During sync-runs:
 Descriptors that cannot be parsed should be dropped.  Or, do we want to
 sync invalid descriptors?
 Imo, syncs should only take valid stuff and there should be sync-runs for
 all descriptors (just not necessarily enabled on every instance).

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/23421#comment:7>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online