[tor-bugs] #18320 [Core Tor/Tor]: Clear old entries from the key-pinning journal file

[Tor Bug Tracker & Wiki]

#18320: Clear old entries from the key-pinning journal file
----------------------------------------+----------------------------------
 Reporter:  teor                        |          Owner:  andrea
     Type:  defect                      |         Status:  assigned
 Priority:  Medium                      |      Milestone:  Tor:
Component:  Core Tor/Tor                |  0.2.9.x-final
 Severity:  Normal                      |        Version:
 Keywords:  tor-dos, TorCoreTeam201606  |     Resolution:
Parent ID:  #17293                      |  Actual Points:
 Reviewer:                              |         Points:  3
                                        |        Sponsor:  SponsorU-can
----------------------------------------+----------------------------------

Comment (by nickm):

 Replying to [comment:13 andrea]:
 > To prune the journal, we should remove duplicates and lines that
 introduce mappings
 > that will subsequently be removed as conflicts.  The easiest approach
 will be to
 > parse the journal, then re-generate it from scratch based on the in-
 memory structure,
 > but this will fail to preserve the comment/reserved lines and the
 ordering of non-
 > conflicting entries.  Do we care?

 I don't care about the ordering of the nonconflicting entries, but the
 comments and reserved lines probably should get preserved; especially
 since we're storing info in the comment.

 > If we do, then the next-easiest approach is probably an in-memory list
 of
 > lines, and hash tables indexing it to query all lines matching
 particular
 > keys, so when we spot a conflicting line during pruning we can find the
 > conflicting previous lines and remove them, then re-emit the file from
 > the list of remaining lines.
 >
 > Since there are no more than a few thousand active routers, a fully
 pruned
 > list cannot become unreasonably large, but this ticket is concerned with
 > running out of disk space due to excessive size.  I think if we remove
 > earlier conflicting lines immediately while parsing to prune we'll keep
 it
 > down to a reasonable size though.
 >
 > Has using tor_mmap_file() in keypin_load_journal() become a problem for
 > space reasons too?

 I don't _think_ so -- on every place where we'd consider running an
 authority, we have mmap. So we'd only need to worry if we were at risk of
 exhausting our address space. I think that if we have any authorities that
 are running on a 32-bit system (where address space exhaustion can
 happen), limiting the keypin file to 1G is probably safe.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/18320#comment:16>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online