[tor-bugs] #2334 [Torouter]: Torouter breaks with large cached-descriptors[.new] files

[Tor Bug Tracker & Wiki]

#2334: Torouter breaks with large cached-descriptors[.new] files
----------------------+-----------------------------------------------------
 Reporter:  karsten   |       Owner:  ioerror      
     Type:  defect    |      Status:  new          
 Priority:  blocker   |   Milestone:               
Component:  Torouter  |     Version:  Tor: 0.2.1.26
 Keywords:            |      Parent:               
----------------------+-----------------------------------------------------

Comment(by karsten):

 Replying to [comment:2 karsten]:
 > As a next step we should investigate how many descriptors we need to
 cache as a bridge (or as a relay). We can probably throw out descriptors
 more quickly.
 >
 > I'm going to set up a new directory mirror to a) take hourly snapshots
 of the cached-descriptors* files and b) log the requested descriptor
 digests (using log granularity of 15 minutes). With these data we can
 answer 1) for what consensuses we're keeping descriptors and 2) which
 consensuses clients use to decide which descriptors to download. Once we
 know that clients don't download old descriptors, we can stop caching
 them.

 I ran a similar experiment to the one described above. I modified a
 moderately fast directory mirror to log which descriptors are requested
 and collected a few hours of those logs. But instead of hourly snapshots I
 took the cached-descriptors* files of a relay that was running for a
 couple of days. No reason to wait a couple of days if we can have results
 right now.

 See the attached graph descriptor-age-2011-01-03.png.

 You'll see that 97.12% of requested, but only 19.61% of cached descriptors
 were last referenced in a consensus at most 3 hours old. Also note that
 15.21% of cached descriptors are not referenced in any consensus at all.

 That means if we threw out all descriptors that are not referenced in one
 of the last 3 consensuses, we wouldn't be able to answer 2.88% of the
 requests, but would reduce the cached-descriptors* files to 1/5 of the
 current size.

 What about these 2.88% of requests? Can the requesting clients even make
 use of descriptors of relays that are very likely not running anymore?

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/2334#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online