[tor-bugs] #25227 [Core Tor/Tor]: Avoid storing all Tor nodes in RAM

[Tor Bug Tracker & Wiki]

#25227: Avoid storing all Tor nodes in RAM
------------------------------+--------------------------------
     Reporter:  teor          |      Owner:  (none)
         Type:  enhancement   |     Status:  new
     Priority:  Medium        |  Milestone:  Tor: 0.3.4.x-final
    Component:  Core Tor/Tor  |    Version:
     Severity:  Normal        |   Keywords:  tor-client
Actual Points:                |  Parent ID:
       Points:  5             |   Reviewer:
      Sponsor:  Sponsor8-can  |
------------------------------+--------------------------------
 {{{
 <teor4> ahf: I've been thinking about reducing Tor's RAM usage by sampling
 from each consensus, rather than keeping it all in RAM
 <teor4> there are ways to stream the consensus, pick a weighted sample,
 and just keep those nodes
 <teor4> https://en.wikipedia.org/wiki/Reservoir_sampling
 <teor4> it would really help the iOS VPN, and other embedded impls
 <teor4> The catch is that if you run out of sampled nodes, you need to re-
 parse the consensus
 <+ahf>  teor4: i had been thinking a bit in the back of my head if we
 could refactor the code that is used for accessing the consensus to be a
 bit more pluggable to experiment with different ways of backing them
 <+ahf>  teor4: on ios i think we can mmap() files without them adding to
 the memory limit of the application too
 <teor4> that would be helpful, because we could only keep the selected
 node queues in RAM
 <teor4> if we got the queue size right, we could regenerate them once per
 consensus
 <+ahf>  do you want to create a ticket with this? maybe add sponsor8-can
 to the keywords
 }}}

 We could be smart about node selection, and just store some weights for
 each node. And then when we use too much RAM for nodes, we could evict
 older nodes.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25227>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online