[tor-bugs] #3406 [Torctl]: High memory usage from parse_ns_body

[Tor Bug Tracker & Wiki]

#3406: High memory usage from parse_ns_body
--------------------+-------------------------------------------------------
 Reporter:  atagar  |          Owner:     
     Type:  defect  |         Status:  new
 Priority:  normal  |      Milestone:     
Component:  Torctl  |        Version:     
 Keywords:          |         Parent:     
   Points:          |   Actualpoints:     
--------------------+-------------------------------------------------------
 From earlier today...

 09:16 < atagar> Figured out my memory issue. It's from calling TorCtl's
 get_network_status function which consumes 3.5 MB of memory at once.
 09:16 < atagar> I immediately discard those results and the memory is
 freed, however because of how large objects are allocated in python 'ps'
 is never aware that the memory's been released:
 09:16 < atagar> http://effbot.org/pyfaq/why-doesnt-python-release-the-
 memory-when-i-delete-a-large-object.htm
 09:16 < atagar> Fix: either ignore it (it's misreported memory usage) or
 implement my own less memory intensive version of get_network_status.
 09:20 < mikeperry> atagar: does python *never* release said memory? in
 otherwords, do you experience this as a leak, or just a consistent 3.5MB
 of unneeded overhead?
 09:20 < mikeperry> s/release/reuse
 09:22 < atagar> I'm not quite sure how to tell the difference. If I call
 get_network_status and discard the results (ie, don't get a reference to
 the nslist object it returns) then ps reports that memory as being used
 for the life of the python process.
 09:23 < atagar> though I'd imagine this memory would be reused by python
 for other activity when needed
 09:25 < mikeperry> hopefully. there is definitely a leak in the bw
 authorities that is proving ridiculously hard to track down. we're down to
 it either being a python leak, an sqlite leak, or a python-sqlite bindings
 leak
 09:26 < atagar> side note: the memory all comes from the NetworkStatus
 objects generated on line 1246 (ie, the problem is definitely because we
 have a list with all that memory allocated at one time, rather than the
 incremental results contributing to it)
 09:27 < atagar> ie, if we were able to have an iterator pattern here then
 there would be no issue
 09:28 < mikeperry> if you want to write an intermediate funtion that only
 gives you them one at a time, you can do that. just don't change the
 external behavior of the current function.
 09:28 < mikeperry> I'd merge that
 09:29 < atagar> Will do. I'm a little worried about a future change
 accidently making a get_network_status call, but I'll edit the comment to
 warn of this.
 09:29 < atagar> btw, do you have any idea when you'll be able to look at
 the five torctl patches I sent on Monday?
 09:30  * nickm is going to be semi- offline for most of today afternoon
 doing errands writing reviews and hacking.  Hoping this is a good idea. :)
 09:34 < atagar> mikeperry: I'll also fix update_consensus so it doesn't
 encounter this issue (no external change but will avoid new consensus
 events from triggering huge memory usage)

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/3406>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online