[tor-bugs] #19249 [Metrics/Onionoo]: Onionoo server runs out of memory when importing a full month of data

[Tor Bug Tracker & Wiki]

#19249: Onionoo server runs out of memory when importing a full month of data
-----------------------------+---------------------
 Reporter:  karsten          |          Owner:
     Type:  defect           |         Status:  new
 Priority:  Medium           |      Milestone:
Component:  Metrics/Onionoo  |        Version:
 Severity:  Normal           |     Resolution:
 Keywords:                   |  Actual Points:
Parent ID:                   |         Points:
 Reviewer:                   |        Sponsor:
-----------------------------+---------------------

Comment (by iwakeh):

 An initial analysis:

 With enough memory provided the archive data import of a month doesn't
 take that long.

 First I altered the code to not run the download part in order to get a
 clearer picture there, i.e.
 the download descriptors part is omitted (hope, this isn't totally
 unreasonable).

 I ran the imports with the tars listed above and an otherwise empty
 Onionoo environment.
 Using 4g to 6g it needs less than two hours for the entire import:
 {{{
 2016-06-05 18:47:47,866 INFO o.t.o.cron.Main:191 Reading descriptors.
 2016-06-05 18:47:47,867 INFO o.t.o.u.DescriptorSource:153 Reading archived
 descriptors...
 ...
 2016-06-05 20:23:10,832 INFO o.t.o.cron.Main:236 Cleaning up.
 2016-06-05 20:23:10,833 INFO o.t.o.cron.Main:242 Done.
 2016-06-05 20:23:10,833 INFO o.t.o.cron.Main:246 Releasing lock.
 2016-06-05 20:23:10,865 INFO o.t.o.cron.Main:248 Released lock
 }}}

 Using way less causes the OOM, e.g. only 2g, which happens pretty quickly.

 When 3g are provided the processing runs for a very, very long time as
 visible in the log excerpt in the description. This happens b/c the jvm is
 garbage collection most of the time to stay running, but there is hardly
 any processing time left for the onionoo updater. With such a scarce
 amount of memory it just hits oom anyway (as in the initial error
 description).

 I assume, that the full onionoo single run will need memory >= 6g to run
 comfortably when importing a month worth of data.
 The 4g are not sufficient and just keep the jvm busy collecting garbage
 and the processing takes ages.

 Hope, it is possible to increase the memory! As an immediate measure at
 least.

 Still other means of reducing the memory consumption should be
 investigated.
 Those should be new trac issues then.

 PS: I also caught the nfe's (#19259).

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/19249#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online