[tor-bugs] #19651 [Metrics/CollecTor]: add missing scripts to collector.git

[Tor Bug Tracker & Wiki]

#19651: add missing scripts to collector.git
-------------------------------+------------------------------
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  task               |         Status:  needs_review
 Priority:  Medium             |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+------------------------------

Comment (by karsten):

 Replying to [comment:5 iwakeh]:
 > The second script is now added to the create tarball one.
 > The reasoning behind is that the current use-case runs the scripts
 together anyway.
 >
 > One variable setting I could only guess: I assume that `archive` is the
 same as the absolute path
 `/srv/collector.torproject.org/collector/archive` on main CollecTor.

 Yep, that's the directory.

 > With the configurable paths I let it run on my mirror and everything
 worked fine, but took ages, well hours.  Maybe, we should up the priority
 on the improve create tarball topic
 ([https://trac.torproject.org/projects/tor/wiki/doc/CollecTor/Improvements#FurtherSketchesofAreasforImprovements
 mentioned here])?

 It does take ages to produce tarballs, but that's also because we're using
 `xz -9e` which saves a lot of disk space and bandwidth, at the price of
 long creation time.

 > In addition, I'd like to integrate this script process in the java
 application, i.e., replace the script and let java handle all.  This could
 also pause the tarball-creation momentarily when new downloads take place
 and start it as soon the download stops, in general there would be more
 control.

 Sure, sounds useful.  In the past I tried to append to existing tarballs,
 which worked for a while but then produced some weird problems.  Happy to
 dig out what that was when we improve this.

 However, I wouldn't want to prioritize this over, say, syncing between
 CollecTor instances.

 > Well, some thoughts while waiting for the script to finish ;-)
 >
 > Please review the extended create tarballs
 [https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?h=task-19651
 -link-script&id=18a4290b58eaa67cd2ca82238789b3919f0d3eca script].

 Looks good, merged!  Thanks.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/19651#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online