[tor-bugs] #4687 [Metrics Data Processor]: Provide metrics data from the last 3--7 days via rsync

[Tor Bug Tracker & Wiki]

#4687: Provide metrics data from the last 3--7 days via rsync
------------------------------------+---------------------------------------
 Reporter:  karsten                 |          Owner:  weasel
     Type:  enhancement             |         Status:  new   
 Priority:  normal                  |      Milestone:        
Component:  Metrics Data Processor  |        Version:        
 Keywords:                          |         Parent:        
   Points:                          |   Actualpoints:        
------------------------------------+---------------------------------------
 metrics-db provides compressed tarballs of its collected and sanitized
 files via rsync, so that others can copy them for their analyses or
 services.  But these tarballs are only updated once per day, and updating
 them more often has performance implications on the metrics-db host.
 Also, having others update their services from potentially very large
 monthly tarballs has performance issues, too.

 Can we make the collected and sanitized descriptors from the last, say, 3
 to 7 days available via rsync?  For a point of reference, in the first 8.5
 days of December, metrics-db collected and sanitized 178,000 files being
 2.4G in size.  Are those numbers totally crazy for rsync?

 We could implement this by having a script copy new descriptors to a
 directory that is then made available via rsync.  The same script would
 also delete files that are older than 3--7 days.

 This ticket is part of decoupling metrics-db from metrics-web and other
 metrics services.  Right now, metrics-web uses the metrics-db output using
 a symbolic link the file system.  That means metrics-db and metrics-web
 rely on running on the same host.  We should change that to allow others
 to run their own metrics-web or or similar service.

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/4687>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online