[tor-bugs] #29425 [Metrics/Statistics]: Write integration tests for data-processing modules

[Tor Bug Tracker & Wiki]

#29425: Write integration tests for data-processing modules
-------------------------------------+--------------------------
 Reporter:  karsten                  |          Owner:  karsten
     Type:  enhancement              |         Status:  accepted
 Priority:  Medium                   |      Milestone:
Component:  Metrics/Statistics       |        Version:
 Severity:  Normal                   |     Resolution:
 Keywords:  metrics-roadmap-2019-q2  |  Actual Points:
Parent ID:                           |         Points:  8
 Reviewer:                           |        Sponsor:
-------------------------------------+--------------------------

Comment (by karsten):

 I finally got it all working! There are still minor differences between
 macOS and Linux:

  - `bandwidth.csv` relies on the import order of descriptors, because
 either the first or last descriptor wins (I did not look at the code in
 detail). The differences in the output are tiny, but detectable. However,
 fixing this requires extensive changes to the bwhist module.
  - `platforms.csv` and `userstats-combined.csv` rely on sorting order,
 which is apparently platform-dependent. After sorting these files, they're
 equivalent. This is unfortunate, but acceptable, because we do not
 guarantee a specific sorting order. We could fix this by changing SQL
 statements to something like ORDER BY LOWER(x).

 I decided to ignore these differences and expect the output as Linux would
 produce it.

 I also made a temporary decision regarding git-annex vs. adding large
 files to git by creating this relatively large repository on GitHub for
 now: https://github.com/kloesing/metrics-test

 I'd like to use these integration tests for some more actual metrics-web
 changes, including the Java 8 Date/Time updates. It's okay for me to just
 keep it on GitHub without review and later get it reviewed and on Tor's
 Git. For example, we could do that in April with more time on our hands.
 Hope that works for you!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29425#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online