[tor-bugs] #29425 [Metrics/Statistics]: Write integration tests for data-processing modules

[Tor Bug Tracker & Wiki]

#29425: Write integration tests for data-processing modules
-------------------------------------+--------------------------------
 Reporter:  karsten                  |          Owner:  metrics-team
     Type:  enhancement              |         Status:  needs_revision
 Priority:  Medium                   |      Milestone:
Component:  Metrics/Statistics       |        Version:
 Severity:  Normal                   |     Resolution:
 Keywords:  metrics-roadmap-2019-q2  |  Actual Points:
Parent ID:                           |         Points:  8
 Reviewer:  irl                      |        Sponsor:
-------------------------------------+--------------------------------
Changes (by irl):

 * status:  needs_review => needs_revision


Comment:

 git-annex is what I have used in the past for storing large test data in
 git repositories. Our git server does not currently support this but we
 could look at adding support. If it requires us to upgrade the gitolite
 then that is not an easy project, if we are running a new enough version
 then it could just be some lines in the config.

 I looked at running the script:

 {{{
 Cloning metrics-web Git repository into metrics-web/ subdirectory...
 Cloning into 'metrics-web'...
 remote: Counting objects: 14553, done.
 remote: Compressing objects: 100% (1213/1213), done.
 remote: Total 14553 (delta 977), reused 503 (delta 233)
 Receiving objects: 100% (14553/14553), 16.55 MiB | 2.64 MiB/s, done.
 Resolving deltas: 100% (8303/8303), done.
 Bootstrapping development environment...
 Submodule 'src/build' (https://git.torproject.org/metrics-base.git)
 registered for path 'src/build'
 Submodule 'src/submods/metrics-lib' (https://git.torproject.org/metrics-
 lib.git) registered for path 'src/submods/metrics-lib'
 Cloning into '/tmp/metrics-web-integ-tests/metrics-web/src/build'...
 Cloning into '/tmp/metrics-web-integ-tests/metrics-web/src/submods
 /metrics-lib'...
 Submodule path 'src/build': checked out
 'e639c697e9e94c6dbb26e946e5247c20a62c0661'
 Submodule path 'src/submods/metrics-lib': checked out
 '23927c2777f273c42ad3e75fc0a2940ed8eb4bf6'
 Submodule 'src/build' (https://git.torproject.org/metrics-base) registered
 for path 'src/build'
 Cloning into '/tmp/metrics-web-integ-tests/metrics-web/src/submods
 /metrics-lib/src/build'...
 Submodule path 'src/build': checked out
 'e639c697e9e94c6dbb26e946e5247c20a62c0661'
 Replacing absolute paths in build.xml with relative paths...
 sed: can't read s/.srv.metrics.torproject.org.metrics/./: No such file or
 directory
 }}}

 I like the concept though. This is the sort of thing we can run in CI or
 Vagrant to be able to run it in disposable environments.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29425#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online