[tor-bugs] #29425 [Metrics/Statistics]: Write integration tests for data-processing modules

[Tor Bug Tracker & Wiki]

#29425: Write integration tests for data-processing modules
-------------------------------------+------------------------------
 Reporter:  karsten                  |          Owner:  metrics-team
     Type:  enhancement              |         Status:  needs_review
 Priority:  Medium                   |      Milestone:
Component:  Metrics/Statistics       |        Version:
 Severity:  Normal                   |     Resolution:
 Keywords:  metrics-roadmap-2019-q2  |  Actual Points:
Parent ID:                           |         Points:  8
 Reviewer:  irl                      |        Sponsor:
-------------------------------------+------------------------------
Changes (by karsten):

 * status:  needs_revision => needs_review


Comment:

 Replying to [comment:5 irl]:
 > git-annex is what I have used in the past for storing large test data in
 git repositories. Our git server does not currently support this but we
 could look at adding support. If it requires us to upgrade the gitolite
 then that is not an easy project, if we are running a new enough version
 then it could just be some lines in the config.

 Maybe we don't have to do this with a dedicated metrics-test repository,
 though. I wouldn't want to dump tons of descriptors into the actual code
 repositories, but if this repository is just for (integration) testing,
 people will have to fetch the data anyway. Don't feel strongly, though.

 > I looked at running the script:
 >
 > {{{
 > Cloning metrics-web Git repository into metrics-web/ subdirectory...
 > Cloning into 'metrics-web'...
 > remote: Counting objects: 14553, done.
 > remote: Compressing objects: 100% (1213/1213), done.
 > remote: Total 14553 (delta 977), reused 503 (delta 233)
 > Receiving objects: 100% (14553/14553), 16.55 MiB | 2.64 MiB/s, done.
 > Resolving deltas: 100% (8303/8303), done.
 > Bootstrapping development environment...
 > Submodule 'src/build' (https://git.torproject.org/metrics-base.git)
 registered for path 'src/build'
 > Submodule 'src/submods/metrics-lib' (https://git.torproject.org/metrics-
 lib.git) registered for path 'src/submods/metrics-lib'
 > Cloning into '/tmp/metrics-web-integ-tests/metrics-web/src/build'...
 > Cloning into '/tmp/metrics-web-integ-tests/metrics-web/src/submods
 /metrics-lib'...
 > Submodule path 'src/build': checked out
 'e639c697e9e94c6dbb26e946e5247c20a62c0661'
 > Submodule path 'src/submods/metrics-lib': checked out
 '23927c2777f273c42ad3e75fc0a2940ed8eb4bf6'
 > Submodule 'src/build' (https://git.torproject.org/metrics-base)
 registered for path 'src/build'
 > Cloning into '/tmp/metrics-web-integ-tests/metrics-web/src/submods
 /metrics-lib/src/build'...
 > Submodule path 'src/build': checked out
 'e639c697e9e94c6dbb26e946e5247c20a62c0661'
 > Replacing absolute paths in build.xml with relative paths...
 > sed: can't read s/.srv.metrics.torproject.org.metrics/./: No such file
 or directory
 > }}}
 >
 > I like the concept though. This is the sort of thing we can run in CI or
 Vagrant to be able to run it in disposable environments.

 Hrmmm, looks like an incompatibility between macOS and Linux sed... Can
 you edit your script and use this line instead?

 {{{
 sed -i.bak 's/.srv.metrics.torproject.org.metrics/./' build.xml
 }}}

 If this works okay, I'll request a metrics-test repository and push the
 current script there. Except if you care more strongly about the git-annex
 part in which case I'd reach out to hiro and see what we can do there.

 Thanks!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29425#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online