lcov test coverage reports
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, Would there be interest in using lcov [0] for test coverage analysis? Obviously, as lcov is an external pacakge, the existing scripts that only use built in gcov commands must remain. However, if I were to create a script that generates a pretty, HTML coverage report using lcov & genhtml, would this be something others would use? And how would you like to see this implemented? As a custom make target (e.g., make htmlcoverage + autotools black magic to silently disable/hide the target if we can't find lcov), or as a standalone script (i.e., ./scripts/test/run_lcov.sh). For what it's worth, I've manually run lcov on tor after enabling coverage through ./configure, and the output seems sane, so I think from a technical standpoint this would work. [0]: http://ltp.sourceforge.net/coverage/lcov.php Cheers, daube -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTnYryAAoJEBfCjrGktO5q+FgP/26ZgVZOAKSOBvWr6z2HYEku 1Yo+xFqTXk/Fw6yilv6Z4Jwc0Ar2B/VjHfLpOL1Vm+hFZ+VYsObSt9JGG2baItbG IrcedfjCuyjfaP1PloZoDM24FEbXqjseQO2EZ8bXbtErsW52Z4uAR3nYYcyXCCYp 1gi6lEwN/Yn0xm+fGgoz6/Z5FXvETdzbWLfhZoT4jFsnQsp/fByld55JLyft/8gA OrH3QQYGgXZkwIsnTkbLO3xWpXbqC5kU/ngCOL9WXXU99cgnW+bec7DA+cHhTqh/ 7oZD54noQ9XYtOA7AhAEOiTWnJUca2vV1KFPVBKSlqfkOhF3BRANHo2cIpsvgeBK 9+tcTyuwcFGAJVmW9l/AN3Y89CepCxnk6B1SSLFxuUVupAz2ilBM7KVtgj2ilJnC 2GjfuQB0RG/FW0CSEaa+DDtHfzCMfZNwGb6Fs1L41P8NnlGVFOz11NLlMG3t5ZpT 0OrncYaeVz74H+63Ig1ZwOjlfgUr9BHMLeYdwn9FmqIggyFRt1Z5PnNCaMuExs/v WQbLMP0oZ9XK1f51vfFVFYk/n1nh1c4J40/3I5B525GANAZ3fxTDLp0oSp/gNr5X lXPaF32KexhOiup5n1d6x8oJPzeNmex5CfcT29bZMPavF8p8v0OdZrUtU40bG7nO PUlA3EFfva1YW0aTRKY5 =+v+h -----END PGP SIGNATURE-----
On Sun, Jun 15, 2014 at 8:00 AM, David Murray <spam@kdmurray.id.au> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi all,
Would there be interest in using lcov [0] for test coverage analysis?
Obviously, as lcov is an external pacakge, the existing scripts that only use built in gcov commands must remain.
However, if I were to create a script that generates a pretty, HTML coverage report using lcov & genhtml, would this be something others would use? And how would you like to see this implemented? As a custom make target (e.g., make htmlcoverage + autotools black magic to silently disable/hide the target if we can't find lcov), or as a standalone script (i.e., ./scripts/test/run_lcov.sh).
So, I think that if I'm reading the documentation right, all that should be necessary would be an additional make target. (I'm assuming that the make target wouldn't call "configure" -- we shouldn't define our make targets that do that, IMO.) (If I'm reading this properly, lcov uses the same command-line arguments to gcc that gcov does, so it could also depend on the --enable-coverage configure flag.) I don't think it's necessary to have the target disabled when there is no lcov, though it would be nice if it gave a useful error message instead of "lcov: command not found". Additionally, one thing that I'd really love to see -- though I don't at all know whether lcov can do this as it stands -- is a semantic diff between two coverage outputs. When writing new unit tests at random, or when checking functions that you know need coverage, it's useful to ask "Which lines are not covered?" But for validating new code and checking patches, the question I want to ask is "Which *new* or *changed* lines are not covered?" We kinda have a script for this now, but learning to read its outputs is not as easy as I'd like. cheers, -- Nick
On Mon, Jun 16, 2014 at 10:55 AM, Nick Mathewson <nickm@freehaven.net> wrote: [...]
Additionally, one thing that I'd really love to see -- though I don't at all know whether lcov can do this as it stands -- is a semantic diff between two coverage outputs. When writing new unit tests at random, or when checking functions that you know need coverage, it's useful to ask "Which lines are not covered?" But for validating new code and checking patches, the question I want to ask is "Which *new* or *changed* lines are not covered?"
We kinda have a script for this now, but learning to read its outputs is not as easy as I'd like.
It would appear at first glance that you can do something *almost* like this using lcov --diff and genhtml --highlight. Those options together seem to be able to let you know about lines that used to be covered but which are no longer covered. (I'm hoping to learn about likes that changed or started to exist which are uncovered.)
On Mon, Jun 16, 2014 at 4:07 PM, Nick Mathewson <nickm@freehaven.net> wrote:
On Mon, Jun 16, 2014 at 10:55 AM, Nick Mathewson <nickm@freehaven.net> wrote: [...]
Additionally, one thing that I'd really love to see -- though I don't at all know whether lcov can do this as it stands -- is a semantic diff between two coverage outputs. When writing new unit tests at random, or when checking functions that you know need coverage, it's useful to ask "Which lines are not covered?" But for validating new code and checking patches, the question I want to ask is "Which *new* or *changed* lines are not covered?"
We kinda have a script for this now, but learning to read its outputs is not as easy as I'd like.
It would appear at first glance that you can do something *almost* like this using lcov --diff and genhtml --highlight. Those options together seem to be able to let you know about lines that used to be covered but which are no longer covered. (I'm hoping to learn about likes that changed or started to exist which are uncovered.)
I asked this question on the ltp list, and got an answer: http://sourceforge.net/p/ltp/mailman/message/32486080/ This is a kludge too, but we should investigate whether it's better than our current kludge. -- Nick
participants (2)
-
David Murray -
Nick Mathewson