[tor-bugs] #29693 [Core Tor/Tor]: Decrease probability of stochastic failures in test-slow

[Tor Bug Tracker & Wiki]

#29693: Decrease probability of stochastic failures in test-slow
----------------------------------------+----------------------------------
 Reporter:  teor                        |          Owner:  asn
     Type:  defect                      |         Status:  assigned
 Priority:  High                        |      Milestone:  Tor:
                                        |  0.4.0.x-final
Component:  Core Tor/Tor                |        Version:  Tor:
                                        |  0.4.0.1-alpha
 Severity:  Normal                      |     Resolution:
 Keywords:  tor-ci, tor-test, 040-must  |  Actual Points:
Parent ID:                              |         Points:  0.5
 Reviewer:                              |        Sponsor:  Sponsor2-must
----------------------------------------+----------------------------------

Comment (by teor):

 Another failure at:
 https://travis-ci.org/tlyu/tor/jobs/508233862#L4244

 Here's what Riastradh said on IRC today:

 > nickm: Hi!  You are welcome to publish the IRC discussion we had earlier
 about stochastic tests.
 > (I don't remember which one, but you have my permission to publish all
 of the discussions we've had about stochastic tests and the distribution
 samplers since November or whenever this all started.)
 > I saw that there was an issue about changing the false positive rate.
 I'm low on energy right now, but here's the three things that I would
 suggest doing, some of which I might do if I had more energy:
 > 1. Write some tests of the tests -- that is, write a _buggy_ sampler for
 a distribution, and apply a stochastic test to it, and confirm the
 stochastic test _fails_.
 > Examples:
 https://github.com/brave/crypto/blob/master/test/randomTest.js,
 https://github.com/probcomp/crosscat/blob/master/cpp_code/tests/test_random_number_generator.cpp
 > You'll want to estimate the false positive rate of these test-tests
 (i.e., the statistical power of the tests to detect the bugs) empirically,
 since for most bugs there will be no neat analytic expression for it.
 > 2. Tweak NTRIALS and NPASSES_MIN so that the false positive rates of the
 usual tests _and_ of the test-tests are acceptable.  The first one you can
 compute analytically as I described in past conversations; the second will
 necessarily be based on the empirical measurements in (1).
 > 3. Teach the CI to report the alarm rates -- not just number of alarms,
 but ratio of alarms to total tests run.  And keep this state continuously
 across CI jobs so it can be aggregated over time.

 In 0.4.0, I think increasing NTRIALS is our best option. I don't have
 access to the previous conversation about NTRIALS. If we can't find it,
 let's ask Riastradh, or just double NTRIALS.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29693#comment:4>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online