[tor-bugs] #29693 [Core Tor/Tor]: Decrease probability of stochastic failures in test-slow

[Tor Bug Tracker & Wiki]

#29693: Decrease probability of stochastic failures in test-slow
----------------------------------------+----------------------------------
 Reporter:  teor                        |          Owner:  asn
     Type:  defect                      |         Status:  needs_review
 Priority:  High                        |      Milestone:  Tor:
                                        |  0.4.0.x-final
Component:  Core Tor/Tor                |        Version:  Tor:
                                        |  0.4.0.1-alpha
 Severity:  Normal                      |     Resolution:
 Keywords:  tor-ci, tor-test, 040-must  |  Actual Points:  0.5
Parent ID:                              |         Points:  0.5
 Reviewer:                              |        Sponsor:  Sponsor2-must
----------------------------------------+----------------------------------

Comment (by asn):

 Replying to [comment:11 teor]:
 > Here are our worst-case scenarios:
 > * a merge forward from 0.4.0 (or earlier) to master:
 >   * 3 (maint-0.4.0, release-0.4.0, master) * 14 jobs (travis, appveyor)
 >   * With N_TRIALS=3 and alpha=1e-06, a travis build will fail with 50%
 chance after 1651 builds.
 > * pull requests to 0.4.0 and master:
 >   * 2 (branch, PR) * 2 (0.4.0, master) * 14 jobs (travis, appveyor)
 >   * With N_TRIALS=3 and alpha=1e-06, a travis build will fail with 50%
 chance after 1238 builds.
 >
 > These error rates are acceptable, but only until we split off
 maint-0.4.1. Then we will need to go to 4 trials, or find some other
 solution.

 ACK. I also pushed https://github.com/torproject/tor/pull/824 which bumps
 it to 4. Let's minimize false positive interruptions for now and think of
 smarter approaches in #29847.

 For the record:
 {{{
 With N_TRIALS=4 and alpha=1e-08 we have failure_rate_for_test_suite
 9.99999959506e-08
 With N_TRIALS=4 and alpha=1e-08 we have travis build fail
 1.39999903326e-06
 With N_TRIALS=4 and alpha=1e-08, a travis build will fail with 50% chance
 after 495106 builds.
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29693#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online