[tor-bugs] #29693 [Core Tor/Tor]: Decrease probability of stochastic failures in test-slow

Tor Bug Tracker & Wiki blackhole at torproject.org
Thu Mar 21 14:38:40 UTC 2019


#29693: Decrease probability of stochastic failures in test-slow
----------------------------------------+----------------------------------
 Reporter:  teor                        |          Owner:  asn
     Type:  defect                      |         Status:  assigned
 Priority:  High                        |      Milestone:  Tor:
                                        |  0.4.0.x-final
Component:  Core Tor/Tor                |        Version:  Tor:
                                        |  0.4.0.1-alpha
 Severity:  Normal                      |     Resolution:
 Keywords:  tor-ci, tor-test, 040-must  |  Actual Points:
Parent ID:                              |         Points:  0.5
 Reviewer:                              |        Sponsor:  Sponsor2-must
----------------------------------------+----------------------------------

Comment (by asn):

 OK, my suggestion is to increase N_TRIALS from 2 to 3 for now, and open a
 ticket for the future to do more advanced stuff like the suggestions from
 comment:4.

 In particular with N_TRIALS=2 now and 14 travis jobs per build, we have
 1.3% probability of a travis build failing because of the stoch tests.
 This means that one travis build will fail after 50 travis builds with 50%
 chance.

 If we bump N_TRIALS to 3, then we have 0.013% probability of a travis
 build failing  because of the stoch tests. This means that one travis
 build will fail after 4952 travis builds with 50% chance. Not so annoying
 anymore.

 Here is a (potentially borken) python script I used to calculate the data
 above, along with the documentation here
 https://github.com/torproject/tor/blob/938d97cb0d4acfdd1ea57ec0a3094bcc2101f13d/src/test/test_prob_distr.c#L866
 :
 {{{
 N_TRIALS = 3
 # Probability of a stochastic test failing
 alpha = pow(0.01, N_TRIALS)

 # number of stochastic tests
 n_stoch_tests = 10
 # Probability of at least one stochastic test failing
 failure_rate_for_test_suite = 1 - pow(1 - alpha, n_stoch_tests)
 print("With N_TRIALS={} and alpha={} we have failure_rate_for_test_suite
 {}".format(N_TRIALS, alpha, failure_rate_for_test_suite))

 # Number of travis jobs per build
 n_travis_jobs = 14
 # Probability of at least one travis job failing
 failure_rate_of_travis_build = 1 - pow(1 - failure_rate_for_test_suite,
 n_travis_jobs)
 print("With N_TRIALS={} and alpha={} we have travis build fail
 {}".format(N_TRIALS,alpha, failure_rate_of_travis_build))

 for n in xrange(5000):
     # Probability of travis build failing after n builds
     p = 1 - pow(1 - failure_rate_of_travis_build, n)
     if p > 0.5:
         print("With N_TRIALS={} and alpha={}, a travis build will fail
 with 50% chance after {} builds.".format(N_TRIALS,alpha,n))
         break
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29693#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online


More information about the tor-bugs mailing list