[tor-bugs] #4086 [Analysis]: Compare performance of TokenBucketRefillInterval params in simulated network

[Tor Bug Tracker & Wiki]

#4086: Compare performance of TokenBucketRefillInterval params in simulated
network
-------------------------------------+--------------------------------------
 Reporter:  arma                     |          Owner:       
     Type:  task                     |         Status:  new  
 Priority:  normal                   |      Milestone:       
Component:  Analysis                 |        Version:       
 Keywords:  performance flowcontrol  |         Parent:  #4465
   Points:                           |   Actualpoints:       
-------------------------------------+--------------------------------------

Comment(by arma):

 Replying to [comment:25 robgjansen]:
 > My previous experiments were run on EC2 and with consensus weights as
 capacity, as opposed to my server with observed bandwidth as capacity. (I
 made the classic mistake here of changing too many variables.) I can turn
 off the CPU delay model, and we can take a look at performance under the
 assumption that CPU will never be a bottleneck, if you'd like.

 Can't hurt, might help?

 > > Do you think there's a lot of variance from one run to the next? Is
 the variance from the choice of topology? You're already averaging lots of
 individual fetches from clients I believe. What else might be big
 contributing factors to variance?
 >
 > It may depends on how clients pick paths, as that directs network
 congestion. Smaller networks may have more variance since if you get
 unlucky and happen to clog up some important nodes, it affects a high
 fraction of clients.

 Ok. I'm increasingly thinking that a "run" should be k runs in a row,
 averaged. Even though it takes longer. Because right now, if I understand
 you right, there's a risk that we look at the output for a given
 experiment and draw a conclusion that the code change is good or bad, when
 in fact there's a good chance that it's just variation in runs that
 produced the difference. Hopefully that will be pretty easy to automate
 too? What's a good value of k -- 5 or 10 maybe?

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/4086#comment:26>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online