<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 30, 2017 at 3:57 PM, David Goulet <span dir="ltr"><<a href="mailto:dgoulet@ev0ke.net" target="_blank">dgoulet@ev0ke.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello everyone!<br>
<br>
DISCLAIMER: The following is enormous and tries to describe in some level of<br>
details the situation in tor with connection<->channel<-><wbr>scheduler. This comes<br>
after we've merged the KIST scheduler, we've realized many things we'ren't what<br>
they were suppose to be or meant for. In the end, I'm asking questions so we<br>
can move forward with development and fixing things.<br>
<br>
Last thing before you start your journey in the depth of Tor, the 3 subsystems<br>
I'm going to talk about and how they interact are kind of very complicated so<br>
it is very possible that I might have gotten things wrong or miss some details.<br>
Please, point them out so we can better document, better be informed and make<br>
good decisions. I plan to document as much as I can from this process for a new<br>
file in torguts.git repository.<br><br></blockquote><div><br></div><div>Snipping the analysis, and going straight to the conclusions. I'll leave one sentence in the analysis because it's such a great summary:</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Many things are problematic currently</blockquote><div><br></div><div>They sure are. :)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
== Part Four - The Conclusion ==<br>
<br>
Through this epic journey, we've discovered some issues as well as design<br>
problems. Now the question is what should and can do about it?<br>
<br>
In a nutshell, there are a couple of questions we should ask our selfves and<br>
try to answer so we can move forward:<br>
<br>
* I believe now that we should seriously discuss the relevance of channels.<br>
Originally, the idea was good that is providing an abstraction layer for the<br>
relay to relay handshake and send/process cells related to the protocol. But,<br>
as of now, they are half doing it.<br>
<br>
There is an important cost in code and maintanance of something that is not<br>
properly implemented/finished (channel abstraction) and also something that<br>
is unused. An abstraction implemented only for one thing is not really useful<br>
except maybe to offer an example for others? But we aren't providing a good<br>
example right now imo...<br>
<br>
That being said, we can spend time fixing the channel subsystem, trying to<br>
turn it in a nicer interface, fixing all the issues I've described above (and<br>
I suspect there might be more) so the cell scheduler can play nicely with<br>
channels. Or, we could rip them off eliminating lots of code and reducing our<br>
technical debt. I would like us to think about what we want seriously because<br>
that channel subsystem is _complicated_ and very few of us fully understands<br>
it afaict.<br>
<br>
Which would bring us back to (which is btw basically what we have now<br>
considering the channel queues are useless):<br>
<br>
conn inbuf -> circ queue -> conn outbuf<br>
<br>
If we don't want to get rid of channel, the fixes are non trivial. For<br>
starter, we have to decide if we want to keep the channel queue or not and if<br>
yes, we need to almost start from square 1 in terms of testing because we<br>
would basically introduce a new layer of queuing cells.<br></blockquote><div><br></div><div>So, this is the question I'm least sure about. Please take the following as tentative.</div><div><br></div><div>I think that the two choices ("refactor channels" and "rip out channels") may be less different than we think. Neither one is going to be trivial to do, and we shouldn't assume that sticking everything together into one big type will actually make the code _simpler_.</div><div><br></div><div>The way I think about the code right now, "channel" is an interface which "connection_or" implements, and there is no meaningful barrier between connection_or and channeltls. I _do_ like the idea of keeping some kind of abstraction barrier, though: a "channel" is "whatever we can send and receive cells from", whereas an "or_connection" has a lot of other baggage that comes with it.<br></div><div><br></div><div><div>From my POV, we *should* definitely abolish the channels' queues, and minimize the amount of logic that channels do on their own. I'm not sure if we should rip them out entirely, or just simplify them a lot. I don't think either necessarily simpler or less bug-prone than the other.</div></div><div><br></div><div>Perhaps we should sketch out what the new interface would look like? Or maybe do an hour or two worth of exploratory hacking on each approach?</div><div><br></div><div>(This reminds me of another change I want someday, which is splitting edge_connection_t into an "edge_connection" type that implements a "stream" interface: right now, we have quite a few streams that aren't actually edge connections, but which use the type anyway.) </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
* Dealing with the DESTROY cell design issue will require a bit more tricky<br>
work I think. We must not starve circuit with a DESTROY cell pending to be<br>
sent else the other side keeps sending data. But we should also not starve<br>
all the circuits because if we ever need to send a gazillion DESTROY cell in<br>
priority, we'll make the relay useless (DoS vector).<br>
<br>
The question is, do we trust our EWMA policy to be wise enough to pick the<br>
circuit in a reasonable amount of time so we can flush the DESTROY cell from<br>
the circuit queue? Or we really need to bypass or prioritize somehow that<br>
cell in order to send them asap in order to avoid load on the network because<br>
the other side of the circuit is still sending?<br></blockquote><div><br></div><div>So, elsewhere in the thread, folks have been discussing whether a circuit that's going to send a DESTROY cell should flush its pending cells first.<br></div><div><br></div><div>The answer is sadly, "it depends". </div><div><br></div><div>Case 1: Suppose Alice is downloading a webpage. Suppose we are the middle relay and we lose our connection to the exit. It would be nice to keep flushing the data we have towards Alice -- maybe. If she can use partial data. But any data that Alice sends to us would be lost, so it would be good if we had some way to tell Alice "stop sending please".</div><div><br class="gmail-Apple-interchange-newline">Case 2: Suppose Alice is uploading something to a webserver. Suppose we are the middle relay and we lose our connection from Alice. In this case, there's no point in sending any more data towards the webserver before we send it a DESTROY cell. (Even if Alice was in the middle of a big upload, she'll need to repeat any part of it that wasn't ACKed, since she won't know what was received and what wasn't.)<br></div><div><br></div><div>Case 3: Suppose we hit our OOM killer. In this case, we had better discard all the data on the circuit we're killing, or we're vulnerable to "sniper attacks" again.</div><div><br></div><div>So it's clear that sometimes we should dump the data, and sometimes we shouldn't. I think this is an independent question from what we're asking here. (My own take is that solving case 1 right requires "RELAY_TRUNCATED" cells, which I believe we don't implement today.)</div><div><br></div><div>What we're asking here is: how can we reintegrate DESTROY cells with the rest of the scheduler logic?</div><div><br></div><div>I think that, from a priority POV, DESTROY cells are in a certain sense the _opposite_ of traffic, and we might actually want to treat them differently from data cells. Consider that if we have a choice between DESTROYing a busy circuit or a quiet one, we will save more bandwidth by destroying the busy circuit first, so that no more data is sent to us over it.<br></div><div><br></div><div>On the other hand, this doesn't mean that the FIFO structure we have today is a good idea at all. It probably makes sense to use the same priority queue-based scheduler thing that we use everywhere else, but possibly with a different (inverted??) priority parameter for destroyed circuits.<br></div><div><br></div><div>One more thing to be aware of: the destroy_cell_queue exists in part because we tear down circuits at the same time that we queue their destroy cells. If we changed Tor so that "destroyed" circuits were kept around somehow until their cells could be sent, then we'd be introducing a new state to our state machine, to represent circuits that were schedulable but not actually usable for traffic. We'd need to be careful to handle that correctly: this kind of "unusable object that still exists" has caused us problems before. (The solution I like best for avoiding this confusion is to make it so the scheduler can schedule two types of "schedule-able" things: circuits, and "pending destroy cells".) </div><div> </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
* In the short term, we should get rid of Vanilla scheduler because it<br>
complefixies a lot the scheduler code by adding uneeded things to channel_t<br>
but also bloated the scheduler interface with pointless function pointers for<br>
instance. And in my opinion, it is not helping performance the way it is done<br>
right now.<br></blockquote><div> </div><div>I agree with Roger here: it's fine to throw away the vanilla scheduler, but we should wait until KIST has been running unproblematically in a stable release for a while. 0.3.4 seems like a good time for this.</div><div><br></div><div>-- </div><div>Nick</div></div></div></div>