<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"><span></span></div><div dir="ltr"><div dir="ltr"><div><br></div></div><div dir="ltr">On 14 Feb 2019, at 02:57, Katharina Kohls <<a href="mailto:katharina.kohls@rub.de">katharina.kohls@rub.de</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr">
<blockquote type="cite" cite="mid:41C8C2AD-7053-47BD-9A72-F84E9B6D26F3@riseup.net">
<div dir="ltr">
<blockquote type="cite">
<div dir="ltr"><span>All nodes bootstrap properly and reach
100%, the authorities both manage to vote and exchange
information. Also the relays and the client bootstrap to
100%.</span></div>
</blockquote>
<div dir="ltr"><br>
</div>
<div dir="ltr">When are these messages logged?</div>
</div>
</blockquote>
Sorry, I must update this: The authorities bootstrap to 100%, relays
and client are stuck with 80% (sometimes reach 85%).</div></blockquote><div><br></div><div>We recently changed the bootstrap percentages and messages in Tor.</div><div>Please paste the log lines that containing these bootstrap messages.</div><div>And any error messages near those lines.</div><div><br></div><div>You might get better bootstrap messages using Tor master.</div><br><blockquote type="cite"><div dir="ltr"><blockquote type="cite" cite="mid:41C8C2AD-7053-47BD-9A72-F84E9B6D26F3@riseup.net"><div dir="ltr">
<blockquote type="cite">
<div dir="ltr"><span>Nevertheless, the consensus seems to lack
relays with guard flags:</span><br>
<span></span><br>
<span>Feb 12 10:35:56.000 [notice] I learned some more
directory information, but not enough to build a circuit:
We need more microdescriptors: we have 2/2,</span></div>
</blockquote>
<div><br>
</div>
<div dir="ltr">This log message says that there are only 2 nodes
in the consensus at that time.<br>
<blockquote type="cite">
<div dir="ltr"><span>and can only build 0% of likely paths.
(We have 0% of guards bw, 100% of midpoint bw, and 100%
of end bw (no exits in consensus,</span></div>
</blockquote>
<div><br>
</div>
<div>This log message say that there are no exits in the
consensus at that time.</div>
</div>
</div>
</blockquote>
Right now there are even less available nodes and bandwidth showing
up in the logs. This changes between runs but never to more
promising numbers.
</div></blockquote><div dir="ltr"><br></div><div dir="ltr">To get good bandwidth numbers, you'll need to pass some traffic</div><div dir="ltr">through your network. To get measured bandwidth in the votes,</div><div dir="ltr">you'll need to run a bandwidth authority, like sbws:</div><div dir="ltr"><a href="https://git.torproject.org/sbws.git">https://git.torproject.org/sbws.git</a></div><br><blockquote type="cite"><div dir="ltr"><blockquote type="cite" cite="mid:41C8C2AD-7053-47BD-9A72-F84E9B6D26F3@riseup.net">
<div dir="ltr">
<div dir="ltr">
<blockquote type="cite">
<div dir="ltr"><span>using mid) = 0% of path bw.)</span></div>
</blockquote>
<div dir="ltr">
<blockquote type="cite"><br>
</blockquote>
</div>
<span style="background-color: rgba(255, 255, 255, 0);"></span>
<blockquote type="cite"><span style="background-color:
rgba(255, 255, 255, 0);">Because of this, no default
circuits can be built in the client or the relays</span></blockquote>
<div dir="ltr"><br>
</div>
<div dir="ltr">When there are only 2 nodes in the network, you
can't build a 3-hop path.</div>
</div>
</div>
</blockquote>
There should be 8 nodes in total so it's kind of strange that only 2
seem to be available in this relay.<br></div></blockquote><div dir="ltr"><br></div><div dir="ltr">It would help to know what's actually in the consensus. (See below.)</div><br><blockquote type="cite"><div dir="ltr">
<blockquote type="cite" cite="mid:41C8C2AD-7053-47BD-9A72-F84E9B6D26F3@riseup.net">
<div dir="ltr">
<div dir="ltr">
<blockquote type="cite"><span style="background-color:
rgba(255, 255, 255, 0);">
…<br>
<br>
In the data_dir/state file I see several guard entries:<br>
Guard in=default rsa_id=[...] nickname=auth01
sampled_on=2019-01-17T18:33:12 sampled_by=0.3.5.7 listed=1<br>
Guard in=default rsa_id=[...] nickname=relay03
sampled_on=2019-01-22T17:<a href="x-apple-data-detectors://4" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="calendar-event" x-apple-data-detectors-result="4" style="-webkit-text-decoration-color: rgba(0, 0, 0,
0.258824);" moz-do-not-send="true">17:10</a>sampled_by=0.3.5.7
unlisted_since=2019-01-27T11:<a href="x-apple-data-detectors://6" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="calendar-event" x-apple-data-detectors-result="6" style="-webkit-text-decoration-color: rgba(0, 0, 0,
0.258824);" moz-do-not-send="true">00:36</a> listed=0<br>…<br>
</span></blockquote>
<div dir="ltr"><br>
</div>
<div dir="ltr">The state file says that there were some nodes
in some previous consensuses. None of these nodes come from
the current consensus at the time of your log messages.</div>
</div>
</div>
</blockquote>
I use a bash script that manages all the VMs. It kills Tor on all
machines, then waits for 5 seconds just to be sure
(ShutdownWaitLength 0),</div></blockquote><div dir="ltr"><br></div><div dir="ltr">Maybe there's a bug in <span style="background-color: rgba(255, 255, 255, 0);">ShutdownWaitLength.</span></div><div dir="ltr"><span style="background-color: rgba(255, 255, 255, 0);">We changed that code recently.</span></div><div dir="ltr">Is Tor actually shut down when you remove the files?</div><div dir="ltr"><br></div><div dir="ltr">When you start Tor, what is actually in the data directory?</div><br><blockquote type="cite"><div dir="ltr">then removes all cached, old logs, the state
file, ... and some more stuff on the authorities (see below).<br>
<br>
<div style="color: #f6f6f4;background-color: #282a36;font-family: 'Droid Sans Mono', 'monospace', monospace, 'Droid Sans Fallback';font-weight: normal;font-size: 14px;line-height: 19px;white-space: pre;"><div><span style="color: #f6f6f4;"> ssh auth01 rm /var/lib/tor/cached</span><span style="color: #f286c4;">*</span></div><div><span style="color: #f6f6f4;"> ssh auth01 rm /var/lib/tor/</span><span style="color: #f286c4;">*</span><span style="color: #f6f6f4;">.log</span></div><div><span style="color: #f6f6f4;"> ssh auth01 rm /var/lib/tor/state</span></div>
<div><span style="color: #f6f6f4;"> ssh auth01 rm -r /var/lib/tor/router-stability</span></div><div><span style="color: #f6f6f4;"> ssh auth01 rm -r /var/lib/tor/sr-state</span></div><div><span style="color: #f6f6f4;"> ssh auth01 rm -r /var/lib/tor/v3-status-votes</span></div><div><span style="color: #f6f6f4;"> ssh auth01 rm -r /var/lib/tor/diff-cache</span></div></div>
<blockquote type="cite" cite="mid:41C8C2AD-7053-47BD-9A72-F84E9B6D26F3@riseup.net">
<div dir="ltr">
<div dir="ltr"><br>
<blockquote type="cite"><span style="background-color:
rgba(255, 255, 255, 0);">The client also seems to receive
a complete consensus, at least all fingerprints of my
setup show up if I fetch the file manually.</span></blockquote>
<div dir="ltr"><br>
</div>
How do you fetch the file manually, and from where?<br>
</div>
</div>
</blockquote>
wget <a class="moz-txt-link-freetext" href="http://authip:7000/tor/server/all">http://authip:7000/tor/server/all</a><br>
<br>
which should be the cached-descriptors.new file on the authority
(which also means it gets deleted on each new startup and must be
fresh).<br></div></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><div dir="ltr"><font color="#000000"><span style="caret-color: rgb(0, 0, 0); background-color: rgba(255, 255, 255, 0);">In this file I see all the fingerprints that are supposed to be there.</span></font></div></blockquote><div><br></div><div>tor/server/all is a list of all relay descriptors that the authority knows about.</div><div><br></div><div>But the consensus is different: it contains the relays from the authorities'</div><div>votes, but only if those relays are reachable from the authorities</div><div>(the Running flag), and the authorities agree on enough info about the</div><div>relays.</div><div><br></div><div>Please check the votes and consensuses on each authority:</div><div><span style="background-color: rgba(255, 255, 255, 0);">http://<hostname>/tor/status-vote/current/authority</span></div><div><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><font face="UICTFontTextStyleTallBody"><span style="white-space: normal; background-color: rgba(255, 255, 255, 0);">http://<hostname>/tor/status-vote/current/consensus</span></font></pre><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><font face="UICTFontTextStyleTallBody"><span style="white-space: normal; background-color: rgba(255, 255, 255, 0);">http://<hostname>/tor/status-vote/current/consensus-microdesc</span></font></pre><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><font face="UICTFontTextStyleTallBody"><span style="white-space: normal; background-color: rgba(255, 255, 255, 0);"><br></span></font></pre><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><font face="UICTFontTextStyleTallBody"><span style="white-space: normal; background-color: rgba(255, 255, 255, 0);">Source:</span></font></pre><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><a href="https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n3867">https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n3867</a></pre></pre><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><font face="UICTFontTextStyleTallBody"><span style="white-space: normal; background-color: rgba(255, 255, 255, 0);"><br></span></font></pre><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><font face="UICTFontTextStyleTallBody"><span style="white-space: normal; background-color: rgba(255, 255, 255, 0);">Then, check the cached consensus-microdesc files on each client.</span></font></pre><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><font face="UICTFontTextStyleTallBody"><span style="white-space: normal; background-color: rgba(255, 255, 255, 0);">(Clients and relays use the microdesc consensus by default.)</span></font></pre></div><br><blockquote type="cite"><div dir="ltr"> It's also possible to connect to the client's control port
and manually build circuits to all relays that should be there. This
is an indicator that the client knows the relays (using a
fingerprint that is not in the consensus would not work).<br></div></blockquote><div dir="ltr"><br></div><div dir="ltr">That's not how Tor works:</div><div dir="ltr"><br></div><div dir="ltr">Clients randomly select relays from the consensus.</div><div dir="ltr"><br></div><div dir="ltr">But when someone else specifies the relay, clients will happily</div><div dir="ltr">connect to that relay by fingerprint and IP address, even if the</div><div dir="ltr">relay isn't in the consensus. (The fingerprint is a hash of the</div><div dir="ltr">relay's identity key, which the client checks when it connects</div><div dir="ltr">to the relay.)</div><div dir="ltr"><br></div><div dir="ltr">This feature exists so that the network still works when clients</div><div dir="ltr">tell relays and onion services about new relays. (There are a few</div><div dir="ltr">valid consensuses on the network at each point in time, and they</div><div dir="ltr">can contain different relays.)</div><div dir="ltr"><br></div><div dir="ltr">Can you copy and paste the code you're using?</div><br><blockquote type="cite"><div dir="ltr">
Again, guards also show up in the state files of the relays <br>
<br>
Guard in=default rsa_id=C122CBB79DC660621E352D401AD7F781F8F6D62D
nickname=relay03 sampled_on=2019-02-07T16:24:21 sampled_by=0.3.5.7
listed=1<br>
Guard in=default rsa_id=2B74825BE33752B21D17713F88D101F3BADC79BC
nickname=relay06 sampled_on=2019-02-03T22:16:29 sampled_by=0.3.5.7
listed=1<br>
Guard in=default rsa_id=E4B1152CDF0E5FE697A3E916716FC363A2A0ACF3
nickname=relay07 sampled_on=2019-02-12T18:51:00 sampled_by=0.3.5.7
listed=1<br>
Guard in=default rsa_id=911EDA6CB639AAE955517F02AA4D651E0F7F6EFD
nickname=relay02 sampled_on=2019-02-11T22:58:28 sampled_by=0.3.5.7
listed=1<br>
Guard in=default rsa_id=8E574F0C428D235782061F44B2D20A66E4336993
nickname=relay05 sampled_on=2019-02-01T17:46:05 sampled_by=0.3.5.7
listed=1<br>
<br>
The dates are still old, but I delete all states in the big cleanup
procedure. Are there some more old caches I need to remove, where
does the date information come from?<br></div></blockquote><div dir="ltr"><br></div><div dir="ltr"><span style="background-color: rgba(255, 255, 255, 0);">The dates are the time when Tor chose the guard.</span></div><div dir="ltr"><span style="background-color: rgba(255, 255, 255, 0);">Maybe you're not actually deleting the state file?</span></div><div dir="ltr"><span style="background-color: rgba(255, 255, 255, 0);">Maybe there's an undocumented state.new file?</span></div><div dir="ltr"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div><div dir="ltr"><span style="background-color: rgba(255, 255, 255, 0);">What's in the directory after you run the script?</span></div><div dir="ltr"><br></div><div dir="ltr">Removing specific files is inherently fragile: future Tor versions</div><div dir="ltr">may add new files.</div><div dir="ltr"><br></div><div dir="ltr">Instead, configure different directories for CacheDirectory,</div><div dir="ltr">DataDirectory, and KeyDirectory. Then, delete and re-create</div><div dir="ltr">CacheDirectory and DataDirectory. Fail and refuse to start Tor</div><div dir="ltr">if the deletion and re-creation fails.</div><div dir="ltr"><br></div><div dir="ltr">(Normally, relay operators want to keep info from previous</div><div dir="ltr">runs in DataDirectory, but your setup is a special case.)</div><div dir="ltr"><br></div><div dir="ltr">You can also safely delete the short and medium term keys</div><div dir="ltr">in KeyDirectory. But it probably doesn't hurt to keep them.</div><div dir="ltr"><br></div><div dir="ltr">For more info, see:</div><div dir="ltr"><a href="https://www.torproject.org/docs/tor-manual.html.en">https://www.torproject.org/docs/tor-manual.html.en</a></div><br><blockquote type="cite"><div dir="ltr">
<blockquote type="cite" cite="mid:41C8C2AD-7053-47BD-9A72-F84E9B6D26F3@riseup.net">
<div dir="ltr">…</div></blockquote></div></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><div dir="ltr"><blockquote type="cite" cite="mid:41C8C2AD-7053-47BD-9A72-F84E9B6D26F3@riseup.net"><div dir="ltr">I suggest that you start again with the same
config, but remove all previous state.</div></blockquote><blockquote type="cite" cite="mid:41C8C2AD-7053-47BD-9A72-F84E9B6D26F3@riseup.net">
<div dir="ltr">(Move the cached state, consensuses, descriptors,
and log files somewhere else. Do not remove the keys.)</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Then you'll know if your current network actually
works.</div>
</blockquote>
Questions are: Why does the client know all the relays' fingerprints
but the network still has problems finishing the bootstrapping and
building a complete circuit? Are there any other things I should
look into and check to understand the problem?</div></blockquote><br></div><div dir="ltr">I think I answered these questions above in context.</div><div dir="ltr"><br></div><div dir="ltr">Let me know if you're still having trouble.</div><div dir="ltr"><br></div><div dir="ltr">T</div></body></html>