Testing Network: Circuit buildup errors and missing guards
Hey guys, currently I am working on a private Tor setup and I repeatedly run into issues with the circuit buildup procedure (it's Tor 0.3.5.7 on linux, the setup consists of several debian jessie VMs). The setup is as follows: 1 Client, 2 V3 Authorities, 6 Relays of which 3 have the ExitRelay 1 option set. In the torrc configs of all relays I define a list of fixed exits TestingDirAuthVoteExit and fixed guards TestingDirAuthVoteGuard and I use DirAuthority to fix the two V3 authorities of my setup. All nodes bootstrap properly and reach 100%, the authorities both manage to vote and exchange information. Also the relays and the client bootstrap to 100%. Nevertheless, the consensus seems to lack relays with guard flags: Feb 12 10:35:56.000 [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 2/2, and can only build 0% of likely paths. (We have 0% of guards bw, 100% of midpoint bw, and 100% of end bw (no exits in consensus, using mid) = 0% of path bw.) Because of this, no default circuits can be built in the client or the relays in all logs the following message appears every second: [warn] Failed to find node for hop #1 of our path. Discarding this circuit. Google says it might be an ntp-sync problem. The VMs are not connected to the Internet (but can talk to each other), so I made sure that all machines are in sync and use the firewall as NTP server. Sync shouldn't be the problem. In the data_dir/state file I see several guard entries: Guard in=default rsa_id=[...] nickname=auth01 sampled_on=2019-01-17T18:33:12 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=[...] nickname=relay03 sampled_on=2019-01-22T17:17:10 sampled_by=0.3.5.7 unlisted_since=2019-01-27T11:00:36 listed=0 Guard in=default rsa_id=[...] nickname=relay02 sampled_on=2019-01-24T22:19:10 sampled_by=0.3.5.7 unlisted_since=2019-01-29T09:08:59 listed=0 Guard in=default rsa_id=[...] nickname=relay03 sampled_on=2019-02-06T21:07:36 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=[...] nickname=relay05 sampled_on=2019-01-27T16:37:38 sampled_by=0.3.5.7 listed=1 The client also seems to receive a complete consensus, at least all fingerprints of my setup show up if I fetch the file manually. Please find below an example of the configs I use for the different nodes. Any help or hints would be great :) Thanks, Katharina # DIRECTORIES, LOGGING SafeLogging 0 ProtocolWarnings 1 DisableDebuggerAttachment 0 DataDirectory /var/lib/tor PidFile /var/lib/tor/pid Log notice file /var/lib/tor/notice.log Log info file /var/lib/tor/info.log # CONTACT ContactInfo ... # GENERAL RunAsDaemon 1 AssumeReachable 1 ConnLimit 60 MaxMemInQueues 1507 MB ShutdownWaitLength 0 HashedControlPassword ... # FIXED AUTH DirAuthority auth01 orport=5000 no-v2 v3ident=... ...:7000 B218B78864CEF4397CEE0AEF61703459EEE64E38 DirAuthority auth02 orport=5000 no-v2 v3ident=... ...:7000 431E50CDBB0B6FFDD0284A45ABEC875136D980E8 TestingDirAuthVoteExit 2B74825BE33752B21D17713F88D101F3BADC79BC,E4B1152CDF0E5FE697A3E916716FC363A2A0ACF3,7353D324677B9E7A9A50240339C2C7366B381F64 TestingDirAuthVoteGuard 911EDA6CB639AAE955517F02AA4D651E0F7F6EFD,C122CBB79DC660621E352D401AD7F781F8F6D62D,8E574F0C428D235782061F44B2D20A66E4336993 # PORTS OrPort 5000 ControlPort 9051 SocksPort 9050 # FLAGS ExitRelay 1 Nickname ... Address ...
Hi,
On 12 Feb 2019, at 21:22, Katharina Kohls <katharina.kohls@rub.de> wrote:
All nodes bootstrap properly and reach 100%, the authorities both manage to vote and exchange information. Also the relays and the client bootstrap to 100%.
When are these messages logged?
Nevertheless, the consensus seems to lack relays with guard flags:
Feb 12 10:35:56.000 [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 2/2,
This log message says that there are only 2 nodes in the consensus at that time.
and can only build 0% of likely paths. (We have 0% of guards bw, 100% of midpoint bw, and 100% of end bw (no exits in consensus,
This log message say that there are no exits in the consensus at that time.
using mid) = 0% of path bw.)
Because of this, no default circuits can be built in the client or the relays
When there are only 2 nodes in the network, you can't build a 3-hop path.
in all logs the following message appears every second:
[warn] Failed to find node for hop #1 of our path. Discarding this circuit.
…
In the data_dir/state file I see several guard entries: Guard in=default rsa_id=[...] nickname=auth01 sampled_on=2019-01-17T18:33:12 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=[...] nickname=relay03 sampled_on=2019-01-22T17:17:10sampled_by=0.3.5.7 unlisted_since=2019-01-27T11:00:36 listed=0 Guard in=default rsa_id=[...] nickname=relay02 sampled_on=2019-01-24T22:19:10sampled_by=0.3.5.7 unlisted_since=2019-01-29T09:08:59 listed=0 Guard in=default rsa_id=[...] nickname=relay03 sampled_on=2019-02-06T21:07:36sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=[...] nickname=relay05 sampled_on=2019-01-27T16:37:38 sampled_by=0.3.5.7 listed=1
The state file says that there were some nodes in some previous consensuses. None of these nodes come from the current consensus at the time of your log messages.
The client also seems to receive a complete consensus, at least all fingerprints of my setup show up if I fetch the file manually.
How do you fetch the file manually, and from where? I'm not sure what is happening here. It looks like some consensuses only have 2 nodes. But other consensuses have most of the nodes. You might have a bug in your network setup, or you may have found a bug in Tor. The most likely explanation is that you had a working network at some time, which gave you the state file. And you had a failed network at some time, which gave you the log messages. I suggest that you start again with the same config, but remove all previous state. (Move the cached state, consensuses, descriptors, and log files somewhere else. Do not remove the keys.) Then you'll know if your current network actually works. T
All nodes bootstrap properly and reach 100%, the authorities both manage to vote and exchange information. Also the relays and the client bootstrap to 100%.
When are these messages logged? Sorry, I must update this: The authorities bootstrap to 100%, relays and client are stuck with 80% (sometimes reach 85%).
Nevertheless, the consensus seems to lack relays with guard flags:
Feb 12 10:35:56.000 [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 2/2,
This log message says that there are only 2 nodes in the consensus at that time.
and can only build 0% of likely paths. (We have 0% of guards bw, 100% of midpoint bw, and 100% of end bw (no exits in consensus,
This log message say that there are no exits in the consensus at that time. Right now there are even less available nodes and bandwidth showing up in the logs. This changes between runs but never to more promising numbers.
using mid) = 0% of path bw.)
Because of this, no default circuits can be built in the client or the relays
When there are only 2 nodes in the network, you can't build a 3-hop path. There should be 8 nodes in total so it's kind of strange that only 2 seem to be available in this relay.
in all logs the following message appears every second:
[warn] Failed to find node for hop #1 of our path. Discarding this circuit.
…
In the data_dir/state file I see several guard entries: Guard in=default rsa_id=[...] nickname=auth01 sampled_on=2019-01-17T18:33:12 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=[...] nickname=relay03 sampled_on=2019-01-22T17:17:10 <x-apple-data-detectors://4>sampled_by=0.3.5.7 unlisted_since=2019-01-27T11:00:36 <x-apple-data-detectors://6> listed=0 Guard in=default rsa_id=[...] nickname=relay02 sampled_on=2019-01-24T22:19:10 <x-apple-data-detectors://7>sampled_by=0.3.5.7 unlisted_since=2019-01-29T09:08:59 <x-apple-data-detectors://9> listed=0 Guard in=default rsa_id=[...] nickname=relay03 sampled_on=2019-02-06T21:07:36 <x-apple-data-detectors://10>sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=[...] nickname=relay05 sampled_on=2019-01-27T16:37:38 sampled_by=0.3.5.7 listed=1
The state file says that there were some nodes in some previous consensuses. None of these nodes come from the current consensus at the time of your log messages. I use a bash script that manages all the VMs. It kills Tor on all machines, then waits for 5 seconds just to be sure (ShutdownWaitLength 0), then removes all cached, old logs, the state file, ... and some more stuff on the authorities (see below).
ssh auth01 rm /var/lib/tor/cached* ssh auth01 rm /var/lib/tor/*.log ssh auth01 rm /var/lib/tor/state ssh auth01 rm -r /var/lib/tor/router-stability ssh auth01 rm -r /var/lib/tor/sr-state ssh auth01 rm -r /var/lib/tor/v3-status-votes ssh auth01 rm -r /var/lib/tor/diff-cache
The client also seems to receive a complete consensus, at least all fingerprints of my setup show up if I fetch the file manually.
How do you fetch the file manually, and from where?
wget http://authip:7000/tor/server/all which should be the cached-descriptors.new file on the authority (which also means it gets deleted on each new startup and must be fresh). In this file I see all the fingerprints that are supposed to be there. It's also possible to connect to the client's control port and manually build circuits to all relays that should be there. This is an indicator that the client knows the relays (using a fingerprint that is not in the consensus would not work). Again, guards also show up in the state files of the relays Guard in=default rsa_id=C122CBB79DC660621E352D401AD7F781F8F6D62D nickname=relay03 sampled_on=2019-02-07T16:24:21 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=2B74825BE33752B21D17713F88D101F3BADC79BC nickname=relay06 sampled_on=2019-02-03T22:16:29 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=E4B1152CDF0E5FE697A3E916716FC363A2A0ACF3 nickname=relay07 sampled_on=2019-02-12T18:51:00 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=911EDA6CB639AAE955517F02AA4D651E0F7F6EFD nickname=relay02 sampled_on=2019-02-11T22:58:28 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=8E574F0C428D235782061F44B2D20A66E4336993 nickname=relay05 sampled_on=2019-02-01T17:46:05 sampled_by=0.3.5.7 listed=1 The dates are still old, but I delete all states in the big cleanup procedure. Are there some more old caches I need to remove, where does the date information come from?
I'm not sure what is happening here. It looks like some consensuses only have 2 nodes. But other consensuses have most of the nodes.
You might have a bug in your network setup, or you may have found a bug in Tor. I think it's a bug somewhere in the setup but I just can't find it :(
The most likely explanation is that you had a working network at some time, which gave you the state file. And you had a failed network at some time, which gave you the log messages.
I suggest that you start again with the same config, but remove all previous state. (Move the cached state, consensuses, descriptors, and log files somewhere else. Do not remove the keys.)
Then you'll know if your current network actually works. Questions are: Why does the client know all the relays' fingerprints but the network still has problems finishing the bootstrapping and building a complete circuit? Are there any other things I should look into and check to understand the problem?
T
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 14 Feb 2019, at 02:57, Katharina Kohls <katharina.kohls@rub.de> wrote:
All nodes bootstrap properly and reach 100%, the authorities both manage to vote and exchange information. Also the relays and the client bootstrap to 100%.
When are these messages logged? Sorry, I must update this: The authorities bootstrap to 100%, relays and client are stuck with 80% (sometimes reach 85%).
We recently changed the bootstrap percentages and messages in Tor. Please paste the log lines that containing these bootstrap messages. And any error messages near those lines. You might get better bootstrap messages using Tor master.
Nevertheless, the consensus seems to lack relays with guard flags:
Feb 12 10:35:56.000 [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 2/2,
This log message says that there are only 2 nodes in the consensus at that time.
and can only build 0% of likely paths. (We have 0% of guards bw, 100% of midpoint bw, and 100% of end bw (no exits in consensus,
This log message say that there are no exits in the consensus at that time. Right now there are even less available nodes and bandwidth showing up in the logs. This changes between runs but never to more promising numbers.
To get good bandwidth numbers, you'll need to pass some traffic through your network. To get measured bandwidth in the votes, you'll need to run a bandwidth authority, like sbws: https://git.torproject.org/sbws.git
using mid) = 0% of path bw.) Because of this, no default circuits can be built in the client or the relays
When there are only 2 nodes in the network, you can't build a 3-hop path. There should be 8 nodes in total so it's kind of strange that only 2 seem to be available in this relay.
It would help to know what's actually in the consensus. (See below.)
…
In the data_dir/state file I see several guard entries: Guard in=default rsa_id=[...] nickname=auth01 sampled_on=2019-01-17T18:33:12 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=[...] nickname=relay03 sampled_on=2019-01-22T17:17:10sampled_by=0.3.5.7 unlisted_since=2019-01-27T11:00:36 listed=0 …
The state file says that there were some nodes in some previous consensuses. None of these nodes come from the current consensus at the time of your log messages. I use a bash script that manages all the VMs. It kills Tor on all machines, then waits for 5 seconds just to be sure (ShutdownWaitLength 0),
Maybe there's a bug in ShutdownWaitLength. We changed that code recently. Is Tor actually shut down when you remove the files? When you start Tor, what is actually in the data directory?
then removes all cached, old logs, the state file, ... and some more stuff on the authorities (see below).
ssh auth01 rm /var/lib/tor/cached* ssh auth01 rm /var/lib/tor/*.log ssh auth01 rm /var/lib/tor/state
ssh auth01 rm -r /var/lib/tor/router-stability ssh auth01 rm -r /var/lib/tor/sr-state ssh auth01 rm -r /var/lib/tor/v3-status-votes ssh auth01 rm -r /var/lib/tor/diff-cache
The client also seems to receive a complete consensus, at least all fingerprints of my setup show up if I fetch the file manually.
How do you fetch the file manually, and from where?
wget http://authip:7000/tor/server/all
which should be the cached-descriptors.new file on the authority (which also means it gets deleted on each new startup and must be fresh).
In this file I see all the fingerprints that are supposed to be there.
tor/server/all is a list of all relay descriptors that the authority knows about. But the consensus is different: it contains the relays from the authorities' votes, but only if those relays are reachable from the authorities (the Running flag), and the authorities agree on enough info about the relays. Please check the votes and consensuses on each authority: http://<hostname>/tor/status-vote/current/authority http://<hostname>/tor/status-vote/current/consensus http://<hostname>/tor/status-vote/current/consensus-microdesc Source: https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n3867 Then, check the cached consensus-microdesc files on each client. (Clients and relays use the microdesc consensus by default.)
It's also possible to connect to the client's control port and manually build circuits to all relays that should be there. This is an indicator that the client knows the relays (using a fingerprint that is not in the consensus would not work).
That's not how Tor works: Clients randomly select relays from the consensus. But when someone else specifies the relay, clients will happily connect to that relay by fingerprint and IP address, even if the relay isn't in the consensus. (The fingerprint is a hash of the relay's identity key, which the client checks when it connects to the relay.) This feature exists so that the network still works when clients tell relays and onion services about new relays. (There are a few valid consensuses on the network at each point in time, and they can contain different relays.) Can you copy and paste the code you're using?
Again, guards also show up in the state files of the relays
Guard in=default rsa_id=C122CBB79DC660621E352D401AD7F781F8F6D62D nickname=relay03 sampled_on=2019-02-07T16:24:21 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=2B74825BE33752B21D17713F88D101F3BADC79BC nickname=relay06 sampled_on=2019-02-03T22:16:29 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=E4B1152CDF0E5FE697A3E916716FC363A2A0ACF3 nickname=relay07 sampled_on=2019-02-12T18:51:00 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=911EDA6CB639AAE955517F02AA4D651E0F7F6EFD nickname=relay02 sampled_on=2019-02-11T22:58:28 sampled_by=0.3.5.7 listed=1 Guard in=default rsa_id=8E574F0C428D235782061F44B2D20A66E4336993 nickname=relay05 sampled_on=2019-02-01T17:46:05 sampled_by=0.3.5.7 listed=1
The dates are still old, but I delete all states in the big cleanup procedure. Are there some more old caches I need to remove, where does the date information come from?
The dates are the time when Tor chose the guard. Maybe you're not actually deleting the state file? Maybe there's an undocumented state.new file? What's in the directory after you run the script? Removing specific files is inherently fragile: future Tor versions may add new files. Instead, configure different directories for CacheDirectory, DataDirectory, and KeyDirectory. Then, delete and re-create CacheDirectory and DataDirectory. Fail and refuse to start Tor if the deletion and re-creation fails. (Normally, relay operators want to keep info from previous runs in DataDirectory, but your setup is a special case.) You can also safely delete the short and medium term keys in KeyDirectory. But it probably doesn't hurt to keep them. For more info, see: https://www.torproject.org/docs/tor-manual.html.en
…
I suggest that you start again with the same config, but remove all previous state. (Move the cached state, consensuses, descriptors, and log files somewhere else. Do not remove the keys.)
Then you'll know if your current network actually works. Questions are: Why does the client know all the relays' fingerprints but the network still has problems finishing the bootstrapping and building a complete circuit? Are there any other things I should look into and check to understand the problem?
I think I answered these questions above in context. Let me know if you're still having trouble. T
We recently changed the bootstrap percentages and messages in Tor. Please paste the log lines that containing these bootstrap messages. And any error messages near those lines.
Feb 14 10:37:46.000 [notice] Bootstrapped 10%: Finishing handshake with directory server Feb 14 10:38:32.000 [notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus. Feb 14 10:38:32.000 [notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus. Feb 14 10:38:32.000 [notice] The current consensus has no exit nodes. Tor can only build internal paths, such as paths to onion services. Feb 14 10:38:32.000 [notice] Bootstrapped 45%: Asking for relay descriptors for internal paths Feb 14 10:38:32.000 [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/9, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of end bw (no exits in consensus, using mid) = 0% of path bw.) Feb 14 10:38:32.000 [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/9, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of end bw (no exits in consensus, using mid) = 0% of path bw.) Feb 14 10:38:32.000 [notice] The current consensus contains exit nodes. Tor can build exit and internal paths. Feb 14 10:38:32.000 [notice] Bootstrapped 50%: Loading relay descriptors Feb 14 10:38:32.000 [notice] Bootstrapped 80%: Connecting to the Tor network Feb 14 10:38:33.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. Feb 14 10:38:33.000 [notice] Our circuit 0 (id: 1) died due to an invalid selected path, purpose General-purpose client. This may be a torrc configuration issue, or a bug. Feb 14 10:38:34.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. Feb 14 10:38:35.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. Feb 14 10:38:36.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit.
To get good bandwidth numbers, you'll need to pass some traffic through your network. To get measured bandwidth in the votes, you'll need to run a bandwidth authority, like sbws: https://git.torproject.org/sbws.git
Is this optional, are there alternative ways of measuring bandwidth?
It would help to know what's actually in the consensus. (See below.)
I pasted one of the microdescriptors at the end of the message (it's fetched from one of the authorities).
Maybe there's a bug in ShutdownWaitLength. We changed that code recently. Is Tor actually shut down when you remove the files?
It is definitely shut down when I start deleting the files.
When you start Tor, what is actually in the data directory?
I followed your idea of defining and deleting directories so now there are data/ cache/ and keys/ defined in the torrc and this is what I remove in the cleanup procedure. I keep the keys dir, the stats file, and the fingerprint file. The issue with old Guard entries in the new state file remains.
tor/server/all is a list of all relay descriptors that the authority knows about.
But the consensus is different: it contains the relays from the authorities' votes, but only if those relays are reachable from the authorities (the Running flag), and the authorities agree on enough info about the relays.
Please check the votes and consensuses on each authority: http://<hostname>/tor/status-vote/current/authority http://<hostname>/tor/status-vote/current/consensus http://<hostname>/tor/status-vote/current/consensus-microdesc
One of the microdesc files is pasted below. I see the Running flag set for several relays, along with Fast, Guard, and sometimes also Exit (for example, relay08 is defined as exit in the torrc and also shows up as Exit Fast Guard HSDir Running Stable V2Dir Valid).
That's not how Tor works:
Clients randomly select relays from the consensus.
Yes, and this is exactly what I need to measure in the private network. My project is about testing the consequences of the DoS features in relays and how the client reacts to being blocked (if it recognizes this at all, that's one of the things I want to find out).
Can you copy and paste the code you're using?
It's just a simple extendcircuit 0 FP,FP,FP,... for testing the network status. If the network starts functioning normally again, I need to use either NEWNYM via the controlport or stem and new_circuit(). But right now I don't get to this point, because NEWNYM results in no new circuit and also the stem implementation does not deliver new circuits. Just to give an example: getinfo circuit-status 250-circuit-status= 250 OK signal newnym 250 OK getinfo circuit-status 250-circuit-status= 250 OK extendcircuit 0 2B74825BE33752B21D17713F88D101F3BADC79BC,7353D324677B9E7A9A50240339C2C7366B381F64 250 EXTENDED 610 getinfo circuit-status 250-circuit-status=610 BUILT $2B74825BE33752B21D17713F88D101F3BADC79BC~relay06,$7353D324677B9E7A9A50240339C2C7366B381F64~relay08 PURPOSE=GENERAL TIME_CREATED=2019-02-14T13:44:27.304736 250 OK
The dates are the time when Tor chose the guard. Maybe you're not actually deleting the state file? Maybe there's an undocumented state.new file?
I'm pretty much sure I delete the file and that there are no .new versions of the state file. Still, after a while the old Guards show up in the state file. What information is used to generate the state file? Maybe there is still some kind of cache left somewhere else?
What's in the directory after you run the script?
Removing specific files is inherently fragile: future Tor versions may add new files.
Instead, configure different directories for CacheDirectory, DataDirectory, and KeyDirectory. Then, delete and re-create CacheDirectory and DataDirectory. Fail and refuse to start Tor if the deletion and re-creation fails.
Great idea, did that! Nothing but the keys and fingerprint in there, as described above.
I think I answered these questions above in context. Yes you did and things are getting clearer :)
Let me know if you're still having trouble. Yes, the setup is still not in the state where the client is able to create "natural" circuits.
network-status-version 3 microdesc vote-status consensus consensus-method 28 valid-after 2019-02-14 11:05:00 fresh-until 2019-02-14 11:10:00 valid-until 2019-02-14 11:20:00 voting-delay 20 20 client-versions server-versions known-flags Authority Exit Fast Guard HSDir NoEdConsensus Running Stable V2Dir Valid recommended-client-protocols Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1 Link=4 Microdesc=1-2 Relay=2 recommended-relay-protocols Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1 Link=4 Microdesc=1-2 Relay=2 required-client-protocols Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1 Link=4 Microdesc=1-2 Relay=2 required-relay-protocols Cons=1 Desc=1 DirCache=1 HSDir=1 HSIntro=3 HSRend=1 Link=3-4 Microdesc=1 Relay=1-2 shared-rand-current-value 0 zxJao+gBmFMSezvz/VXkEWEQJD5b/z+7AXNCGoLFVW0= dir-source auth01 14EA360AE456079B386651CDEA2996A6D48F1798 100.113.5.34 100.113.5.34 7000 5000 contact katharina.kohls@rub.de vote-digest 0E3E3697D3CC785928FE2B27B43E272BB7D52650 dir-source auth01 92E466CD419200DE68A4893EA5A758DAE70EFD9E 100.113.5.29 100.113.5.29 7000 5000 contact katharina.kohls@rub.de vote-digest 3ADC1A8BF96D74C661DF5797D2EE59FAB899DC00 r client01 BhCZoOQt0RnwUth3dUbyVVP9wxM 2019-02-14 09:37:31 100.113.5.28 5000 0 m ZCRB9OyxE3+HOAgd/Sl4ewcNfFkAlq5S/QfCT3HH/wY s Fast Running V2Dir Valid v Tor 0.3.5.7 pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 w Bandwidth=0 Unmeasured=1 r relay06 K3SCW+M3UrIdF3E/iNEB87rcebw 2019-02-14 09:37:30 100.113.5.36 5000 0 m TvN1yotIqxUIb2Pnz8LuVnseijh/NZDqu+ZCpqRDCQA s Exit Fast Guard HSDir Running Stable V2Dir Valid v Tor 0.3.5.7 pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 w Bandwidth=0 Unmeasured=1 r auth01 Qx5QzbsLb/3QKEpFq+yHUTbZgOg 2019-02-14 09:37:46 100.113.5.34 5000 7000 m idNUa5gHlZC03yfL4Wy+NgnPRxl9+9xrFbVL0uNqcr0 s Authority Fast Running V2Dir Valid v Tor 0.3.5.7 pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 w Bandwidth=0 Unmeasured=1 r relay08 c1PTJGd7nnqaUCQDOcLHNms4H2Q 2019-02-14 09:37:30 100.113.5.35 5000 0 m r1L2yrAoe3E/H3LDmGrIzeCiaKAINS17Ddg1a36LXBc s Exit Fast Guard HSDir Running Stable V2Dir Valid v Tor 0.3.5.7 pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 w Bandwidth=0 Unmeasured=1 r relay05 jldPDEKNI1eCBh9EstIKZuQzaZM 2019-02-14 09:37:30 100.113.5.38 5000 0 m /EeM/KjGfYM3wQbSW44UNPhNACHFmdjoyqPtbUmVyJU s Fast Guard HSDir Running Stable V2Dir Valid v Tor 0.3.5.7 pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 w Bandwidth=0 Unmeasured=1 r relay02 kR7abLY5qulVUX8Cqk1lHg9/bv0 2019-02-14 09:37:30 100.113.5.31 5000 0 m wHYw+7ts0/eNWjBtMNynbA+5Abv5tPzvAEjJjxpOtIA s Fast Guard HSDir Running Stable V2Dir Valid v Tor 0.3.5.7 pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 w Bandwidth=0 Unmeasured=1 r auth01 shi3iGTO9Dl87grvYXA0We7mTjg 2019-02-14 09:38:31 100.113.5.29 5000 7000 m rtksCdOu7iyIyQjbnWUWmJiN0o94+KKWR8WLvV9s1rw s Authority Exit Fast Guard Running V2Dir Valid v Tor 0.3.5.7 pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 w Bandwidth=0 Unmeasured=1 r relay03 wSLLt53GYGIeNS1AGtf3gfj21i0 2019-02-14 09:37:31 100.113.5.32 5000 0 m aU27kuVh12aIDWmNl28KULvpijEzNpFkJ09SXpZdOwg s Exit Fast Running V2Dir Valid v Tor 0.3.5.7 pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 w Bandwidth=0 Unmeasured=1 r relay07 5LEVLN8OX+aXo+kWcW/DY6KgrPM 2019-02-14 09:37:30 100.113.5.37 5000 0 m fkaGwNIKBFlluhAhEDKuHsZl/LocsJ6v0YVBzkQip8A s Exit Fast Guard HSDir Running Stable V2Dir Valid v Tor 0.3.5.7 pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 w Bandwidth=0 Unmeasured=1 directory-footer bandwidth-weights Wbd=3333 Wbe=0 Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=3333 Wee=10000 Weg=3333 Wem=10000 Wgb=10000 Wgd=3333 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=3333 Wme=0 Wmg=0 Wmm=10000 directory-signature sha256 14EA360AE456079B386651CDEA2996A6D48F1798 35B12EA25BB95F35BCAB63B5E7321DFE56F6EF32 -----BEGIN SIGNATURE----- YMdJPt/tjwkDIycPVUIMVEeZGrLt6lzoqbiDXu04/6t7Hz6lcgduL6+pJ9cyNqcP +u788vl6eL+yli1KySoPirnAdIIrHBM0xa3P5BN/nmcvjwzQXRx9gg1XG28BRt76 Rmfd+AAKF3PycnQDu8T9wT9bvk48+nPt60VOs0BrMot8L27sJr9+G2vi+KiyaW6b klJuGWYsbOcK/aPX63bx/PfdY4vSgT9W36kXOU8UtzYM9ILdsZFua84Gn6THzyEu 9Da9FlUOfDmEto3TuLppjP2kLjWxG5WDqzC3VQm39rdFgNTAERjMELND9KZqlcrD CHoGkbskkrRoUevUAF1E+g== -----END SIGNATURE----- directory-signature sha256 92E466CD419200DE68A4893EA5A758DAE70EFD9E EBF4ADA3B6557BF648F19B172172C341AE2730E6 -----BEGIN SIGNATURE----- rK70R9PDhm8zL33ThazFutHz6IBRLavI8k96plIR6bayaNLapZUWIbOAYTNW9pYv PNZwRBTTtTkahjYLNp3PSyGAzXOk7+E3iSIhTr8LBBryfihtThYWeyz4CjBTkTXj kQLddWS/knx9ZaovLAMzEXNBtflxSlf5W5/AXCHYzP3i+ssV/rxfBcHHZzayYdBG 175335s1VsjjlQYFHkuilnF++t14nFhL7Qeu7zmFaQ6IMUudwgctKgt+3xM8P8B+ /4N2NRFADSAN8gD7TfS593y1E737qJ/eznlxB4O+aRZw//6MqirNNaB3evNKiYSH /oFqSvEYnGAiVerccLOeVQ== -----END SIGNATURE-----
Hi,
On 14 Feb 2019, at 23:50, Katharina Kohls <katharina.kohls@rub.de> wrote:
We recently changed the bootstrap percentages and messages in Tor. Please paste the log lines that containing these bootstrap messages. And any error messages near those lines.
Feb 14 10:37:46.000 [notice] Bootstrapped 10%: Finishing handshake with directory server Feb 14 10:38:32.000 [notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus. ... Feb 14 10:38:32.000 [notice] Bootstrapped 45%: Asking for relay descriptors for internal paths Feb 14 10:38:32.000 [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/9, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of end bw (no exits in consensus, using mid) = 0% of path bw.) ...
In this consensus, you have 9 relays. (In previous emails, your logs said "0/2".) If a future consensus only has 2 relays, you'll get similar errors to the ones you have right now.
Feb 14 10:38:32.000 [notice] The current consensus contains exit nodes. Tor can build exit and internal paths. Feb 14 10:38:32.000 [notice] Bootstrapped 50%: Loading relay descriptors Feb 14 10:38:32.000 [notice] Bootstrapped 80%: Connecting to the Tor network
Tor logs this message when it has enough relay descriptors. So you have at least 60% of the relays in the network, which in this case is ceil(60% * 9) = 6. So in this network, you do not have a problem with the number of relays in the consensus.
Feb 14 10:38:33.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. Feb 14 10:38:33.000 [notice] Our circuit 0 (id: 1) died due to an invalid selected path, purpose General-purpose client. This may be a torrc configuration issue, or a bug. Feb 14 10:38:34.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit.
... Your client can't find a valid guard. That's probably because all your relays are in the same /16. Tor builds paths from the exit to the guard. It rejects paths where any of the relays are in the same /16. (I don't know why it failed on the guard, rather than the middle.) Try setting: TestingTorNetwork 1 (But this sets a lot of other options that you may not want.) Or: EnforceDistinctSubnets 0 You might also find it useful to set: LogTimeGranularity 1 So your logs are in milliseconds, not seconds.
To get good bandwidth numbers, you'll need to pass some traffic through your network. To get measured bandwidth in the votes, you'll need to run a bandwidth authority, like sbws: https://git.torproject.org/sbws.git
Is this optional, are there alternative ways of measuring bandwidth?
It is optional: try "EnforceDistinctSubnets 0" and see if your network works without bandwidth measurements.
It would help to know what's actually in the consensus. (See below.)
I pasted one of the microdescriptors at the end of the message (it's fetched from one of the authorities).
Maybe there's a bug in ShutdownWaitLength. We changed that code recently. Is Tor actually shut down when you remove the files?
It is definitely shut down when I start deleting the files.
When you start Tor, what is actually in the data directory?
I followed your idea of defining and deleting directories so now there are data/ cache/ and keys/ defined in the torrc and this is what I remove in the cleanup procedure. I keep the keys dir, the stats file, and the fingerprint file. The issue with old Guard entries in the new state file remains.
I think I worked out why the dates are wrong: Tor randomises the dates in the state file, to reduce the accuracy of time-based forensic analysis. So I would expect to see guards spread across a few different dates.
tor/server/all is a list of all relay descriptors that the authority knows about.
But the consensus is different: it contains the relays from the authorities' votes, but only if those relays are reachable from the authorities (the Running flag), and the authorities agree on enough info about the relays.
Please check the votes and consensuses on each authority: http://<hostname>/tor/status-vote/current/authority http://<hostname>/tor/status-vote/current/consensus http://<hostname>/tor/status-vote/current/consensus-microdesc One of the microdesc files is pasted below. I see the Running flag set for several relays, along with Fast, Guard, and sometimes also Exit (for example, relay08 is defined as exit in the torrc and also shows up as Exit Fast Guard HSDir Running Stable V2Dir Valid).
That's not how Tor works:
Clients randomly select relays from the consensus. Yes, and this is exactly what I need to measure in the private network. My project is about testing the consequences of the DoS features in relays and how the client reacts to being blocked (if it recognizes this at all, that's one of the things I want to find out).
Sounds like a great project! Thank you for working on it. ...
The dates are the time when Tor chose the guard. Maybe you're not actually deleting the state file? Maybe there's an undocumented state.new file? I'm pretty much sure I delete the file and that there are no .new versions of the state file. Still, after a while the old Guards show up in the state file. What information is used to generate the state file? Maybe there is still some kind of cache left somewhere else?
(Tor deliberately randomises the dates. See above.) ...
Let me know if you're still having trouble.
Yes, the setup is still not in the state where the client is able to create "natural" circuits.
network-status-version 3 microdesc
...
dir-source auth01 14EA360AE456079B386651CDEA2996A6D48F1798 100.113.5.34 100.113.5.34 7000 5000 contact katharina.kohls@rub.de vote-digest 0E3E3697D3CC785928FE2B27B43E272BB7D52650 dir-source auth01 92E466CD419200DE68A4893EA5A758DAE70EFD9E 100.113.5.29 100.113.5.29 7000 5000 contact katharina.kohls@rub.de vote-digest 3ADC1A8BF96D74C661DF5797D2EE59FAB899DC00
Your authorities have the same nickname: tor won't get confused, but the nicknames in the logs might confuse people.
r client01 BhCZoOQt0RnwUth3dUbyVVP9wxM 2019-02-14 09:37:31 100.113.5.28 5000 0
Your clients should not be in the consensus: clients and relays behave differently when building connections and circuits. Try removing the ORPort, or setting ClientOnly 1. T
participants (2)
-
Katharina Kohls -
teor