Onion self healing / republish
Hey all, I'm sorry if this is something obvious, I've searched for a day and couldn't find satisfactory information. Feel free to point me to relevant resources if I missed them. So, I'd like to run Tor in a P2P setting on mobile phones ( https://github.com/coronanet). In general, everything works as expected. My issues are with the fine grained behaviors of the Tor proxy wrt onion addresses and network connectivity. As long as I allow a stable internet connection to Tor, everything's fine. What I wanted to check is how Tor behaves if I start pulling the rug from underneath it in various states. I've tried keeping Tor offline (network disabled), publish an onion into the service and then enable networking afterwards (SETCONF DisableNetwork 0). This also seems to work, when Tor goes online it publishes the onion that was added offline and I can access it (at least sending a HEARTBEAT signal reports "1 v3 INTRODUCE2 cells and attempted to launch 1 rendezvous circuits"). So far so good. Now comes the more interesting test: while networking is enabled and seemingly everything works, I cut off network access at the router level (i.e. simulate a phone losing WiFi signal / turning off data). Similar to my previous test, I've created a new onion in this scenario too, then reenabled WiFI. Unfortunately, whilst my first onion comes back online, the second one is lost in some black hole. My guess is that Tor **thinks** it's online, fails to register it, then some internal state gets messed up. If I try to list the current onions (GETINFO onions/current), both appear. If I try to list some infos about them via `GETINFO hs/service/desc/id/`, both look the same, reporting their own crypto keys. However, only the first one response, the second never gets router, neither noticed by the local Tor instance. At this point I'd be perfectly happy is this weird scenario was "not meant to work by design", as long as Tor could explicitly tell me somehow that "sorry, onion X is dead, sort it out yourself". I'd be even happier if this worked though. Maybe this is a bug that was fixed in a later version? I'm on 0.3.5.10. Thanks, Peter
Péter Szilágyi <peterke@gmail.com> writes:
Hey all,
I'm sorry if this is something obvious, I've searched for a day and couldn't find satisfactory information. Feel free to point me to relevant resources if I missed them.
So, I'd like to run Tor in a P2P setting on mobile phones ( https://github.com/coronanet). In general, everything works as expected. My issues are with the fine grained behaviors of the Tor proxy wrt onion addresses and network connectivity.
As long as I allow a stable internet connection to Tor, everything's fine. What I wanted to check is how Tor behaves if I start pulling the rug from underneath it in various states.
I've tried keeping Tor offline (network disabled), publish an onion into the service and then enable networking afterwards (SETCONF DisableNetwork 0). This also seems to work, when Tor goes online it publishes the onion that was added offline and I can access it (at least sending a HEARTBEAT signal reports "1 v3 INTRODUCE2 cells and attempted to launch 1 rendezvous circuits"). So far so good.
Now comes the more interesting test: while networking is enabled and seemingly everything works, I cut off network access at the router level (i.e. simulate a phone losing WiFi signal / turning off data). Similar to my previous test, I've created a new onion in this scenario too, then reenabled WiFI. Unfortunately, whilst my first onion comes back online, the second one is lost in some black hole. My guess is that Tor **thinks** it's online, fails to register it, then some internal state gets messed up.
If I try to list the current onions (GETINFO onions/current), both appear. If I try to list some infos about them via `GETINFO hs/service/desc/id/`, both look the same, reporting their own crypto keys. However, only the first one response, the second never gets router, neither noticed by the local Tor instance.
At this point I'd be perfectly happy is this weird scenario was "not meant to work by design", as long as Tor could explicitly tell me somehow that "sorry, onion X is dead, sort it out yourself". I'd be even happier if this worked though. Maybe this is a bug that was fixed in a later version? I'm on 0.3.5.10.
Greetings friend, while I don't have The Solution for you I have a few questions that will take us closer to it: a) What is the purpose of your experiments? Are you trying to simulate a phone losing WiFi signal? If yes, that seems like a fine test to do and something that Tor should handle gracefully (and if it doesn't we should fix it ASAP). In general, hosting onion services over mobile is a use case that we've had trouble in the past, and we've been constantly trying to increase its stability, by getting feedback and help from our friends at the Briar project (cc'ed Michael) and Guardian project (cc'ed Nathan). b) Yes, do use the latest alpha tor instead of 0.3.5.10. There are various logs and general reachability enhancements that might not have been backported that far back. Ideally you would even use Tor with git master since we recently merged a branch that should provide further debugging log messages in cases like yours: https://trac.torproject.org/projects/tor/ticket/33400 c) It would be really useful to see some *info* logs from your service if you manage to reproduce this bug with Tor master. My intuition is that the second service fails to upload its descriptor because of Tor being offline, and then never notices the network going up to try again. Feel free to either upload your logs on Tor's trac (in a new ticket) or either send it to me and David Goulet (dgoulet@torproject.org). Good luck and looking forward to hear back from your experiments! PS: You might want to register on the mailing list because I had to manually approve your post.
Thanks for the quick replies! On Mon, Mar 30, 2020 at 8:18 PM George Kadianakis <desnacked@riseup.net> wrote:
Péter Szilágyi <peterke@gmail.com> writes:
Hey all,
I'm sorry if this is something obvious, I've searched for a day and couldn't find satisfactory information. Feel free to point me to relevant resources if I missed them.
So, I'd like to run Tor in a P2P setting on mobile phones ( https://github.com/coronanet). In general, everything works as expected. My issues are with the fine grained behaviors of the Tor proxy wrt onion addresses and network connectivity.
As long as I allow a stable internet connection to Tor, everything's fine. What I wanted to check is how Tor behaves if I start pulling the rug from underneath it in various states.
I've tried keeping Tor offline (network disabled), publish an onion into the service and then enable networking afterwards (SETCONF DisableNetwork 0). This also seems to work, when Tor goes online it publishes the onion that was added offline and I can access it (at least sending a HEARTBEAT signal reports "1 v3 INTRODUCE2 cells and attempted to launch 1 rendezvous circuits"). So far so good.
Now comes the more interesting test: while networking is enabled and seemingly everything works, I cut off network access at the router level (i.e. simulate a phone losing WiFi signal / turning off data). Similar to my previous test, I've created a new onion in this scenario too, then reenabled WiFI. Unfortunately, whilst my first onion comes back online, the second one is lost in some black hole. My guess is that Tor **thinks** it's online, fails to register it, then some internal state gets messed up.
If I try to list the current onions (GETINFO onions/current), both appear. If I try to list some infos about them via `GETINFO hs/service/desc/id/`, both look the same, reporting their own crypto keys. However, only the first one response, the second never gets router, neither noticed by the local Tor instance.
At this point I'd be perfectly happy is this weird scenario was "not meant to work by design", as long as Tor could explicitly tell me somehow that "sorry, onion X is dead, sort it out yourself". I'd be even happier if this worked though. Maybe this is a bug that was fixed in a later version? I'm on 0.3.5.10.
Greetings friend,
while I don't have The Solution for you I have a few questions that will take us closer to it:
a) What is the purpose of your experiments? Are you trying to simulate a phone losing WiFi signal? If yes, that seems like a fine test to do and something that Tor should handle gracefully (and if it doesn't we should fix it ASAP).
That is indeed what I'm aiming for. In reality I want the phone to be able to switch networks, go from Wifi to data and back, turn everything off and on, etc. I have to make some more meaningful tests. I can confirm that Tor indeed seems to be trying very hard to self heal and I failed to reproduce later the issue. I've built my code around the idea now that it self-heals, so if things go wrong again or in a reproducible way, I'll try to experiment. I'll get back if I can figure out what was the scenario that made things fail.
In general, hosting onion services over mobile is a use case that we've had trouble in the past, and we've been constantly trying to increase its stability, by getting feedback and help from our friends at the Briar project (cc'ed Michael) and Guardian project (cc'ed Nathan).
I'll report any data that I gather then, maybe it will be helpful, though I'm only at the beginning for now.
b) Yes, do use the latest alpha tor instead of 0.3.5.10. There are various logs and general reachability enhancements that might not have been backported that far back.
Ideally you would even use Tor with git master since we recently merged a branch that should provide further debugging log messages in cases like yours: https://trac.torproject.org/projects/tor/ticket/33400
I'm using Tor from Go via https://github.com/ipsn/go-libtor, mostly because when I made the auto-wrappers the 0.4.x branches were fairly new. If I find a bit of time I'll switch the wrappers over to the latest stable releases and see where that takes me. I've tried hacking it and generally it seems to work, but I did see some quirks (possibly control protocol incompatibilities).
c) It would be really useful to see some *info* logs from your service if you manage to reproduce this bug with Tor master. My intuition is that the second service fails to upload its descriptor because of Tor being offline, and then never notices the network going up to try again. Feel free to either upload your logs on Tor's trac (in a new ticket) or either send it to me and David Goulet ( dgoulet@torproject.org).
I'll post any data that I'll gather. My initial email was mostly a reachout to see if this use case was supposed to work or if there's something I'm missing. Just didn't want to put too much effort to figure it out on my own that it's "not working as designed" :)
Good luck and looking forward to hear back from your experiments!
PS: You might want to register on the mailing list because I had to manually approve your post.
I am registered, maybe the first post needs approval? Cheers, Peter
On 30/03/2020 18:18, George Kadianakis wrote:
Péter Szilágyi <peterke@gmail.com> writes:
Hey all,
I'm sorry if this is something obvious, I've searched for a day and couldn't find satisfactory information. Feel free to point me to relevant resources if I missed them.
So, I'd like to run Tor in a P2P setting on mobile phones ( https://github.com/coronanet). In general, everything works as expected. My issues are with the fine grained behaviors of the Tor proxy wrt onion addresses and network connectivity.
As long as I allow a stable internet connection to Tor, everything's fine. What I wanted to check is how Tor behaves if I start pulling the rug from underneath it in various states.
I've tried keeping Tor offline (network disabled), publish an onion into the service and then enable networking afterwards (SETCONF DisableNetwork 0). This also seems to work, when Tor goes online it publishes the onion that was added offline and I can access it (at least sending a HEARTBEAT signal reports "1 v3 INTRODUCE2 cells and attempted to launch 1 rendezvous circuits"). So far so good.
Now comes the more interesting test: while networking is enabled and seemingly everything works, I cut off network access at the router level (i.e. simulate a phone losing WiFi signal / turning off data). Similar to my previous test, I've created a new onion in this scenario too, then reenabled WiFI. Unfortunately, whilst my first onion comes back online, the second one is lost in some black hole. My guess is that Tor **thinks** it's online, fails to register it, then some internal state gets messed up.
If I try to list the current onions (GETINFO onions/current), both appear. If I try to list some infos about them via `GETINFO hs/service/desc/id/`, both look the same, reporting their own crypto keys. However, only the first one response, the second never gets router, neither noticed by the local Tor instance.
At this point I'd be perfectly happy is this weird scenario was "not meant to work by design", as long as Tor could explicitly tell me somehow that "sorry, onion X is dead, sort it out yourself". I'd be even happier if this worked though. Maybe this is a bug that was fixed in a later version? I'm on 0.3.5.10.
Greetings friend,
while I don't have The Solution for you I have a few questions that will take us closer to it:
a) What is the purpose of your experiments? Are you trying to simulate a phone losing WiFi signal? If yes, that seems like a fine test to do and something that Tor should handle gracefully (and if it doesn't we should fix it ASAP).
In general, hosting onion services over mobile is a use case that we've had trouble in the past, and we've been constantly trying to increase its stability, by getting feedback and help from our friends at the Briar project (cc'ed Michael) and Guardian project (cc'ed Nathan).
b) Yes, do use the latest alpha tor instead of 0.3.5.10. There are various logs and general reachability enhancements that might not have been backported that far back.
Ideally you would even use Tor with git master since we recently merged a branch that should provide further debugging log messages in cases like yours: https://trac.torproject.org/projects/tor/ticket/33400
c) It would be really useful to see some *info* logs from your service if you manage to reproduce this bug with Tor master. My intuition is that the second service fails to upload its descriptor because of Tor being offline, and then never notices the network going up to try again. Feel free to either upload your logs on Tor's trac (in a new ticket) or either send it to me and David Goulet (dgoulet@torproject.org).
Good luck and looking forward to hear back from your experiments!
PS: You might want to register on the mailing list because I had to manually approve your post.
Hey Péter, Sorry for the slow reply! Glad to meet a fellow traveller on the road to mobile P2P (mostly uphill, no signposts, many potholes). The way we handle this situation in Briar is to monitor the phone's connectivity status (to the extent that the OS is able to detect it - for example, the phone may not know that the router has lost internet access). When the phone detects that it's lost internet access, we disable Tor's network connection via DisableNetwork, then re-enable it when internet access is restored. We also make connectivity checks when we receive intents that might indicate an undetected connnectivity change - such as ACTION_DEVICE_IDLE_MODE_CHANGED, because network access may be shut off during idle (aka doze) mode without a CONNECTIVITY_ACTION intent being broadcast. We don't use DisableNetwork when the phone roams between wifi and mobile data (unless this results in a connectivity loss). In this situation Tor detects that its guard connections have been closed and republishes the hidden service descriptor when the new guard connections are open. I guess we might want to make this a bit more robust by setting DisableNetwork to 1 and back to 0 even if there's no connectivity loss. This might help if, for example, old guard connections aren't closed promptly on some devices or Android versions. More to the point, it should allow Tor to recover if the phone was previously connected to a wifi network without internet access and didn't know it. I guess there may be other Tor internal state that should be reset if Tor has moved to a new network, and perhaps setting DisableNetwork to 1 and back to 0 would be a reasonable way to achieve that. George, while I've got your attention, does that sound reasonable, or is it better to let Tor detect the change itself? Here's the code if you're interested (Apache license): https://code.briarproject.org/briar/briar/blob/master/bramble-android/src/ma... https://code.briarproject.org/briar/briar/blob/master/bramble-core/src/main/... Cheers, Michael
participants (3)
-
George Kadianakis -
Michael Rogers -
Péter Szilágyi