All,
I am working on a pure Golang relay implementation.
https://github.com/mmcloughlin/pearl/
I have thus far been testing locally with chutney ( https://gitweb.torproject.org/chutney.git). The project is not complete by any stretch, but I believe I am close to the point where it can handle Tor network traffic. I expect I can learn *a lot* by running against real traffic rather than my sandboxed environment.
I wanted to get in touch with the community before going live. Any advice gratefully received.
The implementation identifies itself clearly in the server descriptor, both in the "platform" and "contact" items. If people notice any problems with the relay specifically they will be able to contact me.
Thanks, Mike
Hey
Thanks for doing that! It came many times to my mind.
There were efforts on implementing Tor relay in Golang before, by Tom van der Woerdt: https://github.com/tvdw/gotor Have you looked at the project?
In his blog post https://tvdw.eu/blog/2015/01/24/implementing-a-tor-relay-from-scratch/ Tom described some issues he faced with during this work and a list of lessons learnt.
On Tue, Oct 24, 2017 at 5:18 AM, Michael McLoughlin mmcloughlin@gmail.com wrote:
All,
I am working on a pure Golang relay implementation.
https://github.com/mmcloughlin/pearl/
I have thus far been testing locally with chutney (https://gitweb.torproject.org/chutney.git). The project is not complete by any stretch, but I believe I am close to the point where it can handle Tor network traffic. I expect I can learn *a lot* by running against real traffic rather than my sandboxed environment.
I wanted to get in touch with the community before going live. Any advice gratefully received.
The implementation identifies itself clearly in the server descriptor, both in the "platform" and "contact" items. If people notice any problems with the relay specifically they will be able to contact me.
Thanks, Mike
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Thanks for the encouragement Nagaev :)
Yes I am very aware of Tom van der Woerdt's previous work, and I am attempting to avoid some of the problems he faced. This implementation is pure Go, so I will not have cgo-based issues at least.
As I said the project is still far from complete. It implements a minimal subset of the protocol. But I'd like to learn by handling real traffic as soon as I can.
On Tue, Oct 24, 2017 at 6:59 AM, Nagaev Boris bnagaev@gmail.com wrote:
Hey
Thanks for doing that! It came many times to my mind.
There were efforts on implementing Tor relay in Golang before, by Tom van der Woerdt: https://github.com/tvdw/gotor Have you looked at the project?
In his blog post https://tvdw.eu/blog/2015/01/24/implementing-a-tor-relay-from-scratch/ Tom described some issues he faced with during this work and a list of lessons learnt.
On Tue, Oct 24, 2017 at 5:18 AM, Michael McLoughlin mmcloughlin@gmail.com wrote:
All,
I am working on a pure Golang relay implementation.
https://github.com/mmcloughlin/pearl/
I have thus far been testing locally with chutney (https://gitweb.torproject.org/chutney.git). The project is not
complete by
any stretch, but I believe I am close to the point where it can handle
Tor
network traffic. I expect I can learn *a lot* by running against real traffic rather than my sandboxed environment.
I wanted to get in touch with the community before going live. Any advice gratefully received.
The implementation identifies itself clearly in the server descriptor,
both
in the "platform" and "contact" items. If people notice any problems with the relay specifically they will be able to contact me.
Thanks, Mike
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
-- Best regards, Boris Nagaev _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Tue, Oct 24, 2017 at 10:54:38AM -0700, Michael McLoughlin wrote:
Yes I am very aware of Tom van der Woerdt's previous work, and I am attempting to avoid some of the problems he faced. This implementation is pure Go, so I will not have cgo-based issues at least.
Great! Yes, I think the memory bloat was the main issue that stopped Tom.
Also, you should take a look at Alex's talk about his erlang Tor relay implementation experiences, including lessons learned and why he became less enthusiastic about maintaining a separate Tor relay implementation: https://media.ccc.de/v/SHA2017-304-introducing_talla_an_erlang_implementatio...
--Roger
https://trac.torproject.org/projects/tor/ticket/23981#comment:9 wrote:
Looks like this issue is caused by a server descriptor produced by an alternate Tor implementation that identifies itself as `"platform Pearl 0fd5756 on Linux"`. And that implementation uses a different order of elements in its descriptor.
:)
So it's already finding bugs in other implementations?
That's pretty awesome! ;-)
//Spider
On 25/10/17 21:16, nusenu wrote:
https://trac.torproject.org/projects/tor/ticket/23981#comment:9 wrote:
Looks like this issue is caused by a server descriptor produced by an alternate Tor implementation that identifies itself as `"platform Pearl 0fd5756 on Linux"`. And that implementation uses a different order of elements in its descriptor.
:)
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Is the bug in my descriptor or the parser?
In testing it took a little bit of finagling to get chutney to accept a non-Tor "platform" item in the descriptor. Turned out adding a "proto" line as well is enough. But it did occur to me that other code may assume the "platform" line takes a certain form.
I can easily change the descriptor if necessary? I've included it below for reference:
---
router pearl000 35.203.138.1 9001 0 0 signing-key -----BEGIN RSA PUBLIC KEY----- MIGJAoGBAKzTaN4tZGv1kiQWBzeuOk+ovr2LtIURlaVC38j6j/fQuYfuAZX/XvV1 fQr9EVh+T617dh+frt2D0QDuzLUvP3hpgVozW94w+Ib85pUCne03f4rj3QYu5Qtg GvzShslZI6vgyy0g2jAOGa4jxT/UYAcKE5dQo8CBKA6Qb0P5Joc1AgMBAAE= -----END RSA PUBLIC KEY----- fingerprint 6832 5B4B 1E17 7374 B84D 372F 0304 6351 BEE7 FF6A onion-key -----BEGIN RSA PUBLIC KEY----- MIGJAoGBANN/gLTe05kWKPSEyYJeknuxQst+cVsVmZrZgIYNXuhPn+3XnWhEc10r ICa82FkB7hBH6REuW0ugGDc2QLwENmDiaBiFW1LDujEFeVlV8o0VSDwrL3VCPsPL zC4/zHqR4DmLFXp5V238MKj85Pud04g65piZCIAsy6hiGMDCoGdtAgMBAAE= -----END RSA PUBLIC KEY----- ntor-onion-key ATSN2Q9KwCeRu35agh/ChjX8MsgM/FGFRDUX6o9Sbmk platform Pearl 0fd5756 on Linux contact pearl@m15n.org https://github.com/mmcloughlin/pearl bandwidth 153600 307200 153600 published 2017-10-25 18:00:28 reject *:* proto Link=4 LinkAuth=1 Relay=2 router-signature -----BEGIN SIGNATURE----- OyY0vQc5n2RYdkrXqfn09HoACJBx7GrBHZMnmNtlX5nJIL9N4eyyPvmxhmuC+A94 dDE0u/6w3nCABikFFLHcKaBAdmYBdxrzk3imfVjzYZazHWWr/se8HxK1jibP186A 8K8bdtMih127CGv3mn+g17uXFTbbuylM7r1xf8NpqRs= -----END SIGNATURE-----
On Wed, Oct 25, 2017 at 12:34 PM, D.S. Ljungmark spider@takeit.se wrote:
So it's already finding bugs in other implementations?
That's pretty awesome! ;-)
//Spider
On 25/10/17 21:16, nusenu wrote:
https://trac.torproject.org/projects/tor/ticket/23981#comment:9 wrote:
Looks like this issue is caused by a server descriptor produced by an alternate Tor implementation that identifies itself as `"platform Pearl 0fd5756 on Linux"`. And that implementation uses a different order of elements in its descriptor.
:)
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Michael McLoughlin:
Is the bug in my descriptor or the parser?
First of all it was a problem for the parser and they hotfixed it. (I did not verify if your descriptor follows the specification, if it does you should be fine.)
On 26 Oct 2017, at 06:58, Michael McLoughlin mmcloughlin@gmail.com wrote:
...
In testing it took a little bit of finagling to get chutney to accept a non-Tor "platform" item in the descriptor. Turned out adding a "proto" line as well is enough.
I think this is a feature of the tor directory authority implementation, not chutney. (Chutney doesn't do anything with platform or proto lines.) Directory authorities will only synthesise a proto line for platforms that claim to be Tor, because they can't be sure what features non-Tor platforms support.
But it did occur to me that other code may assume the "platform" line takes a certain form.
It shouldn't, that's what protocol lines are for.
But any alternate implementation will never be used as a v3 HSDir, because it would need to claim to be Tor 0.3.0.8 or later for v3 onion services to use it. This is a poor design decision on our part: just like consensus methods, when we break a protocol version, we should allocate a new number, and check it. (Or we should exclude broken versions from the consensus.)
I've opened this ticket to fix that: https://trac.torproject.org/projects/tor/ticket/23998
I can easily change the descriptor if necessary?
As long as it conforms to the spec, it's fine.
We should really fuzz descriptor parsers better. But that's not an appropriate thing to do on the live network, and some parser code only runs on descriptors on the live network.
T
On 2017-10-26 00:09, teor wrote:
On 26 Oct 2017, at 06:58, Michael McLoughlin <mmcloughlin@gmail.com mailto:mmcloughlin@gmail.com> wrote:
I can easily change the descriptor if necessary?
As long as it conforms to the spec, it's fine.
Agreed. FWIW, the descriptor published by this relay confused Metrics quite a bit. But that's okay, we'll just make Metrics more robust. The good news is that we didn't lose any data in the process.
We should really fuzz descriptor parsers better. But that's not an appropriate thing to do on the live network, and some parser code only runs on descriptors on the live network.
If somebody wants to generate a bunch of fuzzed descriptors that conform to the spec, I'll happily throw them into a local Metrics instance to see if anything else breaks. I could imagine that Damian would do the same with stem and Philipp with zoossh.
All the best, Karsten
On 26 Oct 2017, at 18:07, Karsten Loesing karsten@torproject.org wrote:
On 2017-10-26 00:09, teor wrote:
On 26 Oct 2017, at 06:58, Michael McLoughlin <mmcloughlin@gmail.com mailto:mmcloughlin@gmail.com> wrote:
I can easily change the descriptor if necessary?
As long as it conforms to the spec, it's fine.
Agreed. FWIW, the descriptor published by this relay confused Metrics quite a bit. But that's okay, we'll just make Metrics more robust. The good news is that we didn't lose any data in the process.
We should really fuzz descriptor parsers better. But that's not an appropriate thing to do on the live network, and some parser code only runs on descriptors on the live network.
If somebody wants to generate a bunch of fuzzed descriptors that conform to the spec, I'll happily throw them into a local Metrics instance to see if anything else breaks. I could imagine that Damian would do the same with stem and Philipp with zoossh.
leekspin and maybe stem can produce valid, randomised descriptors.
Tor also has a collection of valid and invalid directory documents in:
https://gitweb.torproject.org/fuzzing-corpora.git/tree/
I'm not sure if there are any others.
-- Tim / teor
PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
leekspin and maybe stem can produce valid, randomised descriptors.
Yup. Stem now supports descriptor creation (and should have all capabilities Leekspin did)...
https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html#can-i-c...
After another look at the spec, I still believe the descriptor I'm publishing conforms, as was my intention. Sorry to have caused all these problems :(
Heads up that there's another (nascent) tor relay implementation in the works. I reached out to them to see if they were interested in collaborating, but I didn't get a response. It's unclear to me what their plans are. However Filippo Valsorda has a strong reputation so it's worth keeping an eye on.
Mike
On Thu, Oct 26, 2017 at 12:07 AM, Karsten Loesing karsten@torproject.org wrote:
On 2017-10-26 00:09, teor wrote:
On 26 Oct 2017, at 06:58, Michael McLoughlin <mmcloughlin@gmail.com mailto:mmcloughlin@gmail.com> wrote:
I can easily change the descriptor if necessary?
As long as it conforms to the spec, it's fine.
Agreed. FWIW, the descriptor published by this relay confused Metrics quite a bit. But that's okay, we'll just make Metrics more robust. The good news is that we didn't lose any data in the process.
We should really fuzz descriptor parsers better. But that's not an appropriate thing to do on the live network, and some parser code only runs on descriptors on the live network.
If somebody wants to generate a bunch of fuzzed descriptors that conform to the spec, I'll happily throw them into a local Metrics instance to see if anything else breaks. I could imagine that Damian would do the same with stem and Philipp with zoossh.
All the best, Karsten
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Thu, Oct 26, 2017 at 02:56:03PM -0700, Michael McLoughlin wrote:
After another look at the spec, I still believe the descriptor I'm publishing conforms, as was my intention. Sorry to have caused all these problems :(
No, don't apologize! It's great that there are people implementing from our specs, and it's great when they find bugs with tools that made bad assumptions and expectations. :)
--Roger
On 27 Oct 2017, at 09:00, Roger Dingledine arma@mit.edu wrote:
On Thu, Oct 26, 2017 at 02:56:03PM -0700, Michael McLoughlin wrote: After another look at the spec, I still believe the descriptor I'm publishing conforms, as was my intention.
Each of the bugs you found are very useful to us, because they help the next person who tries to implement something that works with Tor.
In particular, we almost forgot this one:
If directory authorities treat the "proto" line as mandatory for non-Tor implementations, we should probably document this fact here:
https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n773
I opened a ticket:
https://trac.torproject.org/projects/tor/ticket/24023#ticket
Sorry to have caused all these problems :(
No, don't apologize! It's great that there are people implementing from our specs, and it's great when they find bugs with tools that made bad assumptions and expectations. :)
You should see the list of changes we made to tor-spec.txt when implementing a simple OR protocol client in Python. About half of the last 20 spec changes were triggered by that project:
https://gitweb.torproject.org/torspec.git/log/tor-spec.txt
And here's the project repository:
https://github.com/teor2345/endosome
That spec is much clearer now!
T
Missed a link from my last email: https://github.com/go-tor/gotor
On Thu, Oct 26, 2017 at 2:56 PM, Michael McLoughlin mmcloughlin@gmail.com wrote:
After another look at the spec, I still believe the descriptor I'm publishing conforms, as was my intention. Sorry to have caused all these problems :(
Heads up that there's another (nascent) tor relay implementation in the works. I reached out to them to see if they were interested in collaborating, but I didn't get a response. It's unclear to me what their plans are. However Filippo Valsorda has a strong reputation so it's worth keeping an eye on.
Mike
On Thu, Oct 26, 2017 at 12:07 AM, Karsten Loesing karsten@torproject.org wrote:
On 2017-10-26 00:09, teor wrote:
On 26 Oct 2017, at 06:58, Michael McLoughlin <mmcloughlin@gmail.com mailto:mmcloughlin@gmail.com> wrote:
I can easily change the descriptor if necessary?
As long as it conforms to the spec, it's fine.
Agreed. FWIW, the descriptor published by this relay confused Metrics quite a bit. But that's okay, we'll just make Metrics more robust. The good news is that we didn't lose any data in the process.
We should really fuzz descriptor parsers better. But that's not an appropriate thing to do on the live network, and some parser code only runs on descriptors on the live network.
If somebody wants to generate a bunch of fuzzed descriptors that conform to the spec, I'll happily throw them into a local Metrics instance to see if anything else breaks. I could imagine that Damian would do the same with stem and Philipp with zoossh.
All the best, Karsten
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 26 Oct 2017, at 09:09, teor teor2345@gmail.com wrote:
But it did occur to me that other code may assume the "platform" line takes a certain form.
It shouldn't, that's what protocol lines are for.
But any alternate implementation will never be used as a v3 HSDir, because it would need to claim to be Tor 0.3.0.8 or later for v3 onion services to use it. This is a poor design decision on our part: just like consensus methods, when we break a protocol version, we should allocate a new number, and check it. (Or we should exclude broken versions from the consensus.)
I've opened this ticket to fix that: https://trac.torproject.org/projects/tor/ticket/23998
For the record, I was wrong about this.
We ignore "Tor" implementations on 0.3.0.7 and below when checking for the HSDir protover 2. (Which supports v3 onion services.)
So any non-Tor implementation that advertises support for protover 2 will be accepted as a v3 onion service HSDir.
T
On Tue, Oct 24, 2017 at 12:18 AM, Michael McLoughlin mmcloughlin@gmail.com wrote:
I am working on a pure Golang relay implementation. https://github.com/mmcloughlin/pearl/
Please add your implementation (and any others known, and their locations, as may have been mentioned or updated in this thread) to the list of implementations maintained here...
https://trac.torproject.org/projects/tor/wiki/doc/ListOfTorImplementations
This can help bring participants, review, etc to all.
tor-relays@lists.torproject.org
-
D.S. Ljungmark -
Damian Johnson -
grarpamp -
Karsten Loesing -
Michael McLoughlin -
Nagaev Boris -
nusenu -
Roger Dingledine -
teor