New Fallback Directory File Format
Hi all, Some time in the next few weeks, the Tor fallback directory mirror file format will change. This affects stem and Relay Search, which parse this file. Change Description Here is a list of changes to the file format: * the "weight" line has been removed, and replaced with a Tor config default (#24679, #24681) * the comma that separates fallback C strings is now on its own line * a "nickname" comment has been added (#24600) * an optional "extrainfo" comment has been added (#22759) The added fields will be populated with placeholders until the list is rebuilt (#22271). This will hopefully happen some time in the next few weeks. Requesting More Extra Info Caches There are only a few fallbacks that cache extra-info documents. I checked 67, and only 4 cached extra-info documents. atagar, do you want me to ask some fallback operators to set DownloadExtraInfo 1? What number or proportion would you like? (We allow approximately 25% of fallbacks to go down before we start to rebuild the list. In the worst case, this can mean that ~40% are down at some point.) Example Entries A sample entry in the new format, using actual relay info: "5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB" " ipv6=[2a01:4f8:162:51e2::2]:9001" /* nickname=rueckgrat */ /* extrainfo=1 */ , The current fallback file in the new format, with placeholders: https://github.com/teor2345/tor/blob/ticket22759_tree/src/or/fallback_dirs.i... A small sample fallback file in the new format, with actual relay info: https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_di... Please let me know if you would like any changes to the format. T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
Hi I just added the DownloadExtraInfo in the config file for the nodes "chulak" and "aurora". Greetings teor:
Hi all,
Some time in the next few weeks, the Tor fallback directory mirror file format will change. This affects stem and Relay Search, which parse this file.
Change Description
Here is a list of changes to the file format: * the "weight" line has been removed, and replaced with a Tor config default (#24679, #24681) * the comma that separates fallback C strings is now on its own line * a "nickname" comment has been added (#24600) * an optional "extrainfo" comment has been added (#22759)
The added fields will be populated with placeholders until the list is rebuilt (#22271). This will hopefully happen some time in the next few weeks.
Requesting More Extra Info Caches
There are only a few fallbacks that cache extra-info documents. I checked 67, and only 4 cached extra-info documents.
atagar, do you want me to ask some fallback operators to set DownloadExtraInfo 1?
What number or proportion would you like?
(We allow approximately 25% of fallbacks to go down before we start to rebuild the list. In the worst case, this can mean that ~40% are down at some point.)
Example Entries
A sample entry in the new format, using actual relay info:
"5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB" " ipv6=[2a01:4f8:162:51e2::2]:9001" /* nickname=rueckgrat */ /* extrainfo=1 */ ,
The current fallback file in the new format, with placeholders: https://github.com/teor2345/tor/blob/ticket22759_tree/src/or/fallback_dirs.i...
A small sample fallback file in the new format, with actual relay info: https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_di...
Please let me know if you would like any changes to the format.
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Hi Tim, added preliminary Stem parsing support for the v2 fallback format... https://gitweb.torproject.org/stem.git/commit/?id=ea55eaa Few thoughts are... * It would be nice if the 'extrainfo=' lines were mandatory since I need a delimiter between the entries. * Maybe we should start the document with a format version number? Cheers! -Damian On Fri, Dec 22, 2017 at 6:53 AM, teor <teor2345@gmail.com> wrote:
Hi all,
Some time in the next few weeks, the Tor fallback directory mirror file format will change. This affects stem and Relay Search, which parse this file.
Change Description
Here is a list of changes to the file format: * the "weight" line has been removed, and replaced with a Tor config default (#24679, #24681) * the comma that separates fallback C strings is now on its own line * a "nickname" comment has been added (#24600) * an optional "extrainfo" comment has been added (#22759)
The added fields will be populated with placeholders until the list is rebuilt (#22271). This will hopefully happen some time in the next few weeks.
Requesting More Extra Info Caches
There are only a few fallbacks that cache extra-info documents. I checked 67, and only 4 cached extra-info documents.
atagar, do you want me to ask some fallback operators to set DownloadExtraInfo 1?
What number or proportion would you like?
(We allow approximately 25% of fallbacks to go down before we start to rebuild the list. In the worst case, this can mean that ~40% are down at some point.)
Example Entries
A sample entry in the new format, using actual relay info:
"5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB" " ipv6=[2a01:4f8:162:51e2::2]:9001" /* nickname=rueckgrat */ /* extrainfo=1 */ ,
The current fallback file in the new format, with placeholders: https://github.com/teor2345/tor/blob/ticket22759_tree/src/or/fallback_dirs.i...
A small sample fallback file in the new format, with actual relay info: https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_di...
Please let me know if you would like any changes to the format.
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
On 24 Dec 2017, at 08:16, Damian Johnson <atagar@torproject.org> wrote:
Hi Tim, added preliminary Stem parsing support for the v2 fallback format...
https://gitweb.torproject.org/stem.git/commit/?id=ea55eaa
Few thoughts are...
* It would be nice if the 'extrainfo=' lines were mandatory since I need a delimiter between the entries.
We have a few alternatives here: C requires a comma as the delimiter between entries. I can guarantee there will always be a comma delimiter after every entry, Including the last entry. I will also make sure that any comment fields come before this delimiter. I can guarantee there will never be a comma inside the C string or comments in the entry. Alternately, I can make extrainfo mandatory, and if I can't fetch a relay's descriptor (a rare case?), I will mark is as 0. But I'd like to be able to add extra fields in future without breaking parsers, so I don't want parsers relying on field order. Do you want me to add an explicit end of record comment, or is the comma sufficient? Is there a delimiter you'd like me to add before the first entry?
* Maybe we should start the document with a format version number?
I thought of that, too. Thanks for reminding me. https://trac.torproject.org/projects/tor/ticket/24725
Cheers! -Damian
On Fri, Dec 22, 2017 at 6:53 AM, teor <teor2345@gmail.com> wrote: Hi all,
Some time in the next few weeks, the Tor fallback directory mirror file format will change. This affects stem and Relay Search, which parse this file.
Change Description
Here is a list of changes to the file format: * the "weight" line has been removed, and replaced with a Tor config default (#24679, #24681) * the comma that separates fallback C strings is now on its own line * a "nickname" comment has been added (#24600) * an optional "extrainfo" comment has been added (#22759)
The added fields will be populated with placeholders until the list is rebuilt (#22271). This will hopefully happen some time in the next few weeks.
Requesting More Extra Info Caches
There are only a few fallbacks that cache extra-info documents. I checked 67, and only 4 cached extra-info documents.
atagar, do you want me to ask some fallback operators to set DownloadExtraInfo 1?
What number or proportion would you like?
(We allow approximately 25% of fallbacks to go down before we start to rebuild the list. In the worst case, this can mean that ~40% are down at some point.)
Example Entries
A sample entry in the new format, using actual relay info:
"5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB" " ipv6=[2a01:4f8:162:51e2::2]:9001" /* nickname=rueckgrat */ /* extrainfo=1 */ ,
The current fallback file in the new format, with placeholders: https://github.com/teor2345/tor/blob/ticket22759_tree/src/or/fallback_dirs.i...
A small sample fallback file in the new format, with actual relay info: https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_di...
Please let me know if you would like any changes to the format.
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Do you want me to add an explicit end of record comment, or is the comma sufficient?
Hi Tim. I'd rather not rely on just a comma. I can easily see us tweaking the layout so 'expect a line with only a comma' breaks. I actually like both of the other suggested options: having 'extrainfo=0' so that's explicit *and* a delimiter. For instance... /* ============================== */ ... or whatever. That would both help with parsing and make the file nicer to read. Cheers! -Damian
On 24 Dec 2017, at 11:14, Damian Johnson <atagar@torproject.org> wrote:
Do you want me to add an explicit end of record comment, or is the comma sufficient?
Hi Tim. I'd rather not rely on just a comma. I can easily see us tweaking the layout so 'expect a line with only a comma' breaks.
I actually like both of the other suggested options: having 'extrainfo=0' so that's explicit *and* a delimiter. For instance...
/* ============================== */
... or whatever. That would both help with parsing and make the file nicer to read.
Done! * the file now starts with a type and a version line: /* type=fallback */ /* version=2.0.0 */ * extrainfo is mandatory (occasionally we won't get a descriptor, so we'll warn and mark the relay extrainfo=0) * each fallback entry ends with /* ===== */ Two remaining questions: * is 6 extra info caches (up from 4) enough in a list of 150? * do you want the delimiter before the first fallback entry as well? Sample entry: "5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB" " ipv6=[2a01:4f8:162:51e2::2]:9001" /* nickname=rueckgrat */ /* extrainfo=1 */ /* ===== */ , Sample file: https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_di... T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
Done!
* the file now starts with a type and a version line: /* type=fallback */ /* version=2.0.0 */ * extrainfo is mandatory (occasionally we won't get a descriptor, so we'll warn and mark the relay extrainfo=0) * each fallback entry ends with /* ===== */
Sweet, thanks Tim!
* is 6 extra info caches (up from 4) enough in a list of 150?
Hmmm. Can't say on Stem's side I have an opinion on this. It doesn't rely on fallback directories for anything so extrainfo caches aren't a concern.
* do you want the delimiter before the first fallback entry as well?
Stem doesn't care about a delimiter before the first entry but that seems like a good idea so we have a clear separation between comments and the start of the machine readable section. A detail Stem does care about is that the last entry ends with a delimiter. If it doesn't that's fine, but code is a tad simpler if we ensure it does. :) Cheers! -Damian
Hi, On 24/12/17 20:13, Damian Johnson wrote. As we are planning to also add a parser to metrics-lib (#24434), would it be possible to get a full description of the format of the file possibly in RFC5234 format so that we can check that the generator and parsers all match up to that specification? Thanks, Iain.
On 25 Dec 2017, at 07:13, Damian Johnson <atagar@torproject.org> wrote:
Done!
* the file now starts with a type and a version line: /* type=fallback */ /* version=2.0.0 */ * extrainfo is mandatory (occasionally we won't get a descriptor, so we'll warn and mark the relay extrainfo=0) * each fallback entry ends with /* ===== */
Sweet, thanks Tim!
* is 6 extra info caches (up from 4) enough in a list of 150?
Hmmm. Can't say on Stem's side I have an opinion on this. It doesn't rely on fallback directories for anything so extrainfo caches aren't a concern.
So stem just uses a mix of fallbacks and authorities? Good, that's what Tor does. I think we will be fine then.
* do you want the delimiter before the first fallback entry as well?
Stem doesn't care about a delimiter before the first entry but that seems like a good idea so we have a clear separation between comments and the start of the machine readable section.
I made the header end with a delimiter, and I made the list of entries start with a delimiter: https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_di... This means that parsers can (and should) ingore the human-readable second section.
A detail Stem does care about is that the last entry ends with a delimiter. If it doesn't that's fine, but code is a tad simpler if we ensure it does. :)
It does and it will continue to.
On 25 Dec 2017, at 07:26, Iain Learmonth <irl@torproject.org> wrote:
As we are planning to also add a parser to metrics-lib (#24434), would it be possible to get a full description of the format of the file possibly in RFC5234 format so that we can check that the generator and parsers all match up to that specification?
I have written up a format in the standard torspec style: https://github.com/teor2345/torspec/blob/fallback-format-2/fallback-spec.txt It is deliberately under-specified, please let me know if this causes any trouble when writing the parser, and I will tighten it up. It's not ABNF/RFC5234, it's rather restrictive, and strict ABNF is unreadable for case sensitive strings. I am happy to put an ABNF spec in an appendix, if someone wants to write one. T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
On 26 Dec 2017, at 18:47, teor <teor2345@gmail.com> wrote:
On 25 Dec 2017, at 07:26, Iain Learmonth <irl@torproject.org> wrote:
As we are planning to also add a parser to metrics-lib (#24434), would it be possible to get a full description of the format of the file possibly in RFC5234 format so that we can check that the generator and parsers all match up to that specification?
I have written up a format in the standard torspec style:
https://github.com/teor2345/torspec/blob/fallback-format-2/fallback-spec.txt
It is deliberately under-specified, please let me know if this causes any trouble when writing the parser, and I will tighten it up.
It's not ABNF/RFC5234, it's rather restrictive, and strict ABNF is unreadable for case sensitive strings. I am happy to put an ABNF spec in an appendix, if someone wants to write one.
We've added separators to each section, and a timestamp field in the header. The revised spec is here: https://github.com/teor2345/torspec/blob/fallback-format-2/fallback-spec.txt The revised sample file is here: https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_di... We're tracking the details in this ticket: https://trac.torproject.org/projects/tor/ticket/24742 T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
Hi,
On 27 Dec 2017, at 21:53, teor <teor2345@gmail.com> wrote:
On 26 Dec 2017, at 18:47, teor <teor2345@gmail.com> wrote:
On 25 Dec 2017, at 07:26, Iain Learmonth <irl@torproject.org> wrote:
As we are planning to also add a parser to metrics-lib (#24434), would it be possible to get a full description of the format of the file possibly in RFC5234 format so that we can check that the generator and parsers all match up to that specification?
I have written up a format in the standard torspec style:
https://github.com/teor2345/torspec/blob/fallback-format-2/fallback-spec.txt
It is deliberately under-specified, please let me know if this causes any trouble when writing the parser, and I will tighten it up.
It's not ABNF/RFC5234, it's rather restrictive, and strict ABNF is unreadable for case sensitive strings. I am happy to put an ABNF spec in an appendix, if someone wants to write one.
We've added separators to each section, and a timestamp field in the header.
The revised spec is here:
https://github.com/teor2345/torspec/blob/fallback-format-2/fallback-spec.txt
The revised sample file is here:
https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_di...
We're tracking the details in this ticket:
We have merged the list format change to Tor's master branch. The fallback 2.0.0 spec hasn't been merged yet, but atagar has reviewed it. It's at: https://github.com/teor2345/torspec/blob/fallback-format-2-v2/fallback-spec.... The fallback list is now in the 2.0.0 format, but it has exactly the same fallbacks in it: https://gitweb.torproject.org/tor.git/tree/src/or/fallback_dirs.inc Some time over the next few days, we will generate a new list of fallbacks in the 2.0.0 format, and backport it to Tor 0.2.9 and later. T
Hi, On 05/01/18 22:33, teor wrote:
The fallback 2.0.0 spec hasn't been merged yet, but atagar has reviewed it. It's at: https://github.com/teor2345/torspec/blob/fallback-format-2-v2/fallback-spec....
I've added this link to the Metrics ticket #24434, I started work on this but ran out of time after not getting very far at all.
The fallback list is now in the 2.0.0 format, but it has exactly the same fallbacks in it: https://gitweb.torproject.org/tor.git/tree/src/or/fallback_dirs.inc
Some time over the next few days, we will generate a new list of fallbacks in the 2.0.0 format, and backport it to Tor 0.2.9 and later.
Ok, I'll see if I can do a quick hack workaround to generate the list for Relay Search until the data is available through Tor Metrics. Thanks, Iain.
Hi, On 06/01/18 15:52, Iain Learmonth wrote:
Ok, I'll see if I can do a quick hack workaround to generate the list for Relay Search until the data is available through Tor Metrics.
In fact, my original quick hack workaround still works, as it's only extracting fingerprints. (Tested that there is no lines changed in a diff against the same file in the old format). I'm ready to update as soon as there's a new list available. Thanks, Iain.
Hi, We've merged a new list of fallbacks in the version 2.0.0 format. But we also want to keep a separate list of authorities, rather than hiding them in the middle of config.c: https://trac.torproject.org/projects/tor/ticket/24818 So we are changing the current authority and fallback list formats, to make them consistent with each other. (Sorry about doing the two changes separately, we had to do fallbacks for the 0.3.2.9 release deadline.) Here's what will change in version 3.0.0: https://github.com/teor2345/torspec/commits/dir-list Hopefully, you can use the same parser for both file formats.
On 7 Jan 2018, at 02:52, Iain Learmonth <irl@torproject.org> wrote:
Hi,
On 05/01/18 22:33, teor wrote:
The fallback 2.0.0 spec hasn't been merged yet, but atagar has reviewed it. It's at: https://github.com/teor2345/torspec/blob/fallback-format-2-v2/fallback-spec....
I've added this link to the Metrics ticket #24434, I started work on this but ran out of time after not getting very far at all.
The fallback list spec version 2.0.0 has been merged, it's now at: https://gitweb.torproject.org/torspec.git/tree/dir-list-spec.txt
The fallback list is now in the 2.0.0 format, but it has exactly the same fallbacks in it: https://gitweb.torproject.org/tor.git/tree/src/or/fallback_dirs.inc
Some time over the next few days, we will generate a new list of fallbacks in the 2.0.0 format, and backport it to Tor 0.2.9 and later.
Ok, I'll see if I can do a quick hack workaround to generate the list for Relay Search until the data is available through Tor Metrics.
I'm glad this worked out! Thanks for doing this update. T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
We've merged a new list of fallbacks in the version 2.0.0 format.
Hi Tim. Last couple weeks I've been working on a Fallback and Endosome branch for Stem. Just finished and merged the former... https://gitweb.torproject.org/stem.git/commit/?id=ea2752c Stem now supports the v2 format, has additional test coverage, and is synced with Tor's current fallback file. Cheers! -Damian
participants (4)
-
Damian Johnson -
Iain Learmonth -
teor -
Tyler Durden