[tor-bugs] #20421 [Metrics/CollecTor]: Investigate invalid descriptors in out/ and recent/ subdirectories
Tor Bug Tracker & Wiki
blackhole at torproject.org
Fri Oct 21 09:26:44 UTC 2016
#20421: Investigate invalid descriptors in out/ and recent/ subdirectories
-------------------------------+---------------------
Reporter: karsten | Owner:
Type: defect | Status: new
Priority: High | Milestone:
Component: Metrics/CollecTor | Version:
Severity: Normal | Resolution:
Keywords: | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
-------------------------------+---------------------
Comment (by karsten):
Ah, some more information from the CollecTor logs:
{{{
2016-10-19 04:05:00,001 INFO o.t.c.c.CollecTorMain:54 Starting relaydescs
module of CollecTor.
2016-10-19 04:05:10,870 INFO o.t.c.r.ArchiveWriter:842 Could not parse
relay descriptor out/relay-descriptors/server-
descriptor/2016/10/5/7/57077357a8bc1d431e89e29e3cec7d9a31d59332 before
storing it to disk. Storing anyway.
org.torproject.descriptor.DescriptorParseException: Either keyword
'accept' or 'reject' must be contained at least once.
at
org.torproject.descriptor.impl.ServerDescriptorImpl.<init>(ServerDescriptorImpl.java:49)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.RelayServerDescriptorImpl.<init>(RelayServerDescriptorImpl.java:35)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.RelayServerDescriptorImpl.parseDescriptors(RelayServerDescriptorImpl.java:24)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.DescriptorImpl.parseDescriptors(DescriptorImpl.java:61)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.DescriptorParserImpl.parseDescriptors(DescriptorParserImpl.java:26)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.store(ArchiveWriter.java:836)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.storeServerDescriptor(ArchiveWriter.java:752)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.RelayDescriptorParser.parse(RelayDescriptorParser.java:266)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.CachedRelayDescriptorReader.<init>(CachedRelayDescriptorReader.java:216)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.startProcessing(ArchiveWriter.java:166)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:56)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_111]
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
2016-10-19 04:05:18,322 INFO o.t.c.r.ArchiveWriter:842 Could not parse
relay descriptor out/relay-descriptors/extra-
info/2016/10/4/8/481a2c9b4db3a29e499b9ce0b8bf26ca0c344e0f before storing
it to disk. Storing anyway.
org.torproject.descriptor.DescriptorParseException: Mismatch between
identity-ed25519 and master-key-ed25519.
at
org.torproject.descriptor.impl.ExtraInfoDescriptorImpl.parseIdentityEd25519CryptoBlock(ExtraInfoDescriptorImpl.java:789)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.ExtraInfoDescriptorImpl.parseDescriptorBytes(ExtraInfoDescriptorImpl.java:260)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.ExtraInfoDescriptorImpl.<init>(ExtraInfoDescriptorImpl.java:35)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.RelayExtraInfoDescriptorImpl.<init>(RelayExtraInfoDescriptorImpl.java:35)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.RelayExtraInfoDescriptorImpl.parseDescriptors(RelayExtraInfoDescriptorImpl.java:24)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.DescriptorImpl.parseDescriptors(DescriptorImpl.java:71)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.DescriptorParserImpl.parseDescriptors(DescriptorParserImpl.java:26)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.store(ArchiveWriter.java:836)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.storeExtraInfoDescriptor(ArchiveWriter.java:782)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.RelayDescriptorParser.parse(RelayDescriptorParser.java:305)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.CachedRelayDescriptorReader.<init>(CachedRelayDescriptorReader.java:216)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.startProcessing(ArchiveWriter.java:166)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:56)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_111]
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
2016-10-19 04:05:18,590 INFO o.t.c.r.CachedRelayDescriptorReader:255
Finished importing relay descriptors from local Tor data directories:
}}}
{{{
2016-10-19 07:05:00,001 INFO o.t.c.c.CollecTorMain:54 Starting relaydescs
module of CollecTor.
2016-10-19 07:05:10,500 INFO o.t.c.r.ArchiveWriter:842 Could not parse
relay descriptor out/relay-descriptors/server-
descriptor/2016/10/4/6/4630ff545faaf16f8102311c4a1506a7497a70f2 before
storing it to disk. Storing anyway.
org.torproject.descriptor.DescriptorParseException: Unrecognized character
in keyword 'hidden- at uploaded-at' in line 'hidden- at uploaded-at 2016-10-19
06:53:16'.
at
org.torproject.descriptor.impl.ParseHelper.parseKeyword(ParseHelper.java:30)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.ServerDescriptorImpl.parseDescriptorBytes(ServerDescriptorImpl.java:228)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.ServerDescriptorImpl.<init>(ServerDescriptorImpl.java:30)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.RelayServerDescriptorImpl.<init>(RelayServerDescriptorImpl.java:35)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.RelayServerDescriptorImpl.parseDescriptors(RelayServerDescriptorImpl.java:24)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.DescriptorImpl.parseDescriptors(DescriptorImpl.java:61)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.DescriptorParserImpl.parseDescriptors(DescriptorParserImpl.java:26)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.store(ArchiveWriter.java:836)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.storeServerDescriptor(ArchiveWriter.java:752)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.RelayDescriptorParser.parse(RelayDescriptorParser.java:266)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.CachedRelayDescriptorReader.<init>(CachedRelayDescriptorReader.java:216)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.startProcessing(ArchiveWriter.java:166)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:56)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_111]
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
2016-10-19 07:05:10,978 INFO o.t.c.r.ArchiveWriter:842 Could not parse
relay descriptor out/relay-descriptors/extra-
info/2016/10/3/6/36d5e078af899d327f286e5cd0bdc48189ea2a1d before storing
it to disk. Storing anyway.
org.torproject.descriptor.DescriptorParseException: Keyword 'published' is
contained 2 times, but must be contained exactly once.
at
org.torproject.descriptor.impl.DescriptorImpl.checkExactlyOnceKeywords(DescriptorImpl.java:293)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.ExtraInfoDescriptorImpl.<init>(ExtraInfoDescriptorImpl.java:40)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.RelayExtraInfoDescriptorImpl.<init>(RelayExtraInfoDescriptorImpl.java:35)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.RelayExtraInfoDescriptorImpl.parseDescriptors(RelayExtraInfoDescriptorImpl.java:24)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.DescriptorImpl.parseDescriptors(DescriptorImpl.java:71)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.descriptor.impl.DescriptorParserImpl.parseDescriptors(DescriptorParserImpl.java:26)
~[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.store(ArchiveWriter.java:836)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.storeExtraInfoDescriptor(ArchiveWriter.java:782)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.RelayDescriptorParser.parse(RelayDescriptorParser.java:305)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.CachedRelayDescriptorReader.<init>(CachedRelayDescriptorReader.java:216)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.relaydescs.ArchiveWriter.startProcessing(ArchiveWriter.java:166)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:56)
[collector-1.0.2.jar:1.0.2-c43d0ad]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_111]
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
2016-10-19 07:05:14,710 INFO o.t.c.r.CachedRelayDescriptorReader:255
Finished importing relay descriptors from local Tor data directories:
}}}
So, we can probably expect invalid descriptors to be contained in the
tarballs since we allowed descriptors that we cannot parse. Heh, makes
more sense now, though we probably cannot do much about this. Still, ugh!
These descriptors confuse CollecTor's reference checker which relies on
metrics-lib, making it think it's missing a whole lot of descriptors. I
wonder if we need to make it more robust and continue reading until it
finds the next valid descriptor.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20421#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tor-bugs
mailing list