[tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances
Tor Bug Tracker & Wiki
blackhole at torproject.org
Mon Oct 17 18:04:32 UTC 2016
#18910: distributing descriptors accross CollecTor instances
-------------------------------+---------------------------------
Reporter: iwakeh | Owner: iwakeh
Type: enhancement | Status: needs_review
Priority: High | Milestone: CollecTor 1.1.0
Component: Metrics/CollecTor | Version:
Severity: Normal | Resolution:
Keywords: ctip | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
-------------------------------+---------------------------------
Comment (by iwakeh):
Replying to [comment:46 karsten]:
I found the a good reference for the paths topic: it's in the
[https://gitweb.torproject.org/collector.git/tree/src/main/resources
/create-tarballs.sh#n68 create tars script]:
{{{
$OUTDIR/exit-lists/$YEARONE/$MONTHONE/
$OUTDIR/exit-lists/$YEARTWO/$MONTHTWO/
$OUTDIR/torperf/$YEARONE/$MONTHONE/
$OUTDIR/torperf/$YEARTWO/$MONTHTWO/
$OUTDIR/relay-descriptors/certs/
$OUTDIR/relay-descriptors/microdesc/$YEARONE/$MONTHONE
$OUTDIR/relay-descriptors/microdesc/$YEARTWO/$MONTHTWO
$OUTDIR/relay-descriptors/consensus/$YEARONE/$MONTHONE
$OUTDIR/relay-descriptors/consensus/$YEARTWO/$MONTHTWO
$OUTDIR/relay-descriptors/vote/$YEARONE/$MONTHONE/
$OUTDIR/relay-descriptors/vote/$YEARTWO/$MONTHTWO/
$OUTDIR/relay-descriptors/server-descriptor/$YEARONE/$MONTHONE/
$OUTDIR/relay-descriptors/server-descriptor/$YEARTWO/$MONTHTWO/
$OUTDIR/relay-descriptors/extra-info/$YEARONE/$MONTHONE/
$OUTDIR/relay-descriptors/extra-info/$YEARTWO/$MONTHTWO/
$OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/statuses/
$OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/statuses/
$OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/server-descriptors/
$OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/server-descriptors/
$OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/extra-infos/
$OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/extra-infos/
}}}
The above makes an inconsistency between relay descriptor and bridge
descriptor paths visible.
As the new structure from #20228, i.e. votes also grouped and all dates in
recent paths are derived from the acquisition time, will be in place soon,
I'd like to use the 'new way' already for the merge.
With these as background the test diff has some soon to be outdated
corrections (comments added after '//'):
>
> {{{
> diff --git
a/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
b/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
> index ad5ee6a..cdb90b8 100644
> ---
a/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
> +++
b/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
> @@ -58,15 +58,15 @@ public class SyncPersistenceTest {
> @Parameters
> public static Collection<Object[]> pathFilename() {
> return Arrays.asList(new Object[][] {
> - {"exit-lists/2016-10-05-19-06-17", // expected recent path //
acqu-date, which will be ok
> + {"exit-lists/2016-09-20-13-02-00", // expected recent path
> "exit-lists/2016/09/20/2016-09-20-13-02-00", // expected out
path
> "2016-09-20-13-02-00", // test-filename
> Integer.valueOf(1), // expected recent count of descs files
> Integer.valueOf(1)}, // expected output count of descs files
>
> {"relay-descriptors/microdescs/consensus-microdesc/"
> - + "2016-10-05-19-06-17-consensus-microdesc", // acqu-date,
which will be ok
> - "relay-descriptors/microdescs/2016/10/consensus-microdesc/"
> + + "2016-10-02-17-00-00-consensus-microdesc",
> + "relay-descriptors/microdesc/2016/10/consensus-microdesc/"
> + "02/2016-10-02-17-00-00-consensus-microdesc",
> "2016-10-02-17-00-00-consensus-microdesc",
> Integer.valueOf(1),
> @@ -74,7 +74,7 @@ public class SyncPersistenceTest {
>
> {"bridge-descriptors/server-descriptors/"
> + "2016-10-05-19-06-17-server-descriptors",
> - "bridge-descriptors/server-descriptor/2016/10/"
> + "bridge-descriptors/2016/10/server-descriptor/"
> + "A/8/A8A5509AD1393C8F36ABD2D8F0DE1BB751926872",
> "bridge-2016-10-02-16-09-00-server-descriptors",
> Integer.valueOf(1),
> @@ -88,28 +88,30 @@ public class SyncPersistenceTest {
> Integer.valueOf(1),
> Integer.valueOf(10)},
>
> - {"relay-descriptors/consensuses/2016-10-05-19-06-17-consensus",
// acqu-date, which will be ok
> + {"relay-descriptors/consensuses/2016-09-20-13-00-00-consensus",
> "relay-
descriptors/consensus/2016/09/20/2016-09-20-13-00-00-consensus",
> "2016-09-20-13-00-00-consensus",
> Integer.valueOf(1),
> Integer.valueOf(1)},
>
> {"bridge-descriptors/statuses/"
> - +
"20161005-190617-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1", // acqu-date,
which will be ok
> - "bridge-descriptors/statuses/2016/09/20/"
> + +
"20160920-063816-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1",
> + "bridge-descriptors/2016/09/statuses/20/"
> +
"20160920-063816-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1",
> "20160920-063816-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1",
> Integer.valueOf(1),
> Integer.valueOf(1)},
>
> {"relay-
descriptors/microdescs/micro/2016-10-05-19-06-17-micro",
> - "relay-descriptors/microdescs/2016/10/micro/f/b/"
> + "relay-descriptors/microdesc/2016/10/micro/f/b/"
> +
"fbf0c4cb8216d950d267866ea879880e69216bd0b46878ff0f51941ee4eca707",
> "2016-09-20-15-31-19-micro",
> Integer.valueOf(1),
> Integer.valueOf(5)},
>
> - {"relay-descriptors/votes/2016-10-05-19-06-17-votes", // acqu-
date, which will be ok
> + {"relay-descriptors/votes/2016/10/01/2016-10-01-16-00-00-vote"
> + + "-0232AF901C31A04EE9848595AF9BB7620D4C5B2E"
> + + "-FEE63B4AB7CE5A6BDD09E9A5C4F01BD61EB7E4F1",
> "relay-descriptors/vote/2016/10/01/2016-10-01-16-00-00-vote"
> + "-0232AF901C31A04EE9848595AF9BB7620D4C5B2E"
> + "-FEE63B4AB7CE5A6BDD09E9A5C4F01BD61EB7E4F1",
> @@ -118,7 +120,9 @@ public class SyncPersistenceTest {
> Integer.valueOf(1),
> Integer.valueOf(1)},
>
> - {"relay-descriptors/votes/2016-10-05-19-06-17-votes", // acqu-
date, which will be ok
> + {"relay-descriptors/votes/2016/09/20/2016-09-20-13-00-00-vote-"
> + + "49015F787433103580E3B66A1707A00E60F2D15B"
> + + "-60ADC6BEC262AE921A1037D54C8A3976367DBE87",
> "relay-descriptors/vote/2016/09/20/2016-09-20-13-00-00-vote-"
> + "49015F787433103580E3B66A1707A00E60F2D15B"
> + "-60ADC6BEC262AE921A1037D54C8A3976367DBE87",
> @@ -128,7 +132,7 @@ public class SyncPersistenceTest {
> Integer.valueOf(1)},
>
> {"bridge-descriptors/extra-infos/2016-10-05-19-06-17-extra-
infos",
> - "bridge-descriptors/extra-info/2016/10/9/F/"
> + "bridge-descriptors/2016/10/extra-info/9/F/"
> + "9F88A7C2ABE6665D204137BA8C2661D42E7C2829",
> "bridge-2016-10-02-08-09-00-extra-infos",
> Integer.valueOf(1),
> }}}
When changing the structure now shouldn't relays and bridges be stored in
similar ways? I think the current difference (in 'out') caused some of
the confusion here.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/18910#comment:49>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tor-bugs
mailing list