[tor-bugs] #8799 [Metrics/CollecTor]: collector's relay descriptor downloads: avoid httpurl-connection (was: Add timeouts to collector's relay descriptor downloads)

[Tor Bug Tracker & Wiki]

#8799: collector's relay descriptor downloads: avoid httpurl-connection
-------------------------------+---------------------------------
 Reporter:  karsten            |          Owner:  iwakeh
     Type:  enhancement        |         Status:  assigned
 Priority:  High               |      Milestone:  CollecTor 1.2.0
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:  ctip               |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+---------------------------------
Changes (by iwakeh):

 * priority:  Medium => High


Old description:

> In November 2012, tor26 was very slow in responding to metrics-db, which
> made metrics-db exceed its 30-minute schedule.  This happened again on
> March 2, 2013.  The workaround was to avoid tor26 and other slow
> directories and download descriptors only from the other directory
> authorities.  metrics-db should use a timeout similar to how DocTor does.

New description:

 In November 2012, tor26 was very slow in responding to metrics-db, which
 made metrics-db exceed its 30-minute schedule.  This happened again on
 March 2, 2013.  The workaround was to avoid tor26 and other slow
 directories and download descriptors only from the other directory
 authorities.  metrics-db should use a timeout similar to how DocTor does.

 This should avoid httpurl-connection as suggested in #20323 for metrics-
 lib.

--

Comment:

 Moved from #20323 (comment 3, karsten):

 It might be that this issue killed the relaydescs module 1.5 days ago by
 blocking the downloader indefinitely:

 {{{
 2016-10-30 23:05:00,000 INFO o.t.c.c.CollecTorMain:54 Starting relaydescs
 module of CollecTor.
 2016-10-30 23:05:20,823 INFO o.t.c.r.CachedRelayDescriptorReader:255
 Finished importing relay descriptors from local Tor data directories:
 cached-consensus: 2016-10-30 23:00:00
 cached-descriptors: parsed 0, skipped 24471 server descriptors
 cached-descriptors.new: parsed 753, skipped 6024 server descriptors
 cached-extrainfo: parsed 0, skipped 24508 extra-info descriptors
 cached-extrainfo.new: parsed 745, skipped 1463 extra-info descriptors
 v3-status-votes: parsed 8, skipped 0 votes
 2016-10-30 23:06:00,001 INFO o.t.c.c.CollecTorMain:54 Starting updateindex
 module of CollecTor.
 }}}

 There should be a line like this (copied over from the previous run), but
 there wasn't:

 {{{
 2016-10-30 22:35:19,933 INFO o.t.c.r.RelayDescriptorDownloader:1067
 Finished downloading relay descriptors from the directory authorities.
 }}}

 Raising priority to high.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/8799#comment:7>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online