[tor-bugs] #19791 [Metrics/metrics-lib]: Use CollecTor's index.json for download; adapt current download to use new date format

[Tor Bug Tracker & Wiki]

#19791: Use CollecTor's index.json for download; adapt current download to use new
date format
---------------------------------+-----------------------------------
 Reporter:  karsten              |          Owner:  iwakeh
     Type:  defect               |         Status:  needs_revision
 Priority:  High                 |      Milestone:  metrics-lib 1.4.0
Component:  Metrics/metrics-lib  |        Version:
 Severity:  Normal               |     Resolution:
 Keywords:                       |  Actual Points:
Parent ID:                       |         Points:
 Reviewer:                       |        Sponsor:
---------------------------------+-----------------------------------
Changes (by karsten):

 * status:  merge_ready => needs_revision


Comment:

 Bad news for the 1.4.0 release!  I ran into a bug that I couldn't resolve
 easily.  Or rather, I shouldn't fix this bug quickly and then put out the
 1.4.0 release, because that would make a 1.4.1 release quite likely.
 Let's postpone that 1.4.0 release until we're sure what the fix is.

 Please look at [https://gitweb.torproject.org/user/karsten/metrics-
 lib.git/log/?h=task-19791-2 my task-19791-2 branch] which contains your
 #19791 commit and all subsequent tweaks squashed together, plus some
 tweaks to the change log.  That branch also includes the following fix
 that I made after realizing that `DescriptorIndexCollector` doesn't
 download actual descriptor files but rather the `index.html` file:

 {{{
 diff --git
 a/src/main/java/org/torproject/descriptor/index/DescriptorIndexCollector.java
 b/src/main/java/org/torproject/descriptor/index/DescriptorIndexCollector.java
 index 5d6bcdd..4f62be6 100644
 ---
 a/src/main/java/org/torproject/descriptor/index/DescriptorIndexCollector.java
 +++
 b/src/main/java/org/torproject/descriptor/index/DescriptorIndexCollector.java
 @@ -83,7 +83,8 @@ public class DescriptorIndexCollector implements
 DescriptorCollector {
        }
        File destinationFile = new File(filepath, filename);
        File tempDestinationFile = new File(filepath, "." + filename);
 -      try (InputStream is = new URL(baseUrl).openStream()) {
 +      try (InputStream is = new URL(baseUrl + "/" + filepathname)
 +          .openStream()) {
          Files.copy(is, tempDestinationFile.toPath());
          if (tempDestinationFile.length() == entry.getValue().size) {
            tempDestinationFile.renameTo(destinationFile);
 }}}

 However, that change makes the tests fail, and that makes me less
 optimistic that the change was correct, despite the fact that it's working
 fine in a quickly written test application.  I think you should take a
 look.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/19791#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online