[tor-bugs] #20234 [Metrics/Metrics website]: add CollecTor's file-structure protocol to Metrics-web (was: Define CollecTor's file-structure protocol)

[Tor Bug Tracker & Wiki]

#20234: add CollecTor's file-structure protocol to Metrics-web
-------------------------------------+-----------------------------------
 Reporter:  karsten                  |          Owner:  iwakeh
     Type:  enhancement              |         Status:  needs_information
 Priority:  Low                      |      Milestone:
Component:  Metrics/Metrics website  |        Version:
 Severity:  Normal                   |     Resolution:
 Keywords:                           |  Actual Points:
Parent ID:                           |         Points:
 Reviewer:                           |        Sponsor:
-------------------------------------+-----------------------------------
Changes (by iwakeh):

 * priority:  High => Low
 * component:  Metrics/CollecTor => Metrics/Metrics website


Old description:

> It looks like the CollecTor mirror serves the `index.json` file at a
> different URL (`/index/index.json`) than the main CollecTor instance
> (`/index.json`).  In theory, we could agree on a common place for that
> file and all other files and consider that part of the "protocol".  And
> I'm happy to consider changing paths on the main CollecTor instance if
> different paths make more sense.
>
> Here are the current directories and files on the main CollecTor
> instance:
>
> {{{
> /                # start page with all the content for humans
> /index.html      # same as /
> /css/            # web stuff
> /images/         # web stuff
> /header.html     # used to style directory listings
> /footer.html     # used to style directory listings
> /formats.html    # not used anymore, could go away if we wanted
> /archive/        # archived descriptors
> /recent/         # recent descriptors
> /index.json      # JSON file with all files in archive/ and recent/
> /index.json.bz2  # same as /index.json, but compressed
> /index.json.gz   # same as /index.json, but compressed
> /index.json.xz   # same as /index.json, but compressed
> }}}
>
> I guess my original intention to put `index.json` directly in the root
> directory was to place it next to `index.html` and in the parent
> directory of `archive/` and `recent/` which are further described by
> `index.json`.  But I guess your motivation for putting it in `/index/`
> was to avoid cluttering the root directory any further, right?
>
> What do you think, should we unify this and keep it unified?  And if yes,
> who moves their `index.json` files? ;)  I don't think they're used by
> anything yet, so we're unlikely to break anything.  Again, happy to move
> mine if this makes more sense.  Maybe we can briefly think of other
> files/directories we might be adding in the near future?

New description:

 Transform into appropriate format and also add path descriptions for
 webstats.


 Old description: It looks like the CollecTor mirror serves the
 `index.json` file at a different URL (`/index/index.json`) than the main
 CollecTor instance (`/index.json`).  In theory, we could agree on a common
 place for that file and all other files and consider that part of the
 "protocol".  And I'm happy to consider changing paths on the main
 CollecTor instance if different paths make more sense.

 Here are the current directories and files on the main CollecTor instance:

 {{{
 /                # start page with all the content for humans
 /index.html      # same as /
 /css/            # web stuff
 /images/         # web stuff
 /header.html     # used to style directory listings
 /footer.html     # used to style directory listings
 /formats.html    # not used anymore, could go away if we wanted
 /archive/        # archived descriptors
 /recent/         # recent descriptors
 /index.json      # JSON file with all files in archive/ and recent/
 /index.json.bz2  # same as /index.json, but compressed
 /index.json.gz   # same as /index.json, but compressed
 /index.json.xz   # same as /index.json, but compressed
 }}}

 I guess my original intention to put `index.json` directly in the root
 directory was to place it next to `index.html` and in the parent directory
 of `archive/` and `recent/` which are further described by `index.json`.
 But I guess your motivation for putting it in `/index/` was to avoid
 cluttering the root directory any further, right?

 What do you think, should we unify this and keep it unified?  And if yes,
 who moves their `index.json` files? ;)  I don't think they're used by
 anything yet, so we're unlikely to break anything.  Again, happy to move
 mine if this makes more sense.  Maybe we can briefly think of other
 files/directories we might be adding in the near future?

--

Comment:

 Example paths for webstat webserver logs:
 {{{
  recent/webstats/metrics.torproject.org-meronense.torproject.org-
 access.log-20170905.xz
  archive/webstats/metrics.torproject.org/2017/09/05/metrics.torproject
 .org-meronense.torproject.org-access.log-20170905.xz
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20234#comment:18>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online