[tor-bugs] #31690 [Internal Services/Service - trac]: study trac.torproject.org archival possibilities

[Tor Bug Tracker & Wiki]

#31690: study trac.torproject.org archival possibilities
--------------------------------------------------+--------------------
     Reporter:  anarcat                           |      Owner:  qbi
         Type:  project                           |     Status:  new
     Priority:  Medium                            |  Milestone:
    Component:  Internal Services/Service - trac  |    Version:
     Severity:  Normal                            |   Keywords:
Actual Points:                                    |  Parent ID:  #30857
       Points:                                    |   Reviewer:
      Sponsor:                                    |
--------------------------------------------------+--------------------
 this is a split out of #30857 to discuss specifically the question of
 if/how to archive trac.torproject.org.

 As mentioned in that ticket, there are a few options on how to deal with
 trac, provided we have another system we want to use:

  1. '''the golden redirect set''': every migrated ticket and wiki page has
 a corresponding ticket/wiki page in GitLab and a gigantic set of
 redirection rules makes sure they are mapped correctly. probably
 impractical, but solves the maintenance problem possibly forever.

  2. '''read-only Trac''': user creation is disabled and existing users are
 locked from making any change to the site. only a temporary or
 intermediate measure.

  3. '''fossilization''': Trac is turned into a static HTML site that can
 be mirrored like any other site. can be a long term solution and a good
 compromise with a possibly impossible to design and therefore failing
 (because incomplete) set of redirection rules.

  4. '''destruction''': we hate the web and pretend link rot is not a
 problem and just get rid of the old site, assuming everything is migrated
 and people will find their stuff eventually. probably not an option.

 == Archive team work

 With my archive team hat, I was able to coordinate a first archival of the
 website during the summer of 2019, as documented in #30857. This is an
 attempt at doing "3. '''fossilization'''".

 All those jobs end up populating the wayback machine at web.archive.org,
 but are also available as WARC files, an archival format for web pages.

 A first archival of all tickets up to #30856 has been performed here:

 https://archive.fart.website/archivebot/viewer/job/5vytc

 It's about 600MB of compressed HTML (more or less).

 Then a full archival job of the entire site was performed here:

 https://archive.fart.website/archivebot/viewer/job/bpu6j

 It created about 10GB of WARC files, crawled over 730,000 links (including
 external sites linked from Trac) and 105.34GiB of data. It took over 5
 days:

 {{{
 2019-06-17 01:49:02,514 - wpull.application.tasks.stats - INFO - Duration:
 5 days, 7:32:55. Speed: 0.0 B/s.
 2019-06-17 01:49:02,514 - wpull.application.tasks.stats - INFO -
 Downloaded: 732488 files, 105.4 GiB.
 }}}

 == Other statistics

 Archiving the server itself means dealing with:

  * ~1GB of attachments
  * 4GB PostgreSQL database

 The actual server uses around 25GB of disk space because of random junk
 here and there but that's the very minimum it can be trimmed down to.
 naturally, we can keep *that* data forever, the problem is keeping the app
 running on top of that... That would be some incarnation of "4.
 '''destruction'''".

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/31690>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online