[tor-bugs] #25575 [Internal Services/Tor Sysadmin Team]: Server space request (175 GB total) for hosting Tor Browser downloads

[Tor Bug Tracker & Wiki]

#25575: Server space request (175 GB total) for hosting Tor Browser downloads
-------------------------------------------------+-------------------------
 Reporter:  arthuredelstein                      |          Owner:  tpa
     Type:  project                              |         Status:
                                                 |  needs_information
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by arthuredelstein):

 Replying to [comment:3 weasel]:

 > {{{
 >  5.2M    aus1.torproject.org/
 >  42G     dist.torproject.org/torbrowser/
 >  28G     cdn.torproject.org/aus1/torbrowser/
 > }}}
 >
 > Do you have an estimate how those 175GB would be spread across the
 services?

 Great question. Unfortunately I was ignorant of the cdn.torproject.org
 part. The 175 GB applied to dist.torproject.org only.

 For cdn.torproject.org, I am making a similar calculation. Currently there
 are already 4 releases of Tor Browser mar files on disk. (I don't expect
 more than this.) But we do expect to double the number of locales, and add
 mobile. So that suggests a total of 28 GB * 2 * 1.2 = 67 GB. Adding a
 margin of safety, I would suggest then we could use a total of 80 GB on
 cdn.torproject.org.

 So to summarize, the request of total disk space (including space already
 in use):
 dist.torproject.org: 175 GB
 cdn.torproject.org: 80 GB

 Now, I should mention that boklm explained to me some details of the
 servers that I had not previously been aware of. One important detail he
 explained is that multiple copies of all files are stored on staticiforme.

 As far as I understand (gk and boklm, please correct me if I'm wrong)
 first the files intended for dist.torproject.org are staged in
 `/srv/dist-master.torproject.org`.
 Some of these (the mar files) are then hard-linked from
 `/srv/cdn-master.torproject.org/
 (Hard-links imply no extra space is needed.)

 Then the files from each of these master directories are rsync'd to
 `/srv/static.torproject.org/master/cdn.torproject.org-current-live`
 and
 `/srv/static.torproject.org/master/dist.torproject.org-current-live`
 respectively.

 So, because of this duplication on the disk, it seems we would need
 substantially more storage than I had previously understood. Namely, 175
 GB for the master and then 175 GB + 80 GB for the live directories.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25575#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online