[tor-commits] [collector/master] Split up bridge descriptor tarballs into one per type.

karsten at torproject.org karsten at torproject.org
Mon Sep 19 12:25:25 UTC 2016 

commit f608c94c7f731241bf7ee8e627ca1da98c23d858
Author: Karsten Loesing <karsten.loesing at gmx.net>
Date:   Sun Sep 18 16:09:52 2016 +0200

    Split up bridge descriptor tarballs into one per type.
---
 CHANGELOG.md                          |  4 ++++
 src/main/resources/create-tarballs.sh | 26 ++++++++++++++++++++------
 src/main/webapp/index.html            | 19 +++++++------------
 3 files changed, 31 insertions(+), 18 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index e17abad..99b2918 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,10 @@
      minutes to finish properly, rather than killing them immediately.
    - Replace TCP ports with hashes in @type bridge-network-status 1.1
      and @type bridge-server-descriptor 1.2.
+   - Split up bridge descriptor tarballs into one tarball per month
+     and descriptor type: bridge-statuses-YYYY-MM.tar.xz,
+     bridge-server-descriptors-YYYY-MM.tar.xz and
+     bridge-extra-infos-YYYY-MM.tar.xz.
 
  * Minor changes
    - Remove quotes around base URL in index.json.
diff --git a/src/main/resources/create-tarballs.sh b/src/main/resources/create-tarballs.sh
index cc75219..4f2c6eb 100755
--- a/src/main/resources/create-tarballs.sh
+++ b/src/main/resources/create-tarballs.sh
@@ -56,8 +56,12 @@ TARBALLS=(
   server-descriptors-$YEARTWO-$MONTHTWO
   extra-infos-$YEARONE-$MONTHONE
   extra-infos-$YEARTWO-$MONTHTWO
-  bridge-descriptors-$YEARONE-$MONTHONE
-  bridge-descriptors-$YEARTWO-$MONTHTWO
+  bridge-statuses-$YEARONE-$MONTHONE
+  bridge-statuses-$YEARTWO-$MONTHTWO
+  bridge-server-descriptors-$YEARONE-$MONTHONE
+  bridge-server-descriptors-$YEARTWO-$MONTHTWO
+  bridge-extra-infos-$YEARONE-$MONTHONE
+  bridge-extra-infos-$YEARTWO-$MONTHTWO
 )
 TARBALLS=($(printf "%s\n" "${TARBALLS[@]}" | uniq))
 
@@ -77,8 +81,12 @@ DIRECTORIES=(
   $OUTDIR/relay-descriptors/server-descriptor/$YEARTWO/$MONTHTWO/
   $OUTDIR/relay-descriptors/extra-info/$YEARONE/$MONTHONE/
   $OUTDIR/relay-descriptors/extra-info/$YEARTWO/$MONTHTWO/
-  $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/
-  $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/
+  $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/statuses/
+  $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/statuses/
+  $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/server-descriptors/
+  $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/server-descriptors/
+  $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/extra-infos/
+  $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/extra-infos/
 )
 DIRECTORIES=($(printf "%s\n" "${DIRECTORIES[@]}" | uniq))
 
@@ -109,8 +117,14 @@ cd $CURRENTPATH
 
 echo `date` "Finished tarball creation.  Starting symlink-update ..."
 
-mkdir -p $ARCHIVEDIR/bridge-descriptors/
-ln -f -s -t $ARCHIVEDIR/bridge-descriptors/ $TARBALLTARGETDIR/bridge-descriptors-20??-??.tar.xz
+mkdir -p $ARCHIVEDIR/bridge-descriptors/statuses/
+ln -f -s -t $ARCHIVEDIR/bridge-descriptors/statuses/ $TARBALLTARGETDIR/bridge-statuses-20??-??.tar.xz
+
+mkdir -p $ARCHIVEDIR/bridge-descriptors/server-descriptors/
+ln -f -s -t $ARCHIVEDIR/bridge-descriptors/server-descriptors/ $TARBALLTARGETDIR/bridge-server-descriptors-20??-??.tar.xz
+
+mkdir -p $ARCHIVEDIR/bridge-descriptors/extra-infos/
+ln -f -s -t $ARCHIVEDIR/bridge-descriptors/extra-infos/ $TARBALLTARGETDIR/bridge-extra-infos-20??-??.tar.xz
 
 mkdir -p $ARCHIVEDIR/bridge-pool-assignments/
 ln -f -s -t $ARCHIVEDIR/bridge-pool-assignments/ $TARBALLTARGETDIR/bridge-pool-assignments-20??-??.tar.xz
diff --git a/src/main/webapp/index.html b/src/main/webapp/index.html
index bbc70f2..7ed9145 100644
--- a/src/main/webapp/index.html
+++ b/src/main/webapp/index.html
@@ -168,21 +168,21 @@
   <td>Bridge Network Statuses</td>
   <td><tt>@type bridge-network-status 1.1</tt></td>
   <td><a href="recent/bridge-descriptors/statuses/" class="btn btn-primary btn-xs">recent</a></td>
-  <td><a href="archive/bridge-descriptors/" class="btn btn-primary btn-xs">archive</a></td>
+  <td><a href="archive/bridge-descriptors/statuses/" class="btn btn-primary btn-xs">archive</a></td>
   <td><a href="#type-bridge-network-status" class="btn btn-default btn-xs">format</a></td>
 </tr>
 <tr>
   <td>Bridge Server Descriptors</td>
   <td><tt>@type bridge-server-descriptor 1.2</tt></td>
   <td><a href="recent/bridge-descriptors/server-descriptors/" class="btn btn-primary btn-xs">recent</a></td>
-  <td><a href="archive/bridge-descriptors/" class="btn btn-primary btn-xs">archive</a></td>
+  <td><a href="archive/bridge-descriptors/server-descriptors/" class="btn btn-primary btn-xs">archive</a></td>
   <td><a href="#type-bridge-server-descriptor" class="btn btn-default btn-xs">format</a></td>
 </tr>
 <tr>
   <td>Bridge Extra-info Descriptors</td>
   <td><tt>@type bridge-extra-info 1.3</tt></td>
   <td><a href="recent/bridge-descriptors/extra-infos/" class="btn btn-primary btn-xs">recent</a></td>
-  <td><a href="archive/bridge-descriptors/" class="btn btn-primary btn-xs">archive</a></td>
+  <td><a href="archive/bridge-descriptors/extra-infos/" class="btn btn-primary btn-xs">archive</a></td>
   <td><a href="#type-bridge-extra-info" class="btn btn-default btn-xs">format</a></td>
 </tr>
 <tr>
@@ -500,8 +500,6 @@ descriptor including all signatures.</li>
 Sanitized bridge network statuses are similar to version 2 relay network
 statuses, but with only a <tt>published</tt> line in the header and
 without any lines in the footer.
-The bridge descriptor archive tarballs contain all bridge
-descriptors of a given month, not just network statuses.
 The format has changed over time to accomodate changes to the sanitizing
 process, with earlier versions being:
 </p>
@@ -523,10 +521,8 @@ ports.</li>
 <p>
 Bridge server descriptors follow the same format as relay server
 descriptors, except for the sanitizing steps described above.
-The bridge descriptor archive tarballs contain all bridge
-descriptors of a given month, not just server descriptors.
-These tarballs contain one descriptor per file, whereas recently published bridge descriptor
-files
+The bridge server descriptor archive tarballs contain one descriptor per
+file, whereas recently published bridge server descriptor files
 contain all descriptors collected in an hour concatenated into a single
 file to reduce the number of files.
 The format has changed over time to accomodate changes to the sanitizing
@@ -579,9 +575,8 @@ published by bridges using an Ed25519 master key.</li>
 </ul>
 
 <p>
-The bridge descriptor archive tarballs contain all bridge
-descriptors of a given month, not just extra-info descriptors.
-These tarballs contain one descriptor per file, whereas recently published bridge descriptor
+The bridge extra-info descriptor archive tarballs contain one descriptor
+per file, whereas recently published bridge extra-info descriptor
 files contain all descriptors collected in an hour concatenated into a single
 file to reduce the number of files.
 </p>

More information about the tor-commits mailing list