[tor-dev] Scaling tor for a global population

[Tom Ritter]

On 26 September 2014 22:28, Mike Perry <mikeperry at torproject.org> wrote:
> That's basically what I'm arguing: We can increase the capacity of the
> network by reducing directory waste but adding more high capacity relays
> to replace this waste, causing the overall directory to be the same
> size, but with more capacity.

I'm sure that diffs will make a huge difference, but if you're
focusing on the directory documents why not also change the consensus
and related document formats to be something more efficient than ASCII
text?  Taking the latest consensus and doing some rough estimates, I
found the following:

Original consensus, xz-ed: 407K
Change flags to uint16: ~399K
+Removing names: 363K
+Compressing IPv6 to 16Bytes + 4 Bytes - 360K
+Compressing IPv4 to 4 Bytes + 4Bytes + 4bytes - 315K
+Compressing the Datetime to 4 bytes - 291K
+Compressing the Version string to 4bytes - 288K
+Replacing reject 1-65K to a single byte - 287K
+Replacing Bandwidth=# with a 4 byte - 273K

These numbers are optimistic - you won't see quite this much gain, but
if I'm understanding you correctly that the consensus is painful, it
seems like you could save at least 50K-70K out of 400K with relatively
straightforward changes.

-tom