[tor-dev] Scaling tor for a global population

[Sebastian Hahn]

On 28 Sep 2014, at 02:12, Tom Ritter <tom at ritter.vg> wrote:
> why not also change the consensus
> and related document formats to be something more efficient than ASCII
> text?  Taking the latest consensus and doing some rough estimates, I
> found the following:
> 
> Original consensus, xz-ed: 407K
> Change flags to uint16: ~399K
> +Removing names: 363K
> +Compressing IPv6 to 16Bytes + 4 Bytes - 360K
> +Compressing IPv4 to 4 Bytes + 4Bytes + 4bytes - 315K
> +Compressing the Datetime to 4 bytes - 291K
> +Compressing the Version string to 4bytes - 288K
> +Replacing reject 1-65K to a single byte - 287K
> +Replacing Bandwidth=# with a 4 byte - 273K
> 
> These numbers are optimistic - you won't see quite this much gain, but
> if I'm understanding you correctly that the consensus is painful, it
> seems like you could save at least 50K-70K out of 400K with relatively
> straightforward changes.

This analysis doesn't make much sense, I'm afraid. We use compression
on the wire, so repeating flags as human-readable strings has a much
lower overhead than you estimate, for example. Re-doing your estimates
with actually compressed consensuses might make sense, but probably
you'll see a lot less value.

Cheers
Sebastian