[tor-bugs] #3404 [EFF-HTTPS Everywhere]: Ship rulesets in a more efficient form

[Tor Bug Tracker & Wiki]

#3404: Ship rulesets in a more efficient form
----------------------------------+-----------------------------------------
 Reporter:  pde                   |          Owner:  pde
     Type:  enhancement           |         Status:  new
 Priority:  minor                 |      Milestone:     
Component:  EFF-HTTPS Everywhere  |        Version:     
 Keywords:                        |         Parent:     
   Points:                        |   Actualpoints:     
----------------------------------+-----------------------------------------
 In the current development versions, the ~1,000 ruleset files take up 4+
 MB of disk space on a filesystem with a 4kB blocksize, and more on some
 systems.  Reading them adds a significant amount of time to Firefox's
 startup time.  The extension's .xpi file is currently about 330 kB.

 If/when we wish to, it should be possible to do quite a bit better.
 Rolling all the rulesets into a single giant .xml file at "build" time
 would fix the filesystem block allocation bloat and reduce the IO load for
 parsing them.  We could also strip the comments at build time.

 If there are compression/decrompression hooks in Firefox, we could also
 store this blob on disk compressed, which would make FF load faster.

 Lastly, the ~350kB .xpi installer is currently compressed, but .zip is
 actually pretty terrible for our purposes.  Here is the size of the
 current master ruleset collection as compressed with the default settings
 for zip, gzip, and lzma:

 /tmp/https-everywhere/src/chrome/content$ ls -l rules.*
 -rw-r--r-- 1 pde pde  79128 Jun 15 08:01 rules.tar.lzma
 -rw-r--r-- 1 pde pde  97829 Jun 15 08:00 rules.tgz
 -rw-r--r-- 1 pde pde 325722 Jun 15 08:00 rules.zip

 So if we want a 3-4x smaller .xpi file, all we need to do is work out how
 to read a gzipped or lzma'd file from within the extension.

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/3404>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online