[tor-bugs] #11630 [Tor bundles/installation]: Creating HTTPS-Everywhere's rules.sqlite is non-deterministic

[Tor Bug Tracker & Wiki]

#11630: Creating HTTPS-Everywhere's rules.sqlite is non-deterministic
------------------------------------------+----------------------
     Reporter:  gk                        |      Owner:  erinn
         Type:  defect                    |     Status:  reopened
     Priority:  normal                    |  Milestone:
    Component:  Tor bundles/installation  |    Version:
   Resolution:                            |   Keywords:
Actual Points:                            |  Parent ID:
       Points:                            |
------------------------------------------+----------------------

Comment (by gk):

 Replying to [comment:20 jsha]:
 > Sorry it took me so long to look at this. The first diff that jumps out
 at me is offset 28-31 decimal (0x16-0x19), whose value is 1424 in the
 second file and 1428 (dec) in the first file. Per the docs at
 https://www.sqlite.org/fileformat.html, that's the in-header database size
 in pages. You probably don't still have the two file around, but if you
 do: were they different sizes?

 I only still have the local one it seems which gives me:
 {{{
 -rwxr-xr-x  1 thomas thomas 1462272 Jun 11 10:26 rules_local.sqlite
 }}}
 You can easily compare that one to your own rules.sqlite if you check out
 the 3.5.1 tag, add
 {{{
 conn.execute("VACUUM")
 }}}
 before the `conn.close()`, make sure you don't have an old rules.sqlite
 file in your src/defaults directory and build https-e.

 > The other thing I noticed in this diff is that there is a large chunk of
 the rulesets that show up as a diff, where they are in a different order.
 Looking at the code, that makes sense. At the time of filing this bug, we
 used os.walk, which does not have an order guarantee
 (http://stackoverflow.com/a/5667552). Now we use glob.glob, which also
 does not have an order guarantee (http://stackoverflow.com/a/6774404). I
 will write a change to sort the files before loading them, which should
 hopefully make at least that part deterministic. I suspect it may have
 mysterious add-on effects that could fix the other diffs. Let's try it and
 see.

 Let me know when you are done and I'll give it a try.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/11630#comment:21>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online