[tor-bugs] #14780 [Onionoo]: very large file

[Tor Bug Tracker & Wiki]

#14780: very large file
-------------------------+-----------------
     Reporter:  iwakeh   |      Owner:
         Type:  defect   |     Status:  new
     Priority:  normal   |  Milestone:
    Component:  Onionoo  |    Version:
   Resolution:           |   Keywords:
Actual Points:           |  Parent ID:
       Points:           |
-------------------------+-----------------

Comment (by karsten):

 Ah, that looks exactly like the bug I fixed in 361c56c.

 Here's what I think happens: whenever that relay's details status file is
 read and rewritten, the UTF-8 characters in the country name double in
 size.  That's a problem of the way we read files (using `FileInputStream`)
 and write files (using `FileWriter`).  I didn't fix that bug, though.  I
 just made sure it doesn't get triggered anymore.

 What I fixed in 361c56c was that I escaped UTF-8 characters in details
 status files.  That way, even if a file is read and rewritten it doesn't
 change in size, because there are no UTF-8 characters.

 What this patch doesn't fix, is if there are already UTF-8 characters in
 details status files.  What I did (on the Onionoo mirror that also had
 this problem) was manually remove those JSON fields (there were a few
 dozen of them).

 By the way, this case is rare, because in most cases there will be new
 GeoIP information for details status files.  But in this case, the relay
 was not listed as running for a while (since January 15), but it's still
 publishing descriptors (last on January 25).  And we don't resolve non-
 running relays using the GeoIP database.

 So, you could either make sure you're running 361c56c and edit the JSON
 file to remove the country_name field, or you could try to fix that other
 bug where reading and rewriting a file changes its size.

 I'm attaching the Java class that I used to diagnose the problem.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/14780#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online