[tor-bugs] #14859 [Onionoo]: Adapt Onionoo's parsing code for MaxMind's GeoLite2 City files to their new format

[Tor Bug Tracker & Wiki]

#14859: Adapt Onionoo's parsing code for MaxMind's GeoLite2 City files to their new
format
-------------------------+--------------------------
     Reporter:  karsten  |      Owner:
         Type:  defect   |     Status:  needs_review
     Priority:  normal   |  Milestone:
    Component:  Onionoo  |    Version:
   Resolution:           |   Keywords:
Actual Points:           |  Parent ID:
       Points:           |
-------------------------+--------------------------

Comment (by karsten):

 Nope, it doesn't stop.  Here's another related issue: even if AS names are
 correctly parsed from the database file, any contained non-ASCII
 characters are written to the details documents file as `\\u00F2`.  Which
 is okay.  However, we don't undo that double-escaping and return `\\u00F2`
 to HTTP clients which is six valid characters but not the single escaped
 UTF-8 character we're supposed to return.

 This bug is not limited to AS names, but city names and other fields have
 the same problem.

 Note that including the fields parameter fixes this problem, because it
 leads to rewriting the document and correctly replacing `\\u` with `\u`.
 Compare these two queries:

 https://onionoo.torproject.org/details?search=C13E70D4

 {{{
 ..."city_name":"S\\u00E3o Paulo",...,"as_name":"Servi\\uFFFDos de
 Comunica\\uFFFD\\uFFFDo S.A."...
 }}}

 https://onionoo.torproject.org/details?search=C13E70D4&fields=city_name,as_name

 {{{
 ..."city_name":"S\u00E3o Paulo","as_name":"Servi\uFFFDos de
 Comunica\uFFFD\uFFFDo S.A."}...
 }}}

 Pushed a fix as another commit to my task-14859-2 branch.  Please review.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/14859#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online