[tor-bugs] #14859 [Onionoo]: Adapt Onionoo's parsing code for MaxMind's GeoLite2 City files to their new format

[Tor Bug Tracker & Wiki]

#14859: Adapt Onionoo's parsing code for MaxMind's GeoLite2 City files to their new
format
-------------------------+-----------------
     Reporter:  karsten  |      Owner:
         Type:  defect   |     Status:  new
     Priority:  normal   |  Milestone:
    Component:  Onionoo  |    Version:
   Resolution:           |   Keywords:
Actual Points:           |  Parent ID:
       Points:           |
-------------------------+-----------------

Comment (by karsten):

 Interesting!  I can reproduce this problem with reading a UTF-16 document
 and pretending it's UTF-8:

 {{{
 import java.io.BufferedReader;
 import java.io.BufferedWriter;
 import java.io.File;
 import java.io.FileInputStream;
 import java.io.FileOutputStream;
 import java.io.InputStreamReader;
 import java.io.OutputStreamWriter;
 import java.nio.charset.StandardCharsets;

 public class CharsetDecTest {

   public static void main(String[] args) throws Exception {
     String geo = "GeoLite2-City-Locations-en.csv";
     try {
       BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(
           new FileOutputStream(new File(geo)), "UTF-16"));
       bw.write("Hello world!\n");
       bw.close();
     } catch (Throwable t) {
       System.out.println("e0: " + t);
     }
     try {
       BufferedReader br1 = new BufferedReader(new InputStreamReader(
           new FileInputStream(new File(geo)), "UTF-8"));
       String line;
       while ((line = br1.readLine()) != null) {
         System.out.println(line);
       }
     } catch (Throwable t) {
       System.out.println("e1: " + t);
     }
     try {
       BufferedReader br2 = new BufferedReader(new InputStreamReader(
           new FileInputStream(new File(geo)),
           StandardCharsets.US_ASCII.newDecoder()));
       String line;
       while ((line = br2.readLine()) != null) {
         System.out.println(line);
       }
     } catch (Throwable t) {
       System.out.println("e2: " + t);
     }
   }
 }
 }}}

 Output:

 {{{
 ??

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/14859#comment:9>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online