GeoIP database comparison

[Karsten Loesing]

On 5/6/10 11:49 AM, Roger Dingledine wrote:
> On Mon, May 03, 2010 at 06:40:13PM +0200, Martin Mulazzani wrote:
>> TorStatus is already using the Maxmind GeoLite database.
> 
> I've been wondering for a while why
> http://trunk.torstatus.kgprog.com/index.php
> has so many relays listed as Namibia. They're pretty clearly not in
> Namibia.

You're right, Torstatus is wrong.

I just resolved the IP addresses from the torstatus output at
http://trunk.torstatus.kgprog.com/query_export.php/Tor_query_EXPORT.csv
using Maxmind's free database from May 1. Below you'll find the lines
where Torstatus and Maxmind disagree.

Random thought: does Torstatus use "NA" as abbreviation for "not
available" when resolving IP addresses, but as "Namibia" when choosing a
flag to display?

Also, how recent is the GeoIP database that Torstatus, especially
trunk.torstatus.kgprog.com, uses?

At the end of this mail, you'll also find the Java code to generate this
output. Put the geoip file from current Tor master and the .csv as
stated above in the same directory, compile, run.

Best,
--Karsten


First column = Maxmind
Second column = Torstatus
Third column = nickname

US NA 0x41414141
DE NA 0x42FF
US NA 107167
RU NA a1s2ew4wfedjhc6je8
DE NA agitator
CA NA andromeda
CA NA anona
US NA avarner
PL NA Azhar
DE NA b215c035618f047d
SK NA backslash777
CA US BarkerJrNet
SK NA BecherovHomeunix
DE NA bingsFreeNet
DE NA bmwanon
CA NA buttercup
BY NA ByHomenet
NL NA canon
US NA canoworms
NL NA carpetbagger
DE NA Cerritus
CA NA chuckthecanuck
IE NA cupboard
US NA customer679498
US NA dametenshi
DE NA DerAufbruch2
GB NA deusexmachina
GB FR devasdfasdf
DE NA Diega
US NA DigitalFreedom
DE FR Dinosaur
FR NA djiins
UA NA DLag
SE NA DVBT
GB NA DynoTor
DE FR Edmontosaurus
DE NA Enigma
GB NA F00DD00D
DE NA ferrot04st
RU NA FFA
RU FR Fisto
DE NA flunkey
DE NA Freedom
SE FR frogbeard
DE NA gartentor
US NA greyunknown
RU NA GuyRitchie
UA NA happiness
CA NA harman2
RU NA HazarD
RU NA HellyMotion
DE IL herecomesthesun
RU NA hometoorspbru
US NA hurrdurr
DE NA HWR
RU NA IDDQD
TH NA igor1977
US NA ihas5cat
NL NA ItsHiddenYa
DE IR jefOlewkia
US NA jetddna
US NA jujubee
DE NA justanothernode
DE NA kaon
US NA knegg
GB NA kuzlatko
US NA lapiste
GB NA lego
CA US Link2VoIP
DE NA LittleMao
DE NA ljsilver
US NA lot49
GB NA MaoUK2
FR NA matterhorn
DE NA megaantisanitary1
TW AU MgeniUser
DE NA MicsInTrees
DE NA Mischmaschine
DE NA mullbinde3
DE NA nerdsurf
US NA networkb289be5
DE NA netzfrei2
GB IT nixgeek
US NA nockodotnet
GB NA node13
LK NA NSTHome
US NA oldthink
IL NA OnionFox
DE GB OrionTorNode
HK NA pangu
CA US Pasquino
SE NA PPrivComSweden
GB NA PPrivComUK4
CZ NA PwNibulikvps83
DE NA qeki0i7g3kd8tynseg0
RU LU QLEnDGKmay
DE NA raf
US NA RankaLee
UA NA ReactorRelay
US NA revo
TH NA rsquarersharp
US NA SamuraiPizzaCat
DE NA SedanUK
FR NA servicePublic
CA US Shaft
MN NA Shaman0
DE NA shawnthesheep
DE NA shiven
US NA singulicity
DE NA sork
ID NA speekfree
US NA srv1esctoulous
US NA srv3
US NA srv4
US NA srv5
US NA srv6
US NA stegosaurus
DE NA Suga23
US NA SuperDave
SE NA SwedishBikiniTeam
SE NA swetzsupertor
US NA taoppv
TR GB TinyTurtle
FR NA ToileLibre
DE IR Toni
DE NA TOR2fm1
GB DK toroftheworld
DE NA torschlusspanik2
RO NA trafficlight
GB NA triczlinode
TR GB umbrella
US NA underdonk01
US NA underdonk02
US NA Unnamed
AL NA Unnamed
AT SE Unnamed
DE NA Unnamed
US NA Unnamed
IR NA Unnamed
CA NA Unnamed
DE NA Unnamed
IR NA Unnamed
KR NA Unnamed
JP AU Unnamed
JP AU Unnamed
RU NA uran
GR NA Vagabond2
GB NA vee
CA NA Waldo
DE NA wg362
US NA wingedgods
NL NA worldstr989418
NL NA worldstre0f571
US NA ydobonobody
DE NA zermanes
DE NA Zwiebelschale3



import java.io.*;
import java.util.*;
public class ResolveRelayCountries {
  public static void main(String[] args) throws Exception {
    File geoipFile = new File("geoip");
    File torstatusOutput = new File("Tor_query_EXPORT.csv");
    BufferedReader br = new BufferedReader(new FileReader(geoipFile));
    String line = null;
    SortedMap<Long, String> geoipLines = new TreeMap<Long, String>();
    while ((line = br.readLine()) != null) {
      if (line.startsWith("#")) {
        continue;
      }
      geoipLines.put(Long.parseLong(line.split(",")[0]), line);
    }
    br.close();
    br = new BufferedReader(new FileReader(torstatusOutput));
    br.readLine();
    while ((line = br.readLine()) != null) {
      String nickname = line.split(",")[0];
      String reportedCountryCode = line.split(",")[1];
      String ip = line.split(",")[4];
      String[] octets = ip.split("\\.");
      long ipNum = Long.parseLong(octets[0]) * 256L * 256L * 256L
          + Long.parseLong(octets[1]) * 256L * 256L
          + Long.parseLong(octets[2]) * 256L
          + Long.parseLong(octets[3]);
      SortedMap<Long, String> subMap = geoipLines.headMap(ipNum + 1L);
      if (subMap.isEmpty()) {
        System.out.println("  ?? " + reportedCountryCode + " "
            + nickname);
      } else {
        long intervalStart = subMap.lastKey();
        String geoipLine = geoipLines.get(intervalStart);
        long intervalEnd = Long.parseLong(geoipLine.split(",")[1]);
        if (intervalStart <= ipNum && intervalEnd >= ipNum) {
        String countryCode = geoipLine.split(",")[2];
          System.out.println((countryCode.equals(reportedCountryCode)
              ? "  " : "* ") + countryCode + " " + reportedCountryCode
              + " " + nickname);
        } else {
          System.out.println("  ?? " + reportedCountryCode + " "
              + nickname);
        }
      }
    }
  }
}