[tor-dev] yes hello, internet supervillain here
Fears No One
nachash at observers.net
Sun Nov 9 19:25:39 UTC 2014
I have some news to report, along with more data.
The August DoS attempt appears to have been a crawler bot after all. An
old friend came forward after reading tor-dev and we laughed about his
dumb crawler bot vs my dumb "must-serve-200-codes-at-everything" nginx
config. His user agent string only accounts for the spike in August, and
I see no evidence of a mass crawl from it in my log reports. The
2014-09_24.old file's spike in traffic doesn't match up with his crawl
times in any way, but he theorizes that somebody else maybe used the
same crawler package. For reference, this directory output shows when
each of his mass onion crawls ended:
drwxrwxr-x 3 username group 4096 Jul 27 04:30 onion-1
drwxrwxr-x 3 username group 4096 Jul 28 13:40 onion-2
drwxrwxr-x 3 username group 4096 Jul 28 14:36 onion-3
drwxrwxr-x 3 username group 4096 Jul 31 01:47 onion-4
drwxrwxr-x 3 username group 4096 Jul 31 06:48 onion-5
drwxrwxr-x 3 username group 4096 Aug 17 01:43 onion-6
drwxrwxr-x 3 username group 4096 Aug 28 00:49 onion-7
drwxrwxr-x 3 username group 4096 Sep 13 23:30 onion-8
This is probably the part where I mention that he mass crawled a bunch
of onions, not just mine. To save others the time of grepping for his
user agent string in log reports, I'm going to be slightly rude and
paste my grep command + the results here:
grep -R "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/35.0.1916.153 Safari/537.36" | sort
06/doxbin_2014_06_11.txt: 57 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_12.txt: 186 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_13.txt: 103 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_14.txt: 70 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_15.txt: 106 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_16.txt: 47 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_17.txt: 68 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_18.txt: 51 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_19.txt: 71 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_20.txt: 27 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_21.txt: 32 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_22.txt: 104 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_23.txt: 169 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_24.txt: 68 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_25.txt: 65 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_26.txt: 44 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_27.txt: 86 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_28.txt: 62 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_29.txt: 35 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
06/doxbin_2014_06_30.txt: 97 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_01.txt: 56 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_02.txt: 131 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_03.txt: 86 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_04.txt: 80 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_05.txt: 219 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_06.txt: 140 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_07.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153
Safari/537.36 OPR/22.0.1471.70
07/doxbin_2014_07_07.txt: 40 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_08.txt: 21 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_09.txt: 75 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_10.txt: 94 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_11.txt: 74 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_12.txt: 76 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_13.txt: 83 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_14.txt: 219 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_15.txt: 105 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_16.txt: 108 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_17.txt: 136 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_17.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153
Safari/537.36 OPR/22.0.1471.70
07/doxbin_2014_07_18.txt: 51 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_19.txt: 9 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_20.txt: 21 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_21.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153
Safari/537.36 OPR/22.0.1471.70
07/doxbin_2014_07_21.txt: 9 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_22.txt: 3 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_23.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_24.txt: 2 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_25.txt: 3 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_26.txt: 7 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_27.txt: 2 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_28.txt: 2 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
07/doxbin_2014_07_29.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_01.txt: 5 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_02.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_03.txt: 8 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_04.txt: 3 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_07.txt: 8 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_12.txt: 5 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_15.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_19.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_20.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_21.txt: 125816 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_22.txt: 198649 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_23.txt: 197578 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_24.txt: 220840 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_25.txt: 163170 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_27.txt: 36722 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
08/doxbin_2014_08_30.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
09/doxbin_2014_09_05.txt: 2 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
09/doxbin_2014_09_12.txt: 8 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
09/doxbin_2014_09_17.txt: 14 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
09/doxbin_2014_09_22.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
09/doxbin_2014_09_24.old: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
09/doxbin_2014_09_24.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
09/doxbin_2014_09_25.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_02.txt: 5 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_05.txt: 2 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_08.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_14.txt: 2 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_16.txt: 1 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_18.txt: 10 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_22.txt: 13 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_23.txt: 4 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_28.txt: 4 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10/doxbin_2014_10_29.txt: 7 Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
For reference, the useragent in doxbin_2014_09_24.old's crawl or DoS
attempt was "Wget/1.15 (linux-gnu)" It's probably worth pointing out
again that when this particular hammering of the server happened, I at
first began mitigating with ConstrainedSocks, then switched to 301
redirecting all the requests to The Hidden Wiki's Hard Candy page. His
theory is that his bot broke on doxbin before August and didn't even
crawl it. After he realized that his bot was breaking on doxbin in
August, he added .+hackforums\.net.+' as a crawling exclusion.
In other news, the same guy runs a bot that records uptimes for various
onions, and he gave me output related to up/down times for doxbin,
Cloud9, and Silk Road 2.0.
NOTE: Time zone is GMT+9:30 on all of these. He used sed to replace 0
with down and 1 with up for readability reasons on the doxbin and Silk
Road pastes, but the Cloud9 paste is raw.
doxbin: http://pastebin.com/pVxQDS9u
Cloud9: http://pastebin.com/5uYmpmfQ (0 = down, 1 = up)
Silk Road 2.0: http://pastebin.com/jQvgz0VF
Hoping some of this helps FTP,
- nachash
More information about the tor-dev
mailing list