Available for collaboration - neumon.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi to all, i just discovered this project, and i'm studying it. I'm the creator of the no-profit service http://www.neumon.org . It's a project similar to OONI, but focused only on DNS and HTTP. Basically, i detect DNS around the world (the same technique we rebuild here: http://ipleak.net/ ). NeuMon browse collected DNS servers, and check if can be queried (open and recursive). This because most of DNS ISP are recursive only from it's customers subnet. We maintain a huge list of domains to check (mix of known blocked website, top alexa, etc). Every DNS it's queried for each domains, we collect results, compare against a known good value and discover custom injection (generally that point to blocking page, i published some example here: http://tinyurl.com/pl8znb4 ). So, i have: - - a huge list of DNS servers, with country geolocation. - - lists of domains blocked, country-based. Not exaustive. - - i know many IP address that are destination of DNS redirection, typically IP of servers that show html blocking pages. And DNS servers of ISP that redirect to these addresses. An example of data i know: # dig www.sex.com @203.146.237.237 +short answer: http://203.146.43.133/ i know hundred of domains that redirect to the same answer. I never made public my results. Because i never know anyone interested in that data, only about the lists of domains censored. But i never publish it because contains child pornography domains. We also build a probe software, to allow other activists connected to the ISP directly to launch it and detect censorship not based on DNS. But nobody want to run a software that fetch also child pornography domains, so nobody want to run our probe. In general, the NeuMon project has never attracted the interest of no one, and actually is abandoned and no longer maintained. I'm here to understand if i can help OONI project (or my system & collected data). Ciao! Fabrizio - Clodo -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEbBAEBCAAGBQJVRVFDAAoJEC/ixHrG0m4LyPwH9REkL4RNiVzWKlv4qCxf/YGP t+3U7hOOm95HGZL6VX7XzSBeo3tL32FlppJNdxL0T+B6I2oKPSOizpYGjHSrkTgm ETqdgsOdF5AYWH+BCTS+U+EtOyN6ckIk1+bLI6gfUuFuwxcHC09GisYDIqRqTaX+ X5+ovXuZPkC+lwAZKfKIxfIw7zvrpWKnsfxJskp7pCypNh0vzd4C/erRBg3RcYi9 cExA1yKpbRxzK2dO3nJmQfL0nYwCWEFVWTQmZBJd8v62AbeYg7FDXth8PJZeVJhp WAJx0nTWlR6MzpShTZiJGxSnPuZzuxU+oEDKKSaYw7xBQhaFxQko7McTOTgztg== =NR6I -----END PGP SIGNATURE-----
Hi Clodo, Thank you for your interest in OONI. Clodo wrote:
I'm the creator of the no-profit service http://www.neumon.org . It's a project similar to OONI, but focused only on DNS and HTTP.
Do you have the code published somewhere?
NeuMon browse collected DNS servers, and check if can be queried (open and recursive). This because most of DNS ISP are recursive only from it's customers subnet. We maintain a huge list of domains to check (mix of known blocked website, top alexa, etc). Every DNS it's queried for each domains, we collect results, compare against a known good value and discover custom injection (generally that point to blocking page, i published some example here: http://tinyurl.com/pl8znb4 ).
So, i have: - - a huge list of DNS servers, with country geolocation. - - lists of domains blocked, country-based. Not exaustive. - - i know many IP address that are destination of DNS redirection, typically IP of servers that show html blocking pages. And DNS servers of ISP that redirect to these addresses.
It was quite difficult to find out and interpret the results from [1] could you maybe provide some pointers?
An example of data i know: # dig www.sex.com @203.146.237.237 +short answer: http://203.146.43.133/ i know hundred of domains that redirect to the same answer.
I never made public my results. Because i never know anyone interested in that data, only about the lists of domains censored. But i never publish it because contains child pornography domains.
How did you find out about these domains and why do you think that they contain CP? In any case it would be very interesting to see these results or of the ones that can be made public.
We also build a probe software, to allow other activists connected to the ISP directly to launch it and detect censorship not based on DNS.
It will be very interesting to instruct the probe software submit results to an ooni backend [2]. In any case the probe software can maybe even written as an ooniprobe test [3].
But nobody want to run a software that fetch also child pornography domains, so nobody want to run our probe.
I don't think that is all about CP only. Right now there are so many blacklists and censored websites worldwide and as far as I know people are interested in finding out of what resources are being blocked. Many of these started blocking gambling related websites and later added a bunch of other websites hence opening the door for censorship and blocking of other websites at will [4], [5]. [1] http://www.neumon.org/country/ [2] https://github.com/TheTorProject/ooni-backend [3] https://ooni.torproject.org/docs/writing_tests.html [4] https://edri.org/increased-level-of-online-censorship-in-italy/ [5] https://lists.torproject.org/pipermail/ooni-dev/2015-January/000220.html Cheers ~anadahz
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 05/05/2015 16:19, balooni@espiv.net wrote:
Hi Clodo,
Thank you for your interest in OONI.
Clodo wrote:
I'm the creator of the no-profit service http://www.neumon.org . It's a project similar to OONI, but focused only on DNS and HTTP.
Do you have the code published somewhere? We release the source of the probe here: https://github.com/AirVPN/neumon-probe Written in C#/Mono. I run it from RaspBian on Raspberry PI. But it's not a great piece of software. Simply it fetch from our backend the list of domains to try to resolve/fetch, do it, and resend the results. All detection are server-side based.
The backend it's written in php, sources never released. Contain basically a lot of mysql queries to detect stuffs and generate report.
NeuMon browse collected DNS servers, and check if can be queried (open and recursive). This because most of DNS ISP are recursive only from it's customers subnet. We maintain a huge list of domains to check (mix of known blocked website, top alexa, etc). Every DNS it's queried for each domains, we collect results, compare against a known good value and discover custom injection (generally that point to blocking page, i published some example here: http://tinyurl.com/pl8znb4 ).
So, i have: - - a huge list of DNS servers, with country geolocation. - - lists of domains blocked, country-based. Not exaustive. - - i know many IP address that are destination of DNS redirection, typically IP of servers that show html blocking pages. And DNS servers of ISP that redirect to these addresses.
It was quite difficult to find out and interpret the results from [1] could you maybe provide some pointers?
Mainly because i don't publish all results, generally aggregated stats. I have lists of domains blocked, but isn't available on neumon.org for reason explained below in this mail. http://www.neumon.org/blacklist.html These IP are destination IP of DNS injection. We manage it manually. They hosts the blocking pages. They have virtual-hosts in webserver for domains, so you maybe cannot view the blocking page by viewing the IP directly. The lists contain also private services (like OpenDNS), not related to censorship. The lists may contain blocking page of private services (like adult-filter services), not related to ISP censorship. The recent tweet of Mikko Hypponen: https://twitter.com/mikko/status/595681341334773760 are screenshots of websites of the above IP list. http://www.neumon.org/?view=dns_list&country=it This is an example country list of DNS servers open and recursive we detect in Italy. Note that maybe include a customer of an ISP that have it's own DNS server. Generally, i have a lot of data, catched automatically, that require manual works to obtain nice and clean report, i'm in stall on this kind of works.
How did you find out about these domains and why do you think that they contain CP?
I don't know. Some of them (with domain name with keyword like teen/sex/...) seem like typical porn website, a collection of video and screenshot. Of course i can't know if are real CP. I'm italian, and i know very well the italian situation: here ISP block CP, gambling, proxy, file-sharing, file-hosting, webcam, pharma, escort, drugs, steroid, etc. Sometime, if a CP it's hosted on a public image-hosting, the entire file-hosting services are blocked. ImageShack was DNS blocked for years in Italy for a single CP image. I obtain with my system lists of blocked domain, all of category listed above together, but actually i don't want to publish it (see https://youtu.be/RkmcupFx3FQ?t=1m13s ) because i can't detect CP versus other categories. Anyway, in my system sometime i have the information of what is classified CP. For example, major ISP in Switzerland redirect CP domain to a server hosted by stopp-kinderpornografie.ch .
In any case it would be very interesting to see these results or of the ones that can be made public.
We also build a probe software, to allow other activists connected to the ISP directly to launch it and detect censorship not based on DNS.
It will be very interesting to instruct the probe software submit results to an ooni backend [2]. In any case the probe software can maybe even written as an ooniprobe test [3].
I understand you already have some DNS tests on ooniprobe. I will study them. But actually i don't understand what are the lists of domain tested by OONI, how you detect spoof, and where/if you results are published. My mysql data it's around 25 gb. I think maybe better (for maintenance and independency) not to create OONI tests linked to neumon.org project. I think maybe better if i create some webservices in neumon.org to expose my data, where OONI backend can fetch interesting data for your research. For example, i can provide a list of DNS servers we detect (open to query and with recursion enabled). Or i can provide a list of "open/recursive DNS Server IP -> query domain "xxx" -> the result "ip address" it's probably a blocking page.
But nobody want to run a software that fetch also child pornography domains, so nobody want to run our probe.
I don't think that is all about CP only. Right now there are so many blacklists and censored websites worldwide and as far as I know people are interested in finding out of what resources are being blocked. Many of these started blocking gambling related websites and later added a bunch of other websites hence opening the door for censorship and blocking of other websites at will [4], [5].
A particular example: i know a very important ISP that redirect blocked domain to a fixed IP. Interesting and unbelivable, they specify the reverse-lookup info on that IP. So, a reverse lookup on that IP show thousand of domains, updated frequently. I fetch periodically this list to populate my domains tests list. But, that list contain mixed CP, proxy, and in general all category of blocked domains. Detect what are the domains that may attract mainstream interest for censorship reason, require filter that list by skipping CP, gambling etc, and it's the kind of work that i don't know how to manage with an automatic system. I hope you can understand my poor english. Ciao Fabrizio - Clodo -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJVShOLAAoJEC/ixHrG0m4LZjoH/3ebg0oWuzEac4mNY51qBZ2C MQm/apd0l+M+lYiFxJHY9xS7gmrAr/+YfudUieCfsLTuGAW0moeimU133f/pV+TJ dbCGkUKRjVxKSp7D3ndijiSmFkOEtrNZrq2yEFZXYUAR2sSzz62VREa1XNqVDi6Z hTKjAwwYDUesdD8DSO5UCPQu95s3gHwL4uk/28ExqmPddWu9cpoi3PD7gIhWjRJh Q1wUESqCqnp6eVu4iS4UqUS6Gn/4I0rITOYNdgVZYXlYw0ENaPn8ILyHPEwLLLbY DkWpPcrN+N1tRemAIBKlD5OHkBKSWFR8Qohyc0SUNnTz1qdDT/KS8YOyd6KdFqM= =87nY -----END PGP SIGNATURE-----
Hi, Clodo <clodo@clodo.it> wrote:
On 05/05/2015 16:19, balooni@espiv.net wrote:
Clodo wrote:
I'm the creator of the no-profit service http://www.neumon.org . It's a project similar to OONI, but focused only on DNS and HTTP.
Do you have the code published somewhere?
We release the source of the probe here: https://github.com/AirVPN/neumon-probe Written in C#/Mono. I run it from RaspBian on Raspberry PI.
We are creating a Rasberry Pi image for ooniprobe [1]. You are more than welcome to use ooniprobe and start submiting reports to the OONI backend.. we are about to launch a more effective database solution for the pipeline.
But it's not a great piece of software. Simply it fetch from our backend the list of domains to try to resolve/fetch, do it, and resend the results. All detection are server-side based.
How did you compile this list of domains was reported by other users or you just used some public blacklists?
The backend it's written in php, sources never released. Contain basically a lot of mysql queries to detect stuffs and generate report.
Interesting, care to share these queries?
Generally, i have a lot of data, catched automatically, that require manual works to obtain nice and clean report, i'm in stall on this kind of works.
Let me know if you would like help on this.
In any case it would be very interesting to see these results or of the ones that can be made public.
We also build a probe software, to allow other activists connected to the ISP directly to launch it and detect censorship not based on DNS.
It will be very interesting to instruct the probe software submit results to an ooni backend [2]. In any case the probe software can maybe even written as an ooniprobe test [3]. I understand you already have some DNS tests on ooniprobe. I will study them. But actually i don't understand what are the lists of domain tested by OONI, how you detect spoof, and where/if you results are published.
Currently we use the URL lists maintained by citizenlab [2]. These lists are far from complete but they cover a variety of potential blocked websites per country or globally. But every user can provide its own list of websites/domains. An example of a relevant test is the http_requests test [3] that probes a website/domain from the probes Internet connection and via Tor and then compares and checks if the body proportions matches. The reports are being published here [4]. As I mentioned above we are working in a newer database and pipeline implementation.
My mysql data it's around 25 gb. I think maybe better (for maintenance and independency) not to create OONI tests linked to neumon.org project. I think maybe better if i create some webservices in neumon.org to expose my data, where OONI backend can fetch interesting data for your research.
It will be really nice if you can provide and/or submit this data to the OONI database pipeline. In any case we could collaborate on analyzing these data.
For example, i can provide a list of DNS servers we detect (open to query and with recursion enabled). Or i can provide a list of "open/recursive DNS Server IP -> query domain "xxx" -> the result "ip address" it's probably a blocking page.
Indeed this list will be very useful. [1] https://github.com/anadahz/lepidopter [2] https://github.com/citizenlab/test-lists [3] https://github.com/TheTorProject/ooni-spec/blob/master/test-specs/ts-003-htt... [4] https://ooni.torproject.org/reports/ Cheers ~anadahz
participants (2)
-
balooni＠espiv.net -
Clodo