Dear members of the Tor community, we are a research group at Sapienza University, Rome, Italy. We do research on distributed systems, Tor, and the Dark Web. As part of our work, we have developed OnionGatherer, a service that gives up-to-date information about Dark Web hidden services to Tor users. OnionGatherer is implemented as a Google Chrome extension coupled with a back-end service running on our servers. As the user surfes the Web, OnionGatherer collects all the URLs from the page and adds a green bullet next ot the URL if the hidden service is up and running, an orange one if the system are currently evaluating the address' status or a red one if the hidden service is down. The status of the hidden services is pulled from our servers, which keep track of all the services found by the users and constantly monitor their status. When a new hidden service is found, OnionGatherer checks its status in real time, informs the user accordingly, and adds it to the database. We believe that OnionGatherer can be very useful to Tor users that are interested in surfing the Dark Web. Indeed, hidden services are born and shut down very frequently, and it is often time consuming and frustrating to check manually which services are still active. We kindky ask if you can help disseminate our project ---the largest is the number of users of OnionGatherer, the largest the database and the best the service we can provide. Currently the software is in Beta version and released on GitHub at the following link: client: https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension server: https://github.com/rfidlabsapienza/onionGatherer-Server Any feedback or issue are really appreciated. Thanks in advance. Best regards, The research group: A. Mei, J. Stefa, M. La Morgia, S. Raponi
This is an interesting project, that being said I have a few concerns I'm hoping you can address. From a security standpoint; - The instructions for the webservice don't seem to indicate that it is being served as a hidden service, or even with ssl. See <Virtualhost *:80>. This would mean that, even if chrome is configured properly, when the request is made over Tor it basically sends every link on every page you're viewing, in the clear, over the public internet; and to your server, if one was to actually use it. - Unless you intend to share your onionGatherer service with someone else (you clearly shouldn't) then 'Require All Granted' is unnecessary and inadvisable. - [if(responseData['onions'][portion.text] == 0)](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[(responseData[](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)['](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[onions](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)['](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[][](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[portion](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[.](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[text](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[] ](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[==](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[ ](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[0](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52)[)](https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52) would return an orange circle if portion.text is undefined or null, perhaps stronger typing would be appropriate. From a pure code review standpoint; - ou include the images twice, once in the root, and once in figures. - [You've implemented an XTHML parser in regex](http:// https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master... ); Generally this is inadvisable. - The version of jQuery that was included (2.2.3) is not the most recent (2.2.4) Evan Sent with [ProtonMail](https://protonmail.com) Secure Email. -------- Original Message -------- Subject: [tor-dev] OnionGatherer: evaluating status of hidden services Local Time: 10 March 2017 7:58 AM UTC Time: 10 March 2017 11:58 From: lamorgia@di.uniroma1.it To: tor-dev@lists.torproject.org Julinda Stefa <stefa@di.uniroma1.it>, simone raponi <raponi.1539620@studenti.uniroma1.it>, Alessandro Mei <mei@di.uniroma1.it> Dear members of the Tor community, we are a research group at Sapienza University, Rome, Italy. We do research on distributed systems, Tor, and the Dark Web. As part of our work, we have developed OnionGatherer, a service that gives up-to-date information about Dark Web hidden services to Tor users. OnionGatherer is implemented as a Google Chrome extension coupled with a back-end service running on our servers. As the user surfes the Web, OnionGatherer collects all the URLs from the page and adds a green bullet next ot the URL if the hidden service is up and running, an orange one if the system are currently evaluating the address' status or a red one if the hidden service is down. The status of the hidden services is pulled from our servers, which keep track of all the services found by the users and constantly monitor their status. When a new hidden service is found, OnionGatherer checks its status in real time, informs the user accordingly, and adds it to the database. We believe that OnionGatherer can be very useful to Tor users that are interested in surfing the Dark Web. Indeed, hidden services are born and shut down very frequently, and it is often time consuming and frustrating to check manually which services are still active. We kindky ask if you can help disseminate our project ---the largest is the number of users of OnionGatherer, the largest the database and the best the service we can provide. Currently the software is in Beta version and released on GitHub at the following link: client: https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension server: https://github.com/rfidlabsapienza/onionGatherer-Server Any feedback or issue are really appreciated. Thanks in advance. Best regards, The research group: A. Mei, J. Stefa, M. La Morgia, S. Raponi
2017-03-10 13:28 GMT+01:00 Evan d'Entremont <evan@evandentremont.com>:
This is an interesting project, that being said I have a few concerns I'm hoping you can address.
From a security standpoint;
- The instructions for the webservice don't seem to indicate that it is being served as a hidden service, or even with ssl. See <Virtualhost *:80>. This would mean that, even if chrome is configured properly, when the request is made over Tor it basically sends every link on every page you're viewing, in the clear, over the public internet; and to your server, if one was to actually use it.
No, the webservice is not served as hidden service, but it runs with ssl and requests on port 80 are redirected on port 443 of this URL : https://lamorgiam.redi.uniroma1.it/onionGatherer. The configuration reported with <Virtualhost *:80> on the MD file is for a generic setup of the server.
- - Unless you intend to share your onionGatherer service with someone else (you clearly shouldn't) then 'Require All Granted' is unnecessary and inadvisable. - if(responseData['onions'][portion.text] == 0) <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> (responseData[ <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> ' <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> onions <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> ' <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> ][ <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> portion <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> . <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> text <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52>]
<https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> == <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> 0 <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> ) <https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L52> would return an orange circle if portion.text is undefined or null, perhaps stronger typing would be appropriate.
From a pure code review standpoint;
- ou include the images twice, once in the root, and once in figures. - You've implemented an XTHML parser in regex <http://%C2%A0https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension/blob/master/OnionGatherer.js#L6>; Generally this is inadvisable. - The version of jQuery that was included (2.2.3) is not the most recent (2.2.4)
Thank you for your feedback. Your advices are really appreciated. we will
try to fix asap
Evan
Sent with ProtonMail <https://protonmail.com> Secure Email.
-------- Original Message -------- Subject: [tor-dev] OnionGatherer: evaluating status of hidden services Local Time: 10 March 2017 7:58 AM UTC Time: 10 March 2017 11:58 From: lamorgia@di.uniroma1.it To: tor-dev@lists.torproject.org Julinda Stefa <stefa@di.uniroma1.it>, simone raponi < raponi.1539620@studenti.uniroma1.it>, Alessandro Mei <mei@di.uniroma1.it>
Dear members of the Tor community,
we are a research group at Sapienza University, Rome, Italy. We do research on distributed systems, Tor, and the Dark Web. As part of our work, we have developed OnionGatherer, a service that gives up-to-date information about Dark Web hidden services to Tor users.
OnionGatherer is implemented as a Google Chrome extension coupled with a back-end service running on our servers. As the user surfes the Web, OnionGatherer collects all the URLs from the page and adds a green bullet next ot the URL if the hidden service is up and running, an orange one if the system are currently evaluating the address' status or a red one if the hidden service is down. The status of the hidden services is pulled from our servers, which keep track of all the services found by the users and constantly monitor their status. When a new hidden service is found, OnionGatherer checks its status in real time, informs the user accordingly, and adds it to the database.
We believe that OnionGatherer can be very useful to Tor users that are interested in surfing the Dark Web. Indeed, hidden services are born and shut down very frequently, and it is often time consuming and frustrating to check manually which services are still active.
We kindky ask if you can help disseminate our project ---the largest is the number of users of OnionGatherer, the largest the database and the best the service we can provide. Currently the software is in Beta version and released on GitHub at the following link:
client: https://github.com/rfidlabsapienza/onionGatherer-ChromeExtension server: https://github.com/rfidlabsapienza/onionGatherer-Server
Any feedback or issue are really appreciated. Thanks in advance. Best regards,
The research group: A. Mei, J. Stefa, M. La Morgia, S. Raponi
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On Fri, Mar 10, 2017 at 12:58:55PM +0100, Massimo La Morgia wrote:
we are a research group at Sapienza University, Rome, Italy. We do research on distributed systems, Tor, and the Dark Web. As part of our work, we have developed OnionGatherer, a service that gives up-to-date information about Dark Web hidden services to Tor users.
...and presumably helps you build a crowdsourced list of onion services that you plan to use for some other research purpose? If you're planning a research project on Tor users, you should write to the research safety board and get ideas about how ot do it in a way that minimizes risk. https://research.torproject.org/safetyboard.html This idea seems, to me, to have a lot of privacy problems. You're asking people to use Chrome instead of Tor Browser, which means they will be vulnerable to a lot of fingerprinting and trivial deanonymization attacks. Your extension reports not only the onion domains that it finds, but also the URL of the page you were browsing at the time: var onionsJson = JSON.stringify({onions:onions, website: window.location.href}); You need to at least inform your research subjects/users what of their private data you are storing and what you are doing with it. You're using two different regexes for onion URLs that aren't the same. The one used during replacement doesn't match "https", so I guess it will fail on URLs like https://facebookcorewwwi.onion/. /^(http(s)?:\/\/)?.{16}(\.onion)\/?.*$/ /(http:\/\/)?\b[\w\d]{16}\.onion(\/[\S]*|)/
On Fri, Mar 10, 2017 at 5:39 PM, David Fifield <david@bamsoftware.com> wrote:
On Fri, Mar 10, 2017 at 12:58:55PM +0100, Massimo La Morgia wrote:
we are a research group at Sapienza University, Rome, Italy. We do research on distributed systems, Tor, and the Dark Web. As part of our work, we have developed OnionGatherer, a service that gives up-to-date information about Dark Web hidden services to Tor users.
...and presumably helps you build a crowdsourced list of onion services that you plan to use for some other research purpose?
yes, of course in this way we are building a crowdsourced list of onion services, but is not really different from onion directories. At this time we have no plan for other research that use this crowdsourced list.
If you're planning a research project on Tor users, you should write to the research safety board and get ideas about how ot do it in a way that minimizes risk. https://research.torproject.org/safetyboard.html
thank you for the suggestion.
This idea seems, to me, to have a lot of privacy problems. You're asking people to use Chrome instead of Tor Browser, which means they will be vulnerable to a lot of fingerprinting and trivial deanonymization attacks.
No we are not asking people to use chrome for browsing on tor, but we are offering a service that can help them to know if a onion address is up before start to surf with Tor Browser
Your extension reports not only the onion domains that it finds, but also the URL of the page you were browsing at the time: var onionsJson = JSON.stringify({onions:onions, website: window.location.href}); You need to at least inform your research subjects/users what of their private data you are storing and what you are doing with it.
As you can see from the source code we are not storing any sensitive data like ip or users information. do you think that only URL page can damage user privacy?
You're using two different regexes for onion URLs that aren't the same. The one used during replacement doesn't match "https", so I guess it will fail on URLs like https://facebookcorewwwi.onion/. /^(http(s)?:\/\/)?.{16}(\.onion)\/?.*$/ /(http:\/\/)?\b[\w\d]{16}\.onion(\/[\S]*|)/
Yes, you right, thank you for the feedback.
Massimo La Morgia transcribed 6.7K bytes:
On Fri, Mar 10, 2017 at 5:39 PM, David Fifield <david@bamsoftware.com> wrote:
On Fri, Mar 10, 2017 at 12:58:55PM +0100, Massimo La Morgia wrote:
we are a research group at Sapienza University, Rome, Italy. We do research on distributed systems, Tor, and the Dark Web. As part of our work, we have developed OnionGatherer, a service that gives up-to-date information about Dark Web hidden services to Tor users.
...and presumably helps you build a crowdsourced list of onion services that you plan to use for some other research purpose?
yes, of course in this way we are building a crowdsourced list of onion services, but is not really different from onion directories. At this time we have no plan for other research that use this crowdsourced list.
If you're planning a research project on Tor users, you should write to the research safety board and get ideas about how ot do it in a way that minimizes risk. https://research.torproject.org/safetyboard.html
thank you for the suggestion.
This idea seems, to me, to have a lot of privacy problems. You're asking people to use Chrome instead of Tor Browser, which means they will be vulnerable to a lot of fingerprinting and trivial deanonymization attacks.
No we are not asking people to use chrome for browsing on tor, but we are offering a service that can help them to know if a onion address is up before start to surf with Tor Browser
Having only an extension for Chrome based browsers implies asking users to use Chrome based browsers. If there were a choice between Firefox and Chrome extensions, it would be less clear and not implying.
Your extension reports not only the onion domains that it finds, but also the URL of the page you were browsing at the time: var onionsJson = JSON.stringify({onions:onions, website: window.location.href}); You need to at least inform your research subjects/users what of their private data you are storing and what you are doing with it.
As you can see from the source code we are not storing any sensitive data like ip or users information. do you think that only URL page can damage user privacy?
This aside, do you just check if the page still exists or the top level onion domain you found this page on? If so, this would be an improvement I'd suggest, to only use the toplevel domain. I have not looked at your code.
You're using two different regexes for onion URLs that aren't the same. The one used during replacement doesn't match "https", so I guess it will fail on URLs like https://facebookcorewwwi.onion/. /^(http(s)?:\/\/)?.{16}(\.onion)\/?.*$/ /(http:\/\/)?\b[\w\d]{16}\.onion(\/[\S]*|)/
Yes, you right, thank you for the feedback.
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
2017-03-10 21:13 GMT+01:00 ng0 <contact.ng0@cryptolab.net>:
Massimo La Morgia transcribed 6.7K bytes:
On Fri, Mar 10, 2017 at 5:39 PM, David Fifield <david@bamsoftware.com> wrote:
On Fri, Mar 10, 2017 at 12:58:55PM +0100, Massimo La Morgia wrote:
we are a research group at Sapienza University, Rome, Italy. We do research on distributed systems, Tor, and the Dark Web. As part of our work, we have developed OnionGatherer, a service that gives up-to-date information about Dark Web hidden services to Tor users.
...and presumably helps you build a crowdsourced list of onion services that you plan to use for some other research purpose?
yes, of course in this way we are building a crowdsourced list of onion services, but is not really different from onion directories. At this time we have no plan for other research that use this crowdsourced list.
If you're planning a research project on Tor users, you should write to the research safety board and get ideas about how ot do it in a way
that
minimizes risk. https://research.torproject.org/safetyboard.html
thank you for the suggestion.
This idea seems, to me, to have a lot of privacy problems. You're asking people to use Chrome instead of Tor Browser, which means they will be vulnerable to a lot of fingerprinting and trivial deanonymization attacks.
No we are not asking people to use chrome for browsing on tor, but we are offering a service that can help them to know if a onion address is up before start to surf with Tor Browser
Having only an extension for Chrome based browsers implies asking users to use Chrome based browsers. If there were a choice between Firefox and Chrome extensions, it would be less clear and not implying.
Yes, you're right, but we have created this extension in order to offer a service to people. We chose to start with Chrome because it has a greater number of users. We would be happy if it will be used and also developed for Firefox.
Your extension reports not only the onion domains that it finds, but also the URL of the page you were browsing at the time: var onionsJson = JSON.stringify({onions:onions, website: window.location.href}); You need to at least inform your research subjects/users what of their private data you are storing and what you are doing with it.
As you can see from the source code we are not storing any sensitive data like ip or users information. do you think that only URL page can damage user privacy?
This aside, do you just check if the page still exists or the top level onion domain you found this page on? If so, this would be an improvement I'd suggest, to only use the toplevel domain. I have not looked at your code.
Thank you for the suggestion, we'll improve the website's URL management asap.
You're using two different regexes for onion URLs that aren't the same. The one used during replacement doesn't match "https", so I guess it will fail on URLs like https://facebookcorewwwi.onion/. /^(http(s)?:\/\/)?.{16}(\.onion)\/?.*$/ /(http:\/\/)?\b[\w\d]{16}\.onion(\/[\S]*|)/
Yes, you right, thank you for the feedback.
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
The server is temporarily down due to security improvements, thank you very much for your suggestions. 2017-03-10 21:02 GMT+01:00 simone raponi < raponi.1539620@studenti.uniroma1.it>:
2017-03-10 21:13 GMT+01:00 ng0 <contact.ng0@cryptolab.net>:
Massimo La Morgia transcribed 6.7K bytes:
On Fri, Mar 10, 2017 at 5:39 PM, David Fifield <david@bamsoftware.com> wrote:
On Fri, Mar 10, 2017 at 12:58:55PM +0100, Massimo La Morgia wrote:
we are a research group at Sapienza University, Rome, Italy. We do research on distributed systems, Tor, and the Dark Web. As part of our work, we have developed OnionGatherer, a service that gives up-to-date information about Dark Web hidden services to Tor users.
...and presumably helps you build a crowdsourced list of onion services that you plan to use for some other research purpose?
yes, of course in this way we are building a crowdsourced list of onion services, but is not really different from onion directories. At this time we have no plan for other research that use this crowdsourced list.
If you're planning a research project on Tor users, you should write
to
the research safety board and get ideas about how ot do it in a way that minimizes risk. https://research.torproject.org/safetyboard.html
thank you for the suggestion.
This idea seems, to me, to have a lot of privacy problems. You're asking people to use Chrome instead of Tor Browser, which means they will be vulnerable to a lot of fingerprinting and trivial deanonymization attacks.
No we are not asking people to use chrome for browsing on tor, but we are offering a service that can help them to know if a onion address is up before start to surf with Tor Browser
Having only an extension for Chrome based browsers implies asking users to use Chrome based browsers. If there were a choice between Firefox and Chrome extensions, it would be less clear and not implying.
Yes, you're right, but we have created this extension in order to offer a service to people. We chose to start with Chrome because it has a greater number of users. We would be happy if it will be used and also developed for Firefox.
Your extension reports not only the onion domains that it finds, but also the URL of the page you were browsing at the time: var onionsJson = JSON.stringify({onions:onions, website: window.location.href}); You need to at least inform your research subjects/users what of their private data you are storing and what you are doing with it.
As you can see from the source code we are not storing any sensitive data like ip or users information. do you think that only URL page can damage user privacy?
This aside, do you just check if the page still exists or the top level onion domain you found this page on? If so, this would be an improvement I'd suggest, to only use the toplevel domain. I have not looked at your code.
Thank you for the suggestion, we'll improve the website's URL management asap.
You're using two different regexes for onion URLs that aren't the
same.
The one used during replacement doesn't match "https", so I guess it will fail on URLs like https://facebookcorewwwi.onion/. /^(http(s)?:\/\/)?.{16}(\.onion)\/?.*$/ /(http:\/\/)?\b[\w\d]{16}\.onion(\/[\S]*|)/
Yes, you right, thank you for the feedback.
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On Fri, Mar 10, 2017 at 06:25:04PM +0100, Massimo La Morgia wrote:
On Fri, Mar 10, 2017 at 5:39 PM, David Fifield <david@bamsoftware.com> wrote:
Your extension reports not only the onion domains that it finds, but also the URL of the page you were browsing at the time: var onionsJson = JSON.stringify({onions:onions, website: window.location.href}); You need to at least inform your research subjects/users what of their private data you are storing and what you are doing with it.
As you can see from the source code we are not storing any sensitive data like ip or users information. do you think that only URL page can damage user privacy?
Yes, web applications encode sensitive information in URLs all the time. Usernames, passwords, personal preferences, you name it. Even just the page's domain name reveals a lot about you -- think about somebody visiting google.it versus google.dk.
Hey everybody, I want to announce that our server is up again. Thank you all for your suggestions, we're opening issues for each one in the Github repository. I want to thank David Fifield for reporting us the security issue. Regarding the stored website's url information, we're evaluating the insertion of a checkbox that allows the user to choose if he/she wants to send it or not. 2017-03-15 18:47 GMT+01:00 Philipp Winter <phw@nymity.ch>:
On Fri, Mar 10, 2017 at 06:25:04PM +0100, Massimo La Morgia wrote:
On Fri, Mar 10, 2017 at 5:39 PM, David Fifield <david@bamsoftware.com> wrote:
Your extension reports not only the onion domains that it finds, but also the URL of the page you were browsing at the time: var onionsJson = JSON.stringify({onions:onions, website: window.location.href}); You need to at least inform your research subjects/users what of their private data you are storing and what you are doing with it.
As you can see from the source code we are not storing any sensitive data like ip or users information. do you think that only URL page can damage user privacy?
Yes, web applications encode sensitive information in URLs all the time. Usernames, passwords, personal preferences, you name it. Even just the page's domain name reveals a lot about you -- think about somebody visiting google.it versus google.dk. _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
participants (7)
-
ansijax＠gmail.com -
David Fifield -
Evan d'Entremont -
Massimo La Morgia -
ng0 -
Philipp Winter -
simone raponi