Usefullness of providing TOR mirrors?
Hi! I'm one of the mirror admins for ftp.acc.umu.se. It's nice that the tor mirroring effort is seeing some attention from the project, but I'm still curious on what the plan/purpose/goal with the mirroring effort is from an end-user perspective. Bear in mind that I'm mostly accustomed to "regular" open source projects and not fully updated on all the issues with a privacy-oriented project such as TOR. However, I see issues with the usefulness of providing a TOR mirror. One of the prime motivations for us to provide a mirror is the benefit for users in our local region, and this is especially true for other mirrors in bandwidth-starved regions. If there is more bandwidth consumption by mirroring a project than there are downloads it's kind of hard to motivate a mirror in the first place.
From what I have gathered it's today almost impossible for an end user to find and use a mirror...
1) https://www.torproject.org/ -> Download brings you to https://www.torproject.org/download/download-easy.html.en which seems hard-coded to download from dist.torproject.org ... No mirror usage there. 2) There is no list of alternative/mirror download locations. I can google my way to https://www.torproject.org/getinvolved/mirrors.html.en but that's not updated with the latest mirrors nor sorted in a user-friendly manner (you want the list by country/region). 3) The TOR project itself doesn't seem to have much interest in actually using its mirrors. For example https://trac.torproject.org/projects/tor/ticket/27586 discards using mirrorbits with "not needed at the moment", when the exact opposite is true. We mirror admins WANT the traffic, and the TOR project NEEDS something to enable that together with automatically monitoring/disabling broken mirrors for a useful end-user experience. 4) And finally, if all above is catered for, there is the question on how to do download mirror redirect/selection in a clear yet useful manner for the end user. Should the user always be presented with a list of mirrors to choose from? Should that be generated on the server side or in-browser from a list of current mirrors (ie. json output from mirrorbits or similar)? We are heading into privacy concerns here, but it needs to be addressed in a better way than just assuming that the three magic hosts serving dist.torproject.org is the best choice for a particular end-user... /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- Beam me up, there's no intelligent life here! =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
On Tue, Oct 16, 2018, at 00:19, Niklas Edmundsson wrote:
I'm one of the mirror admins for ftp.acc.umu.se.
I run a mirror, but have no formal involvement with Tor beyond that as a casual user. To me, the only real value is that users in countries with limited access to Tor's main URLs have the opportunity to access a copy of Tor hosted elsewhere. I don't believe bandwidth or other resources are well spent on the mirror infrastructure at this time, at least based on the limited logging I have available.
I don't either especially if the owners don't maintain the list in a timely manner. I requested my mirror be removed and it's still in the list and says Up to date despite having shut this down weeks ago. On 10/16/18, 3:26 PM, "tor-mirrors on behalf of Dave Warren" <tor-mirrors-bounces@lists.torproject.org on behalf of dw@thedave.ca> wrote: On Tue, Oct 16, 2018, at 00:19, Niklas Edmundsson wrote: > > I'm one of the mirror admins for ftp.acc.umu.se. I run a mirror, but have no formal involvement with Tor beyond that as a casual user. To me, the only real value is that users in countries with limited access to Tor's main URLs have the opportunity to access a copy of Tor hosted elsewhere. I don't believe bandwidth or other resources are well spent on the mirror infrastructure at this time, at least based on the limited logging I have available. _______________________________________________ tor-mirrors mailing list tor-mirrors@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-mirrors
Hi, i like where this is going, let me allow to answer inline. On Tue, 16 Oct 2018 09:19:08 +0200 (CEST) Niklas Edmundsson <nikke@acc.umu.se> wrote:
Hi!
I'm one of the mirror admins for ftp.acc.umu.se.
It's nice that the tor mirroring effort is seeing some attention from the project, but I'm still curious on what the plan/purpose/goal with the mirroring effort is from an end-user perspective.
Bear in mind that I'm mostly accustomed to "regular" open source projects and not fully updated on all the issues with a privacy-oriented project such as TOR.
Note that i am not announcing the official position of TPO, instead just my opinion as a volunteer stepping into the mirror component.
However, I see issues with the usefulness of providing a TOR mirror. One of the prime motivations for us to provide a mirror is the benefit for users in our local region, and this is especially true for other mirrors in bandwidth-starved regions. If there is more bandwidth consumption by mirroring a project than there are downloads it's kind of hard to motivate a mirror in the first place.
From what I have gathered it's today almost impossible for an end user to find and use a mirror...
1) https://www.torproject.org/ -> Download brings you to https://www.torproject.org/download/download-easy.html.en which seems hard-coded to download from dist.torproject.org ... No mirror usage there.
This is true for the current website and we are considering a better option for the coming one: https://bugs.torproject.org/21222
2) There is no list of alternative/mirror download locations. I can google my way to https://www.torproject.org/getinvolved/mirrors.html.en but that's not updated with the latest mirrors nor sorted in a user-friendly manner (you want the list by country/region).
This is absolutely true and i'm glad you are pointing out what i thought myself updating the script. Filed and currently working on: https://bugs.torproject.org/28083
3) The TOR project itself doesn't seem to have much interest in actually using its mirrors. For example https://trac.torproject.org/projects/tor/ticket/27586 discards using mirrorbits with "not needed at the moment", when the exact opposite is true. We mirror admins WANT the traffic, and the TOR project NEEDS something to enable that together with automatically monitoring/disabling broken mirrors for a useful end-user experience.
We need a solution for the following problem: 3rd party mirrors introduce privacy issues and TPO hesitates to automatically redirect away from its website. Mirrors not necessarily need to be malicious but can be out of sync or offline. Currently we have no set up that periodicaly monitors the state of mirrors and verifies files. Also what if someone sets up mirrors to investigate who is interested in Tor? Obviously the answer can't be "use tor" here. https://bugs.torproject.org/27998
4) And finally, if all above is catered for, there is the question on how to do download mirror redirect/selection in a clear yet useful manner for the end user. Should the user always be presented with a list of mirrors to choose from? Should that be generated on the server side or in-browser from a list of current mirrors (ie. json output from mirrorbits or similar)?
Is https://gettor.torproject.org/api/mirrors.json what you suggest? I agree that we should make better use of it. https://bugs.torproject.org/16716
We are heading into privacy concerns here, but it needs to be addressed in a better way than just assuming that the three magic hosts serving dist.torproject.org is the best choice for a particular end-user...
The GetTor team dried out and needs your help! https://gettor.torproject.org/ https://trac.torproject.org/projects/tor/query?status=!closed&component=Appl...
/Nikke
There's a big potential to think mirrors and gettor together and i'd like to inspire you to make suggestions how to go into this direction. My answer could be more elaborate but i prefer to delve into code now :) -- traumschule.org gpg fingerprint: 9356 4DED 8546 8D9A C290 3605 12EE 7D70 7111 2056 /otr info OTR: traumschule@irc.indymedia.org fingerprint: OTR: 35AACA83 4564616C B6EBEC66 56B6B2FC C8D572F1 OTR: traumschule@irc.oftc.net fingerprint: OTR: D1CCD207 B60C1866 56A975AE ACE090E9 45E90846 OTR: traumschule@chat.freenode.net fingerprint: OTR: 51BF8BB9 434840CC 24F264BC 76450C27 A6AADB12
Great discussion, I am hosting a mirror for Vietnam (But not big traffic, I had quite a lot when I was helping Tails) On the privacy, i realized that my stats https://mirror.freedif.org/Stats/TorProject.html included the IPs as well of who downloaded Tor.....It was not my intention, but I treated Tor mirroring like any other mirroring project... I had to do some work around to remove those IPs, and may be some others people have the same challenge. It would be great to clarify the position of Tor and guidelines for mirroring for example. Le 17.10.2018 11:33, Traumschule a écrit :
Hi,
i like where this is going, let me allow to answer inline.
On Tue, 16 Oct 2018 09:19:08 +0200 (CEST) Niklas Edmundsson <nikke@acc.umu.se> wrote:
Hi!
I'm one of the mirror admins for ftp.acc.umu.se.
It's nice that the tor mirroring effort is seeing some attention from the project, but I'm still curious on what the plan/purpose/goal with the mirroring effort is from an end-user perspective.
Bear in mind that I'm mostly accustomed to "regular" open source projects and not fully updated on all the issues with a privacy-oriented project such as TOR.
Note that i am not announcing the official position of TPO, instead just my opinion as a volunteer stepping into the mirror component.
However, I see issues with the usefulness of providing a TOR mirror. One of the prime motivations for us to provide a mirror is the benefit for users in our local region, and this is especially true for other mirrors in bandwidth-starved regions. If there is more bandwidth consumption by mirroring a project than there are downloads it's kind of hard to motivate a mirror in the first place.
From what I have gathered it's today almost impossible for an end user to find and use a mirror...
1) https://www.torproject.org/ -> Download brings you to https://www.torproject.org/download/download-easy.html.en which seems hard-coded to download from dist.torproject.org ... No mirror usage there.
This is true for the current website and we are considering a better option for the coming one:
https://bugs.torproject.org/21222
2) There is no list of alternative/mirror download locations. I can google my way to https://www.torproject.org/getinvolved/mirrors.html.en but that's not updated with the latest mirrors nor sorted in a user-friendly manner (you want the list by country/region).
This is absolutely true and i'm glad you are pointing out what i thought myself updating the script. Filed and currently working on:
https://bugs.torproject.org/28083
3) The TOR project itself doesn't seem to have much interest in actually using its mirrors. For example https://trac.torproject.org/projects/tor/ticket/27586 discards using mirrorbits with "not needed at the moment", when the exact opposite is true. We mirror admins WANT the traffic, and the TOR project NEEDS something to enable that together with automatically monitoring/disabling broken mirrors for a useful end-user experience.
We need a solution for the following problem: 3rd party mirrors introduce privacy issues and TPO hesitates to automatically redirect away from its website. Mirrors not necessarily need to be malicious but can be out of sync or offline. Currently we have no set up that periodicaly monitors the state of mirrors and verifies files.
Also what if someone sets up mirrors to investigate who is interested in Tor? Obviously the answer can't be "use tor" here.
https://bugs.torproject.org/27998
4) And finally, if all above is catered for, there is the question on how to do download mirror redirect/selection in a clear yet useful manner for the end user. Should the user always be presented with a list of mirrors to choose from? Should that be generated on the server side or in-browser from a list of current mirrors (ie. json output from mirrorbits or similar)?
Is https://gettor.torproject.org/api/mirrors.json what you suggest?
I agree that we should make better use of it. https://bugs.torproject.org/16716
We are heading into privacy concerns here, but it needs to be addressed in a better way than just assuming that the three magic hosts serving dist.torproject.org is the best choice for a particular end-user...
The GetTor team dried out and needs your help! https://gettor.torproject.org/ https://trac.torproject.org/projects/tor/query?status=!closed&component=Appl...
/Nikke
There's a big potential to think mirrors and gettor together and i'd like to inspire you to make suggestions how to go into this direction.
My answer could be more elaborate but i prefer to delve into code now :)
-- traumschule.org
gpg fingerprint: 9356 4DED 8546 8D9A C290 3605 12EE 7D70 7111 2056
/otr info OTR: traumschule@irc.indymedia.org fingerprint: OTR: 35AACA83 4564616C B6EBEC66 56B6B2FC C8D572F1 OTR: traumschule@irc.oftc.net fingerprint: OTR: D1CCD207 B60C1866 56A975AE ACE090E9 45E90846 OTR: traumschule@chat.freenode.net fingerprint: OTR: 51BF8BB9 434840CC 24F264BC 76450C27 A6AADB12 _______________________________________________ tor-mirrors mailing list tor-mirrors@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-mirrors
Hi, On Thu, 18 Oct 2018 09:57:42 +0700 karibu@freedif.org wrote:
Great discussion, I am hosting a mirror for Vietnam (But not big traffic, I had quite a lot when I was helping Tails) On the privacy, i realized that my stats https://mirror.freedif.org/Stats/TorProject.html included the IPs as well of who downloaded Tor.....It was not my intention, but I treated Tor mirroring like any other mirroring project... I had to do some work around to remove those IPs, and may be some others people have the same challenge.
It would be great to clarify the position of Tor and guidelines for mirroring for example.
I agree we should add something, thanks for pointint this out. For apache you can use the libapache2-mod-removeip. For nginx AFAIK there's no such module. Found this example to modify log_format https://www.hagen-bauer.de/2015/12/nginx-logfiles-anon.html in german. If you know about better instructions in english, let us know! -- traumschule.org gpg fingerprint: 9356 4DED 8546 8D9A C290 3605 12EE 7D70 7111 2056 /otr info OTR: traumschule@irc.indymedia.org fingerprint: OTR: 35AACA83 4564616C B6EBEC66 56B6B2FC C8D572F1 OTR: traumschule@irc.oftc.net fingerprint: OTR: D1CCD207 B60C1866 56A975AE ACE090E9 45E90846 OTR: traumschule@chat.freenode.net fingerprint: OTR: 51BF8BB9 434840CC 24F264BC 76450C27 A6AADB12
On 2018-10-23 23:21, Traumschule wrote:
For apache you can use the libapache2-mod-removeip. For nginx AFAIK there's no such module. Found this example to modify log_format https://www.hagen-bauer.de/2015/12/nginx-logfiles-anon.html in german. If you know about better instructions in english, let us know!
nginx logging is documented here: https://nginx.org/en/docs/http/ngx_http_log_module.html The default log format is given as: log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"'; So if you wanted to suppress IP and user information, while staying compatible with the Common Log Format, and drop the HTTP Referer and User-Agent fields, you could add these lines to your nginx.conf: log_format privacy '- - - [$time_local] "$request" $status $body_bytes_sent'; access_log /var/log/nginx/access.log privacy; You may need to change the second line if your nginx log files are not in /var/log/nginx. Kind regards, Alexander
Hi all, You can surely use mods if need be, but for Apache2, I've found it's easiest just to use the CustomLog directive: CustomLog /path/to/desired/log/location "0.0.0.0 %l %u %t \"%r\" %>s %b" This just makes every user show up as 0.0.0.0. Which makes it plug and play with something like AWstats, or any other log-analysis software. Simple and fast. Works for us. Adam -- Adam Quenneville Founder, The Free Mirror Project <https://FreeMirror.org/> /"Enabling the free exchange of information and ideas."/ On 2018-10-23 6:13 PM, Alexander Dietrich wrote:
On 2018-10-23 23:21, Traumschule wrote:
For apache you can use the libapache2-mod-removeip. For nginx AFAIK there's no such module. Found this example to modify log_format https://www.hagen-bauer.de/2015/12/nginx-logfiles-anon.html in german. If you know about better instructions in english, let us know!
nginx logging is documented here: https://nginx.org/en/docs/http/ngx_http_log_module.html
The default log format is given as:
log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';
So if you wanted to suppress IP and user information, while staying compatible with the Common Log Format, and drop the HTTP Referer and User-Agent fields, you could add these lines to your nginx.conf:
log_format privacy '- - - [$time_local] "$request" $status $body_bytes_sent'; access_log /var/log/nginx/access.log privacy;
You may need to change the second line if your nginx log files are not in /var/log/nginx.
Kind regards, Alexander _______________________________________________ tor-mirrors mailing list tor-mirrors@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-mirrors
Thanks a lot for your input! Please review this commit and correct if necessary: https://github.com/torproject/webwml/pull/67 On Tue, 23 Oct 2018 18:43:26 -0400 Adam Quenneville <adam.quenneville@freemirror.org> wrote:
Hi all,
You can surely use mods if need be, but for Apache2, I've found it's easiest just to use the CustomLog directive:
CustomLog /path/to/desired/log/location "0.0.0.0 %l %u %t \"%r\" %>s %b"
This just makes every user show up as 0.0.0.0. Which makes it plug and play with something like AWstats, or any other log-analysis software.
Simple and fast. Works for us.
Adam
On 2018-10-23 6:13 PM, Alexander Dietrich wrote:
nginx logging is documented here: https://nginx.org/en/docs/http/ngx_http_log_module.html
The default log format is given as:
log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';
So if you wanted to suppress IP and user information, while staying compatible with the Common Log Format, and drop the HTTP Referer and User-Agent fields, you could add these lines to your nginx.conf:
log_format privacy '- - - [$time_local] "$request" $status $body_bytes_sent'; access_log /var/log/nginx/access.log privacy;
You may need to change the second line if your nginx log files are not in /var/log/nginx.
Kind regards, Alexander
-- traumschule.org gpg fingerprint: 9356 4DED 8546 8D9A C290 3605 12EE 7D70 7111 2056 /otr info OTR: traumschule@irc.indymedia.org fingerprint: OTR: 35AACA83 4564616C B6EBEC66 56B6B2FC C8D572F1 OTR: traumschule@irc.oftc.net fingerprint: OTR: D1CCD207 B60C1866 56A975AE ACE090E9 45E90846 OTR: traumschule@chat.freenode.net fingerprint: OTR: 51BF8BB9 434840CC 24F264BC 76450C27 A6AADB12
participants (7)
-
Adam Quenneville -
Alexander Dietrich -
Dave Warren -
karibu＠freedif.org -
Niklas Edmundsson -
Stephen Loeckle -
Traumschule