new ansible-tor features: automatic instance configuration + automatic MyFamily generation (PATCH)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi David, thanks for creating ansible-tor. I added two features that are crucial to me and maybe useful for others as well. If you like it, feel free to merge - this is my first ansible experience and it is lightly tested. Example: lets say you have added a new server to your inventory. The server has 3 public IP addresses (1.1.1.1, 2.2.2.2, 3.3.3.3). After running ansible-tor with the new changes you will have the following 6 tor instances/ORPorts running (without manually specifying IP addresses first): 1.1.1.1:80 1.1.1.1:443 2.2.2.2:80 2.2.2.2:443 3.3.3.3:80 3.3.3.3:443 including MyFamily configuration across all servers/instances. regards, Nusenu changes ======= - - auto instance deployment without manual IP/ORPort configuration (new) starts 2 tor instances per available IP address by default makes manually specifying IP addresses and ORPorts via proc_instances obsolete ORPorts default to 80 and 443 (DirPort not added yet) replace "single.yml" + "instances.yml" -> instance.yml only (handles both cases dynamically) - - MyFamily autogeneration (new) Keeping all relay fingerprints in sync is probably one of the most annoying tasks for a relay operator managing multiple relays, now ansible takes care of this (all relays need to be in the 'relays' group) - - directory structure (changed) defaults: configs -> /etc/tor/<ip>_<orport>.torrc log dir -> /var/log/tor/<ip>_<orport>.log datadir -> /var/lib/tor/<ip>_<orport>/ pid dir -> /var/run/tor/<ip>_<orport>.pid (previously everything was located in /etc) - - added torrc sanity check (tor --verify-config ) (new) - - torrc files are owned by root (previously owned by $tor_user) - - the pid file check has been removed since the file is not required to exist (it will be created when tor starts) open - ----- - - it does not take care of instance removals yet (in case IPs are no longer available or amount of ORPorts have been reduced) - - allow opt-out -> only 1 tor instance per host (even if there are more IPs available) - - DirPort support - - detect RFC1918 IPs (opt-in) -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJU4i+CAAoJEFv7XvVCELh0y+kP/i4Mn/XClgXYloGdgWU9UPR+ Y8yZv97FvJOMPI40tccPKcNPcLQFRvGFYkR96sAOGoMfbJT/tQeH2dOxwAEF31mv afFkLsVPAOpNzlyO2qP1mkLtB/aYXtZ6jb2+JtpAhVBLKOVFBN2nNRiwdgFYZFGy f0ZIp7xyR9XcAhXo4nc0hlETREAnbMOgFGM6vqqIpJfimF3liE6va5HNw2CD+7Zd MmeIOuVNvQh09SiYf48AJpBeBRoybOvmFIPphtXEYlC/y6cd/IyUIYdOBuaLa5td vQnrQOC7TUgp74uarl0yaatOYOEagl0lrNeN6+Vgy5e0e12TgVccWW5ZosM1PBXG VH2FTfjHXUO+VN0p4xn6AS0dhWTRKb7isj3jpznTMsiq0AcvXM6DZjkzkcCPChVz jptdUbNvgpdP7j5X11iZniGpxVe7aFo2wCzgZORY1xMysiigJsL4M/nonr4YO4G9 w7kyNcco9gStklJSvOJXbfX4HrOCuWdq8hp4xubyON+5jpEUgMmG1o/v5NJANV4C CLzlz4kf9l9o351Z7DJQzilxzDEwe6oZwSWnsq+yB65Mgj5sUJnchi40iPLOHSUr DaVSSUxoZ8VVNYqqvGYb2fysYa7DsCgofsF/eXP4QyJp1WFNwc0ft6qIhyAGIDwx RfwQHrA+Lg95mdXDyr0B =QHkD -----END PGP SIGNATURE-----
Hi Nusenu, Thanks for the patch. You've added quite a bit more features than 2. Would you mind telling me which 2 features are critical for your use-case and why? Can you share your ansible-tor playbook? Perhaps a redacted copy if you have sensitive information in it... I'd like for this ansible role to be useful to relay operators like yourself... so I'm very interested in learning about how you'd like to use it. Why do you think the ORPorts should default to 80 and 443? Are you operating an exit relay? This is a good idea -> added torrc sanity check (tor --verify-config ) I think your auto tor instance deployment feature should be an optional feature that is off by default. The collecting fingerprints idea seems great for the myfamily torrc option is definitely a good idea. If using configure_apt_single.yml then the torrc is in fact owned by root... and tor will then drop prives. The other way tor is deployed with this role is using the configure_tor_instance.yml... and i suppose the individual torrc files could be owned as root as long as they are readable by the tor user. But does this matter? What are the implications? I'd be much more likely to merge your patches if they were one feature per patch... instead of this monolithic patch with many features. Furthermore... I hate centralized media and all but github sure would make patch submission and review easier. I definitely do not need any gpg signature on patches submitted. Your code will be reviewed no matter who has signed your key. =-p Sincerely, David On Mon, Feb 16, 2015 at 5:57 PM, Nusenu <nusenu@openmailbox.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Hi David,
thanks for creating ansible-tor. I added two features that are crucial to me and maybe useful for others as well. If you like it, feel free to merge - this is my first ansible experience and it is lightly tested.
Example: lets say you have added a new server to your inventory. The server has 3 public IP addresses (1.1.1.1, 2.2.2.2, 3.3.3.3). After running ansible-tor with the new changes you will have the following 6 tor instances/ORPorts running (without manually specifying IP addresses first):
1.1.1.1:80 1.1.1.1:443 2.2.2.2:80 2.2.2.2:443 3.3.3.3:80 3.3.3.3:443
including MyFamily configuration across all servers/instances.
regards, Nusenu
changes =======
- - auto instance deployment without manual IP/ORPort configuration (new) starts 2 tor instances per available IP address by default makes manually specifying IP addresses and ORPorts via proc_instances obsolete ORPorts default to 80 and 443 (DirPort not added yet) replace "single.yml" + "instances.yml" -> instance.yml only (handles both cases dynamically)
- - MyFamily autogeneration (new) Keeping all relay fingerprints in sync is probably one of the most annoying tasks for a relay operator managing multiple relays, now ansible takes care of this (all relays need to be in the 'relays' group)
- - directory structure (changed) defaults: configs -> /etc/tor/<ip>_<orport>.torrc log dir -> /var/log/tor/<ip>_<orport>.log datadir -> /var/lib/tor/<ip>_<orport>/ pid dir -> /var/run/tor/<ip>_<orport>.pid
(previously everything was located in /etc)
- - added torrc sanity check (tor --verify-config ) (new)
- - torrc files are owned by root (previously owned by $tor_user)
- - the pid file check has been removed since the file is not required to exist (it will be created when tor starts)
open - ----- - - it does not take care of instance removals yet (in case IPs are no longer available or amount of ORPorts have been reduced) - - allow opt-out -> only 1 tor instance per host (even if there are more IPs available) - - DirPort support - - detect RFC1918 IPs (opt-in) -----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJU4i+CAAoJEFv7XvVCELh0y+kP/i4Mn/XClgXYloGdgWU9UPR+ Y8yZv97FvJOMPI40tccPKcNPcLQFRvGFYkR96sAOGoMfbJT/tQeH2dOxwAEF31mv afFkLsVPAOpNzlyO2qP1mkLtB/aYXtZ6jb2+JtpAhVBLKOVFBN2nNRiwdgFYZFGy f0ZIp7xyR9XcAhXo4nc0hlETREAnbMOgFGM6vqqIpJfimF3liE6va5HNw2CD+7Zd MmeIOuVNvQh09SiYf48AJpBeBRoybOvmFIPphtXEYlC/y6cd/IyUIYdOBuaLa5td vQnrQOC7TUgp74uarl0yaatOYOEagl0lrNeN6+Vgy5e0e12TgVccWW5ZosM1PBXG VH2FTfjHXUO+VN0p4xn6AS0dhWTRKb7isj3jpznTMsiq0AcvXM6DZjkzkcCPChVz jptdUbNvgpdP7j5X11iZniGpxVe7aFo2wCzgZORY1xMysiigJsL4M/nonr4YO4G9 w7kyNcco9gStklJSvOJXbfX4HrOCuWdq8hp4xubyON+5jpEUgMmG1o/v5NJANV4C CLzlz4kf9l9o351Z7DJQzilxzDEwe6oZwSWnsq+yB65Mgj5sUJnchi40iPLOHSUr DaVSSUxoZ8VVNYqqvGYb2fysYa7DsCgofsF/eXP4QyJp1WFNwc0ft6qIhyAGIDwx RfwQHrA+Lg95mdXDyr0B =QHkD -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi David, thanks for your quick feedback.
Would you mind telling me which 2 features are critical for your use-case and why?
- - automatic instance deployment (and all the dependencies that comes with that, like ORListenAddress - without it tor0 would block tor2 from starting since they are binding on the same port) - - automatic MyFamily management this is just too annoying to manage manually - - the directory layout change is more cosmetic but your current way (everything in /etc) is rather unusual and as an example would require custom logrotate configuration that wouldn't be required otherwise
Can you share your ansible-tor playbook?
Since everything tor process specific is handled in ansible-tor, my playbook will mainly just references the role.
I'd like for this ansible role to be useful to relay operators like yourself... so I'm very interested in learning about how you'd like to use it.
- From the examples I assumed you are probably not using it mainly for relays, is that correct?
Why do you think the ORPorts should default to 80 and 443?
It is assumed that these well known ports are less likely blocked by outbound firewalls and therefore more useful for clients than on some highports. Once dirport support is added, I would use 80 as dirport. For an example of common ORPorts see https://atlas.torproject.org/#search/contact:torservers ..but since it is very easy to modify the defaults, I've no strong feeling what is actually defined in default/main.yml. One could also add some auto detection to see if the ports are already in use..
I think your auto tor instance deployment feature should be an optional feature that is off by default.
Yes that is fine, configuration changes are easy enough.
If using configure_apt_single.yml then the torrc is in fact owned by root... and tor will then drop prives. The other way tor is deployed with this role is using the configure_tor_instance.yml... and i suppose the individual torrc files could be owned as root as long as they are readable by the tor user. But does this matter? What are the implications?
On a default install they are owned by root, I just reverted the change from owner=tor_user to owner=root to restore defaults. Implication.. tor_user will not be able to rewrite/manipulate its own configuration.
I'd be much more likely to merge your patches if they were one feature per patch... instead of this monolithic patch with many features.
Yes, that is what I expected, but then I thought that the two main changes code wise (autoconfig + directory structure) are dependent on each other anyway. Merging autoconfig without the directory restructuring (or vice versa) wouldn't be much fun since these modifications always touch overlapping areas. If you want to add it as additional option, including it as a separate yml in tasks/main.yml + separate torrc is also a possibility - but probably not the nicest way (duplicate code, multiple torrc's).
Furthermore... I hate centralized media and all but github sure would make patch submission and review easier.
Yes, I'm considering it if this becomes something reoccurring. thanks, Nusenu -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJU4mGiAAoJEFv7XvVCELh0QMAQAIMmbJfJ1cLBdNXeRi+x9CLo LwYbFCP1FkMJHN6wfb/Q5wHejq88WIgUxUeob4KMVl30mCVxGIpOYZDb3rsvB3mi lIzFiZUt/eJo40Gjdx4/BgEArEaAsmOix2O8Ev6E84bl9J/zcp8EIFPcYhtrERJ/ zoV8iHeeJnuGGYa2zTDHS2ONtJ4lE7K/DX9NmVj45jmaHOKX8Y5o1klHSnNZ7yAR qGDRqvWlVeDwVRq+xXAyn1GK43SZPydNxZkHjMPmvKOAVGgIXgnxgtpmzj+87j/W UBB6waNmVfMJvCPreRM7Zgc2rhUAZx1TsIy78v3g7rWns0rd6hzm4lzzaP5aBhH0 8fK++UQMwPz+o7fIyKfiJLmP6TxxGk8AZpcrLXuCuDMfGi56/kHVp9P54xJSbYqV YNjVgVelAyZ62YB3tx3J+CWtOxbn3dHT0yllXDOQes6+/x0h7igod/2OWZBG6O6y SCEO2GPqMHx5ZLxvsCjio1FG1oPcMAT4lNOtNPDNxJPX5J8Vwkq4wmIDlj+zJM+m R8dOcqJ8sKtQx98rADlachXwFEFne9VEvqTj9oyq0O4szPx/jM9QAVhvAjlWqCM/ hJ+opaH/xhLs2Vu+UGqUosU23AO1p5fB+/KA4cY4CrrqJ1Inn687i3OWm6RC3cRi kg52xwIdT354yq6xpMCu =qFmI -----END PGP SIGNATURE-----
responding inline
Would you mind telling me which 2 features are critical for your use-case and why?
- - automatic instance deployment (and all the dependencies that comes with that, like ORListenAddress - without it tor0 would block tor2 from starting since they are binding on the same port)
- - automatic MyFamily management this is just too annoying to manage manually
OK. I'd like for this feature to co-exist with the current configure_tor_instance.yml... because other entities are currently using that... including Mozilla.
- - the directory layout change is more cosmetic but your current way (everything in /etc) is rather unusual and as an example would require custom logrotate configuration that wouldn't be required otherwise
OK... I don't have a strong opinion... and I think the parent directory for all this should be configuration via a role variable so that the user can specify.
I'd like for this ansible role to be useful to relay operators like yourself... so I'm very interested in learning about how you'd like to use it.
- From the examples I assumed you are probably not using it mainly for relays, is that correct?
Yes that is correct. I operate many Tor hidden services. However I initially created this Ansible role to help Moritz of torservers.net and those people that may be working for him; therefore pull requests and feedback helps; for instance Moritz specified several features it should have... and an engineer working for Mozilla chatted with me about the features they needed; then he sent me a pull request on github.
One could also add some auto detection to see if the ports are already in use..
I think the sys admin should just know what they are doing; and should know which ports are available.
If using configure_apt_single.yml then the torrc is in fact owned by root... and tor will then drop prives. The other way tor is deployed with this role is using the configure_tor_instance.yml... and i suppose the individual torrc files could be owned as root as long as they are readable by the tor user. But does this matter? What are the implications?
On a default install they are owned by root, I just reverted the change from owner=tor_user to owner=root to restore defaults. Implication.. tor_user will not be able to rewrite/manipulate its own configuration.
Yes I agree.
I'd be much more likely to merge your patches if they were one feature per patch... instead of this monolithic patch with many features.
Yes, that is what I expected, but then I thought that the two main changes code wise (autoconfig + directory structure) are dependent on each other anyway. Merging autoconfig without the directory restructuring (or vice versa) wouldn't be much fun since these modifications always touch overlapping areas. If you want to add it as additional option, including it as a separate yml in tasks/main.yml + separate torrc is also a possibility - but probably not the nicest way (duplicate code, multiple torrc's).
OK... I agree with you... but let's make this a seperate yml task file; your use is quite different than most of the entities currently using this ansible role. So let's add these as a new task file instead of modifying the existing task file.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, For reference, this thread started here [2]. Arzhel and Moritz, could you comment on whether you prefer to manually create/specify tor instances (via 'proc_instances') as you do here [1], or whether you would make use of an instance auto configuration (two instances per IP - see example here [2])? If you prefer manual configuration: Do your manual steps follow any specific design that could be automated as well or are these steps unpredictable? :) [1] https://github.com/XioNoX/moz-tor-relays/blob/master/host_vars/tor-relay1 [2] https://lists.torproject.org/pipermail/tor-relays/2015-February/006414.html David Stainton wrote:
- - the directory layout change is more cosmetic but your current way (everything in /etc) is rather unusual and as an example would require custom logrotate configuration that wouldn't be required otherwise
OK... I don't have a strong opinion... and I think the parent directory for all this should be configuration via a role variable so that the user can specify.
The user is free to specify the vars in a flexible way. Defaults in the patch are: tor_ConfDir: /etc/tor tor_PidDir: /var/run/tor tor_LogDir: /var/log/tor tor_DataDir: /var/lib/tor
However I initially created this Ansible role to help Moritz of torservers.net and those people that may be working for him; therefore pull requests and feedback helps; for instance Moritz specified several features it should have... and an engineer working for Mozilla chatted with me about the features they needed; then he sent me a pull request on github.
I'm surprised that Moritz didn't ask for automatic MyFamily generation ;)
I'd be much more likely to merge your patches if they were one feature per patch... instead of this monolithic patch with many features.
Yes, that is what I expected, but then I thought that the two main changes code wise (autoconfig + directory structure) are dependent on each other anyway. Merging autoconfig without the directory restructuring (or vice versa) wouldn't be much fun since these modifications always touch overlapping areas. If you want to add it as additional option, including it as a separate yml in tasks/main.yml + separate torrc is also a possibility - but probably not the nicest way (duplicate code, multiple torrc's).
OK... I agree with you... but let's make this a seperate yml task file; your use is quite different than most of the entities currently using this ansible role. So let's add these as a new task file instead of modifying the existing task file.
That is fine with me. regards, Nusenu -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJU43p9AAoJEFv7XvVCELh0slEP/ijRiS3LTdKMyhZU7RVj9Gh9 ixkjt6Kc32naohTXGpRxDB2V9fAMex9z01PnX/1zZVc+9bOcJaCHcVHFm8gEgm5j YDCMcggwbAwBKJHvuD1MtQeVP50LGJE40Fl7IrtpLvsu5d7z6jEl+uNYxbqLVHVQ 5dM0XOfZx7miIFcJDdTe4EoBVxHSCewBLBOmJ9N831q+HTaGdRni4FpGdeiQT/G9 m6Ny3P6qNusBuEBxV4tNLapxTKmq1ntzoWG693SWCXhsVePT4+5kuOCyiBT1Zlmn ghdK+5BCWy5rSsEg0SzIxShcPP62BWiGPqexKzpYOG+JpHEEWScBETJ/r1jGy4M9 tcEA7p/UFBRJ8EVDp5cy+ZuTxfNLdZ4Cq6CdfBUSpLjFbJNJKPnyc2nKEAzLmyW+ XMOIvXNZgjcbBkKl4O5rNzpKvp+APzXtAtbVl6TcvnSm7FErEeKV5FsGN96ae4YM T8bv5WKsYtEo1ofc9kf1V9VYoXcMmsNqH5TZau2vXnke9a0e20YXcW0TXUXlGkiq uLukfbVvl+3FfnaNgEiRsv7N3cVPp9BFsJSAOo061vKSCBqBbNlTtUS1ozHRSeS8 CetkzTZvOzo2qVMEGk21kNyneH8JlfIEOZpEAHPDmQShP4wvONmpbWBH+BeYHVEX S89m257Cp1S2PG8X6Py4 =1MDB -----END PGP SIGNATURE-----
Howdy, I didn't mean to imply that Moritz or any torservers.net people are using my Ansible role... I have no idea how they maintain their tor relays... I just meant to say that when I was initially starting to write this ansible role I asked about what sort of features are needed. I think your features additions here are really excellent... and obviously much less hassle to use... which is why we should add them! Nusenu, I will be in Valencia, Spain for the tor-dev meeting... if you are going to be there too then I hope we can meet up and collaborate in person. Cheers, David On Tue, Feb 17, 2015 at 5:29 PM, Nusenu <nusenu@openmailbox.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Hi,
For reference, this thread started here [2].
Arzhel and Moritz, could you comment on whether you prefer to manually create/specify tor instances (via 'proc_instances') as you do here [1], or whether you would make use of an instance auto configuration (two instances per IP - see example here [2])?
If you prefer manual configuration: Do your manual steps follow any specific design that could be automated as well or are these steps unpredictable? :)
[1] https://github.com/XioNoX/moz-tor-relays/blob/master/host_vars/tor-relay1 [2] https://lists.torproject.org/pipermail/tor-relays/2015-February/006414.html
David Stainton wrote:
- - the directory layout change is more cosmetic but your current way (everything in /etc) is rather unusual and as an example would require custom logrotate configuration that wouldn't be required otherwise
OK... I don't have a strong opinion... and I think the parent directory for all this should be configuration via a role variable so that the user can specify.
The user is free to specify the vars in a flexible way. Defaults in the patch are: tor_ConfDir: /etc/tor tor_PidDir: /var/run/tor tor_LogDir: /var/log/tor tor_DataDir: /var/lib/tor
However I initially created this Ansible role to help Moritz of torservers.net and those people that may be working for him; therefore pull requests and feedback helps; for instance Moritz specified several features it should have... and an engineer working for Mozilla chatted with me about the features they needed; then he sent me a pull request on github.
I'm surprised that Moritz didn't ask for automatic MyFamily generation ;)
I'd be much more likely to merge your patches if they were one feature per patch... instead of this monolithic patch with many features.
Yes, that is what I expected, but then I thought that the two main changes code wise (autoconfig + directory structure) are dependent on each other anyway. Merging autoconfig without the directory restructuring (or vice versa) wouldn't be much fun since these modifications always touch overlapping areas. If you want to add it as additional option, including it as a separate yml in tasks/main.yml + separate torrc is also a possibility - but probably not the nicest way (duplicate code, multiple torrc's).
OK... I agree with you... but let's make this a seperate yml task file; your use is quite different than most of the entities currently using this ansible role. So let's add these as a new task file instead of modifying the existing task file.
That is fine with me.
regards, Nusenu -----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJU43p9AAoJEFv7XvVCELh0slEP/ijRiS3LTdKMyhZU7RVj9Gh9 ixkjt6Kc32naohTXGpRxDB2V9fAMex9z01PnX/1zZVc+9bOcJaCHcVHFm8gEgm5j YDCMcggwbAwBKJHvuD1MtQeVP50LGJE40Fl7IrtpLvsu5d7z6jEl+uNYxbqLVHVQ 5dM0XOfZx7miIFcJDdTe4EoBVxHSCewBLBOmJ9N831q+HTaGdRni4FpGdeiQT/G9 m6Ny3P6qNusBuEBxV4tNLapxTKmq1ntzoWG693SWCXhsVePT4+5kuOCyiBT1Zlmn ghdK+5BCWy5rSsEg0SzIxShcPP62BWiGPqexKzpYOG+JpHEEWScBETJ/r1jGy4M9 tcEA7p/UFBRJ8EVDp5cy+ZuTxfNLdZ4Cq6CdfBUSpLjFbJNJKPnyc2nKEAzLmyW+ XMOIvXNZgjcbBkKl4O5rNzpKvp+APzXtAtbVl6TcvnSm7FErEeKV5FsGN96ae4YM T8bv5WKsYtEo1ofc9kf1V9VYoXcMmsNqH5TZau2vXnke9a0e20YXcW0TXUXlGkiq uLukfbVvl+3FfnaNgEiRsv7N3cVPp9BFsJSAOo061vKSCBqBbNlTtUS1ozHRSeS8 CetkzTZvOzo2qVMEGk21kNyneH8JlfIEOZpEAHPDmQShP4wvONmpbWBH+BeYHVEX S89m257Cp1S2PG8X6Py4 =1MDB -----END PGP SIGNATURE-----
On 02/17/2015 08:15 PM, David Stainton wrote:
I didn't mean to imply that Moritz or any torservers.net people are using my Ansible role... I have no idea how they maintain their tor relays...
The sad story is in a draft email to tor-relays. We barely maintain anything these days. This hasn't changed at all since we last talked. I not only need ansible (or whatever), I need a person (or many) to take over. Apart from writing email, I am out.
Nusenu, I will be in Valencia, Spain for the tor-dev meeting
Excellent. Let's definitely continue this there. -- Moritz Bartl https://www.torservers.net/
On 02/17/2015 06:29 PM, Nusenu wrote:
Arzhel and Moritz, could you comment on whether you prefer to manually create/specify tor instances (via 'proc_instances') as you do here [1], or whether you would make use of an instance auto configuration (two instances per IP - see example here [2])?
Automate away please!! :-) In my ideal world, someone would write a "wizard" as wrapper around the ansible stuff (or whatever really), so people who run it can either specify attributes in a clear CLI language that feels "more native" to the domain, or answer questions asked by the wizard. The knowledge spectrum is "Moritz wants to use this", and "Uh. I have zero knowledge of computers, but I want to run a Tor relay. I have this cheap $5 VPS, please turn it into something useful. I don't know if it should be a bridge, or a relay. I don't even really want to know.".
If you prefer manual configuration: Do your manual steps follow any specific design that could be automated as well or are these steps unpredictable? :)
My algorithm. Hm. This all sucks because Tor still doesn't scale across multiple cores. I understand that many people want "fast relays" (ie. processes as fast as possible) merely for psychological reasons. People look at the top #10 fastest relays, and from a marketing perspective, we should make sure we have all top #10 spots (phrased differently: People want to see their relay(s) at the top!). At the beginning of Torservers.net, when torstatus was still the thing to use, I took screenshots from the top #10-20 and marked in red which relays were "ours". This brought us a lot of positive attention and a lot of donations. We still lack the "gamification" Relay Challenge website that Virgil was talking about. It would just sum up all relays of a family, and then it really does not matter any more. Until then, I think the perfect automation would take that into account. This is roughly my "algorithm": Traffic limit? Line speed? => Calculate mean total speed across a month. I like relays that are up for at least 2 weeks per month. Take RELAY_REQUIRED_MIN_BANDWIDTH/BRIDGE_REQUIRED_MIN_BANDWIDTH into account. ( https://trac.torproject.org/projects/tor/ticket/13822 ) A modern CPU without AES-NI can do ~100 Mbit/s per core, with AES-NI up to 400 Mbit/s. => from that, derive lower limit of how many relay processes we need. Then, monitor closely to see if the CPU (core) can actually handle the amount of traffic. If a core reaches 90% use, split off another relay process. If we have more IPs, I try to spread the processes across IPs and various well-known ports. I keep some IPs clean and unused as " backup". Ideally, you don't want to expose SSH at all (portknock, whitelist allowed source IPs etc), but if there's enough IPs you can at least put it on a separate IP. In practice and for simplicity, I wouldn't mind if you completely ignore the "make processes handle as much traffic as possible". The default should be "as automated as possible, even if suboptimal". As long as you expose the "expert options for tuning", that's totally fine with me. :-) Does this help? Moritz
[1] https://github.com/XioNoX/moz-tor-relays/blob/master/host_vars/tor-relay1 [2] https://lists.torproject.org/pipermail/tor-relays/2015-February/006414.html
David Stainton wrote:
- - the directory layout change is more cosmetic but your current way (everything in /etc) is rather unusual and as an example would require custom logrotate configuration that wouldn't be required otherwise
OK... I don't have a strong opinion... and I think the parent directory for all this should be configuration via a role variable so that the user can specify.
The user is free to specify the vars in a flexible way. Defaults in the patch are: tor_ConfDir: /etc/tor tor_PidDir: /var/run/tor tor_LogDir: /var/log/tor tor_DataDir: /var/lib/tor
However I initially created this Ansible role to help Moritz of torservers.net and those people that may be working for him; therefore pull requests and feedback helps; for instance Moritz specified several features it should have... and an engineer working for Mozilla chatted with me about the features they needed; then he sent me a pull request on github.
I'm surprised that Moritz didn't ask for automatic MyFamily generation ;)
I'd be much more likely to merge your patches if they were one feature per patch... instead of this monolithic patch with many features.
Yes, that is what I expected, but then I thought that the two main changes code wise (autoconfig + directory structure) are dependent on each other anyway. Merging autoconfig without the directory restructuring (or vice versa) wouldn't be much fun since these modifications always touch overlapping areas. If you want to add it as additional option, including it as a separate yml in tasks/main.yml + separate torrc is also a possibility - but probably not the nicest way (duplicate code, multiple torrc's).
OK... I agree with you... but let's make this a seperate yml task file; your use is quite different than most of the entities currently using this ansible role. So let's add these as a new task file instead of modifying the existing task file.
That is fine with me.
regards, Nusenu
-- Moritz Bartl https://www.torservers.net/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Moritz Bartl:
On 02/17/2015 06:29 PM, Nusenu wrote:
Arzhel and Moritz, could you comment on whether you prefer to manually create/specify tor instances (via 'proc_instances') as you do here [1], or whether you would make use of an instance auto configuration (two instances per IP - see example here [2])? Automate away please!! :-)
I somehow expected it - great (this is basically done ;)
In my ideal world, someone would write a "wizard" as wrapper around the ansible stuff (or whatever really), so people who run it can either specify attributes in a clear CLI language that feels "more native" to the domain, or answer questions asked by the wizard. The knowledge spectrum is "Moritz wants to use this", and "Uh. I have zero knowledge of computers, but I want to run a Tor relay. I have this cheap $5 VPS, please turn it into something useful. I don't know if it should be a bridge, or a relay. I don't even really want to know.".
Understood.
In practice and for simplicity, I wouldn't mind if you completely ignore the "make processes handle as much traffic as possible". The default should be "as automated as possible, even if suboptimal".
Agreed :) I'm wondering if simply (blindly) running two tor processes per available IP is any worse than anything else from a pure "lets push as much traffic as we can" (on a host not on a tor process level).
As long as you expose the "expert options for tuning", that's totally fine with me. :-)
Does this help?
Thanks for your answer. -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJU47/UAAoJEFv7XvVCELh0jVkP+QHDYwica6a2GLKhgkuapqOc NmxMMD5vRmoyx51FyUbtVUaQ8sYHeNQHe+axVIgGMWepsH5puIfQT4aH1g18VXle h7DuU+/QYaAij+lHxrGU7UJhaEXZAi6OJ0NxNd67ivf+TBCJKYkybFBUgwBeHgVW 0RYhK24xT9yzeTEOd1C6ppJaIXAI/WzJdpnoQZADYiMfra34rzo8QD9xO/EW+QXw LT8vIegT3NIdc6cDFHnB0+sWiHEn26N8PrHiRnq2FsvbclJqg0fePnlj900ZynF1 YBxXSGQuzJ5Z8K/q+5jj8Q8DikXm45QH55kdqQStD7id9Es2j7ygGb1ywKSW6FjU edrgQUskJuFv0YwuY/pwxyJLuB7dpm4OT4cZNQrEzo3aFM0lPWthfXkQsvrZXItP RCorU6Y3Dwczqn21EEWE1w+m2oit5+Wdm7n3qE/sYI78LB5780UKrvR16n7ujGii QsX8dTei+F6wgvgIsdPfIxycnRL6uKsoUCDTHMGhiSJW8v6ZbxKDVlw5uf+Ex7KQ FX6Ux6q8dgHFZQGGk1q0ImvvrIU5KCUfRWlIOdZgdSlieyhSkur5sTeFDuXxnF9N 2t+Jo4iJPulZ1vd7WOPlO6N2zyQCG4oOO3FxCp8D0iQV4h1tPo/nZBVCDGcCobxR r+VSy8W3vy5+01fBfFuT =PEcd -----END PGP SIGNATURE-----
On 02/17/2015 11:25 PM, Nusenu wrote:
I'm wondering if simply (blindly) running two tor processes per available IP is any worse than anything else from a pure "lets push as much traffic as we can" (on a host not on a tor process level).
I don't see a problem with that either, unless the destination is CPU or memory constrained. We have (and had) servers with 16 IPs -- Dual Core, 2-4GB RAM, 100 Mbit/s and less. Maybe not too nice to spin up 32 Tor relays on such machines? But I think in general it is fair to go with "2 per IP", and expect the server to be configured to listen on a sane number of IPs. -- Moritz Bartl https://www.torservers.net/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Moritz Bartl:
On 02/17/2015 11:25 PM, Nusenu wrote:
I'm wondering if simply (blindly) running two tor processes per available IP is any worse than anything else from a pure "lets push as much traffic as we can" (on a host not on a tor process level).
I don't see a problem with that either, unless the destination is CPU or memory constrained.
We have (and had) servers with 16 IPs -- Dual Core, 2-4GB RAM, 100 Mbit/s and less. Maybe not too nice to spin up 32 Tor relays on such machines? But I think in general it is fair to go with "2 per IP", and expect the server to be configured to listen on a sane number of IPs.
I'm wondering why a relay (not bridge) op would order that many IP addresses on a host were he can not make use of it, but adding a simple check like "stop creating new instances once RAM/instancecount below X" is possible (or "ok lets use the remaining IPs for bridges" ;). -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJU48UzAAoJEFv7XvVCELh0lH0P/2OtKcLJnlQJXZ4tt1eBKFKY FMTpVONNmpPp2S4wQpDNPLT0xTd9UpjHdt70ZZ9UVnmOMaMdvmosQSHbLgsi+9EG hsb5f2PsoJR/GFXEoVM9kdRuACYDcWcNr8I0UnrH4S8Cqyho+UV6u4+cwOtJrekN H4Ni3hHkk1KiDhEXJB3KyWboIIvF3eQOjU9cuwfC8WKN+yuSdQ8pJJhC5fGuOEzb /SPC9j0QmgbHRqUrrkQS2wNcFVTJ8N6hQ8rBNZgK3enEk/jFC0c3EM4SfmpHolWb w//bHJ9Cso4KvGyJ8/CVooFUfvnx6bHlWzWMPGjpYYhhpXx5bIGwqxqv+jdtvMkF 9BHdyy7vvOStcqaTHHE6xORsEOY0FMK4b2Y3QieKraMRpbCAhsbr5FQHQ+s8CG2P fx2hMBBZu+p1ktKvpWrHy94CaBqDf4kNTXztjuf/Z8BpU1Bb9BrN5IUNXcOu4u1n ryjAnj0fh7PQIafF5R9zSZKGNYHAGtXStbD4rgSBOHhBBPUejg3oW2xs6B3s5wEk CRciJizSA5dV9hGFFsOfRDFDSxSKLmc/iVOaKwbPlanV9wnKuxsq74a+mdpWpUPF AK53H/5cNtD+ofdk5gcX4bR2t3h38G0/Sh6u7IikTShocU9+5GGsfnkzZ4alLc1s VzRPf3mUq2HNT3CPyjSE =3MZ0 -----END PGP SIGNATURE-----
On 02/17/2015 11:48 PM, Nusenu wrote:
I'm wondering why a relay (not bridge) op would order that many IP addresses on a host were he can not make use of it, but adding a simple check like "stop creating new instances once RAM/instancecount below X" is possible (or "ok lets use the remaining IPs for bridges" ;).
For exit relays, you want WHOIS reassignment. Many providers only offer that for larger IP ranges, and then by default assign all of them to the machine. I would expect many relay admins to take the default install and simply run ansible on top of that, so they don't even actively set up all the IPs by themselves (or have an interest in using them all), they simply 'inherit' them from the ISP. -- Moritz Bartl https://www.torservers.net/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Moritz Bartl:
For exit relays, you want WHOIS reassignment.
expected answer :) I'll add the "RAM sanity check" (RAM/instance ratio). Yes, "should work out of the box - even on weird machines" is a worthy design goal. -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJU48n1AAoJEFv7XvVCELh0y5sP/iZOE+wsK5N6/8URJIRIuz7X bBiEdZJvC5/93avgfstSVMIHv3ucBa1VbvgDrJ9HvNR5HfepqoAKBwxKePTmJ7uv VGMbkHJrDXnY43nWpV2XbMEIOlQE2Mwq8u0NpNgCD0T1AS/VG7waeaQLoq1SY50R r+UyK0Fsgen6bLq8CS3nw/meXVkspbvo4PunmKpy0VClR6GzgoigC0HTx6oA++U8 jiyBBuSj1LYJsd+Na33z0ujwIgRbyygLiPQSHcFJYhilU+myYZh6PZtIUW1gMMgs 07gWhdBuUnHHF+P3z0auWqGKFC6aVL6cdbaTbn11ixEogSY8f+iH/AvflbxrbYm7 Q5hgSjD+FN9X8A8jQtgdyKIVE3JE6wZL7/yBbkQLxyf5HQdAl7NbrVWOfb1X4SDP djPM4NYixESqr9PUKwd+OF8M1HWQm1wqbdSnVbZehku83fKKcAbuEou39WERT6mv uQPSj7QAtuPYfP/olVvQfanR57WX0WO9tW5uGvwunvokhg7rwGy6r8btf8UBIDWF BVnNBpAtcA4+mzk7of+Qir4ao4TFs4gvbaD0NXMFLl4qDd86KnfhwCoAwWBx34D2 Qvf/fXt79RjLBtsB7/VM5mitnLVDdiMZipWH/dFnRZcfHQyvBNmy5LBQ6TGCqJyG GqQuU2CDKUejIgZ1zRQq =qi0r -----END PGP SIGNATURE-----
On Tue, Feb 17, 2015 at 11:00:02PM +0100, Moritz Bartl wrote:
We still lack the "gamification" Relay Challenge website that Virgil was talking about. It would just sum up all relays of a family, and then it really does not matter any more.
If anybody (ok, it has to be the right people, not quite just anybody :) wants to step up and mentor this, we could add it to the ideas list for GSoC 2015, and see if we get any students who bite: https://www.torproject.org/about/gsoc --Roger
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Bug: Due to the fact that MyFamily is not written to the torrc in the "first round", torrc files will always change which results in tor processes being reloaded unnecessarily often - which is not what we want. - --list-fingerprint is probably the better approach here. -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJU450vAAoJEFv7XvVCELh0+CYP/jqmWwA68k7eNBPggrYuXPgW /Z7Y46ualDdWKV/vo2NP4ztQhzBUGCpdP+Xid700Z244FcuYVbhnezzQR5a0oQmu XUc37oyJ5vpmrQZFyks+HumUZP6iwJORjLyT96TUDYlgCsT8fZOxEB9Bx1LSRSgY juKwu9YhbnarVkYRUW8PCOxXaKHsFV31D0nYUlzFX+rS5In10RVGPpbrizh+CcYs zc/LFHTNqE9SNtad5O3zYbl6bUETkV8R+5K2nahjQEdjYesCNGHG0yB6ywAaL6rD LkpmWOj/C+W58ZMSRte5ytTOkL8S1Q6Srk7TVjUPxTFd0TmrzLd4DobNTSHpTxJW UGxjsHe/mIy2GYrhjVZL4NhlZcYrhcHnxeDjTqoAAp7vMgYXlu4m9zSJ5rD4TMwV YNXbjyfoA8jH73C6XcmNAsET68gRI59TkhkdAyX9ho84qcwWeru+Gt/nrcRw/VAH TIcPgg74EJvEMluGHTwUd6/w661lpcH9swVUDSrBSZaVs74kuSx9+NfZtPtaRFBF o2t38S/w8dLgxsFw75qb+Pk2/oMIvqMnhOozSil7ig2zhtUBcPNLM5b1ZOm33BfW Ug0+D4N5H8x86UZN5xQ/VX5htaVuJl6ewM9ljusB8D51anD4601TSayyBJ6t9fyV xqjA/2z48rkKBPEoINiH =TNST -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Bug:
Due to the fact that MyFamily is not written to the torrc in the "first round", torrc files will always change which results in tor processes being reloaded unnecessarily often - which is not what we want.
--list-fingerprint is probably the better approach here.
Patched. I guess I'll setup a repo and stop sending emails.. -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJU46vvAAoJEFv7XvVCELh01b8QAJ3eFrzOT7K5+ysVTNp/PBC7 +A3msVW5NItQBItQCvSc3EOlzc7EMT66FwZzBVwkPPOAttgAcc/GyAMUB7C3DOG1 dkIx50ewqsEywqLbY6tItI2jiEmt98JXFcNC95DJKvfJaCdhEQGj3T4AQKBW/b2l 6hDm3odjP9rm+GmYVbDaDKR8a2ChkRpLAou1TcKT8T/I6e0B3+ANhPkrxiQSNGgi DdYy8qKSrVK3VJtI17DJxFSvHXpcXiFEeyQXg7U7zMSEAkHf4wAbFXla5B1JkGcN Po7gbwYG0f7xqRSgGl8tC9p5BHPlZRgVUmCzvzAfstwI12HsOPu0X72WuHumXVsP liW+7X2lWvaoztAcNOTIJHlvivCoh1e5qbsMNcJ8f9cmxyK+X/TasgKbWa5pWr8F vvq27RW8oWyBc5VQUPBI6jMelqnuaIrDcwjaDoxzNoqjdSE1gR1pQS/EiTea6vdY 8dWwNVkC580yEndKCZUPfY8NUmdwyaewcY8SSTJeXbZu9ud7JyeWcNlHb0eFjMgj /02JWcZdGOOFuANEdyEQHd7S30I7MZVHSAGvYkxJdlIQ/yLUmj/Tgs2MMz/DBOgX HIISPCGAX+F60u4HHxB99+IJiIz2c2yIp1DXDwlCJBhAiTl3WA2nK7dRQpHSD0r6 H+u5pdAalQxhMnH24qJP =f8+2 -----END PGP SIGNATURE-----
participants (4)
-
David Stainton -
Moritz Bartl -
Nusenu -
Roger Dingledine