Re: [tor-dev] Tor Browser videos Automation
+ tor-dev Sorry for the noise, but thought of expanding the audience a little, maybe someone the time to take look into this, or might know what is going on. Any help is appreciated, we are trying to finish this project and this has been blocking us. Thanks! Sherief Alaa:
On Mon, May 18, 2015 at 3:49 PM, Mark Smith <mcs@pearlcrescent.com> wrote:
On 5/17/15 6:54 AM, Sherief Alaa wrote:
Hi Georg/Mark,
I am working on the new Tor Browser videos this month (or at least a process to produce them in an automated manner). I am contacting you because I am running into a problem that you may know how to solve because I think it's browser related.
Karsten insisted that I have to run a local copy of torproject.org <http://torproject.org> using a web server while the automated script runs since we can't estimate or depend on the connection speed. The major blocker in this is that the browser redirects to https://torproject.org/ whenever I try to map 127.0.0.1 to torproject.org <http://torproject.org> (or www.torproject.org <http://www.torproject.org>) in my /etc/hosts file. Now I've tried everything that will come up to mind that may solve the problem, such as: flushing DNS cache, clearing history, disabling all addons and more but nothing worked.
I also ran Wireshark to inspect DNS packets but the browser doesn't even try to query any DNS server.
I've tested on OS X and Ubuntu and both produce the same results. I've also tested multiple browsers (Safari, Chrome and Firefox). Same results.
Trivial note: During testing, I could map google.com <http://google.com>, ign.com <http://ign.com> and basically all .coms but never for .orgs like eff.org <http://eff.org> or torproject.org <http://torproject.org>
If you have any idea what's going on, please let me know as soon as possible as I am considering running a local nameserver but I am keeping that as a last resort.
I do not know what is going on. Most likely it is a caching problem (both the OS and browser cache DNS info, as you know). But Kathy (CC'd on this reply) and I experimented a little on Mac OS 10.9.5 and found like you did that some hosts worked and some did not, even after doing dscacheutil -flushcache and using Firefox with a new profile. From the command line, ping always seems to respect /etc/hosts but other things such as curl do not. Which I guess means it is an OS caching issue, or some applications have their own DNS resolver that bypasses the OS libraries.
-Mark
Ccing Isabela
-- PM at TorProject.org gpg fingerprint = 8F2A F9B6 D4A1 4D03 FDF1 B298 3224 4994 1506 4C7B @isa
Hi Sherief,
Karsten insisted that I have to run a local copy of torproject.org <http://torproject.org> using a web server while the automated script runs since we can't estimate or depend on the connection speed. The major blocker in this is that the browser redirects to https://torproject.org/ whenever I try to map 127.0.0.1 to torproject.org <http://torproject.org> (or www.torproject.org <http://www.torproject.org>) in my /etc/hosts file.
So you're trying to get http, but you get https, correct? Sounds like it might be the HSTS Preload list. See https://blog.mozilla.org/security/2012/11/01/preloading-hsts/ torproject.org is among the domains on Firefox's preload list: https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/nsSTSPre... I think you can turn off HSTS Preloading by creating an integer pref named "test.currentTimeOffsetSeconds", and setting it to 11491200. (Under about:config, right-click and choose "New > Integer".) Arthur
On Thu, Jul 9, 2015 at 7:03 PM, Arthur D. Edelstein < arthuredelstein@gmail.com> wrote:
Hi Sherief,
Karsten insisted that I have to run a local copy of torproject.org <http://torproject.org> using a web server while the automated script runs since we can't estimate or depend on the connection speed. The major blocker in this is that the browser redirects to https://torproject.org/ whenever I try to map 127.0.0.1 to torproject.org <http://torproject.org> (or www.torproject.org <http://www.torproject.org>) in my /etc/hosts file.
So you're trying to get http, but you get https, correct? Sounds like it might be the HSTS Preload list. See https://blog.mozilla.org/security/2012/11/01/preloading-hsts/ torproject.org is among the domains on Firefox's preload list:
https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/nsSTSPre...
I think you can turn off HSTS Preloading by creating an integer pref named "test.currentTimeOffsetSeconds", and setting it to 11491200. (Under about:config, right-click and choose "New > Integer".)
Arthur
Hi Arthur, I've tested the pref and unfortunately it doesn't work. I still get the https version of torproject.org. -- Sherief Alaa pgp 0x8623B882
Hi Sherief, I gave it a try and to get it to work I had to do the following (Note steps 2, 3 and 4 are very bad for your privacy!): 1. Add "torproject.org" as an alias to localhost in /etc/hosts 2. Deactivate "HTTPS Everywhere" 3. Create "test.currentTimeOffsetSeconds" integer pref and set to 11491200 4. Preferences > Advanced > Network > Settings: Turn "Remote DNS" off Then I could see my localhost page by entering http://torproject.org in the URL bar. Arthur On Sat, Jul 11, 2015 at 9:26 PM, Sherief Alaa <sheriefalaa.w@gmail.com> wrote:
On Thu, Jul 9, 2015 at 7:03 PM, Arthur D. Edelstein <arthuredelstein@gmail.com> wrote:
Hi Sherief,
Karsten insisted that I have to run a local copy of torproject.org <http://torproject.org> using a web server while the automated script runs since we can't estimate or depend on the connection speed. The major blocker in this is that the browser redirects to https://torproject.org/ whenever I try to map 127.0.0.1 to torproject.org <http://torproject.org> (or www.torproject.org <http://www.torproject.org>) in my /etc/hosts file.
So you're trying to get http, but you get https, correct? Sounds like it might be the HSTS Preload list. See https://blog.mozilla.org/security/2012/11/01/preloading-hsts/ torproject.org is among the domains on Firefox's preload list:
https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/nsSTSPre...
I think you can turn off HSTS Preloading by creating an integer pref named "test.currentTimeOffsetSeconds", and setting it to 11491200. (Under about:config, right-click and choose "New > Integer".)
Arthur
Hi Arthur,
I've tested the pref and unfortunately it doesn't work. I still get the https version of torproject.org.
-- Sherief Alaa pgp 0x8623B882
On Jul 12, 2015, at 7:51 AM, Arthur D. Edelstein <arthuredelstein@gmail.com> wrote:
Hi Sherief,
I gave it a try and to get it to work I had to do the following (Note steps 2, 3 and 4 are very bad for your privacy!): 1. Add "torproject.org" as an alias to localhost in /etc/hosts 2. Deactivate "HTTPS Everywhere" 3. Create "test.currentTimeOffsetSeconds" integer pref and set to 11491200 4. Preferences > Advanced > Network > Settings: Turn "Remote DNS" off
Then I could see my localhost page by entering http://torproject.org in the URL bar.
Arthur
Arthur, This actually worked on OS X 10.10.4 using FF 39! Thanks a lot!
On Sat, Jul 11, 2015 at 9:26 PM, Sherief Alaa <sheriefalaa.w@gmail.com> wrote:
On Thu, Jul 9, 2015 at 7:03 PM, Arthur D. Edelstein <arthuredelstein@gmail.com> wrote:
Hi Sherief,
> Karsten insisted that I have to run a local copy of torproject.org > <http://torproject.org> using a web server while the automated script > runs since we can't estimate or depend on the connection speed. The > major blocker in this is that the browser redirects to > https://torproject.org/ whenever I try to map 127.0.0.1 to > torproject.org <http://torproject.org> (or www.torproject.org > <http://www.torproject.org>) in my /etc/hosts file.
So you're trying to get http, but you get https, correct? Sounds like it might be the HSTS Preload list. See https://blog.mozilla.org/security/2012/11/01/preloading-hsts/ torproject.org is among the domains on Firefox's preload list:
https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/nsSTSPre...
I think you can turn off HSTS Preloading by creating an integer pref named "test.currentTimeOffsetSeconds", and setting it to 11491200. (Under about:config, right-click and choose "New > Integer".)
Arthur
Hi Arthur,
I've tested the pref and unfortunately it doesn't work. I still get the https version of torproject.org.
-- Sherief Alaa pgp 0x8623B882
—Sherief
participants (3)
-
Arthur D. Edelstein -
Isabela -
Sherief Alaa