- tor-commits - lists.torproject.org

[flashproxy/master] Just read and delete the first IMAP message until we get an error.
by dcf＠torproject.org 04 Oct '12

04 Oct '12

commit 31033ccfd21cd8c1aff7ae5934ddd9da9e6913da Author: David Fifield <david(a)bamsoftware.com> Date: Wed Oct 3 11:58:12 2012 -0700 Just read and delete the first IMAP message until we get an error. We used EXISTS as a hint to how many messages there will be to process, but this number can change during the loop execution, so we have to check for an error anyway, so we may as well use that as our exit condition. --- facilitator/facilitator-email-poller | 41 +++++++++++++++++---------------- 1 files changed, 21 insertions(+), 20 deletions(-) diff --git a/facilitator/facilitator-email-poller b/facilitator/facilitator-email-poller index 412b5eb..cfae523 100755 --- a/facilitator/facilitator-email-poller +++ b/facilitator/facilitator-email-poller @@ -262,26 +262,27 @@ def imap_loop(imap): typ, data = imap.select("[Gmail]/All Mail") exists = int(data[0]) - for i in range(exists): - # Grab message 1 on each iteration; remaining messages shift down so - # the next message we process is also message 1. - typ, data = imap.fetch(1, "(BODY[])") - if data[0] is None: - break - - try: - msg = email.message_from_string(data[0][1]) - handle_message(msg) - except Exception, e: - log("Error processing message, deleting anyway: %s" % str(e)) - - imap.copy(1, "[Gmail]/Trash") - # Attempting to delete from All Mail here would have no effect. - # Messages disappear from All Mail when they are deleted from Trash - # in the next step. - - # Now delete all messages from Trash to really delete them. - imap_empty_trash(imap) + if exists > 0: + while True: + # Grab message 1 on each iteration; remaining messages shift down so + # the next message we process is also message 1. + typ, data = imap.fetch(1, "(BODY[])") + if data[0] is None: + break + + try: + msg = email.message_from_string(data[0][1]) + handle_message(msg) + except Exception, e: + log("Error processing message, deleting anyway: %s" % str(e)) + + imap.copy(1, "[Gmail]/Trash") + # Attempting to delete from All Mail here would have no effect. + # Messages disappear from All Mail when they are deleted from Trash + # in the next step. + + # Now delete all messages from Trash to really delete them. + imap_empty_trash(imap) time.sleep(POLL_INTERVAL)

1 0

[flashproxy/master] Fix a log message format string.
by dcf＠torproject.org 04 Oct '12

04 Oct '12

commit b5e1e71d5fb7efa07e6ad09bf1c40d8f60c3676a Author: David Fifield <david(a)bamsoftware.com> Date: Wed Oct 3 11:47:52 2012 -0700 Fix a log message format string. --- facilitator/facilitator-email-poller | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/facilitator/facilitator-email-poller b/facilitator/facilitator-email-poller index 056e9a5..3f5c4d8 100755 --- a/facilitator/facilitator-email-poller +++ b/facilitator/facilitator-email-poller @@ -255,7 +255,7 @@ def imap_loop(imap): msg = email.message_from_string(data[0][1]) handle_message(msg) except Exception, e: - log("Error processing message, deleting anyway: %s", str(e)) + log("Error processing message, deleting anyway: %s" % str(e)) imap.store(1, "+FLAGS", "\\Deleted") imap.expunge()

1 0

[flashproxy/master] Instructions for setting up two-factor Gmail auth.
by dcf＠torproject.org 04 Oct '12

04 Oct '12

commit e2925222cc5fe5a8f188b75d0fa8701d34e99c36 Author: David Fifield <david(a)bamsoftware.com> Date: Wed Oct 3 16:31:56 2012 -0700 Instructions for setting up two-factor Gmail auth. --- doc/gmail-setup.txt | 18 ++++++++++++++++++ 1 files changed, 18 insertions(+), 0 deletions(-) diff --git a/doc/gmail-setup.txt b/doc/gmail-setup.txt index e3685e6..44f5124 100644 --- a/doc/gmail-setup.txt +++ b/doc/gmail-setup.txt @@ -41,3 +41,21 @@ POP/IMAP". * Enable IMAP * Auto-Expunge on Click "Save Changes". + +Enable two-factor authentication. We do this not so much for the +two-factor, but because it allows creating an independent password that +is used only for IMAP and does not have access to the web interface of +Gmail. Click the email address in the upper right, then "Account". Click +"Security". By "2-step verification" click "Edit". Click through until +it lets you set up. The phone number you provided when the account was +created will be automatically filled in. Choose "Text message (SMS)" +then click "Send code". Get your text message, type it in, and hit +"Verify". Uncheck "Trust this computer" on the next screen. Finally +"Confirm". On the following summary page, click "Show backup codes" and +save the codes to encrypted storage. Future codes can be generated at +https://www.google.com/accounts/SmsAuthConfig. + +Still on the 2-step summary page, click "Manage application-specific +passwords". Enter "IMAP" for the name and click "Generate password". +Save the password to encrypted storage. This is the password that will +be used with the --pass option of facilitator-email-poller.

1 0

[flashproxy/master] Update gmail-setup.txt.
by dcf＠torproject.org 04 Oct '12

04 Oct '12

commit 4de784b87d11df03039ef111376481a06eed7704 Author: David Fifield <david(a)bamsoftware.com> Date: Wed Oct 3 12:02:50 2012 -0700 Update gmail-setup.txt. Correct some text labels from going through the process again. --- doc/gmail-setup.txt | 64 +++++++++++++++++++++++--------------------------- 1 files changed, 29 insertions(+), 35 deletions(-) diff --git a/doc/gmail-setup.txt b/doc/gmail-setup.txt index 847e5b7..e3685e6 100644 --- a/doc/gmail-setup.txt +++ b/doc/gmail-setup.txt @@ -1,49 +1,43 @@ These are instructions for setting up a Gmail account for use with the email-based rendezvous and flashproxy-reg-email. These instructions were -current as of September 2012. +current as of October 2012. You may have trouble if you are using Tor to create the account, for two reasons. The first is that exit nodes are a source of abuse and Google is more suspicious of them. The second is that Gmail is suspicious and can lock you out of the account when your IP address is changing. While setting up the account, use a single node in your torrc ExitNodes -configuration. Choose a U.S. exit node, one of the ones with the lowest -bandwidth. +configuration. Choose a U.S. exit node, one with low bandwidth. -Go to https://mail.google.com/. Allow JavaScript to run. Click the -"CREATE AN ACCOUNT" button. +Go to https://mail.google.com/. Allow JavaScript to run (even from +youtube.com; it seems to be necessary). Click the "CREATE AN ACCOUNT" +button. -Enter the account details and click "Next Step". +Enter the account details. You don't need to fill in "Your current email +address". Enter a mobile phone number for later activation of two-factor +authentication. Solve the captcha. Click "Next Step". You may have to do +a phone SMS verification here. -What may go wrong is that after entering the account details you see a -screen "Verify your account" that asks for a mobile phone number. Try -again with a different low-bandwidth exit node. exit node, one of the -ones with the lowest bandwidth. Also, it may be my imagination, but -Gmail seems to be checking for user names that appear randomly -generated; try inserting vowels between your randomly generated -consonants to make the word pronouncable. +At this point the Gmail account is created. If you are pushed into +joining Google+, close everything out and go back to +https://mail.google.com/. -You may be gently forced into joining Google+ at this point; I think -that the Gmail account is already created, so you may be able to skip -Google+ by browsing back to https://mail.google.com/. If not, you can -delete the Google+ account by clicking on the account name from inside -of Gmail. - -Log out of the account and log back in again. There will be new text in -the lower right reading "Recent Activity". Click "Details" and turn off -the suspicious activity warnings. This will keep you from getting locked -out when you come from different IP addresses. At this point you should -remove the temporary ExitNodes configuration from torrc. +Log out of the account and then back in again. There will be new text in +the lower right reading "Last account activity". Click "Details" and +turn off the unusual activity alerts. This will keep you from getting +locked out when you come from different IP addresses. At this point you +should remove the temporary ExitNodes configuration from torrc. Add a filter to prevent registrations from being marked as spam. Click -on the gear icon, and select "Settings". Select "Filters" and add a new -filter. For "Containing the words:" insert "in:spam". You will get a -warning that says it will not match any incoming messages; this appears -to be wrong and you can click through it. - -Enable IMAP. Click the gear icon, then "Settings". Click "Forwarding and -POP/IMAP". Then -* Disable POP -* Enable IMAP -* Auto-Expunge on -"Save Changes". +on the gear icon and select "Settings". Select "Filters" then "Create a +new filter". For "Has the words" type "in:spam", then "Create filter +with this search". There will be a warning that filters using "in:" will +never match incoming mail; this appears to be false and you can just +click OK. Check "Never send it to Spam" and click "Create filter". + +Enable IMAP. Click the gear icon, then "Settings", then "Forwarding and +POP/IMAP". + * Disable POP + * Enable IMAP + * Auto-Expunge on +Click "Save Changes".

1 0

[flashproxy/master] Increase version to 0.7.
by dcf＠torproject.org 04 Oct '12

04 Oct '12

commit b1f433d9bd12fa62529bf0457b24a4b5af537bca Author: David Fifield <david(a)bamsoftware.com> Date: Wed Oct 3 17:28:49 2012 -0700 Increase version to 0.7. --- Makefile | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Makefile b/Makefile index 9381f0f..001b3ee 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ PREFIX = /usr/local BINDIR = $(PREFIX)/bin -VERSION = 0.6 +VERSION = 0.7 CLIENT_DIST_FILES = flashproxy-client flashproxy-reg-email flashproxy-reg-http README LICENSE torrc

1 0

[flashproxy/master] Typos.
by dcf＠torproject.org 04 Oct '12

04 Oct '12

commit 692a4213fae4f839dd6d888803e93bfaa77921e9 Author: David Fifield <david(a)bamsoftware.com> Date: Wed Oct 3 16:34:26 2012 -0700 Typos. --- doc/facilitator-howto.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/facilitator-howto.txt b/doc/facilitator-howto.txt index 5157659..1defc9c 100644 --- a/doc/facilitator-howto.txt +++ b/doc/facilitator-howto.txt @@ -177,7 +177,7 @@ it not readable or writable by anyone but its owner. You need an RSA keypair because all client registrations over email are encrypted. - # openssl genrsa /etc/flashproxy/reg-email.key 2048 + # openssl genrsa -out /etc/flashproxy/reg-email.key 2048 # chmod 600 /etc/flashproxy/reg-email.key # openssl rsa -pubout < /etc/flashproxy/reg-email.key > reg-email.pub @@ -185,5 +185,5 @@ You will have to edit flashproxy-reg-email and copy the contents of reg-email.pub into the appropriate place. Install reg-email.key and reg-email.pass to /etc/flashproxy to match -what the init scritp expects, or else use the --key and --pass options +what the init script expects, or else use the --key and --pass options if you have them stored in another place.

1 0

[flashproxy/master] New reg-email address flashproxyreg.a@gmail.com.
by dcf＠torproject.org 04 Oct '12

04 Oct '12

commit 3a4e1a32516cfd7b17bb04ba22e96ec93442453b Author: David Fifield <david(a)bamsoftware.com> Date: Wed Oct 3 16:39:22 2012 -0700 New reg-email address flashproxyreg.a(a)gmail.com. We'll miss you hoddwee. --- facilitator/facilitator-email-poller | 2 +- flashproxy-reg-email | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/facilitator/facilitator-email-poller b/facilitator/facilitator-email-poller index ee44cf9..d6b1b75 100755 --- a/facilitator/facilitator-email-poller +++ b/facilitator/facilitator-email-poller @@ -20,7 +20,7 @@ from M2Crypto import BIO, RSA, X509 DEFAULT_IMAP_HOST = "imap.gmail.com" DEFAULT_IMAP_PORT = 993 -DEFAULT_EMAIL_ADDRESS = "hoddwee(a)gmail.com" +DEFAULT_EMAIL_ADDRESS = "flashproxyreg.a(a)gmail.com" DEFAULT_LOG_FILENAME = "facilitator-email-poller.log" POLL_INTERVAL = 60 diff --git a/flashproxy-reg-email b/flashproxy-reg-email index 7ac913a..9feb38a 100755 --- a/flashproxy-reg-email +++ b/flashproxy-reg-email @@ -19,7 +19,7 @@ except ImportError: DEFAULT_REMOTE_ADDRESS = "" DEFAULT_REMOTE_PORT = 9000 -DEFAULT_EMAIL_ADDRESS = "hoddwee(a)gmail.com" +DEFAULT_EMAIL_ADDRESS = "flashproxyreg.a(a)gmail.com" # dig MX gmail.com DEFAULT_SMTP_HOST = "gmail-smtp-in.l.google.com" DEFAULT_SMTP_PORT = 25 @@ -67,13 +67,13 @@ PUBKEY_SHA1 = "1926a5cb3fd7e48328188946e73ade84f55787dc".decode("hex") # openssl rsa -pubout < reg-email > reg-email.pub FACILITATOR_PUBKEY_PEM = """\ -----BEGIN PUBLIC KEY----- -MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArzOJ0Ya0fVp6pcgIf2GW -CBanFAyDygMFm9WfSwgQMf9XmMoHmI39bM4MftLjnhfmN75IT1DESFprmxca+vNA -SYuN5rnCLcV5hcCFIHgDWRxQkDuRnQfYCZN7/lWgYUiMpU92vdlyYnP+XNVat7qw -06tSHey2FS417zmIsNTcKUk9XhWb/uVHzOw/oCwG1wuxdUXKXLHjgwR5V17NbNwc -FJ1qCmQeswQk44wfJHuXJiIxMgDWF6p4LH07hBrP68L1vtlOVpqVAAoi/wLUhBgZ -WG0m/agJiktYR1qkLRNSH0afmpQn/v8kQg3T/5d0+HLECMfuBqoWEuJBACPSlA06 -5wIDAQAB +MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA44Mt8c599/4N2fgu6ppN +oatPW1GOgZxxObljFtEy0OWM1eHB35OOn+Kn9MxNHTRxVWwCEi0HYxWNVs2qrXxV +84LmWBz6A65d2qBlgltgLXusiXLrpwxVmJeO+GfmbF8ur0U9JSYxA20cGW/kujNg +XYDGQxO1Gvxq2lHK2LQmBpkfKEE1DMFASmIvlHDQgDj3XBb5lYeOsHZmg16UrGAq +1UH238hgJITPGLXBtwLtJkYbrATJvrEcmvI7QSm57SgYGpaB5ZdCbJL5bag5Pgt6 +M5SDDYYY4xxEPzokjFJfCQv+kcyAnzERNMQ9kR41ePTXG62bpngK5iWGeJ5XdkxG +gwIDAQAB -----END PUBLIC KEY----- """

1 0

[flashproxy/master] Delete Gmail messages from Trash to really delete them.
by dcf＠torproject.org 04 Oct '12

04 Oct '12

commit 165c0fcd811c4f1766ff90c04f2b00ca10be9ab0 Author: David Fifield <david(a)bamsoftware.com> Date: Wed Oct 3 11:48:57 2012 -0700 Delete Gmail messages from Trash to really delete them. After being deleted from INBOX, messages were being "archived" in All Mail. We can delete them for good by copying them to Trash and then deleting them. https://support.google.com/mail/bin/answer.py?answer=78755 http://gmailblog.blogspot.com/2008/10/new-in-labs-advanced-imap-controls.ht… --- facilitator/facilitator-email-poller | 33 ++++++++++++++++++++++++++++----- 1 files changed, 28 insertions(+), 5 deletions(-) diff --git a/facilitator/facilitator-email-poller b/facilitator/facilitator-email-poller index 3f5c4d8..412b5eb 100755 --- a/facilitator/facilitator-email-poller +++ b/facilitator/facilitator-email-poller @@ -239,14 +239,32 @@ def handle_message(msg): log(u"registering %s" % safe_str(fac.format_addr(client_addr))) fac.put_reg(FACILITATOR_ADDR, client_addr) +def imap_empty_trash(imap): + typ, data = imap.select("[Gmail]/Trash") + imap.store("1:*", "+FLAGS", "\\Deleted") + imap.expunge() + +# Gmail's IMAP folders are funny: they are not real folders, but actually views +# of messages having a particular label. INBOX consists of messages having the +# INBOX label, for example. Deleting a message from a folder just removes its +# label, but the message itself continues to exist in "[Gmail]/All Mail". +# https://support.google.com/mail/bin/answer.py?answer=78755 +# http://gmailblog.blogspot.com/2008/10/new-in-labs-advanced-imap-controls.ht… +# To really delete a message, you must copy it to "[Gmail]/Trash" and then +# delete it from there. Messages in Trash are deleted automatically after 30 +# days, but we do it immediately. def imap_loop(imap): + # Empty the trash first thing, in case there are messages there from a + # previous failed run of this program or something. + imap_empty_trash(imap) + while True: - typ, data = imap.select() + typ, data = imap.select("[Gmail]/All Mail") exists = int(data[0]) for i in range(exists): - # Grab and delete message 1 on each iteration; remaining messages - # shift down so the next message we process is also message 1. + # Grab message 1 on each iteration; remaining messages shift down so + # the next message we process is also message 1. typ, data = imap.fetch(1, "(BODY[])") if data[0] is None: break @@ -257,8 +275,13 @@ def imap_loop(imap): except Exception, e: log("Error processing message, deleting anyway: %s" % str(e)) - imap.store(1, "+FLAGS", "\\Deleted") - imap.expunge() + imap.copy(1, "[Gmail]/Trash") + # Attempting to delete from All Mail here would have no effect. + # Messages disappear from All Mail when they are deleted from Trash + # in the next step. + + # Now delete all messages from Trash to really delete them. + imap_empty_trash(imap) time.sleep(POLL_INTERVAL)

1 0

[flashproxy/master] Move messages to Trash first thing.
by dcf＠torproject.org 04 Oct '12

04 Oct '12

commit a1a3f4bb238d0dd5bf47272d9ccc659a682cf503 Author: David Fifield <david(a)bamsoftware.com> Date: Wed Oct 3 15:22:31 2012 -0700 Move messages to Trash first thing. Instead of processing messages one by one in All Mail, then deleting from Trash en masse, copy from All Mail to Trash en mass, then delete from Trash one by one. Trash acts more like a real mailbox, in that deleting a message from there really deletes the message immediately. If something goes wrong and we are able to process the messages from Trash, Gmail will delete them anyway after 30 days. --- facilitator/facilitator-email-poller | 33 ++++++++++++++------------------- 1 files changed, 14 insertions(+), 19 deletions(-) diff --git a/facilitator/facilitator-email-poller b/facilitator/facilitator-email-poller index 9c9b7d4..ee44cf9 100755 --- a/facilitator/facilitator-email-poller +++ b/facilitator/facilitator-email-poller @@ -251,11 +251,6 @@ def imap_get_uid(imap, index): assert m is not None return m.group(1) -def imap_empty_trash(imap): - typ, data = imap.select("[Gmail]/Trash") - imap.store("1:*", "+FLAGS", "\\Deleted") - imap.expunge() - # Gmail's IMAP folders are funny: they are not real folders, but actually views # of messages having a particular label. INBOX consists of messages having the # INBOX label, for example. Deleting a message from a folder just removes its @@ -266,14 +261,17 @@ def imap_empty_trash(imap): # delete it from there. Messages in Trash are deleted automatically after 30 # days, but we do it immediately. def imap_loop(imap): - # Empty the trash first thing, in case there are messages there from a - # previous failed run of this program or something. - imap_empty_trash(imap) - while True: + # Copy all messages to Trash, and work on them from there. This is a + # failsafe so that messages will eventually be deleted if we are not + # able to retrieve them. This act of copying also deletes from All Mail. typ, data = imap.select("[Gmail]/All Mail") - exists = int(data[0]) + assert typ == "OK" + imap.copy("1:*", "[Gmail]/Trash") + typ, data = imap.select("[Gmail]/Trash") + assert typ == "OK" + exists = int(data[0]) if exists > 0: while True: # Grab message 1 on each iteration; remaining messages shift down so @@ -284,21 +282,18 @@ def imap_loop(imap): typ, data = imap.uid("FETCH", uid, "(BODY[])") assert typ == "OK" + msg_text = data[0][1] + typ, data = imap.uid("STORE", uid, "+FLAGS", "\\Deleted") + assert typ == "OK" + typ, data = imap.expunge() + assert typ == "OK" try: - msg = email.message_from_string(data[0][1]) + msg = email.message_from_string(msg_text) handle_message(msg) except Exception, e: log("Error processing message, deleting anyway: %s" % str(e)) - imap.uid("COPY", uid, "[Gmail]/Trash") - # Attempting to delete from All Mail here would have no effect. - # Messages disappear from All Mail when they are deleted from Trash - # in the next step. - - # Now delete all messages from Trash to really delete them. - imap_empty_trash(imap) - time.sleep(POLL_INTERVAL) def imap_login():

1 0

[metrics-tasks/master] Suggest larger heap space for #6443.
by karsten＠torproject.org 03 Oct '12

03 Oct '12

commit e8a5a5bf988b34b97b5d7aace9d9dc973b5477e6 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Oct 3 19:34:49 2012 -0400 Suggest larger heap space for #6443. --- task-6443/README | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/task-6443/README b/task-6443/README index 57ff483..1677ac6 100644 --- a/task-6443/README +++ b/task-6443/README @@ -10,7 +10,7 @@ Graph cdf of probability of selecting among the biggest k exits (#6443) $ javac -d bin -cp lib/descriptor.jar src/CalculatePathSelectionProbabilities.java - Run Java class: - $ java -cp bin:lib/descriptor.jar:lib/commons-codec-1.6.jar:lib/commons-compress-1.4.1.jar CalculatePathSelectionProbabilities + $ java -Xmx2g -cp bin:lib/descriptor.jar:lib/commons-codec-1.6.jar:lib/commons-compress-1.4.1.jar CalculatePathSelectionProbabilities - Plot graphs: $ R --slave -f cumulated-weights.R

1 0