May 2020 - tor-commits - lists.torproject.org

[torspec/master] Add proposal 318, to limit the range of subprotocol versions.
by nickm＠torproject.org 11 May '20

11 May '20

commit 4625a59da9b6b27bbc803c271d1b7c94497917e1 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon May 11 16:02:59 2020 -0400 Add proposal 318, to limit the range of subprotocol versions. --- proposals/000-index.txt | 2 ++ proposals/318-limit-protovers.md | 45 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/proposals/000-index.txt b/proposals/000-index.txt index 98dbf6b..ecfa49b 100644 --- a/proposals/000-index.txt +++ b/proposals/000-index.txt @@ -238,6 +238,7 @@ Proposals by number: 315 Updating the list of fields required in directory documents [OPEN] 316 FlashFlow: A Secure Speed Test for Tor (Parent Proposal) [DRAFT] 317 Improve security aspects of DNS name resolution [OPEN] +318 Limit protover values to 0-63 [OPEN] Proposals by status: @@ -281,6 +282,7 @@ Proposals by status: 310 Towards load-balancing in Prop 271 315 Updating the list of fields required in directory documents 317 Improve security aspects of DNS name resolution + 318 Limit protover values to 0-63 ACCEPTED: 188 Bridge Guards and other anti-enumeration defenses 249 Allow CREATE cells with >505 bytes of handshake data diff --git a/proposals/318-limit-protovers.md b/proposals/318-limit-protovers.md new file mode 100644 index 0000000..08f3a16 --- /dev/null +++ b/proposals/318-limit-protovers.md @@ -0,0 +1,45 @@ +``` +Filename: 318-limit-protovers.md +Title: Limit protover values to 0-63. +Author: Nick Mathewson +Created: 11 May 2020 +Status: Open +``` + +# Limit protover values to 0-63. + +I propose that we no longer accept protover values higher than 63, +so that they can all fit nicely into 64-bit fields. + +(This proposal is part of the Walking Onions spec project.) + +## Motivation + +Doing this will simplify our implementations and our protocols. +Right now, an efficient protover implementation needs to use ranges +to represent possible protocol versions, and needs workarounds to +prevent an attacker from constructing a protover line that would +consume too much memory. With Walking Onions, we need lists of +protocol versions to be represented in an extremely compact format, +which also would benefit from a limited set of possible versions. + +I believe that we will lose nothing by making this +change. Currently, after nearly two decades of Tor development +and 3.5 years of experiences with protovers in production, we have +no protocol version high than 5. + +Even if we did someday need to implement higher protocol +versions, we could simply add a new subprotocol name instead. For +example, instead of "HSIntro=64", we could say "HSIntro2=1". + +## Migration + +Immediately, authorities should begin rejecting relays with protocol +versions above 63. (There are no such relays in the consensus right +now.) + +Once this change is deployed to a majority of authorities, we can +remove support in other Tor environments for protocol versions +above 63. + +

1 0

[torspec/master] Add proposal 319: RELAY_FRAGMENT cells
by nickm＠torproject.org 11 May '20

11 May '20

commit 8560a5be217893dd3742d65080a19370ac88a208 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon May 11 16:05:46 2020 -0400 Add proposal 319: RELAY_FRAGMENT cells --- proposals/000-index.txt | 2 + proposals/319-wide-everything.md | 123 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+) diff --git a/proposals/000-index.txt b/proposals/000-index.txt index ecfa49b..fb2fc56 100644 --- a/proposals/000-index.txt +++ b/proposals/000-index.txt @@ -239,6 +239,7 @@ Proposals by number: 316 FlashFlow: A Secure Speed Test for Tor (Parent Proposal) [DRAFT] 317 Improve security aspects of DNS name resolution [OPEN] 318 Limit protover values to 0-63 [OPEN] +319 RELAY_FRAGMENT cells [OPEN] Proposals by status: @@ -283,6 +284,7 @@ Proposals by status: 315 Updating the list of fields required in directory documents 317 Improve security aspects of DNS name resolution 318 Limit protover values to 0-63 + 319 RELAY_FRAGMENT cells ACCEPTED: 188 Bridge Guards and other anti-enumeration defenses 249 Allow CREATE cells with >505 bytes of handshake data diff --git a/proposals/319-wide-everything.md b/proposals/319-wide-everything.md new file mode 100644 index 0000000..a7b9b8a --- /dev/null +++ b/proposals/319-wide-everything.md @@ -0,0 +1,123 @@ +``` +Filename: 319-wide-everything.md +Title: RELAY_FRAGMENT cells +Author: Nick Mathewson +Created: 11 May 2020 +Status: Open +``` + +(This proposal is part of the Walking Onions spec project.) + +# Introduction + +Proposal 249 described a system for `CREATE` cells to become wider, in order to +accommodate hybrid crypto. And in order to send those cell bodies across +circuits, it described a way to split `CREATE` cells into multiple `EXTEND` +cells. + +But there are other cell types that can need to be wider too. For +example, `INTRODUCE` and `RENDEZVOUS` cells also contain key material +used for a handshake: if handshakes need to grow larger, then so do +these cells. + +This proposal describes an encoding for arbitrary "wide" relay cells, +that can be used to send a wide variant of anything. + +To be clear, although this proposal describes a way that all relay +cells can become "wide", I do not propose that wide cells should +actually be _allowed_ for all relay cell types. + +# Proposal + +We add a new relay cell type: `RELAY_FRAGMENT`. This cell type contains part +of another relay cell. A `RELAY_FRAGEMENT` cell can either introduce a new +fragmented cell, or can continue one that is already in progress. + +The format of a RELAY_FRAGMENT body is one the following: + + struct fragment_begin { + // What relay_command is in use for the underlying cell? + u8 relay_command; + // What will the total length of the cell be once it is reassembled? + u16 total_len; + // Bytes for the cell body + u8 body[]; + } + + struct fragment_continued { + // More bytes for the cell body. + u8 body[]; + } + +To send a fragmented cell, first a party sends a RELAY_FRAGMENT cell +containing a "fragment_begin" payload. This payload describes the total +length of the cell, the relay command + +Fragmented cells other than the last one in sequence MUST be sent full of +as much data as possible. Parties SHOULD close a circuit if they receive a +non-full fragmented cell that is not the last fragment in a sequence. + +Fragmented cells MUST NOT be interleaved with other relay cells on a circuit, +other than cells used for flow control. (Currently, this is only SENDME +cells.) If any party receives any cell on a circuit, other than a flow +control cell or a RELAY_FRAGEMENT cell, before the fragmented cell is +complete, than it SHOULD close the circuit. + +Parties MUST NOT send extra data in fragmented cells beyond the amount given +in the first 'total_len' field. + +Not every relay command may be sent in a fragmented cell. In this proposal, +we allow the following cell types to be fragmented: EXTEND2, EXTENDED2, +INTRODUCE1, INTRODUCE2, RENDEZVOUS. Any party receiving a command that they +believe should not be fragmented should close the circuit. + +Not all lengths up to 65535 are valid lengths for a fragmented cell. Any +length under 499 bytes SHOULD cause the circuit to close, since that could +fit into a non-fragmented RELAY cell. Parties SHOULD enforce maximum lengths +for cell types that they understand. + +All `RELAY_FRAGMENT` cells for the fragmented cell must have the +same Stream ID. (For those cells allowed above, the Stream ID is +always zero.) Implementations SHOULD close a circuit if they +receive fragments with mismatched Stream ID. + +# Onion service concerns. + +We allocate a new extension for use in the ESTABLISH_INTRO by onion services, +to indicate that they can receive a wide INTRODUCE2 cell. This extension +contains: + + struct wide_intro2_ok { + u16 max_len; + } + +We allocate a new extension for use in the `ESTABLISH_RENDEZVOUS` +cell, to indicate acceptance of wide `RENDEZVOUS2` cells. This +extension contains: + + struct wide_rend2_ok { + u16 max_len; + } + +(Note that `ESTABLISH_RENDEZVOUS` cells do not currently have a an +extension mechanism. They should be extended to use the same +extension format as `ESTABLISH_INTRO` cells, with extensions placed +after the rendezvous cookie.) + +# Handling RELAY_EARLY + +The first fragment of each EXTEND cell should be tagged with `RELAY_EARLY`. +The remaining fragments should not. Relays should accept `EXTEND` cells if and +only if their _first_ fragment is tagged with `RELAY_EARLY`. + +> Rationale: We could allow any fragment to be tagged, but that would give +> hostile guards an opportunity to move RELAY_EARLY tags around and build a +> covert channel. But if we later move to a relay encryption method that +> lets us authenticate RELAY_EARLY, we could then require only that _any_ +> fragment has RELAY_EARLY set. + +# Compatibility + +This proposal will require the allocation of a new 'Relay' protocol version, +to indicate understanding of the RELAY_FRAGMENTED command. +

1 0

[torspec/master] Proposal 314 has been accepted; proposals can be markdown.
by nickm＠torproject.org 11 May '20

11 May '20

commit 2dd966b784f1f0b572bef9908e720a4aa985faa0 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon May 11 15:57:05 2020 -0400 Proposal 314 has been accepted; proposals can be markdown. --- proposals/000-index.txt | 4 ++-- proposals/314-allow-markdown-proposals.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/000-index.txt b/proposals/000-index.txt index 31d05fd..98dbf6b 100644 --- a/proposals/000-index.txt +++ b/proposals/000-index.txt @@ -234,7 +234,7 @@ Proposals by number: 311 Tor Relay IPv6 Reachability [DRAFT] 312 Tor Relay Automatic IPv6 Address Discovery [DRAFT] 313 Tor Relay IPv6 Statistics [DRAFT] -314 Allow Markdown for proposal format [OPEN] +314 Allow Markdown for proposal format [ACCEPTED] 315 Updating the list of fields required in directory documents [OPEN] 316 FlashFlow: A Secure Speed Test for Tor (Parent Proposal) [DRAFT] 317 Improve security aspects of DNS name resolution [OPEN] @@ -279,7 +279,6 @@ Proposals by status: 299 Preferring IPv4 or IPv6 based on IP Version Failure Count 306 A Tor Implementation of IPv6 Happy Eyeballs 310 Towards load-balancing in Prop 271 - 314 Allow Markdown for proposal format 315 Updating the list of fields required in directory documents 317 Improve security aspects of DNS name resolution ACCEPTED: @@ -290,6 +289,7 @@ Proposals by status: 288 Privacy-Preserving Statistics with Privcount in Tor (Shamir version) 292 Mesh-based vanguards 301 Don't include package fingerprints in consensus documents + 314 Allow Markdown for proposal format META: 000 Index of Tor Proposals 001 The Tor Proposal Process diff --git a/proposals/314-allow-markdown-proposals.md b/proposals/314-allow-markdown-proposals.md index 2d2f653..4dcf888 100644 --- a/proposals/314-allow-markdown-proposals.md +++ b/proposals/314-allow-markdown-proposals.md @@ -3,7 +3,7 @@ Filename: 314-allow-markdown-proposals.md Title: Allow Markdown for proposal format. Author: Nick Mathewson Created: 23 April 2020 -Status: Open +Status: Accepted ``` # Introduction

1 0

[bridgedb/develop] Refactor script.
by phw＠torproject.org 11 May '20

11 May '20

commit 91af6c1bbad6a71663372642c5b6531cd197e806 Author: Philipp Winter <phw(a)nymity.ch> Date: Tue Feb 18 10:59:14 2020 -0800 Refactor script. --- scripts/nagios-email-check | 191 +++++++++++++++++++++++++++++++++++++++++++++ scripts/nagios_email_check | 162 -------------------------------------- 2 files changed, 191 insertions(+), 162 deletions(-) diff --git a/scripts/nagios-email-check b/scripts/nagios-email-check new file mode 100755 index 0000000..7287646 --- /dev/null +++ b/scripts/nagios-email-check @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# This script sends an email request for bridges to BridgeDB and then checks if +# it got a response. The result is written to STATUS_FILE, which is consumed +# by Nagios. Whenever BridgeDB fails to respond with bridges, we will get a +# Nagios alert. +# +# Run this script via crontab every three hours as follows: +# 0 */3 * * * path/to/nagios-email-check $(cat path/to/gmail.key) + + +import sys +import smtplib +import time +import imaplib +import email +import email.utils + +# Standard Nagios return codes +OK, WARNING, CRITICAL, UNKNOWN = range(4) + +FROM_EMAIL = "testbridgestorbrowser(a)gmail.com" +TO_EMAIL = "bridges(a)torproject.org" +SMTP_SERVER = "imap.gmail.com" +SMTP_PORT = 993 + +MESSAGE_FROM = TO_EMAIL +MESSAGE_BODY = "Here are your bridges:" + +STATUS_FILE = "/srv/bridges.torproject.org/check/status" + +# This will contain our test email's message ID. We later make sure that this +# message ID is referenced in the In-Reply-To header of BridgeDB's response. +MESSAGE_ID = None + + +def log(*args, **kwargs): + """ + Generic log function. + """ + + print("[+]", *args, file=sys.stderr, **kwargs) + + +def get_email_response(password): + """ + Open our Gmail inbox and see if we got a response. + """ + + log("Checking for email response.") + mail = imaplib.IMAP4_SSL(SMTP_SERVER) + try: + mail.login(FROM_EMAIL, password) + except Exception as e: + return WARNING, str(e) + + mail.select("INBOX") + + _, data = mail.search(None, "ALL") + email_ids = data[0].split() + if len(email_ids) == 0: + log("Found no response.") + return CRITICAL, "No emails from BridgeDB found" + + return check_email(mail, email_ids) + + +def check_email(mail, email_ids): + """ + Check if we got our expected email response. + """ + + log("Checking {:,} emails.".format(len(email_ids))) + for email_id in email_ids: + _, data = mail.fetch(email_id, "(RFC822)") + + # The variable `data` contains the full email object fetched by imaplib + # <https://docs.python.org/3/library/imaplib.html#imaplib.IMAP4.fetch> + # We are only interested in the response part containing the email + # envelope. + for response_part in data: + if isinstance(response_part, tuple): + m = str(response_part[1], "utf-8") + msg = email.message_from_string(m) + email_from = "{}".format(msg["From"]) + email_body = "{}".format(msg.as_string()) + email_reply_to = "{}".format(msg["In-Reply-To"]) + + if (MESSAGE_FROM == email_from) and \ + (MESSAGE_BODY in email_body) and \ + (MESSAGE_ID == email_reply_to): + mail.store(email_id, '+X-GM-LABELS', '\\Trash') + mail.expunge() + mail.close() + mail.logout() + log("Found correct response (referencing {})." + .format(MESSAGE_ID)) + return OK, "BridgeDB's email responder works" + else: + mail.store(email_id, '+X-GM-LABELS', '\\Trash') + mail.expunge() + mail.close() + mail.logout() + log("Found no response.") + return WARNING, "No emails from BridgeDB found" + + +def send_email_request(password): + """ + Attempt to send a bridge request over Gmail. + """ + + subject = "Bridges" + body = "get bridges" + + log("Sending email.") + global MESSAGE_ID + MESSAGE_ID = email.utils.make_msgid(idstring="test-bridgedb", + domain="gmail.com") + email_text = "From: %s\r\nTo: %s\r\nMessage-ID: %s\r\nSubject: %s\r\n" \ + "\r\n%s" % (FROM_EMAIL, TO_EMAIL, MESSAGE_ID, subject, body) + + try: + mail = smtplib.SMTP_SSL("smtp.gmail.com", 465) + mail.login(FROM_EMAIL, password) + mail.sendmail(FROM_EMAIL, TO_EMAIL, email_text) + mail.close() + log("Email successfully sent (message ID: %s)." % MESSAGE_ID) + return OK, "Sent email bridge request" + except Exception as e: + log("Error while sending email: %s" % err) + return UNKNOWN, str(e) + + +def write_status_file(status, message): + """ + Write the given `status` and `message` to our Nagios status file. + """ + + codes = { + 0: "OK", + 1: "WARNING", + 2: "CRITICAL", + 3: "UNKNOWN" + } + code = codes.get(status, UNKNOWN) + + with open(STATUS_FILE, "w") as fd: + fd.write("{}\n{}: {}".format(code, status, message)) + log("Wrote status='%s', message='%s' to status file." % (status, message)) + + +if __name__ == "__main__": + status, message = None, None + + # Our Gmail password should be in sys.argv[1]. + + if len(sys.argv) == 2: + password = sys.argv[1] + else: + log("No email password provided.") + write_status_file(UNKNOWN, "No email password provided") + sys.exit(1) + + # Send an email request to BridgeDB. + + try: + status, message = send_email_request(password) + except Exception as e: + write_status_file(UNKNOWN, repr(e)) + sys.exit(1) + + wait_time = 60 + log("Waiting %d seconds for a response." % wait_time) + time.sleep(wait_time) + + # Check if we've received an email response. + + try: + status, message = get_email_response(password) + except KeyboardInterrupt: + status, message = CRITICAL, "Caught Control-C..." + except Exception as e: + status = CRITICAL + message = repr(e) + finally: + write_status_file(status, message) + sys.exit(status) diff --git a/scripts/nagios_email_check b/scripts/nagios_email_check deleted file mode 100644 index 767d055..0000000 --- a/scripts/nagios_email_check +++ /dev/null @@ -1,162 +0,0 @@ -#!/usr/bin/env python3 -# -*- coding: utf-8 -*- -# -# This file is part of BridgeDB, a Tor bridge distribution system. -# -# This scripts send an email to bridgedb requesting a bridge and check if the -# service reply with a valid bridge. A status file is written in path specified -# by STATUS_FILE. -# -# The STATUS_FILE is read by nagios that will send an alert whenever bridgedb -# will not send a valid email response. -# -# Run via crontab -# Ex: */10 * * * * <path_to_bridgedb>/scripts/nagios_email_check $(cat <path_to_email_key/gmail.key) -# -# :authors: hiro <hiro(a)torproject.org> -# see also AUTHORS file -# -# :license: This is Free Software. See LICENSE for license information. -# -# - - -import sys -import smtplib -import time -import imaplib -import email -import time - -# Standard Nagios return codes -OK, WARNING, CRITICAL, UNKNOWN = range(4) - -ORG_EMAIL = "@gmail.com" -FROM_EMAIL = "test.bridges.browser" + ORG_EMAIL -SMTP_SERVER = "imap.gmail.com" -SMTP_PORT = 993 - -MESSAGE_FROM = "bridges(a)torproject.org" -MESSAGE_SUBJECT = "Bridges" -MESSAGE_BODY = "Here are your bridges:" - -STATUS_FILE = "/srv/bridgedb.torproject.org/check/status" - -# ------------------------------------------------- -# -# Utility to read email from Gmail Using Python -# -# ------------------------------------------------ - -def test_email_from_gmail(password): - mail = imaplib.IMAP4_SSL(SMTP_SERVER) - try: - mail.login(FROM_EMAIL, password) - except Exception as e: - return WARNING, str(e) - - mail.select('INBOX') - - _, data = mail.search(None, 'ALL') - mail_ids = data[0] - - id_list = mail_ids.split() - - status, message = check_email(id_list) - - return status, message - - -def check_email(id_list): - first_email_id = int(str(id_list[0], 'utf-8')) - latest_email_id = int(str(id_list[-1], 'utf-8')) - - for i in range(int(latest_email_id), int(first_email_id), -1): - _, data = mail.fetch(str(i), '(RFC822)') - - - # The variable data contains the full email object fetched by imaplib - # https://docs.python.org/3/library/imaplib.html#imaplib.IMAP4.fetch - # We are only interested in the response part containing the email envelope. - - for response_part in data: - if isinstance(response_part, tuple): - m = str(response_part[1], 'utf-8') - msg = email.message_from_string(m) - email_subject = "{}".format(msg['subject']) - email_from = "{}".format(msg['from']) - email_body = "{}".format(msg.as_string()) - - if (MESSAGE_FROM == email_from) and (MESSAGE_SUBJECT == email_subject) and (MESSAGE_BODY in email_body): - mail.store(str(i), '+FLAGS', '\\Deleted') - mail.close() - return OK, "Bridgedb is good and sending emails with working bridges" - else: - mail.store(str(i), '+FLAGS', '\\Deleted') - - mail.close() - return WARNING, "No emails from gettor found" - - -def send_email_from_gmail(password): - sent_from = FROM_EMAIL - sent_to = ["{}".format(MESSAGE_FROM)] - subject = 'Bridges' - body = 'get bridges' - - email_text = """From: %s\nTo: %s\nSubject: %s\n\n%s""" % (sent_from, ", ".join(sent_to), subject, body) - - try: - mail = smtplib.SMTP_SSL('smtp.gmail.com', 465) - mail.login(sent_from, password) - mail.sendmail(sent_from, sent_to, email_text) - mail.close() - return OK, "Test email sent" - except Exception as e: - return UNKNOWN, str(e) - -if __name__ == "__main__": - status, message = None, None - - if len(sys.argv) == 2: - password = sys.argv[1] - else: - return UNKNOWN, "Empty email password" - - try: - status_file = open(STATUS_FILE, 'r') - message = status_file.read() - status_file.close() - except OSError: - status = UNKNOWN - message = "Status file has been created {}".format(STATUS_FILE) - status_file = open(STATUS_FILE,'w') - status_file.write("UNKNOWN\n3: %s" % message) - status_file.close() - - try: - status, message = send_email_from_gmail(password) - except Exception as e: - status = UNKNOWN - message = repr(e) - status_file = open(STATUS_FILE,'w') - status_file.write("UNKNOWN\n3: %s" % message) - status_file.close() - - time.sleep(600) - - try: - status, message = test_email_from_gmail(password) - except KeyboardInterrupt: - status, message = CRITICAL, "Caught Control-C..." - except Exception as e: - status = CRITICAL - message = repr(e) - finally: - d = {0: "OK", 1: "WARNING", 2: "CRITICAL", 3: "UNKNOWN"} - status_file = open(STATUS_FILE,'w') - status_file.write("{}\n{}: {}" % (d[status], status, message)) - - status_file.close() - - sys.exit(status)

1 0

[bridgedb/develop] Add review feedback
by phw＠torproject.org 11 May '20

11 May '20

commit 85b4a9cfe41308458b29356b1d771b192b4a7681 Author: hiro <hiro(a)torproject.org> Date: Wed Dec 11 17:31:17 2019 +0100 Add review feedback --- scripts/check_status | 91 ++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 46 deletions(-) diff --git a/scripts/check_status b/scripts/check_status index 8f8f3e5..5032e33 100644 --- a/scripts/check_status +++ b/scripts/check_status @@ -82,7 +82,6 @@ def send_email_from_gmail(password): try: mail = smtplib.SMTP_SSL('smtp.gmail.com', 465) - mail.ehlo() mail.login(sent_from, password) mail.sendmail(sent_from, sent_to, email_text) mail.close() @@ -90,48 +89,48 @@ def send_email_from_gmail(password): except Exception as e: return UNKNOWN, str(e) - if __name__ == "__main__": - status, message = None, None - - if len(sys.argv) == 2: - password = sys.argv[1] - else: - password = "yourPassword" - - status_file = open(STATUS_FILE, 'r') - message = status_file.read() - status_file.close() - - try: - status, message = send_email_from_gmail(password) - except Exception as e: - status = UNKNOWN - message = repr(e) - status_file = open(STATUS_FILE,'w') - status_file.write("UNKNOWN\n3: %s" % message) - status_file.close() - - time.sleep(600) - - try: - status, message = test_email_from_gmail(password) - except KeyboardInterrupt: - status, message = CRITICAL, "Caught Control-C..." - except Exception as e: - status = CRITICAL - message = repr(e) - finally: - status_file = open(STATUS_FILE,'w') - if status == OK: - status_file.write("OK\n0: %s" % message) - elif status == WARNING: - status_file.write("WARNING\n1: %s" % message) - elif status == CRITICAL: - status_file.write("CRITICAL\n2: %s" % message) - else: - status_file.write("UNKNOWN\n3: %s" % message) - status = UNKNOWN - - status_file.close() - - sys.exit(status) +if __name__ == "__main__": + status, message = None, None + + if len(sys.argv) == 2: + password = sys.argv[1] + else: + password = "yourPassword" + + status_file = open(STATUS_FILE, 'r') + message = status_file.read() + status_file.close() + + try: + status, message = send_email_from_gmail(password) + except Exception as e: + status = UNKNOWN + message = repr(e) + status_file = open(STATUS_FILE,'w') + status_file.write("UNKNOWN\n3: %s" % message) + status_file.close() + + time.sleep(600) + + try: + status, message = test_email_from_gmail(password) + except KeyboardInterrupt: + status, message = CRITICAL, "Caught Control-C..." + except Exception as e: + status = CRITICAL + message = repr(e) + finally: + status_file = open(STATUS_FILE,'w') + if status == OK: + status_file.write("OK\n0: %s" % message) + elif status == WARNING: + status_file.write("WARNING\n1: %s" % message) + elif status == CRITICAL: + status_file.write("CRITICAL\n2: %s" % message) + else: + status_file.write("UNKNOWN\n3: %s" % message) + status = UNKNOWN + + status_file.close() + + sys.exit(status)

1 0

[bridgedb/develop] Rename email check script and add information on how to run it
by phw＠torproject.org 11 May '20

11 May '20

commit ae07c0950be2ededc2c81b80137a63d85e68130d Author: hiro <hiro(a)torproject.org> Date: Wed Jan 22 17:44:36 2020 +0100 Rename email check script and add information on how to run it --- scripts/{check_status => nagios_email_check} | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/scripts/check_status b/scripts/nagios_email_check similarity index 91% rename from scripts/check_status rename to scripts/nagios_email_check index 1784e6b..767d055 100644 --- a/scripts/check_status +++ b/scripts/nagios_email_check @@ -3,10 +3,23 @@ # # This file is part of BridgeDB, a Tor bridge distribution system. # +# This scripts send an email to bridgedb requesting a bridge and check if the +# service reply with a valid bridge. A status file is written in path specified +# by STATUS_FILE. +# +# The STATUS_FILE is read by nagios that will send an alert whenever bridgedb +# will not send a valid email response. +# +# Run via crontab +# Ex: */10 * * * * <path_to_bridgedb>/scripts/nagios_email_check $(cat <path_to_email_key/gmail.key) +# # :authors: hiro <hiro(a)torproject.org> # see also AUTHORS file # # :license: This is Free Software. See LICENSE for license information. +# +# + import sys import smtplib

1 0

[bridgedb/develop] Add service status check
by phw＠torproject.org 11 May '20

11 May '20

commit 2ae19e5c25dca6acdb0307c147dc93f73e257f7e Author: hiro <hiro(a)torproject.org> Date: Mon Oct 21 19:54:50 2019 +0200 Add service status check --- scripts/check_status | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 137 insertions(+) diff --git a/scripts/check_status b/scripts/check_status new file mode 100644 index 0000000..8f8f3e5 --- /dev/null +++ b/scripts/check_status @@ -0,0 +1,137 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: hiro <hiro(a)torproject.org> +# see also AUTHORS file +# +# :license: This is Free Software. See LICENSE for license information. + +import sys +import smtplib +import time +import imaplib +import email +import time + +# Standard Nagios return codes +OK, WARNING, CRITICAL, UNKNOWN = range(4) + +ORG_EMAIL = "@gmail.com" +FROM_EMAIL = "test.bridges.browser" + ORG_EMAIL +SMTP_SERVER = "imap.gmail.com" +SMTP_PORT = 993 + +MESSAGE_FROM = "bridges(a)torproject.org" +MESSAGE_SUBJECT = "Bridges" +MESSAGE_BODY = ":443" + +STATUS_FILE = "/srv/bridgedb.torproject.org/check/status" + +# ------------------------------------------------- +# +# Utility to read email from Gmail Using Python +# +# ------------------------------------------------ + +def test_email_from_gmail(password): + try: + mail = imaplib.IMAP4_SSL(SMTP_SERVER) + mail.login(FROM_EMAIL, password) + mail.select('INBOX') + + type, data = mail.search(None, 'ALL') + mail_ids = data[0] + + id_list = mail_ids.split() + first_email_id = int(str(id_list[0], 'utf-8')) + latest_email_id = int(str(id_list[-1], 'utf-8')) + + for i in range(int(latest_email_id), int(first_email_id), -1): + typ, data = mail.fetch(str(i), '(RFC822)') + + for response_part in data: + if isinstance(response_part, tuple): + m = str(response_part[1], 'utf-8') + msg = email.message_from_string(m) + email_subject = "{}".format(msg['subject']) + email_from = "{}".format(msg['from']) + email_body = "{}".format(msg.as_string()) + + if (MESSAGE_FROM == email_from) and (MESSAGE_SUBJECT == email_subject) and (MESSAGE_BODY in email_body): + mail.store(str(i), '+FLAGS', '\\Deleted') + mail.close() + return OK, "Bridgedb is good and sending emails with working bridges" + else: + mail.store(str(i), '+FLAGS', '\\Deleted') + + mail.close() + return WARNING, "No emails from gettor found" + + except Exception as e: + return CRITICAL, str(e) + +def send_email_from_gmail(password): + sent_from = FROM_EMAIL + sent_to = ["{}".format(MESSAGE_FROM)] + subject = 'Bridges' + body = 'get bridges' + + email_text = """From: %s\nTo: %s\nSubject: %s\n\n%s""" % (sent_from, ", ".join(sent_to), subject, body) + + try: + mail = smtplib.SMTP_SSL('smtp.gmail.com', 465) + mail.ehlo() + mail.login(sent_from, password) + mail.sendmail(sent_from, sent_to, email_text) + mail.close() + return OK, "Test email sent" + except Exception as e: + return UNKNOWN, str(e) + + if __name__ == "__main__": + status, message = None, None + + if len(sys.argv) == 2: + password = sys.argv[1] + else: + password = "yourPassword" + + status_file = open(STATUS_FILE, 'r') + message = status_file.read() + status_file.close() + + try: + status, message = send_email_from_gmail(password) + except Exception as e: + status = UNKNOWN + message = repr(e) + status_file = open(STATUS_FILE,'w') + status_file.write("UNKNOWN\n3: %s" % message) + status_file.close() + + time.sleep(600) + + try: + status, message = test_email_from_gmail(password) + except KeyboardInterrupt: + status, message = CRITICAL, "Caught Control-C..." + except Exception as e: + status = CRITICAL + message = repr(e) + finally: + status_file = open(STATUS_FILE,'w') + if status == OK: + status_file.write("OK\n0: %s" % message) + elif status == WARNING: + status_file.write("WARNING\n1: %s" % message) + elif status == CRITICAL: + status_file.write("CRITICAL\n2: %s" % message) + else: + status_file.write("UNKNOWN\n3: %s" % message) + status = UNKNOWN + + status_file.close() + + sys.exit(status)

1 0

[bridgedb/develop] Add more review feedback
by phw＠torproject.org 11 May '20

11 May '20

commit 0e5f3fef90090aba33e73702598bf4c4793bb0b1 Author: hiro <hiro(a)torproject.org> Date: Wed Dec 11 18:17:12 2019 +0100 Add more review feedback --- scripts/check_status | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-) diff --git a/scripts/check_status b/scripts/check_status index 5032e33..1784e6b 100644 --- a/scripts/check_status +++ b/scripts/check_status @@ -25,7 +25,7 @@ SMTP_PORT = 993 MESSAGE_FROM = "bridges(a)torproject.org" MESSAGE_SUBJECT = "Bridges" -MESSAGE_BODY = ":443" +MESSAGE_BODY = "Here are your bridges:" STATUS_FILE = "/srv/bridgedb.torproject.org/check/status" @@ -36,41 +36,54 @@ STATUS_FILE = "/srv/bridgedb.torproject.org/check/status" # ------------------------------------------------ def test_email_from_gmail(password): + mail = imaplib.IMAP4_SSL(SMTP_SERVER) try: - mail = imaplib.IMAP4_SSL(SMTP_SERVER) mail.login(FROM_EMAIL, password) - mail.select('INBOX') + except Exception as e: + return WARNING, str(e) + + mail.select('INBOX') + + _, data = mail.search(None, 'ALL') + mail_ids = data[0] + + id_list = mail_ids.split() + + status, message = check_email(id_list) + + return status, message + + +def check_email(id_list): + first_email_id = int(str(id_list[0], 'utf-8')) + latest_email_id = int(str(id_list[-1], 'utf-8')) - type, data = mail.search(None, 'ALL') - mail_ids = data[0] + for i in range(int(latest_email_id), int(first_email_id), -1): + _, data = mail.fetch(str(i), '(RFC822)') - id_list = mail_ids.split() - first_email_id = int(str(id_list[0], 'utf-8')) - latest_email_id = int(str(id_list[-1], 'utf-8')) - for i in range(int(latest_email_id), int(first_email_id), -1): - typ, data = mail.fetch(str(i), '(RFC822)') + # The variable data contains the full email object fetched by imaplib + # https://docs.python.org/3/library/imaplib.html#imaplib.IMAP4.fetch + # We are only interested in the response part containing the email envelope. - for response_part in data: - if isinstance(response_part, tuple): - m = str(response_part[1], 'utf-8') - msg = email.message_from_string(m) - email_subject = "{}".format(msg['subject']) - email_from = "{}".format(msg['from']) - email_body = "{}".format(msg.as_string()) + for response_part in data: + if isinstance(response_part, tuple): + m = str(response_part[1], 'utf-8') + msg = email.message_from_string(m) + email_subject = "{}".format(msg['subject']) + email_from = "{}".format(msg['from']) + email_body = "{}".format(msg.as_string()) - if (MESSAGE_FROM == email_from) and (MESSAGE_SUBJECT == email_subject) and (MESSAGE_BODY in email_body): - mail.store(str(i), '+FLAGS', '\\Deleted') - mail.close() - return OK, "Bridgedb is good and sending emails with working bridges" - else: - mail.store(str(i), '+FLAGS', '\\Deleted') + if (MESSAGE_FROM == email_from) and (MESSAGE_SUBJECT == email_subject) and (MESSAGE_BODY in email_body): + mail.store(str(i), '+FLAGS', '\\Deleted') + mail.close() + return OK, "Bridgedb is good and sending emails with working bridges" + else: + mail.store(str(i), '+FLAGS', '\\Deleted') mail.close() return WARNING, "No emails from gettor found" - except Exception as e: - return CRITICAL, str(e) def send_email_from_gmail(password): sent_from = FROM_EMAIL @@ -95,11 +108,18 @@ if __name__ == "__main__": if len(sys.argv) == 2: password = sys.argv[1] else: - password = "yourPassword" + return UNKNOWN, "Empty email password" - status_file = open(STATUS_FILE, 'r') - message = status_file.read() - status_file.close() + try: + status_file = open(STATUS_FILE, 'r') + message = status_file.read() + status_file.close() + except OSError: + status = UNKNOWN + message = "Status file has been created {}".format(STATUS_FILE) + status_file = open(STATUS_FILE,'w') + status_file.write("UNKNOWN\n3: %s" % message) + status_file.close() try: status, message = send_email_from_gmail(password) @@ -120,16 +140,9 @@ if __name__ == "__main__": status = CRITICAL message = repr(e) finally: + d = {0: "OK", 1: "WARNING", 2: "CRITICAL", 3: "UNKNOWN"} status_file = open(STATUS_FILE,'w') - if status == OK: - status_file.write("OK\n0: %s" % message) - elif status == WARNING: - status_file.write("WARNING\n1: %s" % message) - elif status == CRITICAL: - status_file.write("CRITICAL\n2: %s" % message) - else: - status_file.write("UNKNOWN\n3: %s" % message) - status = UNKNOWN + status_file.write("{}\n{}: {}" % (d[status], status, message)) status_file.close()

1 0

[bridgedb/develop] Add missing newline at the end of the file.
by phw＠torproject.org 11 May '20

11 May '20

commit 9f8e95de1bc7945869eda4bdde3755eafcdad6bd Author: Philipp Winter <phw(a)nymity.ch> Date: Thu May 7 08:25:22 2020 -0700 Add missing newline at the end of the file. --- scripts/nagios-email-check | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/nagios-email-check b/scripts/nagios-email-check index 7287646..b9fc831 100755 --- a/scripts/nagios-email-check +++ b/scripts/nagios-email-check @@ -149,7 +149,7 @@ def write_status_file(status, message): code = codes.get(status, UNKNOWN) with open(STATUS_FILE, "w") as fd: - fd.write("{}\n{}: {}".format(code, status, message)) + fd.write("{}\n{}: {}\n".format(code, status, message)) log("Wrote status='%s', message='%s' to status file." % (status, message))

1 0

[bridgedb/develop] Fix typos in comment.
by phw＠torproject.org 11 May '20

11 May '20

commit b718662658589c4baaecef7c87aa5c589ed5fb3b Author: Philipp Winter <phw(a)nymity.ch> Date: Tue Apr 21 11:11:15 2020 -0700 Fix typos in comment. --- bridgedb/distributors/https/request.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/bridgedb/distributors/https/request.py b/bridgedb/distributors/https/request.py index 3236bee..0add358 100644 --- a/bridgedb/distributors/https/request.py +++ b/bridgedb/distributors/https/request.py @@ -92,9 +92,8 @@ class HTTPSBridgeRequest(bridgerequest.BridgeRequestBase): :api:`request <twisted.web.http.Request>` ``unblocked=`` HTTP GET parameter will be added to the :data:`notBlockedIn` list. - If :data:`addClientCountryCode` is ``True``, the the client's own - geolocated country code will be added to the to the - :data`notBlockedIn` list. + If :data:`addClientCountryCode` is ``True``, then the client's own + geolocated country code will be added to the :data`notBlockedIn` list. :type request: :api:`twisted.web.http.Request` :param request: A ``Request`` object containing the HTTP method, full

1 0