[tor-commits] [gettor/master] Added specs done at the beginning of gsoc. Not updated to the current status of development, but still useful.
ilv at torproject.org
ilv at torproject.org
Tue Sep 22 23:39:13 UTC 2015
commit 2003220ef45fad665d7c688049266118e9ab9a4f
Author: ilv <ilv at users.noreply.github.com>
Date: Wed Nov 5 20:00:51 2014 -0300
Added specs done at the beginning of gsoc. Not updated to the current status of development, but still useful.
---
spec/design/blacklist.txt | 74 ++++++++++++++++++++++
spec/design/core.txt | 155 +++++++++++++++++++++++++++++++++++++++++++++
spec/design/smtp.txt | 118 ++++++++++++++++++++++++++++++++++
spec/design/twitter.txt | 99 +++++++++++++++++++++++++++++
spec/overview.txt | 94 +++++++++++++++++++++++++++
5 files changed, 540 insertions(+)
diff --git a/spec/design/blacklist.txt b/spec/design/blacklist.txt
new file mode 100644
index 0000000..34ce40e
--- /dev/null
+++ b/spec/design/blacklist.txt
@@ -0,0 +1,74 @@
+ Google Summer of Code 2014 GetTor Revamp - Blacklist module
+ Author: Israel Leiva - <israel.leiva at usach.cl>
+ Last update: 2014-05-16
+ Version: 0.01
+ Changes: First version
+
+1. Preface
+
+ Since GetTor was created it has been a collection of functions and
+ classes separated in various modules. As its main purpose was
+ to serve files over SMTP, almost all current files have SMTP-related
+ procedures, including black and white lists. The proposed design for
+ the blacklist module intends to separate GetTor services from the
+ blacklist procedures.
+
+2. Blacklist module
+
+ The main functionalities the blacklist module should provide are:
+
+ * Check if a given entry is blacklisted for a given service
+ * Add/update/remove entries.
+ * Provide a standard critera to prevent flood.
+
+3. Design
+
+ The new design should consist of the following files, directories and
+ functions:
+
+ * lib/gettor/blacklist/: Directory for storing blacklisted users of
+ different services.
+
+ ----- service1.blacklist: users blocked for service1
+ ----- service2.blacklist: users blocked for service2
+
+ * lib/gettor/blacklist.py: Blacklist module of GetTor.
+
+ isBlacklisted(address, provider)
+ Check if the given address is in the blacklist of services and
+ if it acomplish certain restrictions. For example, it could
+ check when was the last time it was updated on the blacklist.
+
+
+4. Roadmap
+
+ A possible example of how the blacklist module could work:
+
+ a. A service receives a request and call the blacklist module.
+ b. The blacklist module check the blacklist for that service.
+ c. If the user (hashed) is present in the blacklist, it checks when
+ was the last time it was updated. If this date is more than X days
+ ago, updates the entry with the current date and returns false. If
+ not, returns true. If the user is not present in the file, it adds
+ the user with the current date and returns false.
+
+
+5. Discussion
+
+5.1 Mistakes
+
+ Mistakes concerning the package requested should be consider. If I
+ send a message asking for a Linux 'es' request but I write 'en' instead
+ of 'es', should I be able to ask again? or wait until I'm no longer
+ blacklisted?
+
+5.2 Users
+
+ The method presented above (Roadmap) should consider weekly or monthly
+ clean-up of the list.
+
+5.3 SorryMessage
+
+ May be when a user tries to send too many requests we could send him
+ a message saying that he won't be able to ask for packages in the next
+ X days.
diff --git a/spec/design/core.txt b/spec/design/core.txt
new file mode 100644
index 0000000..ce07238
--- /dev/null
+++ b/spec/design/core.txt
@@ -0,0 +1,155 @@
+ Google Summer of Code 2014 GetTor Revamp - Core module
+ Author: Israel Leiva - <israel.leiva at usach.cl, ilv at riseup.net>
+ Last update: 2014-07-15
+ Version: 0.05
+ Changes:
+ [0.05]
+ Added fingerprint support
+ [0.04]
+ Changed log format in 'Design'
+ Added section 'Features'
+ Deleted stuff from 'Discussion' and added some others
+ [0.03]
+ Changed proposed format for link files to RFC 882 (ConfigParser).
+ Read configuration from file with ConfigParser.
+ [0.02]
+ Combine official mirrors with providers links (as another provider).
+ Eliminated on demand link generation. Now it reads from files.
+ Modified description according to PEP-8.
+ [0.01]
+ First version.
+
+
+1. Preface
+
+ Since GetTor was created it has been a collection of functions and
+ classes separated in various modules. As its main purpose was
+ to serve files over SMTP, almost all current files have SMTP-related
+ procedures, including address normalization, message composition, etc.
+ The proposed design for the core module intends to separate GetTor
+ main functionalities which are independent of the service that
+ transports the bundles.
+
+2. Core module
+
+ The main functionalities the core module should provide are:
+
+ * Receive a request with OS version, bundle language, and respond
+ with the respective links.
+ * Read links from providers files.
+ * Log anonymous transactions.
+
+3. Design
+
+ The new design should consist of the following files, directories and
+ methods:
+
+ * gettor.cfg: Configuration values, e.g. base directory.
+
+ * providers/: Directory for generated links. Should be specified on
+ gettor.cfg.
+
+ ----- provider1.links: links from provider1.
+ ----- provider2.links: links from provider2.
+ ----- mirrors.links: links of official mirrors.
+
+ All this data is generated automatically.
+
+ * logs/: Directory for logs. Should be specified on gettor.cfg
+
+ ----- all.log
+ ----- info.log
+ ----- debug.log
+ ----- warn.log
+ ----- error.log
+
+ * Core module of GetTor.
+
+ __init(config_file)__
+ Creates a new Core object. It reads its configuration from
+ the config_file using ConfigParser.
+
+ get_links(operating_system, locale)
+ Public method to obtain the links. Returns links for
+ operating_system in locale language. It checks if the operating
+ system and locale are supported and then calls the private
+ method _get_links()
+
+
+ Example: get_links('linux', 'en')
+
+ _get_links(operating_system, locale)
+ Gets the links for a specific operating system and locale
+ according to the options received. It reads all the .links
+ files inside the providers directory. Each one of these files
+ should follow the ConfigParser's format. There should be a
+ section [provider] with the option 'name' for the provider's
+ name (e.g. Dropbox), and a section [key] with the option
+ 'fingerprint' for the key's fingerprint that signed the
+ uploaded packages.
+
+ Following sections should specify the operating system and
+ its options should be the locale. When more than one link is
+ available per operating system and locale (always) the links
+ should be specified as a multiline value. Each link has the
+ format:
+
+ link link_signature
+
+ Example:
+
+ [provider]
+ name: Dropbox
+
+ [key]
+ fingerprint: 123A 456B 789C 012D 345E 678F 901G 234H 567I 890J
+
+ [linux]
+ en: https://foo.bar https://foo.bar.asc,
+ https://foo.bar https://foo.bar.asc
+
+ es: https://bar.baz https://bar.baz.asc
+
+ PROVIDER NAME
+ operating_system locale link package_signature
+
+ NOTE: For now, the official mirrors are considered just as
+ another provider.
+
+
+ _log_request(operating_system, locale)
+ Log information about the request for future stats (e.g. which
+ OS from which service is the most required). All logging
+ should be done with the logging module.
+
+ * Providers: There should be one module/script per provider in charge
+ of generating the .links file with the proper format, including
+ the official mirrors.
+
+
+4. Roadmap
+
+ An example of how the core module work:
+
+ a. The SMTP service receives a request.
+ b. The SMTP calls core.get_links() with the options sent by the user.
+ c. get_links() calls to _log_request().
+ d. get_links() calls to _get_links().
+ e. get_links() constructs a message with the information obtained.
+ f. get_links() returns the message previously constructed.
+ g. The SMTP service creates a message with the links obtained and
+ send it to the user.
+
+5. Discussion
+
+5.1 Daily logs
+
+ Currently, logs are separated by level of information (debug, info, error).
+ Is it necessary to do this by days/weeks too?
+
+6. Features
+
+ Possible features to be added in the future (open to discussion)
+
+ a. Send HTTP links (currently some official mirrors are HTTP only).
+
diff --git a/spec/design/smtp.txt b/spec/design/smtp.txt
new file mode 100644
index 0000000..e86e864
--- /dev/null
+++ b/spec/design/smtp.txt
@@ -0,0 +1,118 @@
+ Google Summer of Code 2014 GetTor Revamp - SMTP module
+ Author: Israel Leiva - <israel.leiva at usach.cl>
+ Last update: 2014-05-16
+ Version: 0.01
+ Changes: First version
+
+1. Preface
+
+ Since GetTor was created it has been a collection of functions and
+ classes separated in various modules. As its main purpose was
+ to serve files over SMTP, almost all current files have SMTP-related
+ procedures, including address normalization, message composition, etc.
+ The proposed design for the SMTP module intends to separate GetTor
+ main functionalities from the services, in this case, SMTP.
+
+2. SMTP module
+
+ The main functionalities the SMTP module should provide are:
+
+ * Receive requests via mail.
+ * Identify user instructions, such as ask for help or for a specific
+ bundle (OS version, language).
+ * Get the required links from the core module.
+ * Send different types of answers to the user.
+ * Manage black lists to avoid flood.
+ * Log requests for stats (anonymous).
+
+3. Design
+
+ The new design should consist of the following files, directories and
+ functions:
+
+ * lib/gettor/services/smtp.py: SMTP module of GetTor.
+
+ __init__(configuration path)
+ Creates an object according to the configuration values.
+
+ processEmail(email object)
+ Process emails received (by forwarding).
+
+ __parseEmail(email object)
+ Parse the raw email sent by processMail(). Check for multi-part
+ emails and then parse the text part. It also tries to get the
+ locale information from the user's address.
+
+ __parseText(email object)
+ Parse the text part of an email looking for the package requested.
+
+ __getFrom(email object)
+ Returns the from address of an email object.
+
+ __getLocale(address)
+ Tries to get the locale information from an email address.
+
+ __checkBlacklist(address)
+ Check if the given address is blacklisted by comparing the
+ hashed address. If address is not present, it's added. If present,
+ check for the date when it was added. Yet to define how many
+ days will be considered for blacklisting or if another method
+ will be used. For this it uses the blacklist module.
+
+ __sendReply(address)
+ Sends a reply to the user with the links required. It asks for
+ the links to the core module.
+
+ __sendDelayAlert(address)
+ If enabled (on configuration), sends a delay message to the user
+ letting him know that the links are on their way.
+
+ __sendHelp(address)
+ Sends a message to the user with help instructions.
+
+ __createEmail(from, to, subject, body)
+ Creates an email object.
+
+ __logRequest(options)
+ Log information about the request for future stats (e.g. which
+ OS and language are the most required). If this is called
+ after a failure, a copy of the email should be stored.
+
+ * BASE_DIR/logs/: Directory for logs. The BASE_DIR should be in the
+ configuration file.
+
+ ----- yyyy-mm-dd.log: daily log of requests.
+
+
+4. Roadmap
+
+ An example of how the SMTP module should work:
+
+ a. The SMTP service receives a request (via forwarding).
+ b. The email sender is checked for blacklisting (comparing hashes).
+ c. The email is parsed, obtaining the package requested and the
+ locale information.
+ d. If the email was asking for help, a help reply is sent.
+ e. If the email was invalid, the process break. This fail is logged
+ and the email that triggered it, too.
+ f. If the email was valid and the delay alert is set, then a reply
+ informing the links are on their way is sent.
+ g. If the email was valid, the SMTP service asks for the links to the
+ code module.
+ h. After that, a reply is sent to the user.
+ i. In all cases the request is logged (with no user information).
+
+
+5. Discussion
+
+5.1 Email forwarding
+
+ Are we going to support forwarding emails as ForwardPackage() did in
+ the old GetTor?
+
+5.2 Blacklist sublists
+
+ Now with less types of request (two if no forwarding is added), creating
+ sublists for different types of requests necessary to blacklist and
+ email address? Or should we blacklist it if it floods anything?
+
diff --git a/spec/design/twitter.txt b/spec/design/twitter.txt
new file mode 100644
index 0000000..46a178a
--- /dev/null
+++ b/spec/design/twitter.txt
@@ -0,0 +1,99 @@
+ Google Summer of Code 2014 GetTor Revamp - Twitter module
+ Author: Israel Leiva - <israel.leiva at usach.cl>
+ Last update: 2014-05-16
+ Version: 0.01
+ Changes: First version
+
+1. Preface
+
+ Since GetTor was created it has been a collection of functions and
+ classes separated in various modules. As its main purpose was
+ to serve files over SMTP, almost all current files have SMTP-related
+ procedures, including address normalization, message composition, etc.
+ The proposed design for the Twitter module intends to separate GetTor
+ main functionalities from the services, in this case, Twitter.
+
+2. Twitter module
+
+ The main functionalities the Twitter module should provide are:
+
+ * Receive requests via direct messages.
+ * Identify user instructions, such as ask for help or for a specific
+ bundle (OS version, language).
+ * Get the required links from the core module.
+ * Send different types of answers to the user.
+ * Split answers to fit Twitter's format.
+ * Manage black lists to avoid flood.
+ * Log requests for stats (anonymous).
+
+3. Design
+
+ The new design should consist of the following files, directories and
+ functions:
+
+ * lib/gettor/services/Twitter.py: Twitter module of GetTor.
+
+ __init__(configuration path)
+ Creates an object according to the configuration values.
+
+ processDM(message)
+ Process direct messages received.
+
+ __parseDM(message)
+ Parse the direct message received. Check for the package requested
+ and the locale information.
+
+ __getUser(message)
+ Gets the user from the message sent.
+
+ __checkBlacklist(user)
+ Check if the given user is blacklisted by comparing the
+ hashed user. Yet to define how many days will be considered for
+ blacklisting or if another method will be used. For this it uses
+ the blacklist module.
+
+ __sendReply(user)
+ Sends a reply to the user with the links required. It asks for
+ the links to the core module and then split them.
+
+ __sendHelp(user)
+ Sends a message to the user with help instructions.
+
+ __splitMessage(message)
+ Receives the links message and split it according to Twitter's
+ format.
+
+ __CheckNewFollowers()
+ In order to ask for links the user has to follow the GetTor
+ account. The Twitter module will be constantly checking for
+ new followers and follow them back.
+
+ __FollowUser(user)
+ Follow the given user.
+
+ __logRequest(options)
+ Log information about the request for future stats (e.g. which
+ OS and language are the most required). If this is called
+ after a failure, a copy of the DM should be stored.
+
+ * BASE_DIR/logs/: Directory for logs. BASE_DIR should be in the
+ configuration file.
+
+ ----- yyyy-mm-dd.log: daily log of requests.
+
+
+4. Roadmap
+
+ An example of how the Twitter module should work:
+
+ a. The Twitter account receives a DM.
+ b. The Twitter service check if is a valid message and if the user is
+ in the blacklist, and then tries to obtain the package requested and
+ the locale information.
+ c. The Twitter service asks for the links to the core module, then it
+ splits the message received to adopt Twitter's format.
+ d. One or more DMs are sent back to the user.
+ e. For all this, the user must follow the GetTor account. The Twitter
+ service will be constantly checking for new followers and following
+ them back.
+
diff --git a/spec/overview.txt b/spec/overview.txt
new file mode 100644
index 0000000..435d256
--- /dev/null
+++ b/spec/overview.txt
@@ -0,0 +1,94 @@
+ Google Summer of Code 2014 GetTor Revamp - Overview
+ Author: Israel Leiva - <israel.leiva at usach.cl>
+ Last update: 2014-05-16
+ Version: 0.01
+ Changes: First version
+
+1. Background
+
+ GetTor was created as a program for serving Tor and related files over
+ SMTP, thus avoiding direct and indirect censorship of Tor's software,
+ in particular, the Tor Browser Bundle (TBB). Users interact with GetTor
+ by sending emails to a specific email address. In the past, after the
+ user specified his OS and language, GetTor would send him an attachment
+ with the required package. This worked until the bundles were too large
+ to be sent as attachments in most email providers. In order to fix this
+ GetTor started to send links instead.
+
+2. Current status
+
+ The GetTor status can be summarized in the following points:
+
+ * Emails are sent to gettor at torproject.org
+ * The GetTor reply contains: TBB links, signatures (with text guides
+ for verification), mirrors, support instructions in six languages.
+ * Dropbox links are sent to download the TBB and signatures.
+ * Users can not ask for packages in their language.
+ * English-only replies are sent.
+ * Any email directed to GetTor is replied with the same information,
+ there is no recognition of instructions.
+ * Links generation is not fully automated.
+ * All code is written in Python. Various parts are not currently used.
+ * Current repositories are [0] and [1].
+
+3. Proposal
+
+ The accepted proposal [2] for Google Summer of Code (GSoC) 2014 proposes
+ rewriting the current GetTor using a modular design, with a core module
+ that handles the main GetTor functionalities, and several other modules,
+ one for each service (e.g. SMTP), which can interact with the core and
+ send replies to the users. Three modules will be developed for the
+ purposes of GSoC: SMTP, Twitter, Skype|XMPP.
+
+3.1. Goals
+
+ The main goals of this proposal are the following:
+
+ * Provide old GetTor functionalites, such as replies in several
+ languages and recognize user instructions.
+ * Send fewer information in each reply.
+ * Support more providers for uploading the TBB packages.
+ * Automate links generation.
+ * Clearer, modular and well-documented code.
+ * Possibilty to create new modules for other common services.
+
+3.2. Design
+
+ Preliminar designs for the core module and the services can be found
+ in the design/ folder. All services consider creating a python script
+ to add the logic for using them. For example, there should be a script
+ that receives the emails and uses the SMTP module. For simplicity,
+ I've tried to specify mostly the main functions of every module; there
+ are some functions, like opening and writing files, that were not
+ considered in this preliminar phase.
+
+4. Discussion
+
+4.1. Skype
+
+ My co-mentor for GSoC, Nima, has publicly rejected the idea of creating
+ a module for Skype and proposed to implement one for XMPP instead.
+ I've chosen Skype for its popularity, but I have no other main reason
+ to maintain this option, so it's probable that XMPP transport will be
+ implemented.
+
+4.2. Storing links
+
+ My original proposal considered the fact that links could be stored
+ somewhere with restricted access, ideally a git repository. Nima
+ mentioned that ideally the links shouldn't be stored. May be this idea
+ could be used only to the mirrors and providers configuration (see
+ core module design).
+
+4.3. Generating unique URLs
+
+ Nima mentioned that unique URLs could be generated for each request,
+ and in case the user don't have access to SSL, these links could be
+ served and later deleted or recycled. I like this idea.
+
+4. References
+
+ [0] https://gitweb.torproject.org/gettor.git
+ [1] https://gitweb.torproject.org/user/sukhbir/gettor.git
+ [2] https://ileiva.github.io/gettor_proposal.html
+
More information about the tor-commits
mailing list