September 2015 - tor-commits - lists.torproject.org

[gettor/master] Initial commit
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit 94e2e095fb4f684ad9dd004062a198967213cfde Author: ilv <ilv(a)users.noreply.github.com> Date: Thu May 15 23:04:33 2014 -0700 Initial commit --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md new file mode 100644 index 0000000..2236c7b --- /dev/null +++ b/README.md @@ -0,0 +1,4 @@ +gettor +====== + +GetTor

1 0

[gettor/master] Simple description
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit ad9888fa2a4be98cf04221889fb0a00a040c7b80 Author: ilv <ilv(a)users.noreply.github.com> Date: Fri May 16 02:08:33 2014 -0400 Simple description --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2236c7b..ac27501 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ gettor ====== -GetTor +GetTor Revamp. +Google Summer of Code 2014.

1 0

[gettor/master] Created the basis of GetTor's Core class. Modified the proposed design to read links from files instead of generating on demand.
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit eef1f3f1975881f08090b6825369126f10c77031 Author: ilv <ilv(a)users.noreply.github.com> Date: Mon Jun 2 00:42:00 2014 -0400 Created the basis of GetTor's Core class. Modified the proposed design to read links from files instead of generating on demand. --- spec/design/core.txt | 133 +++++++++++++++++++++++---------------- src/core_demo.py | 19 ++++++ src/gettor.py | 145 +++++++++++++++++++++++++++++++++++++++++++ src/gettor.pyc | Bin 0 -> 3934 bytes src/providers/dropbox.links | 5 ++ src/providers/gdrive.links | 5 ++ src/providers/mirrors.links | 5 ++ 7 files changed, 257 insertions(+), 55 deletions(-) diff --git a/spec/design/core.txt b/spec/design/core.txt index ecb6951..f90c6f6 100644 --- a/spec/design/core.txt +++ b/spec/design/core.txt @@ -1,8 +1,12 @@ - Google Summer of Code 2014 GetTor Revamp - Code module + Google Summer of Code 2014 GetTor Revamp - Core module Author: Israel Leiva - <israel.leiva(a)usach.cl> - Last update: 2014-05-16 - Version: 0.01 - Changes: First version + Last update: 2014-06-01 + Version: 0.02 + Changes: Combine official mirrors with providers links (as another provider). + Eliminated on demand link generation. Now it reads from files. + Modified description according to PEP-8. + First version. + 1. Preface @@ -18,90 +22,109 @@ The main functionalities the core module should provide are: - * Receive a request with OS version, architecture, bundle - language, and respond with the respective links. - * Generate links, per request or at demand, depending on if the - former is accepted as part of the new design. + * Receive a request with OS version, bundle language, and respond + with the respective links. + * Read links from providers files. * Log anonymous transactions. 3. Design The new design should consist of the following files, directories and - functions: + methods: * core.conf: Configuration values, e.g. base directory. - * providers/: Directory for providers configuration. + * providers/: Directory for generated links. - ----- providersList.txt: list of valid providers. - ----- provider1.conf: configuration for provider1. - ----- provider2.conf: configuration for provider2. + ----- provider1.links: links from provider1. + ----- provider2.links: links from provider2. + ----- mirrors.links: links of official mirrors. - All this data is added manually. - - * mirrors.txt: Contains official mirrors. One per line. Added manually. + All this data is generated automatically. * logs/: Directory for logs. Added automatically. - ----- yyyy-mm-dd.log: daily log of requests. + ----- core_yyyy-mm-dd.log: daily log of requests. - * lib/gettor/core.py: Core module of GetTor. + * Core module of GetTor. - getLinks(os_version, locale) - Returns links for os_version (in both archs) in locale language. - This will read the providers list and call __generateLinks() - for each one of them, plus calling to __getMirrors(). - - Example: getLinks('OSX', 'en') + get_links(operating_system, locale) + Public method to obtain the links. Returns links for + operating_system in locale language. It checks if the operating + system and locale are supported and then calls the private + method _get_links() + - __generateLinks(options, provider) - Generate links for a specific provider according to the options - received (os_version, locale). This will try to import the - provider module and call the uploadBundle function. + Example: get_links('linux', 'en') - Example (within the module): __generateLinks(options, 'dropbox') + _get_links(operating_system, locale) + Gets the links for a specific operating system and locale + according to the options received. It reads all the .links + files inside the providers directory. Each one of these files + contains the provider's name in the first line and all the + links after. The format of these files is as following: + + PROVIDER NAME + operating_system locale link package_signature key_fingerprint + + Example: + + Dropbox + linux en https://foo.bar https://foo.bar.asc 111-222-333-444 + osx es https://bar.baz https://bar.baz.asc 555-666-777-888 + + The key fingerprint is joined by '-' characters for convenience, + but it's sent back to the service module separated by spaces. + + The official mirrors are considered as just another provider. - __getMirrors() - Obtains mirrors from mirrors.txt. - __logRequest(options) + _log_request(operating_system, locale) Log information about the request for future stats (e.g. which - OS from which service is the most required). + OS from which service is the most required). All logging + should be done with the logging module. - * lib/gettor/providers/provider.py: There should be one module per - provider with the uploadBundle public function. There should be - at least three modules at the end of GSoC: dropbox.py, drive.py, - github.py - - uploadBundle(options) - Calls the provider internal functions to upload the required - bundle according to the options received. This internal - functions will depend solely on the API requirements from - the provider. + * Providers: There should be one module/script per provider in charge + of generating the .links file with the proper format, including + the official mirrors. + 4. Roadmap An example of how the core module work: a. The SMTP service receives a request. - b. The SMTP calls getLinks() with the options sent by the user. - c. getLinks() calls to __generateLinks() and then to __getMirrors() - d. getLinks() constructs a message with the information obtained. - e. getLinks() calls to __logRequest(). - f. getLinks() returns the message previously constructed. + b. The SMTP calls core.get_links() with the options sent by the user. + c. get_links() calls to _log_request(). + d. get_links() calls to _get_links(). + e. get_links() constructs a message with the information obtained. + f. get_links() returns the message previously constructed. g. The SMTP service creates a message with the links obtained and send it to the user. 5. Discussion -5.1 Cache +5.1 On demand generation + + Now we consider links generation over a period of time, say, once a week. + Generating links on demand don't seem to be necessary and it would + certainly be resource consuming. The core module doesn't care about + how links are generated, it just assumes they exist. Is there any case + where on demand link generation should be considered? - The above design was thought for per request links generation. Another - way of doing this would be to maintain a cache of generated links and - call __generateLinks() depending on the cache last modified time. - Reading links from this cache should consider to check if the given - links still exists. +5.2 Core class + + Should the Core be a module on its own, or a class of GetTor? + Currently it's just a class, after realizing that + + core = gettor.core.core() + + Was too much. It's better to do + + core = gettor.Core() + + Isn't? -5.2 Logs +5.3 Logs Should we mantain separate logs for successful and fail requests? diff --git a/src/core_demo.py b/src/core_demo.py new file mode 100644 index 0000000..19ffd58 --- /dev/null +++ b/src/core_demo.py @@ -0,0 +1,19 @@ +#!/usr/bin/python +# +# Dummy script to test GetTore's Core module progress +# + +import gettor + +core = gettor.Core() + +try: + links = core.get_links('linux', 'en') + print links +except ValueError as e: + print "Value error! " + str(e) +except RuntimeError as e: + print "Internal error! " + str(e) + + + diff --git a/src/gettor.py b/src/gettor.py new file mode 100644 index 0000000..573bae7 --- /dev/null +++ b/src/gettor.py @@ -0,0 +1,145 @@ +import os +import re +import inspect +import logging + + +class Core: + + """ + Gets links from providers and delivers them to other modules. + + Public methods: + get_links(operating_system, locale) + """ + + def __init__(self): + """ + Initialize a Core object by reading a configuration file. + + Parameters: none + """ + + # Read configuration file + + + # Dummy info. This should be read from a configuration file. + self.basedir = './providers/' + self.supported_locales = [ 'en', 'es' ] + self.supported_os = [ 'windows', 'linux', 'osx' ] + + def get_links(self, operating_system, locale): + """ + Public method to obtain links. + + Checks for supported locales and operating systems. It returns + ValueError if the locale or operating system is not supported. + It raises RuntimeError if something goes wrong while trying + to obtain the links. It returns a string on success. This + method should be called from the services modules of GetTor + (e.g. SMTP). + """ + + # Log requests + self._log_request(operating_system, locale) + + if locale not in self.supported_locales: + raise ValueError("Locale %s not supported at the moment" % locale) + + if operating_system not in self.supported_os: + raise ValueError("Operating system %s not supported at the moment" + % operating_system) + + # This could change in the future, let's leave it isolated. + links = self._get_links(operating_system, locale) + + if links is None: + raise RuntimeError("Something went wrong with GetTor's core") + + return links + + def _get_links(self, operating_system, locale): + """ + Private method to obtain the links. + + Looks for the links inside each provider file. On success + returns a string with the links. On failure returns None. + This should only be called from get_links() method. + + Parameters: + os: string describing the operating system + locale: string describing the locale + """ + + # There should be a 'providers' directory inside self.basedir + # + # Each .links file begins with a string describing the provider. + # After that, every line should have the following format: + # + # operating_system locale link package_signature key_fingerprint + # + # e.g. + # + # Dropbox + # linux en https://foo.bar https://foo.bar.asc 111-222-333-444 + # osx es https://bar.baz https://bar.baz.asc 555-666-777-888 + # + links = [] + + # Look for files ending with .links + p = re.compile('.*\.links$') + + for name in os.listdir(self.basedir): + path = os.path.abspath(os.path.join(self.basedir, name)) + if os.path.isfile(path) and p.match(path): + links.append(path) + + # Let's create a dictionary linking each provider with the links + # found for operating_system and locale. This way makes it easy + # to check if no links were found. + providers = {} + + # We trust links have been generated properly + for name in links: + link_file = open(name, 'r') + provider_name = link_file.readline() + + for line in link_file: + words = line.split(' ') + if words[0] == operating_system and words[1] == locale: + providers.setdefault(provider_name, []).append( + "%s %s %s" % (words[2], words[3], words[4].replace('-', ' ')) + ) + + link_file.close() + + # Create the final links list with all providers + all_links = [] + + for key in providers.keys(): + all_links.append( + "\n%s\n%s\n" % (key, ''.join(providers[key])) + ) + + if all_links: + return "".join(all_links) + else: + return None + + + + def _log_request(self, operating_system, locale): + """ + Private method to log what service module called to get the links. + + Parameters: none + """ + + caller = inspect.stack()[2] + module = inspect.getmodule(caller[0]) + + # Dummy print for now. Should be done with logging + print "\nCalled by %s\nOS: %s\nLocale: %s\n" % \ + (str(module), operating_system, locale) + + diff --git a/src/gettor.pyc b/src/gettor.pyc new file mode 100644 index 0000000..81b41ac Binary files /dev/null and b/src/gettor.pyc differ diff --git a/src/providers/dropbox.links b/src/providers/dropbox.links new file mode 100644 index 0000000..53d4a04 --- /dev/null +++ b/src/providers/dropbox.links @@ -0,0 +1,5 @@ +Dropbox + +linux en https://foo.bar https://foo.bar.asc 111-222-333-444 +linux es https://bar.baz https://bar.baz.asc 555-666-777-888 +linux en https://baz.foo https://baz.foo.asc 999-111-222-333 diff --git a/src/providers/gdrive.links b/src/providers/gdrive.links new file mode 100644 index 0000000..570c8e2 --- /dev/null +++ b/src/providers/gdrive.links @@ -0,0 +1,5 @@ +Google Drive + +linux en https://foo.bar https://foo.bar.asc 111-222-333-444 +linux es https://bar.baz https://bar.baz.asc 555-666-777-888 +linux en https://baz.foo https://baz.foo.asc 999-111-222-333 diff --git a/src/providers/mirrors.links b/src/providers/mirrors.links new file mode 100644 index 0000000..0eb5b10 --- /dev/null +++ b/src/providers/mirrors.links @@ -0,0 +1,5 @@ +Official Mirrors + +linux en https://foo.bar https://foo.bar.asc 111-222-333-444 +linux es https://bar.baz https://bar.baz.asc 555-666-777-888 +linux en https://baz.foo https://baz.foo.asc 999-111-222-333

1 0

[gettor/master] We're talking about services, not providers
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit 18c3c4c2bc6dc6eb18e24bc74e090256553dc852 Author: ilv <ilv(a)users.noreply.github.com> Date: Fri May 16 02:34:32 2014 -0400 We're talking about services, not providers --- spec/design/blacklist.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/design/blacklist.txt b/spec/design/blacklist.txt index 0cabaa2..34ce40e 100644 --- a/spec/design/blacklist.txt +++ b/spec/design/blacklist.txt @@ -35,7 +35,7 @@ * lib/gettor/blacklist.py: Blacklist module of GetTor. isBlacklisted(address, provider) - Check if the given address is in the blacklist of provider and + Check if the given address is in the blacklist of services and if it acomplish certain restrictions. For example, it could check when was the last time it was updated on the blacklist.

1 0

[gettor/master] Deleted arch paremeter in getLinks(). Get links for both archs and let the user decide
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit 94f9942199b90fc6de444a8068493d189f54af63 Author: ilv <ilv(a)users.noreply.github.com> Date: Fri May 23 01:26:58 2014 -0400 Deleted arch paremeter in getLinks(). Get links for both archs and let the user decide --- spec/design/core.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/design/core.txt b/spec/design/core.txt index 447c93b..ecb6951 100644 --- a/spec/design/core.txt +++ b/spec/design/core.txt @@ -47,7 +47,7 @@ * lib/gettor/core.py: Core module of GetTor. - getLinks(os_version, arch, locale) + getLinks(os_version, locale) Returns links for os_version (in both archs) in locale language. This will read the providers list and call __generateLinks() for each one of them, plus calling to __getMirrors().

1 0

[gettor/master] Initial version for GetYor new spec
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit bd2dd1ea00e79dcf60417f160d65ef062b0eef86 Author: ilv <ilv(a)users.noreply.github.com> Date: Fri May 16 02:09:36 2014 -0400 Initial version for GetYor new spec --- spec/design/blacklist.txt | 74 ++++++++++++++++++++++++++++ spec/design/core.txt | 107 ++++++++++++++++++++++++++++++++++++++++ spec/design/smtp.txt | 118 +++++++++++++++++++++++++++++++++++++++++++++ spec/design/twitter.txt | 99 +++++++++++++++++++++++++++++++++++++ spec/overview.txt | 94 ++++++++++++++++++++++++++++++++++++ 5 files changed, 492 insertions(+) diff --git a/spec/design/blacklist.txt b/spec/design/blacklist.txt new file mode 100644 index 0000000..0cabaa2 --- /dev/null +++ b/spec/design/blacklist.txt @@ -0,0 +1,74 @@ + Google Summer of Code 2014 GetTor Revamp - Blacklist module + Author: Israel Leiva - <israel.leiva(a)usach.cl> + Last update: 2014-05-16 + Version: 0.01 + Changes: First version + +1. Preface + + Since GetTor was created it has been a collection of functions and + classes separated in various modules. As its main purpose was + to serve files over SMTP, almost all current files have SMTP-related + procedures, including black and white lists. The proposed design for + the blacklist module intends to separate GetTor services from the + blacklist procedures. + +2. Blacklist module + + The main functionalities the blacklist module should provide are: + + * Check if a given entry is blacklisted for a given service + * Add/update/remove entries. + * Provide a standard critera to prevent flood. + +3. Design + + The new design should consist of the following files, directories and + functions: + + * lib/gettor/blacklist/: Directory for storing blacklisted users of + different services. + + ----- service1.blacklist: users blocked for service1 + ----- service2.blacklist: users blocked for service2 + + * lib/gettor/blacklist.py: Blacklist module of GetTor. + + isBlacklisted(address, provider) + Check if the given address is in the blacklist of provider and + if it acomplish certain restrictions. For example, it could + check when was the last time it was updated on the blacklist. + + +4. Roadmap + + A possible example of how the blacklist module could work: + + a. A service receives a request and call the blacklist module. + b. The blacklist module check the blacklist for that service. + c. If the user (hashed) is present in the blacklist, it checks when + was the last time it was updated. If this date is more than X days + ago, updates the entry with the current date and returns false. If + not, returns true. If the user is not present in the file, it adds + the user with the current date and returns false. + + +5. Discussion + +5.1 Mistakes + + Mistakes concerning the package requested should be consider. If I + send a message asking for a Linux 'es' request but I write 'en' instead + of 'es', should I be able to ask again? or wait until I'm no longer + blacklisted? + +5.2 Users + + The method presented above (Roadmap) should consider weekly or monthly + clean-up of the list. + +5.3 SorryMessage + + May be when a user tries to send too many requests we could send him + a message saying that he won't be able to ask for packages in the next + X days. diff --git a/spec/design/core.txt b/spec/design/core.txt new file mode 100644 index 0000000..447c93b --- /dev/null +++ b/spec/design/core.txt @@ -0,0 +1,107 @@ + Google Summer of Code 2014 GetTor Revamp - Code module + Author: Israel Leiva - <israel.leiva(a)usach.cl> + Last update: 2014-05-16 + Version: 0.01 + Changes: First version + +1. Preface + + Since GetTor was created it has been a collection of functions and + classes separated in various modules. As its main purpose was + to serve files over SMTP, almost all current files have SMTP-related + procedures, including address normalization, message composition, etc. + The proposed design for the core module intends to separate GetTor + main functionalities which are independent of the service that + transports the bundles. + +2. Core module + + The main functionalities the core module should provide are: + + * Receive a request with OS version, architecture, bundle + language, and respond with the respective links. + * Generate links, per request or at demand, depending on if the + former is accepted as part of the new design. + * Log anonymous transactions. + +3. Design + + The new design should consist of the following files, directories and + functions: + + * core.conf: Configuration values, e.g. base directory. + + * providers/: Directory for providers configuration. + + ----- providersList.txt: list of valid providers. + ----- provider1.conf: configuration for provider1. + ----- provider2.conf: configuration for provider2. + + All this data is added manually. + + * mirrors.txt: Contains official mirrors. One per line. Added manually. + + * logs/: Directory for logs. Added automatically. + + ----- yyyy-mm-dd.log: daily log of requests. + + * lib/gettor/core.py: Core module of GetTor. + + getLinks(os_version, arch, locale) + Returns links for os_version (in both archs) in locale language. + This will read the providers list and call __generateLinks() + for each one of them, plus calling to __getMirrors(). + + Example: getLinks('OSX', 'en') + + __generateLinks(options, provider) + Generate links for a specific provider according to the options + received (os_version, locale). This will try to import the + provider module and call the uploadBundle function. + + Example (within the module): __generateLinks(options, 'dropbox') + + __getMirrors() + Obtains mirrors from mirrors.txt. + + __logRequest(options) + Log information about the request for future stats (e.g. which + OS from which service is the most required). + + * lib/gettor/providers/provider.py: There should be one module per + provider with the uploadBundle public function. There should be + at least three modules at the end of GSoC: dropbox.py, drive.py, + github.py + + uploadBundle(options) + Calls the provider internal functions to upload the required + bundle according to the options received. This internal + functions will depend solely on the API requirements from + the provider. + +4. Roadmap + + An example of how the core module work: + + a. The SMTP service receives a request. + b. The SMTP calls getLinks() with the options sent by the user. + c. getLinks() calls to __generateLinks() and then to __getMirrors() + d. getLinks() constructs a message with the information obtained. + e. getLinks() calls to __logRequest(). + f. getLinks() returns the message previously constructed. + g. The SMTP service creates a message with the links obtained and + send it to the user. + +5. Discussion + +5.1 Cache + + The above design was thought for per request links generation. Another + way of doing this would be to maintain a cache of generated links and + call __generateLinks() depending on the cache last modified time. + Reading links from this cache should consider to check if the given + links still exists. + +5.2 Logs + + Should we mantain separate logs for successful and fail requests? diff --git a/spec/design/smtp.txt b/spec/design/smtp.txt new file mode 100644 index 0000000..e86e864 --- /dev/null +++ b/spec/design/smtp.txt @@ -0,0 +1,118 @@ + Google Summer of Code 2014 GetTor Revamp - SMTP module + Author: Israel Leiva - <israel.leiva(a)usach.cl> + Last update: 2014-05-16 + Version: 0.01 + Changes: First version + +1. Preface + + Since GetTor was created it has been a collection of functions and + classes separated in various modules. As its main purpose was + to serve files over SMTP, almost all current files have SMTP-related + procedures, including address normalization, message composition, etc. + The proposed design for the SMTP module intends to separate GetTor + main functionalities from the services, in this case, SMTP. + +2. SMTP module + + The main functionalities the SMTP module should provide are: + + * Receive requests via mail. + * Identify user instructions, such as ask for help or for a specific + bundle (OS version, language). + * Get the required links from the core module. + * Send different types of answers to the user. + * Manage black lists to avoid flood. + * Log requests for stats (anonymous). + +3. Design + + The new design should consist of the following files, directories and + functions: + + * lib/gettor/services/smtp.py: SMTP module of GetTor. + + __init__(configuration path) + Creates an object according to the configuration values. + + processEmail(email object) + Process emails received (by forwarding). + + __parseEmail(email object) + Parse the raw email sent by processMail(). Check for multi-part + emails and then parse the text part. It also tries to get the + locale information from the user's address. + + __parseText(email object) + Parse the text part of an email looking for the package requested. + + __getFrom(email object) + Returns the from address of an email object. + + __getLocale(address) + Tries to get the locale information from an email address. + + __checkBlacklist(address) + Check if the given address is blacklisted by comparing the + hashed address. If address is not present, it's added. If present, + check for the date when it was added. Yet to define how many + days will be considered for blacklisting or if another method + will be used. For this it uses the blacklist module. + + __sendReply(address) + Sends a reply to the user with the links required. It asks for + the links to the core module. + + __sendDelayAlert(address) + If enabled (on configuration), sends a delay message to the user + letting him know that the links are on their way. + + __sendHelp(address) + Sends a message to the user with help instructions. + + __createEmail(from, to, subject, body) + Creates an email object. + + __logRequest(options) + Log information about the request for future stats (e.g. which + OS and language are the most required). If this is called + after a failure, a copy of the email should be stored. + + * BASE_DIR/logs/: Directory for logs. The BASE_DIR should be in the + configuration file. + + ----- yyyy-mm-dd.log: daily log of requests. + + +4. Roadmap + + An example of how the SMTP module should work: + + a. The SMTP service receives a request (via forwarding). + b. The email sender is checked for blacklisting (comparing hashes). + c. The email is parsed, obtaining the package requested and the + locale information. + d. If the email was asking for help, a help reply is sent. + e. If the email was invalid, the process break. This fail is logged + and the email that triggered it, too. + f. If the email was valid and the delay alert is set, then a reply + informing the links are on their way is sent. + g. If the email was valid, the SMTP service asks for the links to the + code module. + h. After that, a reply is sent to the user. + i. In all cases the request is logged (with no user information). + + +5. Discussion + +5.1 Email forwarding + + Are we going to support forwarding emails as ForwardPackage() did in + the old GetTor? + +5.2 Blacklist sublists + + Now with less types of request (two if no forwarding is added), creating + sublists for different types of requests necessary to blacklist and + email address? Or should we blacklist it if it floods anything? + diff --git a/spec/design/twitter.txt b/spec/design/twitter.txt new file mode 100644 index 0000000..46a178a --- /dev/null +++ b/spec/design/twitter.txt @@ -0,0 +1,99 @@ + Google Summer of Code 2014 GetTor Revamp - Twitter module + Author: Israel Leiva - <israel.leiva(a)usach.cl> + Last update: 2014-05-16 + Version: 0.01 + Changes: First version + +1. Preface + + Since GetTor was created it has been a collection of functions and + classes separated in various modules. As its main purpose was + to serve files over SMTP, almost all current files have SMTP-related + procedures, including address normalization, message composition, etc. + The proposed design for the Twitter module intends to separate GetTor + main functionalities from the services, in this case, Twitter. + +2. Twitter module + + The main functionalities the Twitter module should provide are: + + * Receive requests via direct messages. + * Identify user instructions, such as ask for help or for a specific + bundle (OS version, language). + * Get the required links from the core module. + * Send different types of answers to the user. + * Split answers to fit Twitter's format. + * Manage black lists to avoid flood. + * Log requests for stats (anonymous). + +3. Design + + The new design should consist of the following files, directories and + functions: + + * lib/gettor/services/Twitter.py: Twitter module of GetTor. + + __init__(configuration path) + Creates an object according to the configuration values. + + processDM(message) + Process direct messages received. + + __parseDM(message) + Parse the direct message received. Check for the package requested + and the locale information. + + __getUser(message) + Gets the user from the message sent. + + __checkBlacklist(user) + Check if the given user is blacklisted by comparing the + hashed user. Yet to define how many days will be considered for + blacklisting or if another method will be used. For this it uses + the blacklist module. + + __sendReply(user) + Sends a reply to the user with the links required. It asks for + the links to the core module and then split them. + + __sendHelp(user) + Sends a message to the user with help instructions. + + __splitMessage(message) + Receives the links message and split it according to Twitter's + format. + + __CheckNewFollowers() + In order to ask for links the user has to follow the GetTor + account. The Twitter module will be constantly checking for + new followers and follow them back. + + __FollowUser(user) + Follow the given user. + + __logRequest(options) + Log information about the request for future stats (e.g. which + OS and language are the most required). If this is called + after a failure, a copy of the DM should be stored. + + * BASE_DIR/logs/: Directory for logs. BASE_DIR should be in the + configuration file. + + ----- yyyy-mm-dd.log: daily log of requests. + + +4. Roadmap + + An example of how the Twitter module should work: + + a. The Twitter account receives a DM. + b. The Twitter service check if is a valid message and if the user is + in the blacklist, and then tries to obtain the package requested and + the locale information. + c. The Twitter service asks for the links to the core module, then it + splits the message received to adopt Twitter's format. + d. One or more DMs are sent back to the user. + e. For all this, the user must follow the GetTor account. The Twitter + service will be constantly checking for new followers and following + them back. + diff --git a/spec/overview.txt b/spec/overview.txt new file mode 100644 index 0000000..435d256 --- /dev/null +++ b/spec/overview.txt @@ -0,0 +1,94 @@ + Google Summer of Code 2014 GetTor Revamp - Overview + Author: Israel Leiva - <israel.leiva(a)usach.cl> + Last update: 2014-05-16 + Version: 0.01 + Changes: First version + +1. Background + + GetTor was created as a program for serving Tor and related files over + SMTP, thus avoiding direct and indirect censorship of Tor's software, + in particular, the Tor Browser Bundle (TBB). Users interact with GetTor + by sending emails to a specific email address. In the past, after the + user specified his OS and language, GetTor would send him an attachment + with the required package. This worked until the bundles were too large + to be sent as attachments in most email providers. In order to fix this + GetTor started to send links instead. + +2. Current status + + The GetTor status can be summarized in the following points: + + * Emails are sent to gettor(a)torproject.org + * The GetTor reply contains: TBB links, signatures (with text guides + for verification), mirrors, support instructions in six languages. + * Dropbox links are sent to download the TBB and signatures. + * Users can not ask for packages in their language. + * English-only replies are sent. + * Any email directed to GetTor is replied with the same information, + there is no recognition of instructions. + * Links generation is not fully automated. + * All code is written in Python. Various parts are not currently used. + * Current repositories are [0] and [1]. + +3. Proposal + + The accepted proposal [2] for Google Summer of Code (GSoC) 2014 proposes + rewriting the current GetTor using a modular design, with a core module + that handles the main GetTor functionalities, and several other modules, + one for each service (e.g. SMTP), which can interact with the core and + send replies to the users. Three modules will be developed for the + purposes of GSoC: SMTP, Twitter, Skype|XMPP. + +3.1. Goals + + The main goals of this proposal are the following: + + * Provide old GetTor functionalites, such as replies in several + languages and recognize user instructions. + * Send fewer information in each reply. + * Support more providers for uploading the TBB packages. + * Automate links generation. + * Clearer, modular and well-documented code. + * Possibilty to create new modules for other common services. + +3.2. Design + + Preliminar designs for the core module and the services can be found + in the design/ folder. All services consider creating a python script + to add the logic for using them. For example, there should be a script + that receives the emails and uses the SMTP module. For simplicity, + I've tried to specify mostly the main functions of every module; there + are some functions, like opening and writing files, that were not + considered in this preliminar phase. + +4. Discussion + +4.1. Skype + + My co-mentor for GSoC, Nima, has publicly rejected the idea of creating + a module for Skype and proposed to implement one for XMPP instead. + I've chosen Skype for its popularity, but I have no other main reason + to maintain this option, so it's probable that XMPP transport will be + implemented. + +4.2. Storing links + + My original proposal considered the fact that links could be stored + somewhere with restricted access, ideally a git repository. Nima + mentioned that ideally the links shouldn't be stored. May be this idea + could be used only to the mirrors and providers configuration (see + core module design). + +4.3. Generating unique URLs + + Nima mentioned that unique URLs could be generated for each request, + and in case the user don't have access to SSL, these links could be + served and later deleted or recycled. I like this idea. + +4. References + + [0] https://gitweb.torproject.org/gettor.git + [1] https://gitweb.torproject.org/user/sukhbir/gettor.git + [2] https://ileiva.github.io/gettor_proposal.html +

1 0

[gettor/master] Deleted unnecesary files
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit 5c94549b6d6c028017e3afe80adf2e4d34fff49f Author: ilv <ilv(a)users.noreply.github.com> Date: Mon Jun 2 20:50:01 2014 -0400 Deleted unnecesary files --- src/gettor.pyc | Bin 3934 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/src/gettor.pyc b/src/gettor.pyc deleted file mode 100644 index 81b41ac..0000000 Binary files a/src/gettor.pyc and /dev/null differ

1 0

[gettor/master] Deleted extra whitespaces and other formatting (using pep8)
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit db667b37730eb79bb7bea28cfbd9e5b1a7dcfbb5 Author: ilv <ilv(a)users.noreply.github.com> Date: Tue Jun 3 00:20:04 2014 -0400 Deleted extra whitespaces and other formatting (using pep8) --- src/core_demo.py | 3 -- src/gettor.py | 92 ++++++++++++++++++++++++++---------------------------- 2 files changed, 44 insertions(+), 51 deletions(-) diff --git a/src/core_demo.py b/src/core_demo.py index 19ffd58..3513e46 100644 --- a/src/core_demo.py +++ b/src/core_demo.py @@ -14,6 +14,3 @@ except ValueError as e: print "Value error! " + str(e) except RuntimeError as e: print "Internal error! " + str(e) - - - diff --git a/src/gettor.py b/src/gettor.py index 573bae7..02f5eb2 100644 --- a/src/gettor.py +++ b/src/gettor.py @@ -5,33 +5,32 @@ import logging class Core: - + """ Gets links from providers and delivers them to other modules. - - Public methods: + + Public methods: get_links(operating_system, locale) """ - + def __init__(self): """ Initialize a Core object by reading a configuration file. - + Parameters: none - """ - + """ + # Read configuration file - # Dummy info. This should be read from a configuration file. self.basedir = './providers/' - self.supported_locales = [ 'en', 'es' ] - self.supported_os = [ 'windows', 'linux', 'osx' ] - + self.supported_locales = ['en', 'es'] + self.supported_os = ['windows', 'linux', 'osx'] + def get_links(self, operating_system, locale): """ - Public method to obtain links. - + Public method to obtain links. + Checks for supported locales and operating systems. It returns ValueError if the locale or operating system is not supported. It raises RuntimeError if something goes wrong while trying @@ -39,107 +38,104 @@ class Core: method should be called from the services modules of GetTor (e.g. SMTP). """ - + # Log requests self._log_request(operating_system, locale) - + if locale not in self.supported_locales: raise ValueError("Locale %s not supported at the moment" % locale) - + if operating_system not in self.supported_os: - raise ValueError("Operating system %s not supported at the moment" + raise ValueError("Operating system %s not supported at the moment" % operating_system) - + # This could change in the future, let's leave it isolated. links = self._get_links(operating_system, locale) - + if links is None: - raise RuntimeError("Something went wrong with GetTor's core") - + raise RuntimeError("Something went wrong with GetTor's core") + return links - + def _get_links(self, operating_system, locale): """ Private method to obtain the links. - - Looks for the links inside each provider file. On success - returns a string with the links. On failure returns None. + + Looks for the links inside each provider file. On success + returns a string with the links. On failure returns None. This should only be called from get_links() method. - + Parameters: os: string describing the operating system locale: string describing the locale """ - + # There should be a 'providers' directory inside self.basedir # # Each .links file begins with a string describing the provider. # After that, every line should have the following format: - # + # # operating_system locale link package_signature key_fingerprint # - # e.g. + # e.g. # # Dropbox # linux en https://foo.bar https://foo.bar.asc 111-222-333-444 # osx es https://bar.baz https://bar.baz.asc 555-666-777-888 # links = [] - + # Look for files ending with .links p = re.compile('.*\.links$') - + for name in os.listdir(self.basedir): path = os.path.abspath(os.path.join(self.basedir, name)) if os.path.isfile(path) and p.match(path): links.append(path) - + # Let's create a dictionary linking each provider with the links # found for operating_system and locale. This way makes it easy # to check if no links were found. providers = {} - + # We trust links have been generated properly for name in links: link_file = open(name, 'r') provider_name = link_file.readline() - + for line in link_file: words = line.split(' ') if words[0] == operating_system and words[1] == locale: providers.setdefault(provider_name, []).append( - "%s %s %s" % (words[2], words[3], words[4].replace('-', ' ')) + "%s %s %s" % + (words[2], words[3], words[4].replace('-', ' ')) ) - + link_file.close() - + # Create the final links list with all providers all_links = [] - + for key in providers.keys(): all_links.append( "\n%s\n%s\n" % (key, ''.join(providers[key])) ) - + if all_links: return "".join(all_links) else: return None - - - + def _log_request(self, operating_system, locale): """ Private method to log what service module called to get the links. - + Parameters: none """ - + caller = inspect.stack()[2] module = inspect.getmodule(caller[0]) - + # Dummy print for now. Should be done with logging print "\nCalled by %s\nOS: %s\nLocale: %s\n" % \ - (str(module), operating_system, locale) - - + (str(module), operating_system, locale)

1 0

[gettor/master] Changed proposed format for link files to RFC 882. Read configuration from config file with ConfigParser
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit c6ed9c758f08c3d2000814a6189ca889d5478a29 Author: ilv <ilv(a)users.noreply.github.com> Date: Fri Jun 6 13:01:24 2014 -0400 Changed proposed format for link files to RFC 882. Read configuration from config file with ConfigParser --- spec/design/core.txt | 55 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 40 insertions(+), 15 deletions(-) diff --git a/spec/design/core.txt b/spec/design/core.txt index f90c6f6..cb7945f 100644 --- a/spec/design/core.txt +++ b/spec/design/core.txt @@ -1,10 +1,16 @@ Google Summer of Code 2014 GetTor Revamp - Core module - Author: Israel Leiva - <israel.leiva(a)usach.cl> - Last update: 2014-06-01 - Version: 0.02 - Changes: Combine official mirrors with providers links (as another provider). + Author: Israel Leiva - <israel.leiva(a)usach.cl, ilv(a)riseup.net> + Last update: 2014-06-06 + Version: 0.03 + Changes: [0.03] + Changed proposed format for link files to RFC 882 (ConfigParser). + Read configuration from file with ConfigParser. + Added logging. + [0.02] + Combine official mirrors with providers links (as another provider). Eliminated on demand link generation. Now it reads from files. Modified description according to PEP-8. + [0.01] First version. @@ -32,9 +38,10 @@ The new design should consist of the following files, directories and methods: - * core.conf: Configuration values, e.g. base directory. + * gettor.cfg: Configuration values, e.g. base directory. - * providers/: Directory for generated links. + * providers/: Directory for generated links. Should be specified on + gettor.cfg. ----- provider1.links: links from provider1. ----- provider2.links: links from provider2. @@ -42,12 +49,16 @@ All this data is generated automatically. - * logs/: Directory for logs. Added automatically. + * logs/: Directory for logs. Should be specified on gettor.cfg ----- core_yyyy-mm-dd.log: daily log of requests. * Core module of GetTor. + __init(config_file)__ + Creates a new Core object. It reads its configuration from + the config_file using ConfigParser. + get_links(operating_system, locale) Public method to obtain the links. Returns links for operating_system in locale language. It checks if the operating @@ -61,17 +72,31 @@ Gets the links for a specific operating system and locale according to the options received. It reads all the .links files inside the providers directory. Each one of these files - contains the provider's name in the first line and all the - links after. The format of these files is as following: - - PROVIDER NAME - operating_system locale link package_signature key_fingerprint + should follow the ConfigParser's format. There should be a + section [provider] with the option 'name' for the provider's + name (e.g. Dropbox) + + Following sections should specify the operating system and + its options should be the locale. When more than one link is + available per operating system and locale (always) the links + should be specified as a multiline value. Each link has the + format: + + link link_signature key_fingerprint Example: + + [provider] + name: Dropbox + + [linux] + en: https://foo.bar https://foo.bar.asc 111-222-333-444, + https://foo.bar https://foo.bar.asc 555-666-777-888 + + es: https://bar.baz https://bar.baz.asc 555-666-777-888 - Dropbox - linux en https://foo.bar https://foo.bar.asc 111-222-333-444 - osx es https://bar.baz https://bar.baz.asc 555-666-777-888 + PROVIDER NAME + operating_system locale link package_signature key_fingerprint The key fingerprint is joined by '-' characters for convenience, but it's sent back to the service module separated by spaces.

1 0

[gettor/master] New format for links. It follows RFC 882, so it can be read with ConfigParser
by ilv＠torproject.org 22 Sep '15

22 Sep '15

commit cdaa4018c17b5ffee26ab8928a2803dce1ee3b2e Author: ilv <ilv(a)users.noreply.github.com> Date: Fri Jun 6 13:04:16 2014 -0400 New format for links. It follows RFC 882, so it can be read with ConfigParser --- src/providers/dropbox.links | 11 +++++++---- src/providers/gdrive.links | 11 +++++++---- src/providers/mirrors.links | 11 +++++++---- 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/src/providers/dropbox.links b/src/providers/dropbox.links index 53d4a04..6790349 100644 --- a/src/providers/dropbox.links +++ b/src/providers/dropbox.links @@ -1,5 +1,8 @@ -Dropbox +[provider] +name: Dropbox -linux en https://foo.bar https://foo.bar.asc 111-222-333-444 -linux es https://bar.baz https://bar.baz.asc 555-666-777-888 -linux en https://baz.foo https://baz.foo.asc 999-111-222-333 +[linux] +en: https://foo.bar https://foo.bar.asc 111-222-333-444, + https://baz.foo https://baz.foo.asc 999-111-222-333 + +es: https://bar.baz https://bar.baz.asc 555-666-777-888 diff --git a/src/providers/gdrive.links b/src/providers/gdrive.links index 570c8e2..d221863 100644 --- a/src/providers/gdrive.links +++ b/src/providers/gdrive.links @@ -1,5 +1,8 @@ -Google Drive +[provider] +name: Google Drive -linux en https://foo.bar https://foo.bar.asc 111-222-333-444 -linux es https://bar.baz https://bar.baz.asc 555-666-777-888 -linux en https://baz.foo https://baz.foo.asc 999-111-222-333 +[linux] +en: https://foo.bar https://foo.bar.asc 111-222-333-444, + https://baz.foo https://baz.foo.asc 999-111-222-333 + +es: https://bar.baz https://bar.baz.asc 555-666-777-888 diff --git a/src/providers/mirrors.links b/src/providers/mirrors.links index 0eb5b10..553de9d 100644 --- a/src/providers/mirrors.links +++ b/src/providers/mirrors.links @@ -1,5 +1,8 @@ -Official Mirrors +[provider] +name: Official mirrors -linux en https://foo.bar https://foo.bar.asc 111-222-333-444 -linux es https://bar.baz https://bar.baz.asc 555-666-777-888 -linux en https://baz.foo https://baz.foo.asc 999-111-222-333 +[linux] +en: https://foo.bar https://foo.bar.asc 111-222-333-444, + https://baz.foo https://baz.foo.asc 999-111-222-333 + +es: https://bar.baz https://bar.baz.asc 555-666-777-888

1 0