commit 744934523a64e312ce42eaaa31ef936beeae5992 Author: hiro hiro@torproject.org Date: Thu Oct 31 15:48:05 2019 +0100
Update readme Breakdown strings for translations Delete unused scripts --- README.md | 159 ++++----------- dev_scripts/blacklist.py | 111 ----------- dev_scripts/core_demo.py | 19 -- dev_scripts/create_api_mirror.py | 305 ---------------------------- dev_scripts/create_gh_mirrors.py | 207 ------------------- dev_scripts/distribution_methods.txt | 19 -- dev_scripts/get_mirrors.py | 353 --------------------------------- dev_scripts/process_chat.py | 22 -- dev_scripts/process_email.py | 41 ---- dev_scripts/process_http.py | 14 -- dev_scripts/process_tweets.py | 35 ---- dev_scripts/report.py | 129 ------------ dev_scripts/sample-htaccess-api-mirror | 21 -- dev_scripts/stats.py | 123 ------------ gettor/main.py | 4 +- gettor/services/email/sendmail.py | 15 +- gettor/services/twitter/twitterdm.py | 14 +- gettor/utils/options.py | 5 +- share/locale/en.json | 17 +- share/locale/es.json | 19 +- share/locale/pt.json | 17 +- tests/test_email_service.py | 2 +- tests/test_locales.py | 2 +- tests/test_twitter.py | 2 +- 24 files changed, 102 insertions(+), 1553 deletions(-)
diff --git a/README.md b/README.md index f7e130d..a3df81b 100644 --- a/README.md +++ b/README.md @@ -32,144 +32,55 @@ Here is a list of the main goals the new GetTor should accomplish:
Installing GetTor ================= -```sh -WORKON_HOME=${HOME}/venv -export WORKON_HOME -mkdir -p $WORKON_HOME -source $(which virtualenvwrapper.sh) -git clone https://git.torproject.org/gettor.git && cd gettor -mkvirtualenv -a $PWD -r requirements.txt --unzip-setuptools --setuptools gettor -``` - -From now on, to use GetTor's virtualenv, just do ``$ workon gettor`` -(after sourcing virtualenvwrapper.sh, as before). To exit the virtualenv -without exiting the shell, do ``$ deactivate``. - -```sh -export PYTHONPATH=$PYTHONPATH:${VIRTUAL_ENV}/lib/python/site-packages -``` - -```sh -$ ./scripts/create_db -$ ./bin/gettor_service start -``` - -Running tests -================= - -GetTor includes PyTest unit tests. To run the tests, first install some dependencies: - -```sh -pip3 install -r .test.requirements.txt -``` - -Then you can run `pytest` against the `tests/` directory. - -```sh -pytest tests/ -``` - - -How does the new GetTor works? -==============================
-Below are some specifications and core concepts on how the new GetTor works. +To install gettor locally please install the following packages (on debian):
-*Links files*: Currently links are saved in files with the '.links' extension, -using the ConfigParser format (RFC 882). A sample link file should look like -this: +python3-coverage +python3-dkim +python3-dns +python3-internetarchive +python3-openssl +python3-pytest +python3-requests-oauthlib +python3-service-identity +python3-twisted +sqlite3
---- BEGIN FILE --- +The following packages are needed to run a gettor instance:
- [provider] - name = CoolCloudProvider +internetarchive +jq +rclone
- [key] - fingerprint = AAAA BBBB CCCC DDDD EEEE FFFF GGGG HHHH IIII JJJJ +Specifically: +internetarchive is needed to send Tor Browser files via command line to the internet archive. +jq is a json parser that is used to find out about the new tor browser releases. +Both internetarchive and jq are used in: scripts/update_files
- [linux] - en = Package (64-bit): https://cool.cloud.link64 - ASC signature (64-bit): https://cool.cloud.link64.asc - Package SHA256 checksum (64-bit): superhash64, - Package (32-bit): https://cool.cloud.link32 - ASC signature (32-bit): https://cool.cloud.link32.asc - Package SHA256 checksum (32-bit): superhash32 +The following packages are instead needed to deploy gettor via ansible:
- [windows] - ... +ansible +ansible-lint
- [osx] - ... +Gettor ansible playbooks live at: https://gitweb.torproject.org/admin/services/gettor.git/
---- END FILE --- +Finally the following package is used store Tor Browser files via git and support large files: +git-lfs
-You can also check providers/dropbox.links for a better example. -WORKON_HOME=${HOME}/.virtualenvs - export WORKON_HOME - mkdir -p $WORKON_HOME - source $(which virtualenvwrapper.sh) -*Core*: the heart of GetTor. Receives requests for links for a certain OS and -language and respond accordingly. It also presents an easy way for scripts -to create links file.
-*SMTP*: Receives requests via email, process them, contact the core module if -necessary and respond to the user in the specified language. People can send -blank or dummy emails to it to receive a help message describing how to ask -for links. Email forwarding is used to redirect the emails to GetTor.
-*XMPP*: Same as above, but via XMPP (account needed). It has been tested with -dukgo.com, jabber.ccc.de, riseup.net. It doesn't seem to be able to interact -with gtalk users. +Once gettor is installed you can run it with:
-*Twitter*: Receive requests via Twitter direct messages, contact the core module -if necessary and respond to the user in the specified language. Unfinished. - -*DB*: Store anonymous info about the people that interact with GetTor in order -to keep count of the number of requests per person and avoid malicious users -that try to collapse the service. It also keeps count of how many requests -GetTor has received during its lifetime. A lot of other data was being saved -in the original gsoc project, but it was changed to save the minimum. - -*Blacklist*: Provide a mechanism to avoid flood and deny interaction to -malicious users. - -*Providers scripts*: every supported provider should have a script to -automatically upload packages to 'the cloud' and create the corresponding -links files. The script should consider the following steps: - - * Upload the packages. - * Get the sha256 checksum of the files uploaded. - * Get the PGP key fingerprint that signed the files. - * Check for .asc file for every package uploaded. - * Put all together in a '.link' file (using the core module). - - -What is the current status of the new GetTor? -============================================= - -Deployed and working. - - -How can I help? -================ - -If you have ideas to improve GetTor and/or add new providers, please tell us! -I'm currently the lead developer on this, so if you have any comments/doubts/ -ideas you can send me an e-mail to ilv _at_ riseup _dot_ net or ping me (ilv), -or sukhe or mrphs at #tor-dev in the OFTC IRC network. For openning tickets you -should use the trac[0] and select the GetTor component. Some neat ideas we -could use are the following: +``` +$ ./bin/gettor_service start +```
- * Report bugs! - * Create script for new providers, namely: Google Drive, Github. Check -providers.txt - * Create a new module for distributing links. Check distribution_methods.txt - * Finish the Twitter module. - * Propose code/behaviour improvements. - * Update the specs. +Running tests +=================
+GetTor includes PyTest unit tests. To run the tests, first install some dependencies:
-References -===========
-[0] https://trac.torproject.org/projects/tor/query?status=accepted&status=as... +``` +$ pytest-3 tests/ +``` diff --git a/dev_scripts/blacklist.py b/dev_scripts/blacklist.py deleted file mode 100644 index 1aefe1a..0000000 --- a/dev_scripts/blacklist.py +++ /dev/null @@ -1,111 +0,0 @@ -#!/usr/bin/python -# -*- coding: utf-8 -*- -# -# This file is part of GetTor, a Tor Browser distribution system. -# -# :authors: Israel Leiva ilv@riseup.net -# see also AUTHORS file -# -# :copyright: (c) 2008-2014, The Tor Project, Inc. -# (c) 2014, Israel Leiva -# -# :license: This is Free Software. See LICENSE for license information. - -import sys -import time -import sqlite3 -import argparse - - -def main(): - """Script for managing blacklisting of users. - - See argparse usage for more details. - - """ - parser = argparse.ArgumentParser(description='Utility for GetTor' - ' blacklisting') - parser.add_argument('database', metavar='database.db', type=str, - help='the database file') - parser.add_argument('-u', '--user', default=None, - help='filter by user hash') - parser.add_argument('-s', '--service', default=None, - help='filter by service') - parser.add_argument('-b', '--blocked', default=None, - help='filter by blocked users') - parser.add_argument('-a', '--add', default=None, nargs=3, - metavar=('USER', 'SERVICE', 'BLOCKED'), - help='add user') - parser.add_argument('-c', '--clean', default=None, const='c', nargs='?', - metavar='user hash', - help='clean table (delete expired blacklistings)') - parser.add_argument('-r', '--requests', default=None, - help='number of requests; everyone with number of' - ' requests greather than this will be cleaned up') - - args = parser.parse_args() - query = '' - con = sqlite3.connect(args.database) - - if args.add: - # add new entry, useful for adding users permanently blocked - query = "INSERT INTO users VALUES('%s', '%s', 1, %s, %s)"\ - % (args.add[0], args.add[1], args.add[2], time.time()) - with con: - cur = con.cursor() - cur.execute(query) - print "Query execute successfully" - elif args.clean: - if args.clean == 'c': - if args.requests: - # delete by number of times - query = "DELETE FROM users WHERE times > %s" % args.requests - with con: - cur = con.cursor() - cur.execute(query) - print "Query executed successfully." - else: - sys.exit("Number of requests missing. See --help.") - else: - # delete by id - query = "DELETE FROM users WHERE id='%s'" % args.clean - with con: - cur = con.cursor() - cur.execute(query) - print "Query execute succcessfully." - else: - query = "SELECT * FROM users" - has_where = False - # filter - if args.service: - query = "%s %s" % (query, "WHERE service='%s'" % args.service) - has_where = True - if args.user: - if has_where: - query = "%s %s" % (query, "AND id='%s'" % args.user) - else: - query = "%s %s" % (query, "WHERE id='%s'" % args.user) - has_where = True - if args.blocked: - if has_where: - query = "%s %s" % (query, "AND blocked=%s" % args.blocked) - has_where = True - else: - query = "%s %s" % (query, "WHERE blocked=%s" % args.blocked) - - with con: - cur = con.cursor() - cur.execute(query) - rows = cur.fetchall() - # show it nice - print "\nNumber of results: %s\n" % len(rows) - cns = [cn[0] for cn in cur.description] - print "%-70s %-10s %-10s %-10s %-s" % (cns[0], cns[1], cns[2], - cns[3], cns[4]) - - for row in rows: - print "%-70s %-10s %-10s %-10s %s" % (row[0], row[1], row[2], - row[3], row[4]) - -if __name__ == "__main__": - main() diff --git a/dev_scripts/core_demo.py b/dev_scripts/core_demo.py deleted file mode 100644 index 88f725d..0000000 --- a/dev_scripts/core_demo.py +++ /dev/null @@ -1,19 +0,0 @@ -#!/usr/bin/python -# -# Dummy script to test GetTore's Core module -# - -import gettor.core - -try: - core = gettor.core.Core() - links = core.get_links('dummy service', 'linux', 'en') - print links -except gettor.core.ConfigError as e: - print "Misconfiguration: " + str(e) -except gettor.core.UnsupportedOSError as e: - print "Unsupported OS: " + str(e) -except gettor.core.UnsupportedLocaleError as e: - print "Unsupported Locale: " + str(e) -except gettor.core.InternalError as e: - print "Internal error: " + str(e) diff --git a/dev_scripts/create_api_mirror.py b/dev_scripts/create_api_mirror.py deleted file mode 100644 index e80ec4a..0000000 --- a/dev_scripts/create_api_mirror.py +++ /dev/null @@ -1,305 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of GetTor. -# -# :authors: Israel Leiva ilv@torproject.org -# see also AUTHORS file -# -# :copyright: (c) 2008-2015, The Tor Project, Inc. -# (c) 2015, Israel Leiva -# -# :license: This is Free Software. See LICENSE for license information. - -import os -import re -import json -import codecs -import urllib2 -import ConfigParser - -from time import gmtime, strftime - - -"""Script to build a static mirror of GetTor RESTful API""" - -# currently supported locales for Tor Browser -LC = ['ar', 'de', 'en-US', 'es-ES', 'fa', 'fr', 'it', 'ko', 'nl', 'pl', - 'pt-PT', 'ru', 'tr', 'vi', 'zh-CN'] - -# https://gitweb.tpo/tor-browser-spec.git/tree/processes/VersionNumbers -# does not say anything about operating systems, so it's possible the -# notation might change in the future. We should always use the same three -# strings though: linux, windows, osx. -OS = { - 'Linux': 'linux', - 'Windows': 'windows', - 'MacOS': 'osx' -} - -# based on -# https://gitweb.tpo.org/tor-browser-spec.git/tree/processes/VersionNumbers -# except for the first one, which is based on current RecommendedTBBVersions -RE = { - 'os': '(.*)-(\w+)', - 'alpha': '\d.\d(.\d)*a\d+', - 'beta': '\d.\d(.\d)*b\d+', - 'stable': '\d.\d(.\d)*' -} - -# strings to build names of packages depending on OS. -PKG = { - 'windows': 'torbrowser-install-%s_%s.exe', - 'linux': 'tor-browser-linux%s-%s_%s.tar.xz', - 'osx': 'TorBrowser-%s-osx64_%s.dmg' -} - -# bin and asc are used to build the download links for each version, os and lc -URL = { - 'version': 'https://gettor.torproject.org/api/latest.json', - 'mirrors': 'https://gettor.torproject.org/api/mirrors.json', - 'providers': 'https://gettor.torproject.org/api/providers.json' -} - -GETTOR_URL = 'http://gettor.torproject.org' -SERVER_URL = 'https://yourprivateserver.com' -API_TREE = '/home/ilv/Proyectos/tor/static_api/' - -class ConfigError(Exception): - pass - - -class InternalError(Exception): - pass - - -class APIMirror(object): - """ Provide useful resources via RESTful API. """ - def __init__(self, cfg=None): - self.url = SERVER_URL - self.tree = API_TREE - - def _is_json(self, my_json): - """ Check if json generated is valid. - - :param: my_json (string) data to ve verified. - - :return: (bool) true if data is json-valid, false otherwise. - - """ - try: - json_object = json.loads(my_json) - except ValueError, e: - return False - return True - - def _write_json(self, path, content): - """ - """ - try: - with codecs.open( - path, - "w", - "utf-8" - ) as jsonfile: - # Make pretty json - json.dump( - content, - jsonfile, - sort_keys=True, - indent=4, - separators=(',', ': '), - encoding="utf-8", - ) - except IOError as e: - #logging.error("Couldn't write json: %s" % str(e)) - print "Error building %s: %s" % (path, str(e)) - print "%s built" % path - - def _get_provider_name(self, p): - """ Return simplified version of provider's name. - - :param: p (string) provider's name. - - :return: (string) provider's name in lowercase and without spaces. - - """ - p = p.replace(' ', '-') - return p.lower() - - - def _load_latest_version(self): - """ Load latest version data from GetTor API. """ - response = urllib2.urlopen(URL['version']) - json_response = json.load(response) - self.lv = json_response - - def _load_links(self): - """ Load links and providers data. """ - - response = urllib2.urlopen(URL['providers']) - json_response = json.load(response) - - links = {} - - for provider in json_response: - if provider != 'updated_at': - provider_url = json_response[provider] - - provider_response = urllib2.urlopen("%s.json" % provider_url) - provider_content = json.load(provider_response) - - json_response[provider] = provider_url.replace( - GETTOR_URL, - self.url - ) - - - links[provider] = provider_content - - self.providers = json_response - self.links = links - - def _load_mirrors(self): - """ Load mirrors data from GetTor API. """ - response = urllib2.urlopen(URL['mirrors']) - json_response = json.load(response) - self.mirrors = json_response - - def _load_resources(self): - """ Load available resources data. """ - - self.resources = { - 'providers': '%s/providers' % self.url, - 'mirrors': '%s/mirrors' % self.url, - 'latest_version': '%s/latest' % self.url, - 'updated_at': strftime("%Y-%m-%d %H:%M:%S", gmtime()) - } - - def load_data(self): - """ Load all data.""" - self._load_links() - self._load_mirrors() - self._load_resources() - self._load_latest_version() - - def build(self): - """ Build RESTful API. """ - - print "Building API mirror" - - # resources - self._write_json( - os.path.join(self.tree, 'api'), - self.resources - ) - - api_path = os.path.join(self.tree, 'api-content') - if not os.path.isdir(api_path): - os.mkdir(api_path) - - # providers - self._write_json( - os.path.join(api_path, 'providers'), - self.providers - ) - - providers_path = os.path.join(api_path, 'providers-content') - if not os.path.isdir(providers_path): - os.mkdir(providers_path) - - for provider in self.links: - if provider == 'updated_at': - continue - - self._write_json( - os.path.join(providers_path, provider), - self.links[provider] - ) - - provider_path = os.path.join( - providers_path, - "%s-content" % provider - ) - - if not os.path.isdir(provider_path): - os.mkdir(provider_path) - - for osys in self.links[provider]: - self._write_json( - os.path.join(provider_path, osys), - self.links[provider][osys] - ) - - provider_os_path = os.path.join( - provider_path, "%s-content" % osys - ) - - if not os.path.isdir(provider_os_path): - os.mkdir(provider_os_path) - - for lc in self.links[provider][osys]: - self._write_json( - os.path.join(provider_os_path, lc), - self.links[provider][osys][lc] - ) - - # latest version - self._write_json( - os.path.join(api_path, 'latest'), - self.lv - ) - - lv_path = os.path.join(api_path, 'latest-content') - if not os.path.isdir(lv_path): - os.mkdir(lv_path) - - for release in self.lv: - if release == 'updated_at': - continue - - self._write_json( - os.path.join(lv_path, release), - self.lv[release] - ) - - release_path = os.path.join( - lv_path, - "%s-content" % release - ) - - if not os.path.isdir(release_path): - os.mkdir(release_path) - - for osys in self.lv[release]['downloads']: - self._write_json( - os.path.join(release_path, osys), - self.lv[release]['downloads'][osys] - ) - - release_os_path = os.path.join( - release_path, - "%s-content" % osys - ) - - if not os.path.isdir(release_os_path): - os.mkdir(release_os_path) - - for lc in self.lv[release]['downloads'][osys]: - self._write_json( - os.path.join(release_os_path, lc), - self.lv[release]['downloads'][osys][lc] - ) - - # mirrors - self._write_json( - os.path.join(api_path, 'mirrors'), - self.mirrors - ) - -def main(): - api = APIMirror() - api.load_data() - api.build() - -if __name__ == '__main__': - main() diff --git a/dev_scripts/create_gh_mirrors.py b/dev_scripts/create_gh_mirrors.py deleted file mode 100644 index e555488..0000000 --- a/dev_scripts/create_gh_mirrors.py +++ /dev/null @@ -1,207 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of GetTor -# -# :authors: Israel Leiva ilv@torproject.org -# see also AUTHORS file -# -# :license: This is Free Software. See LICENSE for license information. - -"""create_gh_mirrors -- Create landing page and readme for Github.""" - -import os -import ConfigParser - -import github3 - -import gettor.core - - -def create_readme(tpl_path, md_path, tb_version, links): - """Create README file with links stored in github.links. - - :param: tpl_path (string) path to file used as template. - :param: md_path (string) path to file generated. - :param: tb_version (string) tor browser version. - :param: links (object) github links. - - """ - lcs = ['FA', 'ZH', 'TR', 'EN'] - - md_file = open(md_path, 'w') - with open(tpl_path, 'r') as tpl_file: - content_md = tpl_file.read() - - for lc in lcs: - win_link = links.get('windows', lc.lower()) - win_pkg, win_sig, win_sha = [e for e in win_link.split("$") if e] - - osx_link = links.get('osx', lc.lower()) - osx_pkg, osx_sig, osx_sha = [e for e in osx_link.split("$") if e] - - linux_links = links.get('linux', lc.lower()) - linux32_link, linux64_link = linux_links.split(',') - linux32_pkg, linux32_sig, linux32_sha = [ - e for e in linux32_link.split("$") if e - ] - linux64_pkg, linux64_sig, linux64_sha = [ - e for e in linux64_link.split("$") if e - ] - - content_md = content_md.replace( - "%WINDOWS_{}%".format(lc), win_pkg - ) - content_md = content_md.replace( - "%WINDOWS_{}_SIG%".format(lc), win_sig - ) - - content_md = content_md.replace( - "%OSX_{}%".format(lc), osx_pkg - ) - content_md = content_md.replace( - "%OSX_{}_SIG%".format(lc), osx_sig - ) - - content_md = content_md.replace( - "%LINUX32_{}%".format(lc), linux32_pkg - ) - content_md = content_md.replace( - "%LINUX32_{}_SIG%".format(lc), linux32_sig - ) - content_md = content_md.replace( - "%LINUX64_{}%".format(lc), linux64_pkg - ) - content_md = content_md.replace( - "%LINUX64_{}_SIG%".format(lc), linux64_sig - ) - - content_md = content_md.replace("%TB_VERSION%", tb_version) - md_file.write(content_md) - - print "README generated with Tor Browser %s" % tb_version - - -def create_landing_html(tpl_path, html_path, tb_version, links): - """Create README file with links stored in github.links. - - :param: tpl_path (string) path to file used as template. - :param: html_path (string) path to file generated. - :param: tb_version (string) tor browser version. - :param: links (object) github links. - - """ - lcs = ['FA', 'ZH', 'TR', 'EN'] - - html_file = open(html_path, 'w') - with open(tpl_path, 'r') as tpl_file: - content_html = tpl_file.read().replace('\n', '') - - for lc in lcs: - win_link = links.get('windows', lc.lower()) - win_pkg, win_sig, win_sha = [e for e in win_link.split("$") if e] - - osx_link = links.get('osx', lc.lower()) - osx_pkg, osx_sig, osx_sha = [e for e in osx_link.split("$") if e] - - linux_links = links.get('linux', lc.lower()) - linux32_link, linux64_link = linux_links.split(',') - linux32_pkg, linux32_sig, linux32_sha = [ - e for e in linux32_link.split("$") if e - ] - linux64_pkg, linux64_sig, linux64_sha = [ - e for e in linux64_link.split("$") if e - ] - - content_html = content_html.replace( - "%WINDOWS_{}%".format(lc), win_pkg - ) - """ - content_html = content_html.replace( - "%WINDOWS_{}_SIG%".format(lc), win_sig - ) - """ - content_html = content_html.replace( - "%OSX_{}%".format(lc), osx_pkg - ) - """ - content_html = content_html.replace( - "%OSX_{}_SIG%".format(lc), osx_sig - ) - """ - content_html = content_html.replace( - "%LINUX32_{}%".format(lc), linux32_pkg - ) - """ - content_html = content_html.replace( - "%LINUX32_{}_SIG%".format(lc), linux32_sig - ) - """ - content_html = content_html.replace( - "%LINUX64_{}%".format(lc), linux64_pkg - ) - """ - content_html = content_html.replace( - "%LINUX64_{}_SIG%".format(lc), linux64_sig - ) - """ - - content_html = content_html.replace( - "%TB_VERSION%", tb_version - ) - html_file.write(content_html) - - print "HTML generated with Tor Browser %s" % tb_version - - -def main(): - """Generate HTML and md files and update it in Github.""" - github_links = 'providers/github.links' - tb_version_path = 'latest_torbrowser.cfg' - md_path = 'upload/readme_gh.md' - html_path = 'upload/landing_gh.html' - md_tpl_path = 'upload/readme_gh.tpl' - html_tpl_path = 'upload/landing_gh.tpl' - github_access_token = '' - - tb_version_config = ConfigParser.ConfigParser() - tb_version_config.read(tb_version_path) - tb_version = tb_version_config.get('version', 'current') - - links = ConfigParser.ConfigParser() - links.read(github_links) - - create_landing_html(html_tpl_path, html_path, tb_version, links) - create_readme(md_tpl_path, md_path, tb_version, links) - - landing = open(html_path, 'r') - content_landing = landing.read().replace('\n', '') - - readme = open(md_path, 'r') - content_readme = readme.read() - - gh = github3.login(token=github_access_token) - repo_landing = gh.repository('thetorproject', 'gettor') - repo_readme = gh.repository('thetorproject', 'gettorbrowser') - - file_landing_gh = repo_landing.file_contents('index.html', 'gh-pages') - file_readme_gh = repo_readme.file_contents('README.md') - - data_landing = { - 'message': 'Updating landing page.', - 'content': content_landing, - 'branch': 'gh-pages' - } - - data_readme = { - 'message': 'Updating README.', - 'content': content_readme - } - - file_landing_gh.update(**data_landing) - print "Landing page updated in gettor" - - file_readme_gh.update(**data_readme) - print "README updated in gettorbrowser" - -if __name__ == "__main__": - main() diff --git a/dev_scripts/distribution_methods.txt b/dev_scripts/distribution_methods.txt deleted file mode 100644 index 8bbdd4b..0000000 --- a/dev_scripts/distribution_methods.txt +++ /dev/null @@ -1,19 +0,0 @@ -================================== CURRENT ==================================== - -* SMTP: this one is ready and tested. - -* XMPP: this one is ready and tested. We'll probably deploy SMTP first, tough. - -================================== IN DEVELOPMENT ============================= - -* Twitter: This one is half done. I forked a twitter bot from wfn's gsoc work -on BridgeDB, but I got confused with some Twitter policies. I thought that the -bot *had* to follow someone back to send him/her DMs, but it turns out that you -just need to be followed, not follow back (thanks to mrphs for pointing out -this). - -================================== (CRAZY) IDEAS ============================== - -* Duckduckgo: This is just a thought, I haven't done proper research. What if -someone could ask for gettor links like this: "!gettor"? - diff --git a/dev_scripts/get_mirrors.py b/dev_scripts/get_mirrors.py deleted file mode 100644 index afa4cc9..0000000 --- a/dev_scripts/get_mirrors.py +++ /dev/null @@ -1,353 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of GetTor -# -# :authors: Israel Leiva ilv@torproject.org -# see also AUTHORS file -# -# :license: This is Free Software. See LICENSE for license information. - -"""get_mirrors -- Download the list of tpo's mirrors for GetTor.""" - -import os -import json -import codecs -import logging -import argparse - - -from OpenSSL import SSL -from OpenSSL import crypto - -from twisted.web import client -from twisted.python import log -from twisted.internet import ssl -from twisted.internet import defer -from twisted.internet import protocol -from twisted.internet import reactor -from twisted.internet.error import TimeoutError -from twisted.internet.error import DNSLookupError -from twisted.internet.error import ConnectionRefusedError - -from gettor import utils - -# Associate each protocol with its column name in tor-mirrors.csv -PROTOS = { - 'https': 'httpsWebsiteMirror', - 'http': 'httpWebsiteMirror', - 'rsync': 'rsyncWebsiteMirror', - 'https-dist': 'httpsDistMirror', - 'http-dist': 'httpDistMirror', - 'rsync-dist': 'rsyncDistMirror', - 'ftp': 'ftpWebsiteMirror', - 'onion': 'hiddenServiceMirror', -} - -# Tor Project's website certificate -# $ openssl s_client -showcerts -connect tpo:443 < /dev/null > tpo.pem -CERT_TPO = '/path/to/gettor/tpo.pem' - - -# Taken from get-tor-exits (BridgeDB) -class FileWriter(protocol.Protocol): - """Read a downloaded file incrementally and write to file.""" - def __init__(self, finished, file): - """Create a FileWriter. - - .. warning:: We currently only handle the first 2MB of a file. Files - over 2MB will be truncated prematurely. *note*: this should be - enough for the mirrors file. - - :param finished: A :class:`~twisted.internet.defer.Deferred` which - will fire when another portion of the download is complete. - """ - self.finished = finished - self.remaining = 1024 * 1024 * 2 - self.fh = file - - def dataReceived(self, bytes): - """Write a portion of the download with ``bytes`` size to disk.""" - if self.remaining: - display = bytes[:self.remaining] - self.fh.write(display) - self.fh.flush() - self.remaining -= len(display) - - def connectionLost(self, reason): - """Called when the download is complete.""" - logging.info('Finished receiving mirrors list: %s' - % reason.getErrorMessage()) - self.finished.callback(None) - - -# Based in tor2web.utils.ssl (Tor2web) -class HTTPSVerifyingContextFactory(ssl.ClientContextFactory): - def __init__(self, cn): - self.cn = cn - # - # From https://docs.python.org/2/library/ssl.html#ssl-security - # - # "SSL versions 2 and 3 are considered insecure and are therefore - # dangerous to use. If you want maximum compatibility between clients - # and servers, it is recommended to use PROTOCOL_SSLv23 as the protocol - # version and then disable SSLv2 and SSLv3 explicitly" - # - self.method = SSL.SSLv23_METHOD - - def getContext(self, hostname, port): - """Get this connection's OpenSSL context. - - We disable SSLv2 and SSLv3. We also check the certificate received - is the one we expect (using the "common name"). - - """ - ctx = self._contextFactory(self.method) - ctx.set_options(SSL.OP_NO_SSLv2) - ctx.set_options(SSL.OP_NO_SSLv3) - ctx.set_verify( - SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT, - self.verifyCN - ) - - return ctx - - def verifyCN(self, connection, x509, errno, depth, preverifyOK): - # DEBUG: print "%s == %s ?" % (self.cn, x509.get_subject().commonName) - - # Somehow, if I don't set this to true, the verifyCN doesn't go - # down in the chain, I don't know if this is OK - verify = True - if depth == 0: - if self.cn == x509.get_subject().commonName: - verify = True - else: - verify = False - return verify - - -# Based in get-tor-exits (BridgeDB) -def handle(failure): - """Handle a **failure**.""" - if failure.type == ConnectionRefusedError: - logging.error("Couldn't download file; connection was refused.") - elif failure.type == DNSLookupError: - logging.error("Couldn't download file; domain resolution failed.") - elif failure.type == TimeoutError: - logging.error("Couldn't download file; connection timed out.") - else: - logging.error("Couldn't download file.") - - print "Couldn't download file. Check the log." - os._exit(-1) - - -# Taken from get-tor-exits (BridgeDB) -def writeToFile(response, filename): - """Write requested content to filename.""" - finished = defer.Deferred() - response.deliverBody(FileWriter(finished, filename)) - return finished - - -def is_json(my_json): - """Check if json generated is valid.""" - try: - json_object = json.loads(my_json) - except ValueError, e: - return False - return True - - -def add_tpo_link(url): - """Add the download link for Tor Browser.""" - uri = 'projects/torbrowser.html.en#downloads' - if url.endswith('/'): - return "%s%s" % (url, uri) - else: - return "%s/%s" % (url, uri) - - -def add_entry(mirrors, columns, elements): - """Add entry to mirrors list.""" - entry = {} - count = 0 - for e in elements: - e = e.replace("\n", '') - entry[columns[count]] = e - count = count + 1 - - mirrors.append(entry) - - -def add_mirror(file, entry, proto): - """Add mirror to mirrors list.""" - # if proto requested is http(s), we add link to download section - if PROTOS[proto] == 'http' or PROTOS[proto] == 'https': - uri = add_tpo_link(entry[proto]) - else: - uri = entry[proto] - - file.write( - "%s - by %s (%s)\n" % ( - uri, - entry['orgName'], - entry['subRegion'], - ) - ) - - -def main(): - """Script to get the list of tpo's mirrors from tpo and adapt it to - be used by GetTor. - - Usage: python2.7 get_mirrors.py [-h] [--proto protocol] - - By default, the protocol is 'https'. Possible values of protcol are: - - http, https, rsync, ftp, onion, http-dist, https-dist, rsync-dist. - - """ - parser = argparse.ArgumentParser( - description="Utility to download tpo's mirrors and make it usable\ - by GetTor." - ) - - parser.add_argument( - '-p', '--proto', - default='https', - help='Protocol filter. Possible values: http, https, rsync, ftp, onion\ - http-dist, https-dist, rsync-dist. Default to https.') - - args = parser.parse_args() - p = args.proto - - gettor_path = '/path/to/gettor/' - csv_path = os.path.join(gettor_path, 'tor-mirrors.csv') - json_path = os.path.join(gettor_path, 'tor-mirrors') - mirrors_list = os.path.join(gettor_path, 'mirrors-list.txt') - - # Load tpo certificate and extract common name, we'll later compare this - # with the certificate sent by tpo to check we're really taltking to it - try: - data = open(CERT_TPO).read() - x509 = crypto.load_certificate(crypto.FILETYPE_PEM, data) - cn_tpo = x509.get_subject().commonName - except Exception as e: - logging.error("Error with certificate: %s" % str(e)) - return - - # While we wait the json of mirrors to be implemented in tpo, we need - # to download the csv file and transform it to json - - # The code below is based in get-tor-exits script from BridgeDB and - # the tor2web.utils.ssl module from Tor2web - url = 'https://www.torproject.org/include/tor-mirrors.csv' - - try: - fh = open(csv_path, 'w') - except IOError as e: - logging.error("Could not open %s" % csv_path) - return - - logging.info("Requesting %s..." % url) - - # If certificate don't match an exception will be raised - # this is my first experience with twisted, maybe I'll learn to handle - # this better some time in the future... - contextFactory = HTTPSVerifyingContextFactory(cn_tpo) - agent = client.Agent(reactor, contextFactory) - d = agent.request("GET", url) - d.addCallback(writeToFile, fh) - d.addErrback(handle) - d.addCallbacks(log.msg, log.err) - - if not reactor.running: - d.addCallback(lambda ignored: reactor.stop()) - reactor.run() - - logging.info("File downloaded!") - - # Now transform it to json -- I couldn't find out how to use a - # two-character delimiter with the csv package, so I decided to handle - # the csv data by hand. We are doing this until #16601 gets deployed. - # https://trac.torproject.org/projects/tor/ticket/16601 - - # Code below is based in update-mirrors-json.py script from tpo - - # These are the names of each column e.g. adminContact - columns = [] - # List of mirrors to be built - mirrors = [] - - logging.info("Transforming csv data into json...") - logging.info("Getting data from csv") - try: - with codecs.open(csv_path, "rb", "utf-8") as csvfile: - for line in csvfile: - elements = line.split(", ") - # first entry have the names of the columns - if not columns: - columns = elements - else: - add_entry(mirrors, columns, elements) - except IOError as e: - logging.error("Couldn't read csv file: %s" % str(e)) - return - - logging.info("Creating json") - if is_json(json.dumps(mirrors)): - try: - with codecs.open(json_path, "w", "utf-8") as jsonfile: - # Make pretty json - json.dump( - mirrors, - jsonfile, - sort_keys=True, - indent=4, - separators=(',', ': '), - encoding="utf-8", - ) - except IOError as e: - logging.error("Couldn't write json: %s" % str(e)) - return - else: - logging.error("Invalid json file") - return - - # Now make the mirrors list to be used by GetTor - logging.info("Reading json") - try: - mirrors_json = codecs.open(json_path, "rb", "utf-8") - mirrors = json.load(mirrors_json) - except IOError as e: - logging.error("Couldn't open %s" % json_path) - return - - logging.info("Creating new list with protocol: %s" % p) - try: - list = codecs.open(mirrors_list, "w", "utf-8") - for entry in mirrors: - if args.proto is not 'all': - for e in entry: - if e == PROTOS[p] and entry[PROTOS[p]]: - add_mirror(list, entry, PROTOS[p]) - else: - for k, v in PROTOS: - if entry[v]: - add_mirror(list, entry, v) - logging.info("List created: %s" % mirrors_list) - except IOError as e: - logging.error("Couldn't open %s" % mirrors_list) - return - -if __name__ == "__main__": - logging_format = utils.get_logging_format() - logging.basicConfig( - filename='/path/to/gettor/log/get-mirrors.log', - format=logging_format, - datefmt="%Y-%m-%d %H:%M:%S", - level=logging.INFO - ) - logging.info("Started") - main() - logging.info("Finished") diff --git a/dev_scripts/process_chat.py b/dev_scripts/process_chat.py deleted file mode 100644 index 6ad7b59..0000000 --- a/dev_scripts/process_chat.py +++ /dev/null @@ -1,22 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -import sys -import logging - -import gettor.xmpp - -def main(): - try: - bot = gettor.xmpp.XMPP() - bot.start_bot() - except gettor.xmpp.ConfigError as e: - print "Configuration error: %s" % str(e) - except gettor.xmpp.InternalError as e: - print "Core module not working: %s" % str(e) - except Exception as e: - # in case something unexpected happens - print "Unexpected error: %s" % str(e) - -if __name__ == '__main__': - main() diff --git a/dev_scripts/process_email.py b/dev_scripts/process_email.py deleted file mode 100644 index 504b5dd..0000000 --- a/dev_scripts/process_email.py +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -import sys -import logging - -import gettor.smtp - -def main(): - logging_level = 'INFO' - logging_file = '/path/to/gettor/log/process_email.log' - logging_format = '[%(levelname)s] %(asctime)s - %(message)s' - date_format = "%Y-%m-%d" # %H:%M:%S - - logging.basicConfig( - format=logging_format, - datefmt=date_format, - filename = logging_file, - level = logging_level - ) - - logging.debug("New email received") - logging.debug("Creating new SMTP object") - - try: - service = gettor.smtp.SMTP('/path/to/gettor/smtp.cfg') - incoming = sys.stdin.read() - service.process_email(incoming) - logging.debug("Email processed sucessfully") - except gettor.smtp.ConfigError as e: - logging.error("Configuration error: %s" % str(e)) - except gettor.smtp.SendEmailError as e: - logging.error("SMTP not working: %s" % str(e)) - except gettor.smtp.InternalError as e: - logging.error("Core module not working: %s" % str(e)) - except Exception as e: - # in case something unexpected happens - logging.critical("Unexpected error: %s" % str(e)) - -if __name__ == '__main__': - main() diff --git a/dev_scripts/process_http.py b/dev_scripts/process_http.py deleted file mode 100644 index ec20d5f..0000000 --- a/dev_scripts/process_http.py +++ /dev/null @@ -1,14 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -import gettor.http2 - - -def main(): - api = gettor.http2.HTTP('http.cfg') - api.load_data() - # api.run() - api.build() - -if __name__ == '__main__': - main() diff --git a/dev_scripts/process_tweets.py b/dev_scripts/process_tweets.py deleted file mode 100644 index 59c2f48..0000000 --- a/dev_scripts/process_tweets.py +++ /dev/null @@ -1,35 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -import sys -import logging - -import gettor.twitter - -def main(): - logging_level = 'DEBUG' - logging_file = '/home/ilv/Proyectos/tor/gettor/log/process_tweets.log' - logging_format = '[%(levelname)s] %(asctime)s - %(message)s' - date_format = "%Y-%m-%d" # %H:%M:%S - - logging.basicConfig( - format=logging_format, - datefmt=date_format, - filename = logging_file, - level = logging_level - ) - - logging.debug("Starting bot") - try: - bot = gettor.twitter.TwitterBot() - bot.start() - except gettor.twitter.ConfigError as e: - logging.error("Configuration error: %s" % str(e)) - except gettor.twitter.InternalError as e: - logging.error("Core module not working: %s" % str(e)) - except Exception as e: - # in case something unexpected happens - logging.error("Unexpected error: %s" % str(e)) - -if __name__ == '__main__': - main() diff --git a/dev_scripts/report.py b/dev_scripts/report.py deleted file mode 100644 index b75d7c8..0000000 --- a/dev_scripts/report.py +++ /dev/null @@ -1,129 +0,0 @@ -#!/usr/bin/env python - -# This file is part of GetTor, a Tor Browser distribution system. - -# :authors: Sukhbir Singh sukhbir@torproject.org -# see also AUTHORS file -# -# :copyright: (c) 2008-2015, The Tor Project, Inc. - -# To read the log files and populate the db, run the script with `--logs' once -# every day and then clear the log files. To generate the report, run it with -# `--report', once (or twice?) a month. - -# TODO: Logging? argparse? - -import os -import sys -import ConfigParser -from datetime import datetime as dt - -import sqlite3 - -CORE_CFG = "core.cfg" - -CHANNELS = ["smtp", "xmpp", "twitter"] -LOG_FILES = [channel + ".log" for channel in CHANNELS] - -DB = "gettor2.db" - -OUTPUT = "report.log" - -REPORT = """@ GetTor Report for {} - -We received a total of {} requests in {}, with a peak of {} requests on {}. - -[*] Request -{} - -[*] OS -{} - -[*] Language -{} - -[*] Channel -{} - -""" - - -class Report(object): - def __init__(self): - self.config = ConfigParser.ConfigParser() - self.config.read(CORE_CFG) - - self.log_dir = self.config.get("log", "dir") - # This is defined in the sample log file. - self.log_format = ["date", "request", "os", "locale"] - - self.conn = sqlite3.connect(DB) - - def db_write(self): - self.cursor = self.conn.cursor() - for each in self.logs: - self.cursor.execute("""INSERT INTO requests - (date, request, os, locale, channel) VALUES - (:date, :request, :os, :locale, :channel)""", - each) - self.conn.commit() - self.conn.close() - - def get_logs(self): - self.logs = [] - for each in LOG_FILES: - with open(os.path.join(self.log_dir, each)) as f: - for line in f: - if line.startswith("[INFO] "): - # The "7" here is for the "[INFO] " text. - logs = [log.strip() for log in line[7:].split(";")] - log_data = {key: value for key, value in - zip(self.log_format, logs)} - # We also need the channel. - log_data["channel"] = each.split(".")[0] - self.logs.append(log_data) - self.db_write() - self.conn.close() - - def generate_report(self): - self.cursor = self.conn.cursor() - self.requests = [] - - self.cursor.execute("SELECT COUNT(*) FROM requests") - self.requests.append(self.cursor.fetchone()) - - self.cursor.execute("""SELECT date, COUNT(*) FROM requests -GROUP BY date -ORDER BY COUNT(date) -DESC""") - self.requests.append(self.cursor.fetchone()) - - self.columns = ["request", "os", "locale", "channel"] - for each in self.columns: - self.cursor.execute("""SELECT {0}, COUNT(*) FROM requests -GROUP BY {0} -ORDER BY COUNT({0}) -DESC""".format(each)) - result = "\n".join(["{0:>16}: {1}".format(each[0], each[1]) - for each in self.cursor.fetchall() - if not each[0] == "none"]) - self.requests.append(result) - - with open(OUTPUT, "a") as f: - f.write(REPORT.format(dt.now().strftime("%B %Y"), - self.requests[0][0], - dt.now().strftime("%B"), - self.requests[1][1], - dt.strptime(self.requests[1][0], "%Y-%m-%d") - .strftime("%B %-d"), - *self.requests[2:])) - self.conn.close() - -if __name__ == "__main__": - try: - if sys.argv[1] == "--logs": - Report().get_logs() - if sys.argv[1] == "--report": - Report().generate_report() - except IndexError: - sys.exit("You need either `--logs' or `--report.'") diff --git a/dev_scripts/sample-htaccess-api-mirror b/dev_scripts/sample-htaccess-api-mirror deleted file mode 100644 index 41594db..0000000 --- a/dev_scripts/sample-htaccess-api-mirror +++ /dev/null @@ -1,21 +0,0 @@ -Options +FollowSymLinks -RewriteEngine On - -# Operating System and Locale filters for providers -RewriteCond %{QUERY_STRING} os=([a-z]+)&lc=([a-z]+) -RewriteRule ^api/providers/([a-z-]+)$ api-content/providers-content/$1-content/%1-content/%2.json -RewriteCond %{QUERY_STRING} os=([a-z]+) -RewriteRule ^api/providers/([a-z-]+)$ api-content/providers-content/$1-content/%1 - -# Operating System and Locale filters for TPO downloads -RewriteCond %{QUERY_STRING} os=([a-z]+)&lc=([a-zA-Z-]+) -RewriteRule ^api/latest/([a-z]+)$ api-content/latest-content/$1-content/%1-content/%2.json -RewriteCond %{QUERY_STRING} os=([a-z]+) -RewriteRule ^api/latest/([a-z]+)$ api-content/latest-content/$1-content/%1 - -# General rules -RewriteRule ^api/providers/([a-z-]+)$ api-content/providers-content/$1 [NC] -RewriteRule ^api/latest/([a-z]+)$ api-content/latest-content/$1 [NC] -RewriteRule ^api/([a-z]+)$ api-content/$1 [NC] - -ErrorDocument 404 404-file diff --git a/dev_scripts/stats.py b/dev_scripts/stats.py deleted file mode 100644 index 7fca2af..0000000 --- a/dev_scripts/stats.py +++ /dev/null @@ -1,123 +0,0 @@ -#!/usr/bin/python -# -*- coding: utf-8 -*- -# -# This file is part of GetTor, a Tor Browser distribution system. -# -# :authors: Israel Leiva ilv@riseup.net -# see also AUTHORS file -# -# :copyright: (c) 2008-2014, The Tor Project, Inc. -# (c) 2014, Israel Leiva -# -# :license: This is Free Software. See LICENSE for license information. - - -import sqlite3 -import argparse - - -def main(): - """Script for showing stats. - - See argparse usage for more details. - - """ - parser = argparse.ArgumentParser(description='Utility for GetTor stats') - parser.add_argument('database', metavar='database.db', type=str, - help='the database file') - parser.add_argument('-s', '--service', default=None, - help='filter by service') - parser.add_argument('-t', '--type', default=None, - help='filter by type of request') - parser.add_argument('-o', '--os', default=None, - help='filter by OS') - parser.add_argument('-l', '--lc', default=None, - help='filter by locale') - parser.add_argument('-p', '--pt', default=None, - help='filter by PT requests') - parser.add_argument('-y', '--year', default=None, - help='filter by year') - parser.add_argument('-m', '--month', default=None, - help='filter by month') - parser.add_argument('-d', '--day', default=None, - help='filter by day') - parser.add_argument('-u', '--status', default=None, - help='filter by status of the request') - - args = parser.parse_args() - query = 'SELECT * FROM requests' - has_where = False - - # we build the query piece by piece - if args.service: - query = "%s %s" % (query, "WHERE service = '%s'" % args.service) - has_where = True - if args.type: - if has_where: - query = "%s %s" % (query, "AND type = '%s'" % args.type) - else: - query = "%s %s" % (query, "WHERE type = '%s'" % args.type) - has_where = True - if args.os: - if has_where: - query = "%s %s" % (query, "AND os = '%s'" % args.os) - else: - query = "%s %s" % (query, "WHERE os = '%s'" % args.os) - has_where = True - if args.lc: - if has_where: - query = "%s %s" % (query, "AND lc = '%s'" % args.lc) - else: - query = "%s %s" % (query, "WHERE lc = '%s'" % args.lc) - has_where = True - if args.pt: - if has_where: - query = "%s %s" % (query, "AND pt = %s" % args.pt) - else: - query = "%s %s" % (query, "WHERE pt = %s" % args.pt) - has_where = True - if args.year: - if has_where: - query = "%s %s" % (query, "AND year = %s" % args.year) - else: - query = "%s %s" % (query, "WHERE year = %s" % args.year) - has_where = True - if args.month: - if has_where: - query = "%s %s" % (query, "AND month = %s" % args.month) - else: - query = "%s %s" % (query, "WHERE month = %s" % args.month) - has_where = True - if args.day: - if has_where: - query = "%s %s" % (query, "AND day = %s" % args.day) - else: - query = "%s %s" % (query, "WHERE day = %s" % args.day) - has_where = True - if args.status: - if has_where: - query = "%s %s" % (query, "AND status = '%s'" % args.status) - else: - query = "%s %s" % (query, "WHERE status = '%s'" % args.status) - has_where = True - - con = sqlite3.connect(args.database) - - with con: - cur = con.cursor() - cur.execute(query) - rows = cur.fetchall() - # show it nice - print "\nNumber of results: %s\n" % len(rows) - cns = [cn[0] for cn in cur.description] - print "%-10s %-10s %-10s %-10s %-10s %-10s %-10s %-10s"\ - " %-15s %s"\ % (cns[0], cns[1], cns[2], cns[3], cns[4], cns[5], - cns[6], cns[7], cns[8], cns[9]) - - for row in rows: - print "%-10s %-10s %-10s %-10s %-10s %-10s %-10s %-10s"\ - " %-15s %s" % (row[0], row[1], row[2], row[3], row[4], - row[5], row[6], row[7], row[8], row[9]) - -if __name__ == "__main__": - main() diff --git a/gettor/main.py b/gettor/main.py index 9cd18b7..d59f66c 100644 --- a/gettor/main.py +++ b/gettor/main.py @@ -25,7 +25,9 @@ def run(gettor, app): """ This is GetTor's main entry point and main runtime loop. """ - settings = options.parse_settings() + config = "/home/gettor/gettor/gettor.conf.json" + + settings = options.parse_settings("en", config)
sendmail = Sendmail(settings) twitterdm = Twitterdm(settings) diff --git a/gettor/services/email/sendmail.py b/gettor/services/email/sendmail.py index 545fbb8..4644cb5 100644 --- a/gettor/services/email/sendmail.py +++ b/gettor/services/email/sendmail.py @@ -132,10 +132,14 @@ class Sendmail(object): ) )
+ body_msg = strings._("help_body_intro") + body_msg += strings._("help_body_paragraph") + body_msg += strings._("help_body_support") + yield self.sendmail( email_addr=id, subject=strings._("help_subject"), - body=strings._("help_body") + body=body_msg )
yield self.conn.update_stats( @@ -197,7 +201,14 @@ class Sendmail(object): else: link_msg = link_str
- body_msg = strings._("links_body").format(platform, link_msg, file) + body_msg = strings._("links_body_platform").format(platform) + body_msg += trings._("links_body_links").format(link_msg) + body_msg += trings._("links_body_archive") + body_msg += trings._("links_body_internet_archive") + body_msg += trings._("links_body_google_drive") + body_msg += trings._("links_body_internet_archive").format(file) + body_msg += trings._("links_body_ending") + subject_msg = strings._("links_subject")
hid = hashlib.sha256(id.encode('utf-8')) diff --git a/gettor/services/twitter/twitterdm.py b/gettor/services/twitter/twitterdm.py index d78604a..629fa5f 100644 --- a/gettor/services/twitter/twitterdm.py +++ b/gettor/services/twitter/twitterdm.py @@ -144,9 +144,13 @@ class Twitterdm(object): ) )
+ body_msg = strings._("help_body_intro") + body_msg += strings._("help_body_paragraph") + body_msg += strings._("help_body_support") + yield self.twitterdm( twitter_id=twitter_id, - message=strings._("help_body") + message=body_msg )
yield self.conn.update_stats( @@ -210,7 +214,13 @@ class Twitterdm(object): else: link_msg = link_str
- body_msg = strings._("links_body").format(platform, link_msg, file) + body_msg = strings._("links_body_platform").format(platform) + body_msg += trings._("links_body_links").format(link_msg) + body_msg += trings._("links_body_archive") + body_msg += trings._("links_body_internet_archive") + body_msg += trings._("links_body_google_drive") + body_msg += trings._("links_body_internet_archive").format(file) + body_msg += trings._("links_body_ending")
hid = hashlib.sha256(twitter_id.encode('utf-8')) log.info( diff --git a/gettor/utils/options.py b/gettor/utils/options.py index d313876..8504f42 100644 --- a/gettor/utils/options.py +++ b/gettor/utils/options.py @@ -24,12 +24,11 @@ def load_settings(config): settings.load() return settings
-def parse_settings(): +def parse_settings(locale, config): """ Parse settings and loads strings in a given locale This function needs to be rewritten considering passing a locale and returing translated strings """ - strings.load_strings("en") - config = "/home/gettor/gettor/gettor.conf.json" + strings.load_strings(locale) return load_settings(config) diff --git a/share/locale/en.json b/share/locale/en.json index 8ed6722..be7dd33 100644 --- a/share/locale/en.json +++ b/share/locale/en.json @@ -1,7 +1,15 @@ { - "links_body": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following archives:\n\n- Internet Archive: https://archive.org/details/@gettor%5Cn%5Cn- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU%5Cn%5CnDo... the file: {}\n\n \n--\nGetTor", + "links_body_platform": "You requested Tor Browser for {}.\n\n", + "links_body_links": "You will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\n", + "links_body_archive": "Should you have issues with any of the links above you can access the following archives:\n\n", + "links_body_internet_archive": "- Internet Archive: https://archive.org/details/@gettor%5Cn%5Cn", + "links_body_google_drive": "- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU%5Cn%5Cn", + "links_body_internet_archive": "Download the file: {}\n\n", + "links_body_ending": "\n--\nGetTor", "links_subject": "[GetTor] Links for your request", - "help_body": "This is how you can request a tor browser bundle link.\n\n Send an email to: gettor@torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\nWe only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n", + "help_body_intro": "This is how you can request a tor browser bundle link.\n\n", + "help_body_paragraph": "Send an email to: gettor@torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\n", + "help_body_support": "We only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n", "help_subject": "[GetTor] Help Email", "help_debug": "Log application errors to stdout", "help_config": "Custom config file location (optional)", @@ -9,8 +17,5 @@ "smtp_mirrors_subject": "[GetTor] Mirrors", "smtp_help_subject": "[GetTor] Help", "smtp_unsupported_locale_subject": "[GetTor] Unsupported locale", - "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported.", - "smtp_vlinks_msg": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU%5Cn%5Cn Download the file: {}\n\n \n--\nGetTor", - "smtp_mirrors_msg": "Hi! this is the GetTor robot.\n\nThank you for your request. Attached to this email you will find\nan updated list of mirrors of Tor Project's website.", - "smtp_help_msg": "Hi! This is the GetTor robot. I am here to help you download the\nlatest version of Tor Browser.\n\nPlease reply to this message with one of the options below:\n\nwindows\nlinux\nosx\nmirrors\n\nI will then send you the download instructions.\n\nIf you are unsure, just send a blank reply to this message." + "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported." } diff --git a/share/locale/es.json b/share/locale/es.json index 4d18f61..be7dd33 100644 --- a/share/locale/es.json +++ b/share/locale/es.json @@ -1,16 +1,21 @@ { - "links_body": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following archives:\n\n- Internet Archive: https://archive.org/details/@gettor%5Cn%5Cn- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU%5Cn%5CnDo... the file: {}\n\n \n--\nGetTor", + "links_body_platform": "You requested Tor Browser for {}.\n\n", + "links_body_links": "You will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\n", + "links_body_archive": "Should you have issues with any of the links above you can access the following archives:\n\n", + "links_body_internet_archive": "- Internet Archive: https://archive.org/details/@gettor%5Cn%5Cn", + "links_body_google_drive": "- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU%5Cn%5Cn", + "links_body_internet_archive": "Download the file: {}\n\n", + "links_body_ending": "\n--\nGetTor", "links_subject": "[GetTor] Links for your request", - "help_body": "This is how you can request a tor browser bundle link.\n\n Send an email to: gettor@torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\nWe only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n", + "help_body_intro": "This is how you can request a tor browser bundle link.\n\n", + "help_body_paragraph": "Send an email to: gettor@torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\n", + "help_body_support": "We only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n", "help_subject": "[GetTor] Help Email", "help_debug": "Log application errors to stdout", "help_config": "Custom config file location (optional)", "smtp_links_subject": "[GetTor] Links for your request", "smtp_mirrors_subject": "[GetTor] Mirrors", - "smtp_help_subject": "[GetTor] Ayuda", + "smtp_help_subject": "[GetTor] Help", "smtp_unsupported_locale_subject": "[GetTor] Unsupported locale", - "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported.", - "smtp_vlinks_msg": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU%5Cn%5Cn Download the file: {}\n\n \n--\nGetTor", - "smtp_mirrors_msg": "Hi! this is the GetTor robot.\n\nThank you for your request. Attached to this email you will find\nan updated list of mirrors of Tor Project's website.", - "smtp_help_msg": "Hi! This is the GetTor robot. I am here to help you download the\nlatest version of Tor Browser.\n\nPlease reply to this message with one of the options below:\n\nwindows\nlinux\nosx\nmirrors\n\nI will then send you the download instructions.\n\nIf you are unsure, just send a blank reply to this message." + "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported." } diff --git a/share/locale/pt.json b/share/locale/pt.json index 8ed6722..be7dd33 100644 --- a/share/locale/pt.json +++ b/share/locale/pt.json @@ -1,7 +1,15 @@ { - "links_body": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following archives:\n\n- Internet Archive: https://archive.org/details/@gettor%5Cn%5Cn- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU%5Cn%5CnDo... the file: {}\n\n \n--\nGetTor", + "links_body_platform": "You requested Tor Browser for {}.\n\n", + "links_body_links": "You will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\n", + "links_body_archive": "Should you have issues with any of the links above you can access the following archives:\n\n", + "links_body_internet_archive": "- Internet Archive: https://archive.org/details/@gettor%5Cn%5Cn", + "links_body_google_drive": "- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU%5Cn%5Cn", + "links_body_internet_archive": "Download the file: {}\n\n", + "links_body_ending": "\n--\nGetTor", "links_subject": "[GetTor] Links for your request", - "help_body": "This is how you can request a tor browser bundle link.\n\n Send an email to: gettor@torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\nWe only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n", + "help_body_intro": "This is how you can request a tor browser bundle link.\n\n", + "help_body_paragraph": "Send an email to: gettor@torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\n", + "help_body_support": "We only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n", "help_subject": "[GetTor] Help Email", "help_debug": "Log application errors to stdout", "help_config": "Custom config file location (optional)", @@ -9,8 +17,5 @@ "smtp_mirrors_subject": "[GetTor] Mirrors", "smtp_help_subject": "[GetTor] Help", "smtp_unsupported_locale_subject": "[GetTor] Unsupported locale", - "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported.", - "smtp_vlinks_msg": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU%5Cn%5Cn Download the file: {}\n\n \n--\nGetTor", - "smtp_mirrors_msg": "Hi! this is the GetTor robot.\n\nThank you for your request. Attached to this email you will find\nan updated list of mirrors of Tor Project's website.", - "smtp_help_msg": "Hi! This is the GetTor robot. I am here to help you download the\nlatest version of Tor Browser.\n\nPlease reply to this message with one of the options below:\n\nwindows\nlinux\nosx\nmirrors\n\nI will then send you the download instructions.\n\nIf you are unsure, just send a blank reply to this message." + "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported." } diff --git a/tests/test_email_service.py b/tests/test_email_service.py index 4c8ff10..8e60f7a 100644 --- a/tests/test_email_service.py +++ b/tests/test_email_service.py @@ -13,7 +13,7 @@ class EmailServiceTests(unittest.TestCase): # Fail any tests which take longer than 15 seconds. timeout = 15 def setUp(self): - self.settings = conftests.options.parse_settings() + self.settings = conftests.options.parse_settings("en","./gettor.conf.json") self.sm_client = conftests.sendmail.Sendmail(self.settings) self.locales = conftests.strings.get_locales()
diff --git a/tests/test_locales.py b/tests/test_locales.py index bea7e9f..d4e52c5 100644 --- a/tests/test_locales.py +++ b/tests/test_locales.py @@ -11,7 +11,7 @@ class LocalesTests(unittest.TestCase): # Fail any tests which take longer than 15 seconds. timeout = 15 def setUp(self): - self.settings = conftests.options.parse_settings() + self.settings = conftests.options.parse_settings("en","./gettor.conf.json") self.locales = conftests.strings.get_locales()
diff --git a/tests/test_twitter.py b/tests/test_twitter.py index fe155cc..7458cfc 100644 --- a/tests/test_twitter.py +++ b/tests/test_twitter.py @@ -10,7 +10,7 @@ class TwitterTests(unittest.TestCase): # Fail any tests which take longer than 15 seconds. timeout = 15 def setUp(self): - self.settings = conftests.options.parse_settings() + self.settings = conftests.options.parse_settings("en","./gettor.conf.json") self.tw_client = conftests.twitter.Twitter(self.settings)