[tor-commits] [gettor/master] Update readme
hiro at torproject.org
hiro at torproject.org
Wed Dec 11 16:26:00 UTC 2019
commit 744934523a64e312ce42eaaa31ef936beeae5992
Author: hiro <hiro at torproject.org>
Date: Thu Oct 31 15:48:05 2019 +0100
Update readme
Breakdown strings for translations
Delete unused scripts
---
README.md | 159 ++++-----------
dev_scripts/blacklist.py | 111 -----------
dev_scripts/core_demo.py | 19 --
dev_scripts/create_api_mirror.py | 305 ----------------------------
dev_scripts/create_gh_mirrors.py | 207 -------------------
dev_scripts/distribution_methods.txt | 19 --
dev_scripts/get_mirrors.py | 353 ---------------------------------
dev_scripts/process_chat.py | 22 --
dev_scripts/process_email.py | 41 ----
dev_scripts/process_http.py | 14 --
dev_scripts/process_tweets.py | 35 ----
dev_scripts/report.py | 129 ------------
dev_scripts/sample-htaccess-api-mirror | 21 --
dev_scripts/stats.py | 123 ------------
gettor/main.py | 4 +-
gettor/services/email/sendmail.py | 15 +-
gettor/services/twitter/twitterdm.py | 14 +-
gettor/utils/options.py | 5 +-
share/locale/en.json | 17 +-
share/locale/es.json | 19 +-
share/locale/pt.json | 17 +-
tests/test_email_service.py | 2 +-
tests/test_locales.py | 2 +-
tests/test_twitter.py | 2 +-
24 files changed, 102 insertions(+), 1553 deletions(-)
diff --git a/README.md b/README.md
index f7e130d..a3df81b 100644
--- a/README.md
+++ b/README.md
@@ -32,144 +32,55 @@ Here is a list of the main goals the new GetTor should accomplish:
Installing GetTor
=================
-```sh
-WORKON_HOME=${HOME}/venv
-export WORKON_HOME
-mkdir -p $WORKON_HOME
-source $(which virtualenvwrapper.sh)
-git clone https://git.torproject.org/gettor.git && cd gettor
-mkvirtualenv -a $PWD -r requirements.txt --unzip-setuptools --setuptools gettor
-```
-
-From now on, to use GetTor's virtualenv, just do ``$ workon gettor``
-(after sourcing virtualenvwrapper.sh, as before). To exit the virtualenv
-without exiting the shell, do ``$ deactivate``.
-
-```sh
-export PYTHONPATH=$PYTHONPATH:${VIRTUAL_ENV}/lib/python/site-packages
-```
-
-```sh
-$ ./scripts/create_db
-$ ./bin/gettor_service start
-```
-
-Running tests
-=================
-
-GetTor includes PyTest unit tests. To run the tests, first install some dependencies:
-
-```sh
-pip3 install -r .test.requirements.txt
-```
-
-Then you can run `pytest` against the `tests/` directory.
-
-```sh
-pytest tests/
-```
-
-
-How does the new GetTor works?
-==============================
-Below are some specifications and core concepts on how the new GetTor works.
+To install gettor locally please install the following packages (on debian):
-*Links files*: Currently links are saved in files with the '.links' extension,
-using the ConfigParser format (RFC 882). A sample link file should look like
-this:
+python3-coverage
+python3-dkim
+python3-dns
+python3-internetarchive
+python3-openssl
+python3-pytest
+python3-requests-oauthlib
+python3-service-identity
+python3-twisted
+sqlite3
---- BEGIN FILE ---
+The following packages are needed to run a gettor instance:
- [provider]
- name = CoolCloudProvider
+internetarchive
+jq
+rclone
- [key]
- fingerprint = AAAA BBBB CCCC DDDD EEEE FFFF GGGG HHHH IIII JJJJ
+Specifically:
+internetarchive is needed to send Tor Browser files via command line to the internet archive.
+jq is a json parser that is used to find out about the new tor browser releases.
+Both internetarchive and jq are used in: scripts/update_files
- [linux]
- en = Package (64-bit): https://cool.cloud.link64
- ASC signature (64-bit): https://cool.cloud.link64.asc
- Package SHA256 checksum (64-bit): superhash64,
- Package (32-bit): https://cool.cloud.link32
- ASC signature (32-bit): https://cool.cloud.link32.asc
- Package SHA256 checksum (32-bit): superhash32
+The following packages are instead needed to deploy gettor via ansible:
- [windows]
- ...
+ansible
+ansible-lint
- [osx]
- ...
+Gettor ansible playbooks live at: https://gitweb.torproject.org/admin/services/gettor.git/
---- END FILE ---
+Finally the following package is used store Tor Browser files via git and support large files:
+git-lfs
-You can also check providers/dropbox.links for a better example.
-WORKON_HOME=${HOME}/.virtualenvs
- export WORKON_HOME
- mkdir -p $WORKON_HOME
- source $(which virtualenvwrapper.sh)
-*Core*: the heart of GetTor. Receives requests for links for a certain OS and
-language and respond accordingly. It also presents an easy way for scripts
-to create links file.
-*SMTP*: Receives requests via email, process them, contact the core module if
-necessary and respond to the user in the specified language. People can send
-blank or dummy emails to it to receive a help message describing how to ask
-for links. Email forwarding is used to redirect the emails to GetTor.
-*XMPP*: Same as above, but via XMPP (account needed). It has been tested with
-dukgo.com, jabber.ccc.de, riseup.net. It doesn't seem to be able to interact
-with gtalk users.
+Once gettor is installed you can run it with:
-*Twitter*: Receive requests via Twitter direct messages, contact the core module
-if necessary and respond to the user in the specified language. Unfinished.
-
-*DB*: Store anonymous info about the people that interact with GetTor in order
-to keep count of the number of requests per person and avoid malicious users
-that try to collapse the service. It also keeps count of how many requests
-GetTor has received during its lifetime. A lot of other data was being saved
-in the original gsoc project, but it was changed to save the minimum.
-
-*Blacklist*: Provide a mechanism to avoid flood and deny interaction to
-malicious users.
-
-*Providers scripts*: every supported provider should have a script to
-automatically upload packages to 'the cloud' and create the corresponding
-links files. The script should consider the following steps:
-
- * Upload the packages.
- * Get the sha256 checksum of the files uploaded.
- * Get the PGP key fingerprint that signed the files.
- * Check for .asc file for every package uploaded.
- * Put all together in a '.link' file (using the core module).
-
-
-What is the current status of the new GetTor?
-=============================================
-
-Deployed and working.
-
-
-How can I help?
-================
-
-If you have ideas to improve GetTor and/or add new providers, please tell us!
-I'm currently the lead developer on this, so if you have any comments/doubts/
-ideas you can send me an e-mail to ilv _at_ riseup _dot_ net or ping me (ilv),
-or sukhe or mrphs at #tor-dev in the OFTC IRC network. For openning tickets you
-should use the trac[0] and select the GetTor component. Some neat ideas we
-could use are the following:
+```
+$ ./bin/gettor_service start
+```
- * Report bugs!
- * Create script for new providers, namely: Google Drive, Github. Check
-providers.txt
- * Create a new module for distributing links. Check distribution_methods.txt
- * Finish the Twitter module.
- * Propose code/behaviour improvements.
- * Update the specs.
+Running tests
+=================
+GetTor includes PyTest unit tests. To run the tests, first install some dependencies:
-References
-===========
-[0] https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&component=GetTor&col=id&col=summary&col=component&col=status&col=type&col=priority&col=milestone&order=priority
+```
+$ pytest-3 tests/
+```
diff --git a/dev_scripts/blacklist.py b/dev_scripts/blacklist.py
deleted file mode 100644
index 1aefe1a..0000000
--- a/dev_scripts/blacklist.py
+++ /dev/null
@@ -1,111 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-#
-# This file is part of GetTor, a Tor Browser distribution system.
-#
-# :authors: Israel Leiva <ilv at riseup.net>
-# see also AUTHORS file
-#
-# :copyright: (c) 2008-2014, The Tor Project, Inc.
-# (c) 2014, Israel Leiva
-#
-# :license: This is Free Software. See LICENSE for license information.
-
-import sys
-import time
-import sqlite3
-import argparse
-
-
-def main():
- """Script for managing blacklisting of users.
-
- See argparse usage for more details.
-
- """
- parser = argparse.ArgumentParser(description='Utility for GetTor'
- ' blacklisting')
- parser.add_argument('database', metavar='database.db', type=str,
- help='the database file')
- parser.add_argument('-u', '--user', default=None,
- help='filter by user hash')
- parser.add_argument('-s', '--service', default=None,
- help='filter by service')
- parser.add_argument('-b', '--blocked', default=None,
- help='filter by blocked users')
- parser.add_argument('-a', '--add', default=None, nargs=3,
- metavar=('USER', 'SERVICE', 'BLOCKED'),
- help='add user')
- parser.add_argument('-c', '--clean', default=None, const='c', nargs='?',
- metavar='user hash',
- help='clean table (delete expired blacklistings)')
- parser.add_argument('-r', '--requests', default=None,
- help='number of requests; everyone with number of'
- ' requests greather than this will be cleaned up')
-
- args = parser.parse_args()
- query = ''
- con = sqlite3.connect(args.database)
-
- if args.add:
- # add new entry, useful for adding users permanently blocked
- query = "INSERT INTO users VALUES('%s', '%s', 1, %s, %s)"\
- % (args.add[0], args.add[1], args.add[2], time.time())
- with con:
- cur = con.cursor()
- cur.execute(query)
- print "Query execute successfully"
- elif args.clean:
- if args.clean == 'c':
- if args.requests:
- # delete by number of times
- query = "DELETE FROM users WHERE times > %s" % args.requests
- with con:
- cur = con.cursor()
- cur.execute(query)
- print "Query executed successfully."
- else:
- sys.exit("Number of requests missing. See --help.")
- else:
- # delete by id
- query = "DELETE FROM users WHERE id='%s'" % args.clean
- with con:
- cur = con.cursor()
- cur.execute(query)
- print "Query execute succcessfully."
- else:
- query = "SELECT * FROM users"
- has_where = False
- # filter
- if args.service:
- query = "%s %s" % (query, "WHERE service='%s'" % args.service)
- has_where = True
- if args.user:
- if has_where:
- query = "%s %s" % (query, "AND id='%s'" % args.user)
- else:
- query = "%s %s" % (query, "WHERE id='%s'" % args.user)
- has_where = True
- if args.blocked:
- if has_where:
- query = "%s %s" % (query, "AND blocked=%s" % args.blocked)
- has_where = True
- else:
- query = "%s %s" % (query, "WHERE blocked=%s" % args.blocked)
-
- with con:
- cur = con.cursor()
- cur.execute(query)
- rows = cur.fetchall()
- # show it nice
- print "\nNumber of results: %s\n" % len(rows)
- cns = [cn[0] for cn in cur.description]
- print "%-70s %-10s %-10s %-10s %-s" % (cns[0], cns[1], cns[2],
- cns[3], cns[4])
-
- for row in rows:
- print "%-70s %-10s %-10s %-10s %s" % (row[0], row[1], row[2],
- row[3], row[4])
-
-if __name__ == "__main__":
- main()
diff --git a/dev_scripts/core_demo.py b/dev_scripts/core_demo.py
deleted file mode 100644
index 88f725d..0000000
--- a/dev_scripts/core_demo.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#!/usr/bin/python
-#
-# Dummy script to test GetTore's Core module
-#
-
-import gettor.core
-
-try:
- core = gettor.core.Core()
- links = core.get_links('dummy service', 'linux', 'en')
- print links
-except gettor.core.ConfigError as e:
- print "Misconfiguration: " + str(e)
-except gettor.core.UnsupportedOSError as e:
- print "Unsupported OS: " + str(e)
-except gettor.core.UnsupportedLocaleError as e:
- print "Unsupported Locale: " + str(e)
-except gettor.core.InternalError as e:
- print "Internal error: " + str(e)
diff --git a/dev_scripts/create_api_mirror.py b/dev_scripts/create_api_mirror.py
deleted file mode 100644
index e80ec4a..0000000
--- a/dev_scripts/create_api_mirror.py
+++ /dev/null
@@ -1,305 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# This file is part of GetTor.
-#
-# :authors: Israel Leiva <ilv at torproject.org>
-# see also AUTHORS file
-#
-# :copyright: (c) 2008-2015, The Tor Project, Inc.
-# (c) 2015, Israel Leiva
-#
-# :license: This is Free Software. See LICENSE for license information.
-
-import os
-import re
-import json
-import codecs
-import urllib2
-import ConfigParser
-
-from time import gmtime, strftime
-
-
-"""Script to build a static mirror of GetTor RESTful API"""
-
-# currently supported locales for Tor Browser
-LC = ['ar', 'de', 'en-US', 'es-ES', 'fa', 'fr', 'it', 'ko', 'nl', 'pl',
- 'pt-PT', 'ru', 'tr', 'vi', 'zh-CN']
-
-# https://gitweb.tpo/tor-browser-spec.git/tree/processes/VersionNumbers
-# does not say anything about operating systems, so it's possible the
-# notation might change in the future. We should always use the same three
-# strings though: linux, windows, osx.
-OS = {
- 'Linux': 'linux',
- 'Windows': 'windows',
- 'MacOS': 'osx'
-}
-
-# based on
-# https://gitweb.tpo.org/tor-browser-spec.git/tree/processes/VersionNumbers
-# except for the first one, which is based on current RecommendedTBBVersions
-RE = {
- 'os': '(.*)-(\w+)',
- 'alpha': '\d\.\d(\.\d)*a\d+',
- 'beta': '\d\.\d(\.\d)*b\d+',
- 'stable': '\d\.\d(\.\d)*'
-}
-
-# strings to build names of packages depending on OS.
-PKG = {
- 'windows': 'torbrowser-install-%s_%s.exe',
- 'linux': 'tor-browser-linux%s-%s_%s.tar.xz',
- 'osx': 'TorBrowser-%s-osx64_%s.dmg'
-}
-
-# bin and asc are used to build the download links for each version, os and lc
-URL = {
- 'version': 'https://gettor.torproject.org/api/latest.json',
- 'mirrors': 'https://gettor.torproject.org/api/mirrors.json',
- 'providers': 'https://gettor.torproject.org/api/providers.json'
-}
-
-GETTOR_URL = 'http://gettor.torproject.org'
-SERVER_URL = 'https://yourprivateserver.com'
-API_TREE = '/home/ilv/Proyectos/tor/static_api/'
-
-class ConfigError(Exception):
- pass
-
-
-class InternalError(Exception):
- pass
-
-
-class APIMirror(object):
- """ Provide useful resources via RESTful API. """
- def __init__(self, cfg=None):
- self.url = SERVER_URL
- self.tree = API_TREE
-
- def _is_json(self, my_json):
- """ Check if json generated is valid.
-
- :param: my_json (string) data to ve verified.
-
- :return: (bool) true if data is json-valid, false otherwise.
-
- """
- try:
- json_object = json.loads(my_json)
- except ValueError, e:
- return False
- return True
-
- def _write_json(self, path, content):
- """
- """
- try:
- with codecs.open(
- path,
- "w",
- "utf-8"
- ) as jsonfile:
- # Make pretty json
- json.dump(
- content,
- jsonfile,
- sort_keys=True,
- indent=4,
- separators=(',', ': '),
- encoding="utf-8",
- )
- except IOError as e:
- #logging.error("Couldn't write json: %s" % str(e))
- print "Error building %s: %s" % (path, str(e))
- print "%s built" % path
-
- def _get_provider_name(self, p):
- """ Return simplified version of provider's name.
-
- :param: p (string) provider's name.
-
- :return: (string) provider's name in lowercase and without spaces.
-
- """
- p = p.replace(' ', '-')
- return p.lower()
-
-
- def _load_latest_version(self):
- """ Load latest version data from GetTor API. """
- response = urllib2.urlopen(URL['version'])
- json_response = json.load(response)
- self.lv = json_response
-
- def _load_links(self):
- """ Load links and providers data. """
-
- response = urllib2.urlopen(URL['providers'])
- json_response = json.load(response)
-
- links = {}
-
- for provider in json_response:
- if provider != 'updated_at':
- provider_url = json_response[provider]
-
- provider_response = urllib2.urlopen("%s.json" % provider_url)
- provider_content = json.load(provider_response)
-
- json_response[provider] = provider_url.replace(
- GETTOR_URL,
- self.url
- )
-
-
- links[provider] = provider_content
-
- self.providers = json_response
- self.links = links
-
- def _load_mirrors(self):
- """ Load mirrors data from GetTor API. """
- response = urllib2.urlopen(URL['mirrors'])
- json_response = json.load(response)
- self.mirrors = json_response
-
- def _load_resources(self):
- """ Load available resources data. """
-
- self.resources = {
- 'providers': '%s/providers' % self.url,
- 'mirrors': '%s/mirrors' % self.url,
- 'latest_version': '%s/latest' % self.url,
- 'updated_at': strftime("%Y-%m-%d %H:%M:%S", gmtime())
- }
-
- def load_data(self):
- """ Load all data."""
- self._load_links()
- self._load_mirrors()
- self._load_resources()
- self._load_latest_version()
-
- def build(self):
- """ Build RESTful API. """
-
- print "Building API mirror"
-
- # resources
- self._write_json(
- os.path.join(self.tree, 'api'),
- self.resources
- )
-
- api_path = os.path.join(self.tree, 'api-content')
- if not os.path.isdir(api_path):
- os.mkdir(api_path)
-
- # providers
- self._write_json(
- os.path.join(api_path, 'providers'),
- self.providers
- )
-
- providers_path = os.path.join(api_path, 'providers-content')
- if not os.path.isdir(providers_path):
- os.mkdir(providers_path)
-
- for provider in self.links:
- if provider == 'updated_at':
- continue
-
- self._write_json(
- os.path.join(providers_path, provider),
- self.links[provider]
- )
-
- provider_path = os.path.join(
- providers_path,
- "%s-content" % provider
- )
-
- if not os.path.isdir(provider_path):
- os.mkdir(provider_path)
-
- for osys in self.links[provider]:
- self._write_json(
- os.path.join(provider_path, osys),
- self.links[provider][osys]
- )
-
- provider_os_path = os.path.join(
- provider_path, "%s-content" % osys
- )
-
- if not os.path.isdir(provider_os_path):
- os.mkdir(provider_os_path)
-
- for lc in self.links[provider][osys]:
- self._write_json(
- os.path.join(provider_os_path, lc),
- self.links[provider][osys][lc]
- )
-
- # latest version
- self._write_json(
- os.path.join(api_path, 'latest'),
- self.lv
- )
-
- lv_path = os.path.join(api_path, 'latest-content')
- if not os.path.isdir(lv_path):
- os.mkdir(lv_path)
-
- for release in self.lv:
- if release == 'updated_at':
- continue
-
- self._write_json(
- os.path.join(lv_path, release),
- self.lv[release]
- )
-
- release_path = os.path.join(
- lv_path,
- "%s-content" % release
- )
-
- if not os.path.isdir(release_path):
- os.mkdir(release_path)
-
- for osys in self.lv[release]['downloads']:
- self._write_json(
- os.path.join(release_path, osys),
- self.lv[release]['downloads'][osys]
- )
-
- release_os_path = os.path.join(
- release_path,
- "%s-content" % osys
- )
-
- if not os.path.isdir(release_os_path):
- os.mkdir(release_os_path)
-
- for lc in self.lv[release]['downloads'][osys]:
- self._write_json(
- os.path.join(release_os_path, lc),
- self.lv[release]['downloads'][osys][lc]
- )
-
- # mirrors
- self._write_json(
- os.path.join(api_path, 'mirrors'),
- self.mirrors
- )
-
-def main():
- api = APIMirror()
- api.load_data()
- api.build()
-
-if __name__ == '__main__':
- main()
diff --git a/dev_scripts/create_gh_mirrors.py b/dev_scripts/create_gh_mirrors.py
deleted file mode 100644
index e555488..0000000
--- a/dev_scripts/create_gh_mirrors.py
+++ /dev/null
@@ -1,207 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# This file is part of GetTor
-#
-# :authors: Israel Leiva <ilv at torproject.org>
-# see also AUTHORS file
-#
-# :license: This is Free Software. See LICENSE for license information.
-
-"""create_gh_mirrors -- Create landing page and readme for Github."""
-
-import os
-import ConfigParser
-
-import github3
-
-import gettor.core
-
-
-def create_readme(tpl_path, md_path, tb_version, links):
- """Create README file with links stored in github.links.
-
- :param: tpl_path (string) path to file used as template.
- :param: md_path (string) path to file generated.
- :param: tb_version (string) tor browser version.
- :param: links (object) github links.
-
- """
- lcs = ['FA', 'ZH', 'TR', 'EN']
-
- md_file = open(md_path, 'w')
- with open(tpl_path, 'r') as tpl_file:
- content_md = tpl_file.read()
-
- for lc in lcs:
- win_link = links.get('windows', lc.lower())
- win_pkg, win_sig, win_sha = [e for e in win_link.split("$") if e]
-
- osx_link = links.get('osx', lc.lower())
- osx_pkg, osx_sig, osx_sha = [e for e in osx_link.split("$") if e]
-
- linux_links = links.get('linux', lc.lower())
- linux32_link, linux64_link = linux_links.split(',')
- linux32_pkg, linux32_sig, linux32_sha = [
- e for e in linux32_link.split("$") if e
- ]
- linux64_pkg, linux64_sig, linux64_sha = [
- e for e in linux64_link.split("$") if e
- ]
-
- content_md = content_md.replace(
- "%WINDOWS_{}%".format(lc), win_pkg
- )
- content_md = content_md.replace(
- "%WINDOWS_{}_SIG%".format(lc), win_sig
- )
-
- content_md = content_md.replace(
- "%OSX_{}%".format(lc), osx_pkg
- )
- content_md = content_md.replace(
- "%OSX_{}_SIG%".format(lc), osx_sig
- )
-
- content_md = content_md.replace(
- "%LINUX32_{}%".format(lc), linux32_pkg
- )
- content_md = content_md.replace(
- "%LINUX32_{}_SIG%".format(lc), linux32_sig
- )
- content_md = content_md.replace(
- "%LINUX64_{}%".format(lc), linux64_pkg
- )
- content_md = content_md.replace(
- "%LINUX64_{}_SIG%".format(lc), linux64_sig
- )
-
- content_md = content_md.replace("%TB_VERSION%", tb_version)
- md_file.write(content_md)
-
- print "README generated with Tor Browser %s" % tb_version
-
-
-def create_landing_html(tpl_path, html_path, tb_version, links):
- """Create README file with links stored in github.links.
-
- :param: tpl_path (string) path to file used as template.
- :param: html_path (string) path to file generated.
- :param: tb_version (string) tor browser version.
- :param: links (object) github links.
-
- """
- lcs = ['FA', 'ZH', 'TR', 'EN']
-
- html_file = open(html_path, 'w')
- with open(tpl_path, 'r') as tpl_file:
- content_html = tpl_file.read().replace('\n', '')
-
- for lc in lcs:
- win_link = links.get('windows', lc.lower())
- win_pkg, win_sig, win_sha = [e for e in win_link.split("$") if e]
-
- osx_link = links.get('osx', lc.lower())
- osx_pkg, osx_sig, osx_sha = [e for e in osx_link.split("$") if e]
-
- linux_links = links.get('linux', lc.lower())
- linux32_link, linux64_link = linux_links.split(',')
- linux32_pkg, linux32_sig, linux32_sha = [
- e for e in linux32_link.split("$") if e
- ]
- linux64_pkg, linux64_sig, linux64_sha = [
- e for e in linux64_link.split("$") if e
- ]
-
- content_html = content_html.replace(
- "%WINDOWS_{}%".format(lc), win_pkg
- )
- """
- content_html = content_html.replace(
- "%WINDOWS_{}_SIG%".format(lc), win_sig
- )
- """
- content_html = content_html.replace(
- "%OSX_{}%".format(lc), osx_pkg
- )
- """
- content_html = content_html.replace(
- "%OSX_{}_SIG%".format(lc), osx_sig
- )
- """
- content_html = content_html.replace(
- "%LINUX32_{}%".format(lc), linux32_pkg
- )
- """
- content_html = content_html.replace(
- "%LINUX32_{}_SIG%".format(lc), linux32_sig
- )
- """
- content_html = content_html.replace(
- "%LINUX64_{}%".format(lc), linux64_pkg
- )
- """
- content_html = content_html.replace(
- "%LINUX64_{}_SIG%".format(lc), linux64_sig
- )
- """
-
- content_html = content_html.replace(
- "%TB_VERSION%", tb_version
- )
- html_file.write(content_html)
-
- print "HTML generated with Tor Browser %s" % tb_version
-
-
-def main():
- """Generate HTML and md files and update it in Github."""
- github_links = 'providers/github.links'
- tb_version_path = 'latest_torbrowser.cfg'
- md_path = 'upload/readme_gh.md'
- html_path = 'upload/landing_gh.html'
- md_tpl_path = 'upload/readme_gh.tpl'
- html_tpl_path = 'upload/landing_gh.tpl'
- github_access_token = ''
-
- tb_version_config = ConfigParser.ConfigParser()
- tb_version_config.read(tb_version_path)
- tb_version = tb_version_config.get('version', 'current')
-
- links = ConfigParser.ConfigParser()
- links.read(github_links)
-
- create_landing_html(html_tpl_path, html_path, tb_version, links)
- create_readme(md_tpl_path, md_path, tb_version, links)
-
- landing = open(html_path, 'r')
- content_landing = landing.read().replace('\n', '')
-
- readme = open(md_path, 'r')
- content_readme = readme.read()
-
- gh = github3.login(token=github_access_token)
- repo_landing = gh.repository('thetorproject', 'gettor')
- repo_readme = gh.repository('thetorproject', 'gettorbrowser')
-
- file_landing_gh = repo_landing.file_contents('index.html', 'gh-pages')
- file_readme_gh = repo_readme.file_contents('README.md')
-
- data_landing = {
- 'message': 'Updating landing page.',
- 'content': content_landing,
- 'branch': 'gh-pages'
- }
-
- data_readme = {
- 'message': 'Updating README.',
- 'content': content_readme
- }
-
- file_landing_gh.update(**data_landing)
- print "Landing page updated in gettor"
-
- file_readme_gh.update(**data_readme)
- print "README updated in gettorbrowser"
-
-if __name__ == "__main__":
- main()
diff --git a/dev_scripts/distribution_methods.txt b/dev_scripts/distribution_methods.txt
deleted file mode 100644
index 8bbdd4b..0000000
--- a/dev_scripts/distribution_methods.txt
+++ /dev/null
@@ -1,19 +0,0 @@
-================================== CURRENT ====================================
-
-* SMTP: this one is ready and tested.
-
-* XMPP: this one is ready and tested. We'll probably deploy SMTP first, tough.
-
-================================== IN DEVELOPMENT =============================
-
-* Twitter: This one is half done. I forked a twitter bot from wfn's gsoc work
-on BridgeDB, but I got confused with some Twitter policies. I thought that the
-bot *had* to follow someone back to send him/her DMs, but it turns out that you
-just need to be followed, not follow back (thanks to mrphs for pointing out
-this).
-
-================================== (CRAZY) IDEAS ==============================
-
-* Duckduckgo: This is just a thought, I haven't done proper research. What if
-someone could ask for gettor links like this: "!gettor"?
-
diff --git a/dev_scripts/get_mirrors.py b/dev_scripts/get_mirrors.py
deleted file mode 100644
index afa4cc9..0000000
--- a/dev_scripts/get_mirrors.py
+++ /dev/null
@@ -1,353 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# This file is part of GetTor
-#
-# :authors: Israel Leiva <ilv at torproject.org>
-# see also AUTHORS file
-#
-# :license: This is Free Software. See LICENSE for license information.
-
-"""get_mirrors -- Download the list of tpo's mirrors for GetTor."""
-
-import os
-import json
-import codecs
-import logging
-import argparse
-
-
-from OpenSSL import SSL
-from OpenSSL import crypto
-
-from twisted.web import client
-from twisted.python import log
-from twisted.internet import ssl
-from twisted.internet import defer
-from twisted.internet import protocol
-from twisted.internet import reactor
-from twisted.internet.error import TimeoutError
-from twisted.internet.error import DNSLookupError
-from twisted.internet.error import ConnectionRefusedError
-
-from gettor import utils
-
-# Associate each protocol with its column name in tor-mirrors.csv
-PROTOS = {
- 'https': 'httpsWebsiteMirror',
- 'http': 'httpWebsiteMirror',
- 'rsync': 'rsyncWebsiteMirror',
- 'https-dist': 'httpsDistMirror',
- 'http-dist': 'httpDistMirror',
- 'rsync-dist': 'rsyncDistMirror',
- 'ftp': 'ftpWebsiteMirror',
- 'onion': 'hiddenServiceMirror',
-}
-
-# Tor Project's website certificate
-# $ openssl s_client -showcerts -connect tpo:443 < /dev/null > tpo.pem
-CERT_TPO = '/path/to/gettor/tpo.pem'
-
-
-# Taken from get-tor-exits (BridgeDB)
-class FileWriter(protocol.Protocol):
- """Read a downloaded file incrementally and write to file."""
- def __init__(self, finished, file):
- """Create a FileWriter.
-
- .. warning:: We currently only handle the first 2MB of a file. Files
- over 2MB will be truncated prematurely. *note*: this should be
- enough for the mirrors file.
-
- :param finished: A :class:`~twisted.internet.defer.Deferred` which
- will fire when another portion of the download is complete.
- """
- self.finished = finished
- self.remaining = 1024 * 1024 * 2
- self.fh = file
-
- def dataReceived(self, bytes):
- """Write a portion of the download with ``bytes`` size to disk."""
- if self.remaining:
- display = bytes[:self.remaining]
- self.fh.write(display)
- self.fh.flush()
- self.remaining -= len(display)
-
- def connectionLost(self, reason):
- """Called when the download is complete."""
- logging.info('Finished receiving mirrors list: %s'
- % reason.getErrorMessage())
- self.finished.callback(None)
-
-
-# Based in tor2web.utils.ssl (Tor2web)
-class HTTPSVerifyingContextFactory(ssl.ClientContextFactory):
- def __init__(self, cn):
- self.cn = cn
- #
- # From https://docs.python.org/2/library/ssl.html#ssl-security
- #
- # "SSL versions 2 and 3 are considered insecure and are therefore
- # dangerous to use. If you want maximum compatibility between clients
- # and servers, it is recommended to use PROTOCOL_SSLv23 as the protocol
- # version and then disable SSLv2 and SSLv3 explicitly"
- #
- self.method = SSL.SSLv23_METHOD
-
- def getContext(self, hostname, port):
- """Get this connection's OpenSSL context.
-
- We disable SSLv2 and SSLv3. We also check the certificate received
- is the one we expect (using the "common name").
-
- """
- ctx = self._contextFactory(self.method)
- ctx.set_options(SSL.OP_NO_SSLv2)
- ctx.set_options(SSL.OP_NO_SSLv3)
- ctx.set_verify(
- SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
- self.verifyCN
- )
-
- return ctx
-
- def verifyCN(self, connection, x509, errno, depth, preverifyOK):
- # DEBUG: print "%s == %s ?" % (self.cn, x509.get_subject().commonName)
-
- # Somehow, if I don't set this to true, the verifyCN doesn't go
- # down in the chain, I don't know if this is OK
- verify = True
- if depth == 0:
- if self.cn == x509.get_subject().commonName:
- verify = True
- else:
- verify = False
- return verify
-
-
-# Based in get-tor-exits (BridgeDB)
-def handle(failure):
- """Handle a **failure**."""
- if failure.type == ConnectionRefusedError:
- logging.error("Couldn't download file; connection was refused.")
- elif failure.type == DNSLookupError:
- logging.error("Couldn't download file; domain resolution failed.")
- elif failure.type == TimeoutError:
- logging.error("Couldn't download file; connection timed out.")
- else:
- logging.error("Couldn't download file.")
-
- print "Couldn't download file. Check the log."
- os._exit(-1)
-
-
-# Taken from get-tor-exits (BridgeDB)
-def writeToFile(response, filename):
- """Write requested content to filename."""
- finished = defer.Deferred()
- response.deliverBody(FileWriter(finished, filename))
- return finished
-
-
-def is_json(my_json):
- """Check if json generated is valid."""
- try:
- json_object = json.loads(my_json)
- except ValueError, e:
- return False
- return True
-
-
-def add_tpo_link(url):
- """Add the download link for Tor Browser."""
- uri = 'projects/torbrowser.html.en#downloads'
- if url.endswith('/'):
- return "%s%s" % (url, uri)
- else:
- return "%s/%s" % (url, uri)
-
-
-def add_entry(mirrors, columns, elements):
- """Add entry to mirrors list."""
- entry = {}
- count = 0
- for e in elements:
- e = e.replace("\n", '')
- entry[columns[count]] = e
- count = count + 1
-
- mirrors.append(entry)
-
-
-def add_mirror(file, entry, proto):
- """Add mirror to mirrors list."""
- # if proto requested is http(s), we add link to download section
- if PROTOS[proto] == 'http' or PROTOS[proto] == 'https':
- uri = add_tpo_link(entry[proto])
- else:
- uri = entry[proto]
-
- file.write(
- "%s - by %s (%s)\n" % (
- uri,
- entry['orgName'],
- entry['subRegion'],
- )
- )
-
-
-def main():
- """Script to get the list of tpo's mirrors from tpo and adapt it to
- be used by GetTor.
-
- Usage: python2.7 get_mirrors.py [-h] [--proto protocol]
-
- By default, the protocol is 'https'. Possible values of protcol are:
-
- http, https, rsync, ftp, onion, http-dist, https-dist, rsync-dist.
-
- """
- parser = argparse.ArgumentParser(
- description="Utility to download tpo's mirrors and make it usable\
- by GetTor."
- )
-
- parser.add_argument(
- '-p', '--proto',
- default='https',
- help='Protocol filter. Possible values: http, https, rsync, ftp, onion\
- http-dist, https-dist, rsync-dist. Default to https.')
-
- args = parser.parse_args()
- p = args.proto
-
- gettor_path = '/path/to/gettor/'
- csv_path = os.path.join(gettor_path, 'tor-mirrors.csv')
- json_path = os.path.join(gettor_path, 'tor-mirrors')
- mirrors_list = os.path.join(gettor_path, 'mirrors-list.txt')
-
- # Load tpo certificate and extract common name, we'll later compare this
- # with the certificate sent by tpo to check we're really taltking to it
- try:
- data = open(CERT_TPO).read()
- x509 = crypto.load_certificate(crypto.FILETYPE_PEM, data)
- cn_tpo = x509.get_subject().commonName
- except Exception as e:
- logging.error("Error with certificate: %s" % str(e))
- return
-
- # While we wait the json of mirrors to be implemented in tpo, we need
- # to download the csv file and transform it to json
-
- # The code below is based in get-tor-exits script from BridgeDB and
- # the tor2web.utils.ssl module from Tor2web
- url = 'https://www.torproject.org/include/tor-mirrors.csv'
-
- try:
- fh = open(csv_path, 'w')
- except IOError as e:
- logging.error("Could not open %s" % csv_path)
- return
-
- logging.info("Requesting %s..." % url)
-
- # If certificate don't match an exception will be raised
- # this is my first experience with twisted, maybe I'll learn to handle
- # this better some time in the future...
- contextFactory = HTTPSVerifyingContextFactory(cn_tpo)
- agent = client.Agent(reactor, contextFactory)
- d = agent.request("GET", url)
- d.addCallback(writeToFile, fh)
- d.addErrback(handle)
- d.addCallbacks(log.msg, log.err)
-
- if not reactor.running:
- d.addCallback(lambda ignored: reactor.stop())
- reactor.run()
-
- logging.info("File downloaded!")
-
- # Now transform it to json -- I couldn't find out how to use a
- # two-character delimiter with the csv package, so I decided to handle
- # the csv data by hand. We are doing this until #16601 gets deployed.
- # https://trac.torproject.org/projects/tor/ticket/16601
-
- # Code below is based in update-mirrors-json.py script from tpo
-
- # These are the names of each column e.g. adminContact
- columns = []
- # List of mirrors to be built
- mirrors = []
-
- logging.info("Transforming csv data into json...")
- logging.info("Getting data from csv")
- try:
- with codecs.open(csv_path, "rb", "utf-8") as csvfile:
- for line in csvfile:
- elements = line.split(", ")
- # first entry have the names of the columns
- if not columns:
- columns = elements
- else:
- add_entry(mirrors, columns, elements)
- except IOError as e:
- logging.error("Couldn't read csv file: %s" % str(e))
- return
-
- logging.info("Creating json")
- if is_json(json.dumps(mirrors)):
- try:
- with codecs.open(json_path, "w", "utf-8") as jsonfile:
- # Make pretty json
- json.dump(
- mirrors,
- jsonfile,
- sort_keys=True,
- indent=4,
- separators=(',', ': '),
- encoding="utf-8",
- )
- except IOError as e:
- logging.error("Couldn't write json: %s" % str(e))
- return
- else:
- logging.error("Invalid json file")
- return
-
- # Now make the mirrors list to be used by GetTor
- logging.info("Reading json")
- try:
- mirrors_json = codecs.open(json_path, "rb", "utf-8")
- mirrors = json.load(mirrors_json)
- except IOError as e:
- logging.error("Couldn't open %s" % json_path)
- return
-
- logging.info("Creating new list with protocol: %s" % p)
- try:
- list = codecs.open(mirrors_list, "w", "utf-8")
- for entry in mirrors:
- if args.proto is not 'all':
- for e in entry:
- if e == PROTOS[p] and entry[PROTOS[p]]:
- add_mirror(list, entry, PROTOS[p])
- else:
- for k, v in PROTOS:
- if entry[v]:
- add_mirror(list, entry, v)
- logging.info("List created: %s" % mirrors_list)
- except IOError as e:
- logging.error("Couldn't open %s" % mirrors_list)
- return
-
-if __name__ == "__main__":
- logging_format = utils.get_logging_format()
- logging.basicConfig(
- filename='/path/to/gettor/log/get-mirrors.log',
- format=logging_format,
- datefmt="%Y-%m-%d %H:%M:%S",
- level=logging.INFO
- )
- logging.info("Started")
- main()
- logging.info("Finished")
diff --git a/dev_scripts/process_chat.py b/dev_scripts/process_chat.py
deleted file mode 100644
index 6ad7b59..0000000
--- a/dev_scripts/process_chat.py
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-import sys
-import logging
-
-import gettor.xmpp
-
-def main():
- try:
- bot = gettor.xmpp.XMPP()
- bot.start_bot()
- except gettor.xmpp.ConfigError as e:
- print "Configuration error: %s" % str(e)
- except gettor.xmpp.InternalError as e:
- print "Core module not working: %s" % str(e)
- except Exception as e:
- # in case something unexpected happens
- print "Unexpected error: %s" % str(e)
-
-if __name__ == '__main__':
- main()
diff --git a/dev_scripts/process_email.py b/dev_scripts/process_email.py
deleted file mode 100644
index 504b5dd..0000000
--- a/dev_scripts/process_email.py
+++ /dev/null
@@ -1,41 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-import sys
-import logging
-
-import gettor.smtp
-
-def main():
- logging_level = 'INFO'
- logging_file = '/path/to/gettor/log/process_email.log'
- logging_format = '[%(levelname)s] %(asctime)s - %(message)s'
- date_format = "%Y-%m-%d" # %H:%M:%S
-
- logging.basicConfig(
- format=logging_format,
- datefmt=date_format,
- filename = logging_file,
- level = logging_level
- )
-
- logging.debug("New email received")
- logging.debug("Creating new SMTP object")
-
- try:
- service = gettor.smtp.SMTP('/path/to/gettor/smtp.cfg')
- incoming = sys.stdin.read()
- service.process_email(incoming)
- logging.debug("Email processed sucessfully")
- except gettor.smtp.ConfigError as e:
- logging.error("Configuration error: %s" % str(e))
- except gettor.smtp.SendEmailError as e:
- logging.error("SMTP not working: %s" % str(e))
- except gettor.smtp.InternalError as e:
- logging.error("Core module not working: %s" % str(e))
- except Exception as e:
- # in case something unexpected happens
- logging.critical("Unexpected error: %s" % str(e))
-
-if __name__ == '__main__':
- main()
diff --git a/dev_scripts/process_http.py b/dev_scripts/process_http.py
deleted file mode 100644
index ec20d5f..0000000
--- a/dev_scripts/process_http.py
+++ /dev/null
@@ -1,14 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-import gettor.http2
-
-
-def main():
- api = gettor.http2.HTTP('http.cfg')
- api.load_data()
- # api.run()
- api.build()
-
-if __name__ == '__main__':
- main()
diff --git a/dev_scripts/process_tweets.py b/dev_scripts/process_tweets.py
deleted file mode 100644
index 59c2f48..0000000
--- a/dev_scripts/process_tweets.py
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-import sys
-import logging
-
-import gettor.twitter
-
-def main():
- logging_level = 'DEBUG'
- logging_file = '/home/ilv/Proyectos/tor/gettor/log/process_tweets.log'
- logging_format = '[%(levelname)s] %(asctime)s - %(message)s'
- date_format = "%Y-%m-%d" # %H:%M:%S
-
- logging.basicConfig(
- format=logging_format,
- datefmt=date_format,
- filename = logging_file,
- level = logging_level
- )
-
- logging.debug("Starting bot")
- try:
- bot = gettor.twitter.TwitterBot()
- bot.start()
- except gettor.twitter.ConfigError as e:
- logging.error("Configuration error: %s" % str(e))
- except gettor.twitter.InternalError as e:
- logging.error("Core module not working: %s" % str(e))
- except Exception as e:
- # in case something unexpected happens
- logging.error("Unexpected error: %s" % str(e))
-
-if __name__ == '__main__':
- main()
diff --git a/dev_scripts/report.py b/dev_scripts/report.py
deleted file mode 100644
index b75d7c8..0000000
--- a/dev_scripts/report.py
+++ /dev/null
@@ -1,129 +0,0 @@
-#!/usr/bin/env python
-
-# This file is part of GetTor, a Tor Browser distribution system.
-
-# :authors: Sukhbir Singh <sukhbir at torproject.org>
-# see also AUTHORS file
-#
-# :copyright: (c) 2008-2015, The Tor Project, Inc.
-
-# To read the log files and populate the db, run the script with `--logs' once
-# every day and then clear the log files. To generate the report, run it with
-# `--report', once (or twice?) a month.
-
-# TODO: Logging? argparse?
-
-import os
-import sys
-import ConfigParser
-from datetime import datetime as dt
-
-import sqlite3
-
-CORE_CFG = "core.cfg"
-
-CHANNELS = ["smtp", "xmpp", "twitter"]
-LOG_FILES = [channel + ".log" for channel in CHANNELS]
-
-DB = "gettor2.db"
-
-OUTPUT = "report.log"
-
-REPORT = """@ GetTor Report for {}
-
-We received a total of {} requests in {}, with a peak of {} requests on {}.
-
-[*] Request
-{}
-
-[*] OS
-{}
-
-[*] Language
-{}
-
-[*] Channel
-{}
-
-"""
-
-
-class Report(object):
- def __init__(self):
- self.config = ConfigParser.ConfigParser()
- self.config.read(CORE_CFG)
-
- self.log_dir = self.config.get("log", "dir")
- # This is defined in the sample log file.
- self.log_format = ["date", "request", "os", "locale"]
-
- self.conn = sqlite3.connect(DB)
-
- def db_write(self):
- self.cursor = self.conn.cursor()
- for each in self.logs:
- self.cursor.execute("""INSERT INTO requests
- (date, request, os, locale, channel) VALUES
- (:date, :request, :os, :locale, :channel)""",
- each)
- self.conn.commit()
- self.conn.close()
-
- def get_logs(self):
- self.logs = []
- for each in LOG_FILES:
- with open(os.path.join(self.log_dir, each)) as f:
- for line in f:
- if line.startswith("[INFO] "):
- # The "7" here is for the "[INFO] " text.
- logs = [log.strip() for log in line[7:].split(";")]
- log_data = {key: value for key, value in
- zip(self.log_format, logs)}
- # We also need the channel.
- log_data["channel"] = each.split(".")[0]
- self.logs.append(log_data)
- self.db_write()
- self.conn.close()
-
- def generate_report(self):
- self.cursor = self.conn.cursor()
- self.requests = []
-
- self.cursor.execute("SELECT COUNT(*) FROM requests")
- self.requests.append(self.cursor.fetchone())
-
- self.cursor.execute("""SELECT date, COUNT(*) FROM requests
-GROUP BY date
-ORDER BY COUNT(date)
-DESC""")
- self.requests.append(self.cursor.fetchone())
-
- self.columns = ["request", "os", "locale", "channel"]
- for each in self.columns:
- self.cursor.execute("""SELECT {0}, COUNT(*) FROM requests
-GROUP BY {0}
-ORDER BY COUNT({0})
-DESC""".format(each))
- result = "\n".join(["{0:>16}: {1}".format(each[0], each[1])
- for each in self.cursor.fetchall()
- if not each[0] == "none"])
- self.requests.append(result)
-
- with open(OUTPUT, "a") as f:
- f.write(REPORT.format(dt.now().strftime("%B %Y"),
- self.requests[0][0],
- dt.now().strftime("%B"),
- self.requests[1][1],
- dt.strptime(self.requests[1][0], "%Y-%m-%d")
- .strftime("%B %-d"),
- *self.requests[2:]))
- self.conn.close()
-
-if __name__ == "__main__":
- try:
- if sys.argv[1] == "--logs":
- Report().get_logs()
- if sys.argv[1] == "--report":
- Report().generate_report()
- except IndexError:
- sys.exit("You need either `--logs' or `--report.'")
diff --git a/dev_scripts/sample-htaccess-api-mirror b/dev_scripts/sample-htaccess-api-mirror
deleted file mode 100644
index 41594db..0000000
--- a/dev_scripts/sample-htaccess-api-mirror
+++ /dev/null
@@ -1,21 +0,0 @@
-Options +FollowSymLinks
-RewriteEngine On
-
-# Operating System and Locale filters for providers
-RewriteCond %{QUERY_STRING} os=([a-z]+)&lc=([a-z]+)
-RewriteRule ^api/providers/([a-z\-]+)$ api-content/providers-content/$1-content/%1-content/%2.json
-RewriteCond %{QUERY_STRING} os=([a-z]+)
-RewriteRule ^api/providers/([a-z\-]+)$ api-content/providers-content/$1-content/%1
-
-# Operating System and Locale filters for TPO downloads
-RewriteCond %{QUERY_STRING} os=([a-z]+)&lc=([a-zA-Z\-]+)
-RewriteRule ^api/latest/([a-z]+)$ api-content/latest-content/$1-content/%1-content/%2.json
-RewriteCond %{QUERY_STRING} os=([a-z]+)
-RewriteRule ^api/latest/([a-z]+)$ api-content/latest-content/$1-content/%1
-
-# General rules
-RewriteRule ^api/providers/([a-z\-]+)$ api-content/providers-content/$1 [NC]
-RewriteRule ^api/latest/([a-z]+)$ api-content/latest-content/$1 [NC]
-RewriteRule ^api/([a-z]+)$ api-content/$1 [NC]
-
-ErrorDocument 404 404-file
diff --git a/dev_scripts/stats.py b/dev_scripts/stats.py
deleted file mode 100644
index 7fca2af..0000000
--- a/dev_scripts/stats.py
+++ /dev/null
@@ -1,123 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-#
-# This file is part of GetTor, a Tor Browser distribution system.
-#
-# :authors: Israel Leiva <ilv at riseup.net>
-# see also AUTHORS file
-#
-# :copyright: (c) 2008-2014, The Tor Project, Inc.
-# (c) 2014, Israel Leiva
-#
-# :license: This is Free Software. See LICENSE for license information.
-
-
-import sqlite3
-import argparse
-
-
-def main():
- """Script for showing stats.
-
- See argparse usage for more details.
-
- """
- parser = argparse.ArgumentParser(description='Utility for GetTor stats')
- parser.add_argument('database', metavar='database.db', type=str,
- help='the database file')
- parser.add_argument('-s', '--service', default=None,
- help='filter by service')
- parser.add_argument('-t', '--type', default=None,
- help='filter by type of request')
- parser.add_argument('-o', '--os', default=None,
- help='filter by OS')
- parser.add_argument('-l', '--lc', default=None,
- help='filter by locale')
- parser.add_argument('-p', '--pt', default=None,
- help='filter by PT requests')
- parser.add_argument('-y', '--year', default=None,
- help='filter by year')
- parser.add_argument('-m', '--month', default=None,
- help='filter by month')
- parser.add_argument('-d', '--day', default=None,
- help='filter by day')
- parser.add_argument('-u', '--status', default=None,
- help='filter by status of the request')
-
- args = parser.parse_args()
- query = 'SELECT * FROM requests'
- has_where = False
-
- # we build the query piece by piece
- if args.service:
- query = "%s %s" % (query, "WHERE service = '%s'" % args.service)
- has_where = True
- if args.type:
- if has_where:
- query = "%s %s" % (query, "AND type = '%s'" % args.type)
- else:
- query = "%s %s" % (query, "WHERE type = '%s'" % args.type)
- has_where = True
- if args.os:
- if has_where:
- query = "%s %s" % (query, "AND os = '%s'" % args.os)
- else:
- query = "%s %s" % (query, "WHERE os = '%s'" % args.os)
- has_where = True
- if args.lc:
- if has_where:
- query = "%s %s" % (query, "AND lc = '%s'" % args.lc)
- else:
- query = "%s %s" % (query, "WHERE lc = '%s'" % args.lc)
- has_where = True
- if args.pt:
- if has_where:
- query = "%s %s" % (query, "AND pt = %s" % args.pt)
- else:
- query = "%s %s" % (query, "WHERE pt = %s" % args.pt)
- has_where = True
- if args.year:
- if has_where:
- query = "%s %s" % (query, "AND year = %s" % args.year)
- else:
- query = "%s %s" % (query, "WHERE year = %s" % args.year)
- has_where = True
- if args.month:
- if has_where:
- query = "%s %s" % (query, "AND month = %s" % args.month)
- else:
- query = "%s %s" % (query, "WHERE month = %s" % args.month)
- has_where = True
- if args.day:
- if has_where:
- query = "%s %s" % (query, "AND day = %s" % args.day)
- else:
- query = "%s %s" % (query, "WHERE day = %s" % args.day)
- has_where = True
- if args.status:
- if has_where:
- query = "%s %s" % (query, "AND status = '%s'" % args.status)
- else:
- query = "%s %s" % (query, "WHERE status = '%s'" % args.status)
- has_where = True
-
- con = sqlite3.connect(args.database)
-
- with con:
- cur = con.cursor()
- cur.execute(query)
- rows = cur.fetchall()
- # show it nice
- print "\nNumber of results: %s\n" % len(rows)
- cns = [cn[0] for cn in cur.description]
- print "%-10s %-10s %-10s %-10s %-10s %-10s %-10s %-10s"\
- " %-15s %s"\ % (cns[0], cns[1], cns[2], cns[3], cns[4], cns[5],
- cns[6], cns[7], cns[8], cns[9])
-
- for row in rows:
- print "%-10s %-10s %-10s %-10s %-10s %-10s %-10s %-10s"\
- " %-15s %s" % (row[0], row[1], row[2], row[3], row[4],
- row[5], row[6], row[7], row[8], row[9])
-
-if __name__ == "__main__":
- main()
diff --git a/gettor/main.py b/gettor/main.py
index 9cd18b7..d59f66c 100644
--- a/gettor/main.py
+++ b/gettor/main.py
@@ -25,7 +25,9 @@ def run(gettor, app):
"""
This is GetTor's main entry point and main runtime loop.
"""
- settings = options.parse_settings()
+ config = "/home/gettor/gettor/gettor.conf.json"
+
+ settings = options.parse_settings("en", config)
sendmail = Sendmail(settings)
twitterdm = Twitterdm(settings)
diff --git a/gettor/services/email/sendmail.py b/gettor/services/email/sendmail.py
index 545fbb8..4644cb5 100644
--- a/gettor/services/email/sendmail.py
+++ b/gettor/services/email/sendmail.py
@@ -132,10 +132,14 @@ class Sendmail(object):
)
)
+ body_msg = strings._("help_body_intro")
+ body_msg += strings._("help_body_paragraph")
+ body_msg += strings._("help_body_support")
+
yield self.sendmail(
email_addr=id,
subject=strings._("help_subject"),
- body=strings._("help_body")
+ body=body_msg
)
yield self.conn.update_stats(
@@ -197,7 +201,14 @@ class Sendmail(object):
else:
link_msg = link_str
- body_msg = strings._("links_body").format(platform, link_msg, file)
+ body_msg = strings._("links_body_platform").format(platform)
+ body_msg += trings._("links_body_links").format(link_msg)
+ body_msg += trings._("links_body_archive")
+ body_msg += trings._("links_body_internet_archive")
+ body_msg += trings._("links_body_google_drive")
+ body_msg += trings._("links_body_internet_archive").format(file)
+ body_msg += trings._("links_body_ending")
+
subject_msg = strings._("links_subject")
hid = hashlib.sha256(id.encode('utf-8'))
diff --git a/gettor/services/twitter/twitterdm.py b/gettor/services/twitter/twitterdm.py
index d78604a..629fa5f 100644
--- a/gettor/services/twitter/twitterdm.py
+++ b/gettor/services/twitter/twitterdm.py
@@ -144,9 +144,13 @@ class Twitterdm(object):
)
)
+ body_msg = strings._("help_body_intro")
+ body_msg += strings._("help_body_paragraph")
+ body_msg += strings._("help_body_support")
+
yield self.twitterdm(
twitter_id=twitter_id,
- message=strings._("help_body")
+ message=body_msg
)
yield self.conn.update_stats(
@@ -210,7 +214,13 @@ class Twitterdm(object):
else:
link_msg = link_str
- body_msg = strings._("links_body").format(platform, link_msg, file)
+ body_msg = strings._("links_body_platform").format(platform)
+ body_msg += trings._("links_body_links").format(link_msg)
+ body_msg += trings._("links_body_archive")
+ body_msg += trings._("links_body_internet_archive")
+ body_msg += trings._("links_body_google_drive")
+ body_msg += trings._("links_body_internet_archive").format(file)
+ body_msg += trings._("links_body_ending")
hid = hashlib.sha256(twitter_id.encode('utf-8'))
log.info(
diff --git a/gettor/utils/options.py b/gettor/utils/options.py
index d313876..8504f42 100644
--- a/gettor/utils/options.py
+++ b/gettor/utils/options.py
@@ -24,12 +24,11 @@ def load_settings(config):
settings.load()
return settings
-def parse_settings():
+def parse_settings(locale, config):
"""
Parse settings and loads strings in a given locale
This function needs to be rewritten considering passing a locale and
returing translated strings
"""
- strings.load_strings("en")
- config = "/home/gettor/gettor/gettor.conf.json"
+ strings.load_strings(locale)
return load_settings(config)
diff --git a/share/locale/en.json b/share/locale/en.json
index 8ed6722..be7dd33 100644
--- a/share/locale/en.json
+++ b/share/locale/en.json
@@ -1,7 +1,15 @@
{
- "links_body": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following archives:\n\n- Internet Archive: https://archive.org/details/@gettor\n\n- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU\n\nDownload the file: {}\n\n \n--\nGetTor",
+ "links_body_platform": "You requested Tor Browser for {}.\n\n",
+ "links_body_links": "You will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\n",
+ "links_body_archive": "Should you have issues with any of the links above you can access the following archives:\n\n",
+ "links_body_internet_archive": "- Internet Archive: https://archive.org/details/@gettor\n\n",
+ "links_body_google_drive": "- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU\n\n",
+ "links_body_internet_archive": "Download the file: {}\n\n",
+ "links_body_ending": "\n--\nGetTor",
"links_subject": "[GetTor] Links for your request",
- "help_body": "This is how you can request a tor browser bundle link.\n\n Send an email to: gettor at torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\nWe only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n",
+ "help_body_intro": "This is how you can request a tor browser bundle link.\n\n",
+ "help_body_paragraph": "Send an email to: gettor at torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\n",
+ "help_body_support": "We only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n",
"help_subject": "[GetTor] Help Email",
"help_debug": "Log application errors to stdout",
"help_config": "Custom config file location (optional)",
@@ -9,8 +17,5 @@
"smtp_mirrors_subject": "[GetTor] Mirrors",
"smtp_help_subject": "[GetTor] Help",
"smtp_unsupported_locale_subject": "[GetTor] Unsupported locale",
- "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported.",
- "smtp_vlinks_msg": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU\n\n Download the file: {}\n\n \n--\nGetTor",
- "smtp_mirrors_msg": "Hi! this is the GetTor robot.\n\nThank you for your request. Attached to this email you will find\nan updated list of mirrors of Tor Project's website.",
- "smtp_help_msg": "Hi! This is the GetTor robot. I am here to help you download the\nlatest version of Tor Browser.\n\nPlease reply to this message with one of the options below:\n\nwindows\nlinux\nosx\nmirrors\n\nI will then send you the download instructions.\n\nIf you are unsure, just send a blank reply to this message."
+ "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported."
}
diff --git a/share/locale/es.json b/share/locale/es.json
index 4d18f61..be7dd33 100644
--- a/share/locale/es.json
+++ b/share/locale/es.json
@@ -1,16 +1,21 @@
{
- "links_body": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following archives:\n\n- Internet Archive: https://archive.org/details/@gettor\n\n- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU\n\nDownload the file: {}\n\n \n--\nGetTor",
+ "links_body_platform": "You requested Tor Browser for {}.\n\n",
+ "links_body_links": "You will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\n",
+ "links_body_archive": "Should you have issues with any of the links above you can access the following archives:\n\n",
+ "links_body_internet_archive": "- Internet Archive: https://archive.org/details/@gettor\n\n",
+ "links_body_google_drive": "- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU\n\n",
+ "links_body_internet_archive": "Download the file: {}\n\n",
+ "links_body_ending": "\n--\nGetTor",
"links_subject": "[GetTor] Links for your request",
- "help_body": "This is how you can request a tor browser bundle link.\n\n Send an email to: gettor at torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\nWe only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n",
+ "help_body_intro": "This is how you can request a tor browser bundle link.\n\n",
+ "help_body_paragraph": "Send an email to: gettor at torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\n",
+ "help_body_support": "We only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n",
"help_subject": "[GetTor] Help Email",
"help_debug": "Log application errors to stdout",
"help_config": "Custom config file location (optional)",
"smtp_links_subject": "[GetTor] Links for your request",
"smtp_mirrors_subject": "[GetTor] Mirrors",
- "smtp_help_subject": "[GetTor] Ayuda",
+ "smtp_help_subject": "[GetTor] Help",
"smtp_unsupported_locale_subject": "[GetTor] Unsupported locale",
- "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported.",
- "smtp_vlinks_msg": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU\n\n Download the file: {}\n\n \n--\nGetTor",
- "smtp_mirrors_msg": "Hi! this is the GetTor robot.\n\nThank you for your request. Attached to this email you will find\nan updated list of mirrors of Tor Project's website.",
- "smtp_help_msg": "Hi! This is the GetTor robot. I am here to help you download the\nlatest version of Tor Browser.\n\nPlease reply to this message with one of the options below:\n\nwindows\nlinux\nosx\nmirrors\n\nI will then send you the download instructions.\n\nIf you are unsure, just send a blank reply to this message."
+ "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported."
}
diff --git a/share/locale/pt.json b/share/locale/pt.json
index 8ed6722..be7dd33 100644
--- a/share/locale/pt.json
+++ b/share/locale/pt.json
@@ -1,7 +1,15 @@
{
- "links_body": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following archives:\n\n- Internet Archive: https://archive.org/details/@gettor\n\n- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU\n\nDownload the file: {}\n\n \n--\nGetTor",
+ "links_body_platform": "You requested Tor Browser for {}.\n\n",
+ "links_body_links": "You will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\n",
+ "links_body_archive": "Should you have issues with any of the links above you can access the following archives:\n\n",
+ "links_body_internet_archive": "- Internet Archive: https://archive.org/details/@gettor\n\n",
+ "links_body_google_drive": "- Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU\n\n",
+ "links_body_internet_archive": "Download the file: {}\n\n",
+ "links_body_ending": "\n--\nGetTor",
"links_subject": "[GetTor] Links for your request",
- "help_body": "This is how you can request a tor browser bundle link.\n\n Send an email to: gettor at torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\nWe only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n",
+ "help_body_intro": "This is how you can request a tor browser bundle link.\n\n",
+ "help_body_paragraph": "Send an email to: gettor at torproject.org\n\nIn the body of the email only write: <operating system> <language>.\n\n",
+ "help_body_support": "We only support windows, osx and linux as operating systems.\n\nAt the moment please only use en as language.\n\n",
"help_subject": "[GetTor] Help Email",
"help_debug": "Log application errors to stdout",
"help_config": "Custom config file location (optional)",
@@ -9,8 +17,5 @@
"smtp_mirrors_subject": "[GetTor] Mirrors",
"smtp_help_subject": "[GetTor] Help",
"smtp_unsupported_locale_subject": "[GetTor] Unsupported locale",
- "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported.",
- "smtp_vlinks_msg": "You requested Tor Browser for {}.\n\nYou will need only one of the links below to download the bundle. If a link does not work for you, try the next one.\n\n{}\n\nShould you have issues with any of the links above you can access the following Google Drive folder: https://drive.google.com/open?id=13CADQTsCwrGsIID09YQbNz2DfRMUoxUU\n\n Download the file: {}\n\n \n--\nGetTor",
- "smtp_mirrors_msg": "Hi! this is the GetTor robot.\n\nThank you for your request. Attached to this email you will find\nan updated list of mirrors of Tor Project's website.",
- "smtp_help_msg": "Hi! This is the GetTor robot. I am here to help you download the\nlatest version of Tor Browser.\n\nPlease reply to this message with one of the options below:\n\nwindows\nlinux\nosx\nmirrors\n\nI will then send you the download instructions.\n\nIf you are unsure, just send a blank reply to this message."
+ "smtp_unsupported_locale_msg": "The locale you requested '{}' is not supported."
}
diff --git a/tests/test_email_service.py b/tests/test_email_service.py
index 4c8ff10..8e60f7a 100644
--- a/tests/test_email_service.py
+++ b/tests/test_email_service.py
@@ -13,7 +13,7 @@ class EmailServiceTests(unittest.TestCase):
# Fail any tests which take longer than 15 seconds.
timeout = 15
def setUp(self):
- self.settings = conftests.options.parse_settings()
+ self.settings = conftests.options.parse_settings("en","./gettor.conf.json")
self.sm_client = conftests.sendmail.Sendmail(self.settings)
self.locales = conftests.strings.get_locales()
diff --git a/tests/test_locales.py b/tests/test_locales.py
index bea7e9f..d4e52c5 100644
--- a/tests/test_locales.py
+++ b/tests/test_locales.py
@@ -11,7 +11,7 @@ class LocalesTests(unittest.TestCase):
# Fail any tests which take longer than 15 seconds.
timeout = 15
def setUp(self):
- self.settings = conftests.options.parse_settings()
+ self.settings = conftests.options.parse_settings("en","./gettor.conf.json")
self.locales = conftests.strings.get_locales()
diff --git a/tests/test_twitter.py b/tests/test_twitter.py
index fe155cc..7458cfc 100644
--- a/tests/test_twitter.py
+++ b/tests/test_twitter.py
@@ -10,7 +10,7 @@ class TwitterTests(unittest.TestCase):
# Fail any tests which take longer than 15 seconds.
timeout = 15
def setUp(self):
- self.settings = conftests.options.parse_settings()
+ self.settings = conftests.options.parse_settings("en","./gettor.conf.json")
self.tw_client = conftests.twitter.Twitter(self.settings)
More information about the tor-commits
mailing list