tor-commits May 2015

tor-commits@lists.torproject.org

22 participants
990 discussions

[stem/master] Merge version 1.4.1 hotfix
by atagar＠torproject.org 18 May '15

18 May '15

commit 9c0268a007c48b7ac4ad4c14950c3d976340f67c Merge: 1ee755f 3884d37 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon May 18 12:24:10 2015 -0700 Merge version 1.4.1 hotfix docs/change_log.rst | 4 +++ docs/download.rst | 4 +-- docs/index.rst | 2 +- stem/__init__.py | 2 +- stem/descriptor/__init__.py | 6 ++++ stem/descriptor/router_status_entry.py | 18 ++++++++--- test/unit/descriptor/server_descriptor.py | 48 +++++++++++++++++++---------- 7 files changed, 60 insertions(+), 24 deletions(-) diff --cc stem/__init__.py index 7229453,9ff1263..ead8224 --- a/stem/__init__.py +++ b/stem/__init__.py @@@ -441,7 -441,7 +441,7 @@@ Library for working with the tor proces ================= =========== """ - __version__ = '1.4.0-dev' -__version__ = '1.4.1' ++__version__ = '1.4.1-dev' __author__ = 'Damian Johnson' __contact__ = 'atagar(a)torproject.org' __url__ = 'https://stem.torproject.org/'

1 0

[torspec/master] Add NETWORK_LIVENESS event type and network-liveness GETINFO key
by nickm＠torproject.org 18 May '15

18 May '15

commit 44aac630f69fc1bc1ab8cd33645f9f461de116a4 Author: Andrea Shepard <andrea(a)persephoneslair.org> Date: Sun May 17 13:46:18 2015 +0000 Add NETWORK_LIVENESS event type and network-liveness GETINFO key --- control-spec.txt | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/control-spec.txt b/control-spec.txt index cf2608c..0cc46b9 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -854,6 +854,10 @@ HSAddress [New in Tor 0.2.7.1-alpha.] + "network-liveness" + The string "up" or "down", indicating whether we currently believe the + network is reachable. + Examples: C: GETINFO version desc/name/moria1 S: 250+desc/name/moria= @@ -2782,6 +2786,16 @@ [HS_DESC_CONTENT was added in Tor 0.2.7.1-alpha] +4.1.27. Network liveness has changed + + Syntax: + "650" SP "NETWORK_LIVENESS" SP Status ... CRLF + Status = "UP" | "DOWN" + + The Status values are: + "UP" -- The network now seems to be reachable. + "DOWN" -- The network now seems to be unreachable. + 5. Implementation notes 5.1. Authentication

1 0

[torspec/master] Clarify that we can expand NETWORK_LIVENESS in the future
by nickm＠torproject.org 18 May '15

18 May '15

commit 7ec5d394de1aa1d43e58141de9f6d3cb5822b4ca Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon May 18 14:47:54 2015 -0400 Clarify that we can expand NETWORK_LIVENESS in the future --- control-spec.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/control-spec.txt b/control-spec.txt index 0cc46b9..c9f871b 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -2796,6 +2796,9 @@ "UP" -- The network now seems to be reachable. "DOWN" -- The network now seems to be unreachable. + Controllers MUST ignore unrecognized Status values, and MUST ignore + unrecognized material at the end of the event line. + 5. Implementation notes 5.1. Authentication

1 0

[nyx/master] Drop advise that we run as tor's user
by atagar＠torproject.org 18 May '15

18 May '15

commit 0a58a28b3456aa761c2c7693fa72ba3b87b5d98a Author: Damian Johnson <atagar(a)torproject.org> Date: Sun May 17 14:07:07 2015 -0700 Drop advise that we run as tor's user This is something Roger has long advised against... https://trac.torproject.org/projects/tor/ticket/10702 I don't recall what in particular we needed to run as the tor user for, so plan is to drop this advise. If anything breaks we'll cross that bridge when we come to it. --- nyx/config/strings.cfg | 2 +- nyx/starter.py | 7 ++----- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/nyx/config/strings.cfg b/nyx/config/strings.cfg index 7660bfb..9cd217f 100644 --- a/nyx/config/strings.cfg +++ b/nyx/config/strings.cfg @@ -26,7 +26,7 @@ msg.panel.header.fd_used_at_sixty_percent Tor's file descriptor usage is at {per msg.panel.header.fd_used_at_ninety_percent Tor's file descriptor usage is at {percentage}%. If you run out Tor will be unable to continue functioning. msg.panel.log.read_from_log_file Read {count} entries from tor's log file: {path} (read limit: {read_limit}, runtime: {runtime}) -msg.setup.nyx_is_running_as_root Nyx is currently running with root permissions. This isn't a good idea, nor should it be necessary. Try starting nyx with "sudo -u {tor_user} nyx" instead. +msg.setup.nyx_is_running_as_root Nyx is currently running with root permissions. This isn't a good idea, nor should it be necessary. msg.setup.chroot_doesnt_exist The chroot path set in your config ({path}) doesn't exist. msg.setup.set_freebsd_chroot Adjusting paths to account for Tor running in a FreeBSD jail at: {path} msg.setup.tor_is_running_as_root Tor is currently running with root permissions. This isn't a good idea, nor should it be necessary. See the 'User UID' option on Tor's man page for an easy method of reducing its permissions after startup. diff --git a/nyx/starter.py b/nyx/starter.py index 3b56742..87937ed 100644 --- a/nyx/starter.py +++ b/nyx/starter.py @@ -174,13 +174,10 @@ def _warn_if_root(controller): Give a notice if tor or nyx are running with root. """ - tor_user = controller.get_user(None) - - if tor_user == 'root': + if controller.get_user(None) == 'root': log.notice('setup.tor_is_running_as_root') elif os.getuid() == 0: - tor_user = tor_user if tor_user else '<tor user>' - log.notice('setup.nyx_is_running_as_root', tor_user = tor_user) + log.notice('setup.nyx_is_running_as_root') def _warn_if_unable_to_get_pid(controller):

1 0

[tor/master] Implement EVENT_NETWORK_LIVENESS
by nickm＠torproject.org 18 May '15

18 May '15

commit dce9e915c77ccf42e74a5cccca309e8085738d75 Author: Andrea Shepard <andrea(a)torproject.org> Date: Wed May 6 12:37:13 2015 +0000 Implement EVENT_NETWORK_LIVENESS --- src/or/circuitstats.c | 6 ++++++ src/or/control.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ src/or/control.h | 4 +++- 3 files changed, 56 insertions(+), 1 deletion(-) diff --git a/src/or/circuitstats.c b/src/or/circuitstats.c index 7b3ad56..3ced5af 100644 --- a/src/or/circuitstats.c +++ b/src/or/circuitstats.c @@ -1232,6 +1232,9 @@ circuit_build_times_network_is_live(circuit_build_times_t *cbt) } cbt->liveness.network_last_live = now; cbt->liveness.nonlive_timeouts = 0; + + /* Tell control.c */ + control_event_network_liveness_update(1); } /** @@ -1316,6 +1319,9 @@ circuit_build_times_network_close(circuit_build_times_t *cbt, "Tor has not observed any network activity for the past %d " "seconds. Disabling circuit build timeout recording.", (int)(now - cbt->liveness.network_last_live)); + + /* Tell control.c */ + control_event_network_liveness_update(0); } else { log_info(LD_CIRC, "Got non-live timeout. Current count is: %d", diff --git a/src/or/control.c b/src/or/control.c index 950e989..a64822e 100644 --- a/src/or/control.c +++ b/src/or/control.c @@ -959,6 +959,7 @@ static const struct control_event_t control_event_table[] = { { EVENT_TRANSPORT_LAUNCHED, "TRANSPORT_LAUNCHED" }, { EVENT_HS_DESC, "HS_DESC" }, { EVENT_HS_DESC_CONTENT, "HS_DESC_CONTENT" }, + { EVENT_NETWORK_LIVENESS, "NETWORK_LIVENESS" }, { 0, NULL }, }; @@ -4992,6 +4993,52 @@ control_event_or_authdir_new_descriptor(const char *action, return 0; } +/** Cached liveness for network liveness events and GETINFO + */ + +static int network_is_live = 0; + +static int +get_cached_network_liveness(void) +{ + return network_is_live; +} + +static void +set_cached_network_liveness(int liveness) +{ + network_is_live = liveness; +} + +/** The network liveness has changed; this is called from circuitstats.c + * whenever we receive a cell, or when timeout expires and we assume the + * network is down. */ +int +control_event_network_liveness_update(int liveness) +{ + if (liveness > 0) { + if (get_cached_network_liveness() <= 0) { + /* Update cached liveness */ + set_cached_network_liveness(1); + log_debug(LD_CONTROL, "Sending NETWORK_LIVENESS UP"); + send_control_event_string(EVENT_NETWORK_LIVENESS, ALL_FORMATS, + "650 NETWORK_LIVENESS UP\r\n"); + } + /* else was already live, no-op */ + } else { + if (get_cached_network_liveness() > 0) { + /* Update cached liveness */ + set_cached_network_liveness(0); + log_debug(LD_CONTROL, "Sending NETWORK_LIVENESS DOWN"); + send_control_event_string(EVENT_NETWORK_LIVENESS, ALL_FORMATS, + "650 NETWORK_LIVENESS DOWN\r\n"); + } + /* else was already dead, no-op */ + } + + return 0; +} + /** Helper function for NS-style events. Constructs and sends an event * of type event with string event_string out of the set of * networkstatuses statuses. Currently it is used for NS events diff --git a/src/or/control.h b/src/or/control.h index cb0468f..43a95b3 100644 --- a/src/or/control.h +++ b/src/or/control.h @@ -67,6 +67,7 @@ int control_event_or_authdir_new_descriptor(const char *action, size_t desclen, const char *msg); int control_event_my_descriptor_changed(void); +int control_event_network_liveness_update(int liveness); int control_event_networkstatus_changed(smartlist_t *statuses); int control_event_newconsensus(const networkstatus_t *consensus); @@ -166,7 +167,8 @@ void control_free_all(void); #define EVENT_TRANSPORT_LAUNCHED 0x0020 #define EVENT_HS_DESC 0x0021 #define EVENT_HS_DESC_CONTENT 0x0022 -#define EVENT_MAX_ 0x0022 +#define EVENT_NETWORK_LIVENESS 0x0023 +#define EVENT_MAX_ 0x0023 /* sizeof(control_connection_t.event_mask) in bits, currently a uint64_t */ #define EVENT_CAPACITY_ 0x0040

1 0

[tor/master] Merge remote-tracking branch 'andrea/ticket15358_squashed_2'
by nickm＠torproject.org 18 May '15

18 May '15

commit 2308f917f99384640c5b0d1ddc9f2a3dffc68e5a Merge: 0d3b3a4 79f7721 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon May 18 14:44:28 2015 -0400 Merge remote-tracking branch 'andrea/ticket15358_squashed_2' changes/ticket15358 | 3 +++ src/or/circuitstats.c | 6 +++++ src/or/control.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ src/or/control.h | 4 ++- 4 files changed, 82 insertions(+), 1 deletion(-)

1 0

[tor/master] Add GETINFO network-liveness to control protocol
by nickm＠torproject.org 18 May '15

18 May '15

commit 4cbc9c53131573a8edcbe1748b5ed220959db0b6 Author: Andrea Shepard <andrea(a)torproject.org> Date: Fri May 15 19:44:13 2015 +0000 Add GETINFO network-liveness to control protocol --- src/or/control.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/or/control.c b/src/or/control.c index a64822e..c76ade0 100644 --- a/src/or/control.c +++ b/src/or/control.c @@ -179,6 +179,9 @@ static int write_stream_target_to_buf(entry_connection_t *conn, char *buf, static void orconn_target_get_name(char *buf, size_t len, or_connection_t *conn); +static int get_cached_network_liveness(void); +static void set_cached_network_liveness(int liveness); + /** Given a control event code for a message event, return the corresponding * log severity. */ static INLINE int @@ -2207,6 +2210,24 @@ getinfo_helper_onions(control_connection_t *control_conn, return 0; } +/** Implementation helper for GETINFO: answers queries about network + * liveness. */ +static int +getinfo_helper_liveness(control_connection_t *control_conn, + const char *question, char **answer, + const char **errmsg) +{ + if (strcmp(question, "network-liveness") == 0) { + if (get_cached_network_liveness()) { + *answer = tor_strdup("up"); + } else { + *answer = tor_strdup("down"); + } + } + + return 0; +} + /** Callback function for GETINFO: on a given control connection, try to * answer the question q and store the newly-allocated answer in * *a. If an internal error occurs, return -1 and optionally set @@ -2291,6 +2312,8 @@ static const getinfo_item_t getinfo_items[] = { "Information about and from the ns consensus."), ITEM("network-status", dir, "Brief summary of router status (v1 directory format)"), + ITEM("network-liveness", liveness, + "Current opinion on whether the network is live"), ITEM("circuit-status", events, "List of current circuits originating here."), ITEM("stream-status", events,"List of current streams."), ITEM("orconn-status", events, "A list of current OR connections."),

1 0

[tor/master] Changes file for ticket 15358
by nickm＠torproject.org 18 May '15

18 May '15

commit 79f7721a7e339783d6d0005def77abe3d7e23e95 Author: Andrea Shepard <andrea(a)torproject.org> Date: Sun May 17 13:58:05 2015 +0000 Changes file for ticket 15358 --- changes/ticket15358 | 3 +++ 1 file changed, 3 insertions(+) diff --git a/changes/ticket15358 b/changes/ticket15358 new file mode 100644 index 0000000..8c85d51 --- /dev/null +++ b/changes/ticket15358 @@ -0,0 +1,3 @@ + o Features (control protocl): + - Support network-liveness GETINFO key and NETWORK_LIVENESS events in the + control protocol. Resolves ticket #15358.

1 0

[metrics-db/master] Rewrite documentation.
by karsten＠torproject.org 18 May '15

18 May '15

commit 94c7158834ad5068c101f9b1ec95408ed1d46cd7 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Sat May 16 11:40:25 2015 +0200 Rewrite documentation. --- .gitignore | 2 - INSTALL.md | 80 +++++++++ README | 7 - doc/manual.pdf | Bin 156056 -> 0 bytes doc/manual.tex | 548 -------------------------------------------------------- 5 files changed, 80 insertions(+), 557 deletions(-) diff --git a/.gitignore b/.gitignore index 10681a4..f7d045e 100644 --- a/.gitignore +++ b/.gitignore @@ -5,8 +5,6 @@ config data/ in/ out/ -doc/manual.aux -doc/manual.log log/ rsync/ stats/ diff --git a/INSTALL.md b/INSTALL.md new file mode 100644 index 0000000..05db7ba --- /dev/null +++ b/INSTALL.md @@ -0,0 +1,80 @@ +CollecTor -- Operator's Guide +============================= + +Welcome to the Operator's Guide of CollecTor. This guide explains how +to set up a new CollecTor instance to download relay descriptors from the +Tor directory authorities. + + +Requirements +------------ + +You'll need a Linux host with at least 50G disk space and 2G RAM. + +In the following we'll assume that the host runs Debian stable as +operating system, but it should work on any other Linux or possibly even +*BSD. Though you'll be mostly on your own with those. + + +Prepare the system +------------------ + +Create a working directory for CollecTor. In this guide, we'll assume +that you're using `/srv/collector.torproject.org/` as working directory, +but feel free to use another directory that better suits your needs. + +$ sudo mkdir -p /srv/collector.torproject.org/ +$ sudo chown vagrant:vagrant /srv/collector.torproject.org/ + +Install a few packages: + +$ sudo apt-get install openjdk-6-jdk ant libcommons-codec-java \ + libcommons-compress-java + + +Clone the metrics-db repository +------------------------------- + +$ cd /srv/collector.torproject.org/ +$ git clone https://git.torproject.org/metrics-db + + +Clone required submodule metrics-lib +------------------------------------ + +$ git submodule init +$ git submodule update + + +Compile CollecTor +----------------- + +$ ant compile + + +Configure the relay descriptor downloader +----------------------------------------- + +Edit the config file and uncomment and edit at least the following line: + +DownloadRelayDescriptors 1 + + +Run the relay descriptor downloader +----------------------------------- + +$ bin/run-relaydescs + + +Set up an hourly cronjob for the relay descriptor downloader +------------------------------------------------------------ + +Ideally, run the relay descriptor downloader once per hour by adding a +crontab entry like the following: + +6 * * * * cd /srv/collector.torproject.org/db/ && bin/run-relaydescs + +Watch out for INFO-level logs in the `log/` directory. In particular, the +lines following "Statistics on the completeness of written relay +descriptors:" is quite important. + diff --git a/README b/README deleted file mode 100644 index 97a7a7b..0000000 --- a/README +++ /dev/null @@ -1,7 +0,0 @@ -ERNIE is the Enhanced R-based tor Network Intelligence Engine - (why ERNIE? because nobody liked BIRT; sorry for misspelling Tor) - --------------------------------------------------------------------------- - -Please find documentation in doc/ . - diff --git a/doc/manual.pdf b/doc/manual.pdf deleted file mode 100644 index 4c8b375..0000000 Binary files a/doc/manual.pdf and /dev/null differ diff --git a/doc/manual.tex b/doc/manual.tex deleted file mode 100644 index e50b1e4..0000000 --- a/doc/manual.tex +++ /dev/null @@ -1,548 +0,0 @@ -\documentclass{article} -\begin{document} -\title{ERNIE: a tool to study the Tor network\\-- User's Guide --} -\author{by Karsten Loesing \texttt{<karsten(a)torproject.org>}} -\maketitle - -\section{Overview} - -Welcome to ERNIE! -ERNIE is a tool to study the Tor network. -ERNIE has been designed to process all kinds of data about the Tor network -and visualize them or prepare them for further analysis. -ERNIE is also the software behind the Tor Metrics Portal -\verb+http://metrics.torproject.org/+. - -The acronym ERNIE stands for the \emph{Enhanced R-based tor Network -Intelligence Engine} (sorry for misspelling Tor). -Why ERNIE? -Because nobody liked BIRT (Business Intelligence and Reporting Tools) that -we used for visualizing statistics about the Tor network before writing -our own software. -By the way, reasons were that BIRT made certain people's browsers crash -and requires JavaScript that most Tor user have turned off. - -If you want to learn more about the Tor network, regardless of whether you -want to present your findings on a website (like ERNIE does) or include -them in your next Tor paper, this user's guide is for you! - -\section{Installation instructions} - -ERNIE depends on various other software tools. ERNIE is developed in a -\emph{Git} repository which is currently the only way to download it. -ERNIE uses \emph{Java} for parsing data, \emph{R} for plotting graphs, -and \emph{PostgreSQL} for importing data into a database. -Which of these tools you need depends on what tasks you are planning to -use ERNIE for. -In most cases it is not required to install all these tools. -For this tutorial, we assume Debian GNU/Linux 5.0 as operating system. -Installation instructions for other platforms may vary. - -\subsection{Git 1.5.6.5} - -Currently, the only way to download ERNIE is to clone its Git branch. - -Install Git 1.5.6.5 (or higher) and check that it's working: - -\begin{verbatim} -$ sudo apt-get install git-core -$ git --version -\end{verbatim} - -\subsection{Java 6} - -ERNIE requires Java to parse data from various data sources and write them -to one or more data sinks. Java is required for most use cases of ERNIE. - -Add the non-free repository to the apt sources in -\verb+/etc/apt/sources.list+ by changing the line (mirror URL may vary): - -\begin{verbatim} -deb http://ftp.ca.debian.org/debian/ lenny main -\end{verbatim} - -to - -\begin{verbatim} -deb http://ftp.ca.debian.org/debian/ lenny main non-free -\end{verbatim} - -Fetch the package list, install Sun Java 6, and set it as system default: - -\begin{verbatim} -$ sudo apt-get update -$ sudo apt-get install sun-java6-jdk -$ sudo update-alternatives --set java \ - /usr/lib/jvm/java-6-sun/jre/bin/java -$ sudo update-alternatives --set javac \ - /usr/lib/jvm/java-6-sun/bin/javac -\end{verbatim} - -Check that Java 6 is installed and selected as default: - -\begin{verbatim} -$ java -version -$ javac -version -\end{verbatim} - -\subsection{Ant 1.7} - -ERNIE comes with an Ant build file that facilitates common build tasks. -If you want to use Ant to build and run ERNIE, install Ant and check its -installed version (tested with 1.7): - -\begin{verbatim} -$ sudo apt-get install ant -$ ant -version -\end{verbatim} - -\subsection{R 2.8 and ggplot2} - -ERNIE uses R and the R library \emph{ggplot2} to visualize anaylsis -results for presentation on a website or for inclusion in publications. -ggplot2 requires at least R version 2.8 to be installed. - -Add a new line to \verb+/etc/apt/sources.list+: - -\begin{verbatim} -deb http://cran.cnr.berkeley.edu/bin/linux/debian lenny-cran/ -\end{verbatim} - -Download the package maintainer's public key (``Johannes Ranke (CRAN Debian -archive) $<$jranke(a)uni-bremen.de$>$''): - -\begin{verbatim} -$ gpg --keyserver pgpkeys.pca.dfn.de --recv-key 381BA480 -$ gpg --export 381BA480 | sudo apt-key add - -\end{verbatim} - -Install the most recent R version: - -\begin{verbatim} -$ sudo apt-get update -$ sudo apt-get -t unstable install r-base -\end{verbatim} - -Start R to check its version (must be 2.8 or higher) and install ggplot2. -Do this as root, so that the installed package is available to all system -users: - -\begin{verbatim} -$ sudo R -> install.packages("ggplot2") -> q() -\end{verbatim} - -Confirm that R and ggplot2 are installed: - -\begin{verbatim} -$ R -> library(ggplot2) -> q() -\end{verbatim} - -\subsection{PostgreSQL 8.3} -\label{sec-install-postgres} - -ERNIE uses PostgreSQL to import data into a database for later analysis. -This feature is not required for most use cases of ERNIE, but only for -people who prefer having the network data in a database to execute custom -queries. - -Install PostgreSQL 8.3 using apt-get: - -\begin{verbatim} -$ sudo apt-get install postgresql-8.3 -\end{verbatim} - -Create a new database user \verb+ernie+ to insert data and run queries. -This command is executed as unix user \verb+postgres+ and therefore as -database superuser \verb+postgres+ via ident authentication. The -\verb+-P+ flag issues a password prompt for the new user. -There is no need to give the new user superuser privileges or allow it to -create databases or new roles. - -\begin{verbatim} -$ sudo -u postgres createuser -P ernie -\end{verbatim} - -Create a new database schema \verb+tordir+ owned by user \verb+ernie+ -(using option \verb+-O+). -Again, this command is executed as \verb+postgres+ system user to make use -of ident authentication. - -\begin{verbatim} -$ sudo -u postgres createdb -O ernie tordir -\end{verbatim} - -Log into the database schema as user \verb+ernie+ to check that it's -working. -This time, ident authentication is not available, since there is no system -user \verb+ernie+. -Instead, we will use password authentication via a TCP connection to -localhost (using option \verb+-h+) as database user \verb+ernie+ (using -option \verb+-U+). - -\begin{verbatim} -$ psql -h localhost -U ernie tordir -tordir=> \q -\end{verbatim} - -\subsection{ERNIE} - -Finally, you can install ERNIE by cloning its Git branch: - -\begin{verbatim} -$ git clone git://git.torproject.org/ernie -\end{verbatim} - -This command should create a directory \verb+ernie/+ which we will -consider the working directory of ERNIE. - -\section{Getting started with ERNIE} - -The ERNIE project was started as a simple tool to parse Tor relay -descriptors and plot graphs on Tor network usage for a website. -Since then, ERNIE has grown to a tool that can process all kinds of Tor -network data for various purposes, including but not limited to -visualization. - -We think that the easiest way to get started with ERNIE is to walk through -typical use cases in a tutorial style and explain what is required to set -up ERNIE. -These use cases have been chosen from what we think are typical -applications of ERNIE. - -\subsection{Visualizing network statistics} - -{\it Write me.} - -\subsection{Importing relay descriptors into a database} - -As of February 2010, the relays and directories in the Tor network -generate more than 1 GB of descriptors every month. -There are two approaches to process these amounts of data: -extract only the relevant data for the analysis and write them to files, -or import all data to a database and run queries on the database. -ERNIE currently takes the file-based approach for the Metrics Portal, -which works great for standardized analyses. -But the more flexible way to research the Tor network is to work with a -database. - -This tutorial describes how to import relay descriptors into a database -and run a few example queries. -Note that the presented database schema is limited to answering basic -questions about the Tor network. -In order to answer more complex questions, one would have to extend the -database schema and Java classes which is sketched at the end of this -tutorial. - -\subsubsection{Preparing database for data import} - -The first step in importing relay descriptors into a database is to -install a database management system. -See Section \ref{sec-install-postgres} for installation instructions of -PostgreSQL 8.3 on Debian GNU/Linux 5.0. -Note that in theory, any other relational database that has a working JDBC -4 driver should work, too, possibly with minor modifications to ERNIE. - -Import the database schema from file \verb+db/tordir.sql+ containing two -tables that we need for importing relay descriptors plus two indexes to -accelerate queries. Check that tables have been created using \verb+\dt+. -You should see a list containing the two tables \verb+descriptor+ and -\verb+statusentry+. - -\begin{verbatim} -$ psql -h localhost -U ernie -f db/tordir.sql tordir -$ psql -h localhost -U ernie tordir -tordir=> \dt -tordir=> \q -\end{verbatim} - -A row in the \verb+statusentry+ table contains the information that a -given relay (that has published the server descriptor with ID -\verb+descriptor+) was contained in the network status consensus published -at time \verb+validafter+. -These two fields uniquely identify a row in the \verb+statusentry+ table. -The other fields contain boolean values for the flags that the directory -authorities assigned to the relay in this consensus, e.g., the Exit flag -in \verb+isexit+. -Note that for the 24 network status consensuses of a given day with each -of them containing 2000 relays, there will be $24 \times 2000$ rows in the -\verb+statusentry+ table. - -The \verb+descriptor+ table contains some portion of the information that -a relay includes in its server descriptor. -Descriptors are identified by the \verb+descriptor+ field which -corresponds to the \verb+descriptor+ field in the \verb+statusentry+ -table. -The other fields contain further data of the server descriptor that might -be relevant for analyses, e.g., the platform line with the Tor software -version and operating system of the relay. - -Obviously, this data schema doesn't match everyone's needs. -See the instructions below for extending ERNIE to import other data into -the database. - -\subsubsection{Downloading relay descriptors from the metrics website} - -In the next step you will probably want to download relay descriptors from -the metrics website -\verb+http://metrics.torproject.org/data.html#relaydesc+. -Download the \verb+v3 consensuses+ and/or \verb+server descriptors+ of the -months you want to analyze. -The server descriptors are the documents that relays publish at least -every 18 hours describing their capabilities, whereas the v3 consensuses -are views of the directory authorities on the available relays at a given -time. -For this tutorial you need both v3 consensuses and server descriptors. -You might want to start with a single month of data, experiment with it, -and import more data later on. -Extract the tarballs to a new directory \verb+archives/+ in the ERNIE -working directory. - -\subsubsection{Configuring ERNIE to import relay descriptors into a -database} - -ERNIE can be used to read data from one or more data sources and write -them to one or more data sinks. -You need to configure ERNIE so that it knows to use the downloaded relay -descriptors as data source and the database as data sink. -Add the following two lines to your \verb+config+ file: - -\begin{verbatim} -ImportDirectoryArchives 1 -WriteRelayDescriptorDatabase 1 -\end{verbatim} - -You further need to provide the JDBC string that ERNIE shall use to access -the database schema \verb+tordir+ that we created above. -The config option with the JDBC string for a local PostgreSQL database -might be (without line break): - -\begin{verbatim} -RelayDescriptorDatabaseJDBC - jdbc:postgresql://localhost/tordir?user=ernie&password=password -\end{verbatim} - -\subsubsection{Importing relay descriptors using ERNIE} - -Now you are ready to actually import relay descriptors using ERNIE. -Create a directory for Java class files, compile the Java source files, -and run ERNIE. All these steps are performed by the default target in the -provided Ant task. - -\begin{verbatim} -$ ant -\end{verbatim} - -Note that the import process might take between a few minutes and an hour, -depending on your hardware. -You will notice that ERNIE doesn't write progress messages to the standard -output, which is useful for unattended installations with only warnings -being mailed out by cron. -You can change this behavior and make messages on the standard output more -verbose by setting -\verb+java.util.logging.ConsoleHandler.level+ in -\verb+logging.properties+ to \verb+INFO+ or \verb+FINE+. -Alternately, you can look at the log file \verb+log.0+ that is created by -ERNIE. - -If ERNIE finishes after a few seconds, you have probably put the relay -descriptors at the wrong place. -Make sure that you extract the relay descriptors to sub directories of -\verb+archives/+ in the ERNIE working directory. - -If you interrupt ERNIE, or if ERNIE terminates uncleanly for some reason, -you will have problems starting it the next time. -ERNIE uses a local lock file called \verb+lock+ to make sure that only a -single instance of ERNIE is running at a time. -If you are sure that the last ERNIE instance isn't running anymore, you -can delete the lock file and start ERNIE again. - -If all goes well, you should now have the relay descriptors of 1 month in -your database. - -\subsubsection{Example queries} - -In this tutorial, we want to give you a few examples for using the -database schema with the imported relay descriptors to extract some useful -statistics about the Tor network. - -In the first example we want to find out how many relays have been running -on average per day and how many of these relays were exit relays. -We only need the \verb+statusentry+ table for this evaluation, because -the information we are interested in is contained in the network status -consensuses. - -The SQL statement that we need for this evaluation consists of two parts: -First, we find out how many network status consensuses have been published -on any given day. -Second, we count all relays and those with the Exit flag and divide these -numbers by the number of network status consensuses per day. - -\begin{verbatim} -$ psql -h localhost -U ernie tordir -tordir=> SELECT DATE(validafter), - COUNT(*) / relay_statuses_per_day.count AS avg_running, - SUM(CASE WHEN isexit IS TRUE THEN 1 ELSE 0 END) / - relay_statuses_per_day.count AS avg_exit - FROM statusentry, - (SELECT COUNT(*) AS count, DATE(validafter) AS date - FROM (SELECT DISTINCT validafter FROM statusentry) - distinct_consensuses - GROUP BY DATE(validafter)) relay_statuses_per_day - WHERE DATE(validafter) = relay_statuses_per_day.date - GROUP BY DATE(validafter), relay_statuses_per_day.count - ORDER BY DATE(validafter); -tordir=> \q -\end{verbatim} - -Executing this query should finish within a few seconds to one minute, -again depending on your hardware. -The result might start like this (truncated here): - -\begin{verbatim} - date | avg_running | avg_exit -------------+-------------+---------- - 2010-02-01 | 1583 | 627 - 2010-02-02 | 1596 | 638 - 2010-02-03 | 1600 | 654 -: -\end{verbatim} - -In the second example we want to find out what Tor software versions the -relays have been running. -More precisely, we want to know how many relays have been running what Tor -versions on micro version granularity (e.g., 0.2.2) on average per day? - -We need to combine network status consensuses with server descriptors to -find out this information, because the version information is not -contained in the consensuses (or at least, it's optional to be contained -in there; and after all, this is just an example). -Note that we cannot focus on server descriptors only and leave out the -consensuses for this analysis, because we want our analysis to be limited -to running relays as confirmed by the directory authorities and not -include all descriptors that happened to be published at a given day. - -The SQL statement again determines the number of consensuses per day in a -sub query. -In the next step, we join the \verb+statusentry+ table with the -\verb+descriptor+ table for all rows contained in the \verb+statusentry+ -table. -The left join means that we include \verb+statusentry+ rows even if we do -not have corresponding lines in the \verb+descriptor+ table. -We determine the version by skipping the first 4 characters of the platform -string that should contain \verb+"Tor "+ (without quotes) and cutting off -after another 5 characters. -Obviously, this approach is prone to errors if the platform line format -changes, but it should be sufficient for this example. - -\begin{verbatim} -$ psql -h localhost -U ernie tordir -tordir=> SELECT DATE(validafter) AS date, - SUBSTRING(platform, 5, 5) AS version, - COUNT(*) / relay_statuses_per_day.count AS count - FROM - (SELECT COUNT(*) AS count, DATE(validafter) AS date - FROM (SELECT DISTINCT validafter - FROM statusentry) distinct_consensuses - GROUP BY DATE(validafter)) relay_statuses_per_day - JOIN statusentry - ON relay_statuses_per_day.date = DATE(validafter) - LEFT JOIN descriptor - ON statusentry.descriptor = descriptor.descriptor - GROUP BY DATE(validafter), SUBSTRING(platform, 5, 5), - relay_statuses_per_day.count, relay_statuses_per_day.date - ORDER BY DATE(validafter), SUBSTRING(platform, 5, 5); -tordir=> \q -\end{verbatim} - -Running this query takes longer than the first query, which can be a few -minutes to half an hour. -The main reason is that joining the two tables is an expensive database -operation. -If you plan to perform many evaluations like this one, you might want to -create a third table that holds the results of joining the two tables of -this tutorial. -Creating such a table to speed up queries is not specific to ERNIE and -beyond the scope of this tutorial. - -The (truncated) result of the query might look like this: - -\begin{verbatim} - date | version | count -------------+---------+------- - 2010-02-01 | 0.1.2 | 10 - 2010-02-01 | 0.2.0 | 217 - 2010-02-01 | 0.2.1 | 774 - 2010-02-01 | 0.2.2 | 75 - 2010-02-01 | | 505 - 2010-02-02 | 0.1.2 | 14 - 2010-02-02 | 0.2.0 | 328 - 2010-02-02 | 0.2.1 | 1143 - 2010-02-02 | 0.2.2 | 110 -: -\end{verbatim} - -Note that, in the fifth line, we are missing the server descriptors of 505 -relays contained in network status consensuses published on 2010-02-01. -If you want to avoid such missing values, you'll have to import the server -descriptors of the previous month, too. - -\subsubsection{Extending ERNIE to import further data into the database} - -In this tutorial we have explained how to prepare a database, download -relay descriptors, configure ERNIE, import the descriptors, and execute -example queries. -This description is limited to a few examples by the very nature of a -tutorial. -If you want to extend ERNIE to import further data into your database, -you will have to perform at least two steps: -extend the database schema and modify the Java classes used for parsing. - -The first step, extending the database schema, is not specific to ERNIE. -Just add the fields and tables to the schema definition. - -The second step, modifying the Java classes used for parsing, is of course -specific to ERNIE. -You will have to look at two classes in particular: -The first class, \verb+RelayDescriptorDatabaseImporter+, contains the -prepared statements and methods used to add network status consensus -entries and server descriptors to the database. -The second class, \verb+RelayDescriptorParser+, contains the parsing logic -for the relay descriptors and decides what information to add to the -database, among other things. - -This ends the tutorial on importing relay descriptors into a database. -Happy researching! - -\subsection{Aggregating relay and bridge descriptors} - -{\it Write me.} - -\section{Software architecture} - -{\it Write me. In particular, include overview of components: - -\begin{itemize} -\item Data sources and data sinks -\item Java classes with data sources and data sinks -\item R scripts to process CSV output -\item Website -\end{itemize} -} - -\section{Tor Metrics Portal setup} - -{\it -Write me. In particular, include documentation of deployed ERNIE that -runs the metrics website. -This documentation has two purposes: -First, a reference setup can help others creating their own ERNIE -configuration that goes beyond the use cases as described above. -Second, we need to remember how things are configured anyway, so we can -as well document them here.} - -\end{document} -

1 0

[tor/master] Fix crash on HUP with mixed ephemeral services
by nickm＠torproject.org 18 May '15

18 May '15

commit 6f9e90101e64040bda38c34de884f3d996677ac9 Author: John Brooks <john.brooks(a)dereferenced.net> Date: Sat May 16 19:50:18 2015 -0600 Fix crash on HUP with mixed ephemeral services Ephemeral services will be listed in rend_services_list at the end of rend_config_services, so it must check whether directory is non-NULL before comparing. This crash happens when reloading config on a tor with mixed configured and ephemeral services. Fixes bug #16060. Bugfix on 0.2.7.1-alpha. --- changes/bug16060 | 4 ++++ src/or/rendservice.c | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/changes/bug16060 b/changes/bug16060 new file mode 100644 index 0000000..eb0b50f --- /dev/null +++ b/changes/bug16060 @@ -0,0 +1,4 @@ + o Minor bugfixes (hidden service): + - Fix a crash when reloading configuration while at least one + configured and one ephemeral hidden service exists. Fixes bug 16060; + bugfix on 0.2.7.1-alpha. diff --git a/src/or/rendservice.c b/src/or/rendservice.c index e170760..daca4cc 100644 --- a/src/or/rendservice.c +++ b/src/or/rendservice.c @@ -709,7 +709,8 @@ rend_config_services(const or_options_t *options, int validate_only) * probably ok? */ SMARTLIST_FOREACH_BEGIN(rend_service_list, rend_service_t *, new) { SMARTLIST_FOREACH_BEGIN(old_service_list, rend_service_t *, old) { - if (!strcmp(old->directory, new->directory)) { + if (new->directory && old->directory && + !strcmp(old->directory, new->directory)) { smartlist_add_all(new->intro_nodes, old->intro_nodes); smartlist_clear(old->intro_nodes); smartlist_add(surviving_services, old);

1 0

← Newer
1
...
36
37
38
39
40
41
42
...
99
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits May 2015