July 2016 - tor-commits - lists.torproject.org

[torspec/master] Merge remote-tracking branch 'teor/bug18456'
by nickm＠torproject.org 06 Jul '16

06 Jul '16

commit 8104db8f26f64a5436425dd07b0ebd34d34e969e Merge: 75dc767 19df578 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Jul 5 19:19:03 2016 -0400 Merge remote-tracking branch 'teor/bug18456' control-spec.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)

1 0

[torspec/master] Update and clarify exit-policy/reject-private/relay for #18456
by nickm＠torproject.org 06 Jul '16

06 Jul '16

commit 19df57848e1923b196c9c579e476af3a5d019100 Author: teor (Tim Wilson-Brown) <teor2345(a)gmail.com> Date: Fri Jul 1 15:43:47 2016 +1000 Update and clarify exit-policy/reject-private/relay for #18456 Add ExitPolicyRejectLocalInterfaces option. Make it clear that the current option values determine the output. --- control-spec.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/control-spec.txt b/control-spec.txt index 9529d35..37c6484 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -522,8 +522,9 @@ ExitPolicyRejectPrivate is 1. "exit-policy/reject-private/relay" -- The relay-specific exit policy - lines that Tor will *prepend* to the ExitPolicy config option when - ExitPolicyRejectPrivate is 1. These lines are based on the public + lines that Tor will *prepend* to the ExitPolicy config option based + on the current values of ExitPolicyRejectPrivate and + ExitPolicyRejectLocalInterfaces. These lines are based on the public addresses configured in the torrc and present on the relay's interfaces.

1 0

[tor/release-0.2.8] Fold 19271 into changelog
by nickm＠torproject.org 06 Jul '16

06 Jul '16

commit 79213afd3a31aefebdf7d3f4dfe0c3583a959c1f Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Jul 5 18:45:44 2016 -0400 Fold 19271 into changelog --- ChangeLog | 3 +++ changes/19271 | 2 -- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index 06577e5..f6b5ef8 100644 --- a/ChangeLog +++ b/ChangeLog @@ -4,6 +4,9 @@ Changes in version 0.2.8.5-rc - 2016-07-0? 0.2.8 release will be identical to it. It has a few small bugfixes against previous versions. + o Directory authority changes: + - Urras is no longer a directory authority. Closes ticket 19271. + o Major bugfixes (heartbeat): - Fix regression that crashes Tor when disabling heartbeats. Fixes bug 19454; bugfix on tor-0.2.8.1-alpha. Reported by "kubaku". diff --git a/changes/19271 b/changes/19271 deleted file mode 100644 index dc06ead..0000000 --- a/changes/19271 +++ /dev/null @@ -1,2 +0,0 @@ - o Directory authority changes: - - Urras is no longer a directory authority. Closes ticket 19271.

1 0

[tor/master] Test: fix shared random test checking bad errno
by nickm＠torproject.org 05 Jul '16

05 Jul '16

commit 245c10de074cb61ec75eaee152c51253678039cc Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Jul 5 13:54:37 2016 -0400 Test: fix shared random test checking bad errno The test was checking for EISDIR which is a Linux-ism making other OSes unhappy. Instead of checking for a negative specific errno value, just make sure it's negative indicating an error. We don't need more for this test. Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- src/test/test_shared_random.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/src/test/test_shared_random.c b/src/test/test_shared_random.c index 4f05e75..d6787e4 100644 --- a/src/test/test_shared_random.c +++ b/src/test/test_shared_random.c @@ -638,11 +638,7 @@ test_state_load_from_disk(void *arg) /* Try to load the directory itself. Should fail. */ ret = disk_state_load_from_disk_impl(dir); -#ifdef _WIN32 - tt_int_op(ret, OP_EQ, -EACCES); -#else - tt_int_op(ret, OP_EQ, -EISDIR); -#endif + tt_int_op(ret, OP_LT, 0); /* State should be non-existent at this point. */ the_sr_state = get_sr_state();

1 0

[tor/master] Merge remote-tracking branch 'dgoulet/sr-test-fix'
by nickm＠torproject.org 05 Jul '16

05 Jul '16

commit dfa41ac0f32729f9d5a731681d826ec3006f6fb5 Merge: 85c39b6 245c10d Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Jul 5 17:53:13 2016 -0400 Merge remote-tracking branch 'dgoulet/sr-test-fix' src/test/test_shared_random.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)

1 0

[metrics-web/master] Update paths and instructions to Debian Jessie.
by karsten＠torproject.org 05 Jul '16

05 Jul '16

commit 87283048d247b100fa135db7ee76ce76ea7fac4d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jun 30 15:13:53 2016 +0200 Update paths and instructions to Debian Jessie. --- .gitignore | 12 +-- .gitmodules | 3 - README | 205 +++++++++++++---------------------------- deps/metrics-lib | 1 - modules/advbwdist/build.xml | 19 ++-- modules/clients/build.xml | 17 ++-- modules/collectdescs/build.xml | 15 +-- modules/connbidirect/build.xml | 17 ++-- modules/disagreement/build.xml | 7 +- modules/hidserv/build.xml | 17 ++-- modules/legacy/build.xml | 22 ++--- website/build.xml | 36 ++++---- website/lib/REngine.jar | Bin 30637 -> 0 bytes website/lib/RserveEngine.jar | Bin 27827 -> 0 bytes website/lib/jstl.jar | Bin 16905 -> 0 bytes website/lib/standard.jar | Bin 293750 -> 0 bytes website/rserve/shutdown.sh | 5 - 17 files changed, 130 insertions(+), 246 deletions(-) diff --git a/.gitignore b/.gitignore index 0323a32..8f49baf 100644 --- a/.gitignore +++ b/.gitignore @@ -1,16 +1,16 @@ # Eclipse files -.classpath -.project +/.classpath +/.project # Generated web archive -website/metrics.war +/website/metrics.war # Compiled class files -classes/ +*.class # Possibly modified config file -etc/context.xml +/website/etc/context.xml # Copied and possibly modified config file -config +/website/etc/config diff --git a/.gitmodules b/.gitmodules index c6b6388..e69de29 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +0,0 @@ -[submodule "deps/metrics-lib"] - path = deps/metrics-lib - url = https://git.torproject.org/metrics-lib diff --git a/README b/README index 205e8d4..06204bd 100644 --- a/README +++ b/README @@ -1,18 +1,17 @@ -Tor Metrics Database and Website -================================ +Tor Metrics +=========== -The metrics database stores publicly available data about the Tor network -which are visualized by the metrics website. +Tor Metrics aggregates publicly available data about the Tor network and +visualizes that data on a website. -This software package, metrics-web, contains (1) the code to import Tor -network data into a database, (2) the code to generate graphs and .CSV -output, and (3) the code for a dynamic web application. metrics-web is -based on Java, Ant, PostgreSQL, R, Apache HTTP Server, and Apache Tomcat. +This software package, metrics-web, contains (1) the code to aggregate Tor +network data, (2) the code to generate graphs and .CSV output, and (3) the +code for a dynamic web application. metrics-web is based on Java, Ant, +PostgreSQL, R, Apache HTTP Server, and Apache Tomcat. This README explains all necessary steps to install metrics-web including -the database (Section 1), the graphing engine (Section 2), and the web -application (Section 3). It is possible to install only the database part -or only the database and the graphing engine, if desired. +any databases (Section 1), the graphing engine (Section 2), and the web +application (Section 3). 1. Installing the metrics database @@ -27,7 +26,7 @@ performance measurement installations, and others. =================================== This README describes the steps for installing metrics-web on a Debian -GNU/Linux Squeeze server. Instructions for other operating systems may +GNU/Linux Jessie server. Instructions for other operating systems may vary. In the following it is assumed that root privileges are available. @@ -40,43 +39,34 @@ $ below. # adduser metrics The database importer and website sources will be installed in -/srv/metrics-web/ that is created as follows: +/srv/metrics.torproject.org/ that is created as follows: -# mkdir /srv/metrics-web/ -# chmod g+ws /srv/metrics-web/ -# chown metrics:metrics /srv/metrics-web/ +# mkdir /srv/metrics.torproject.org/ +# chmod g+ws /srv/metrics.torproject.org/ +# chown metrics:metrics /srv/metrics.torproject.org/ -Either extract the metrics-web source tarball... +Clone the metrics-web Git repository: -$ tar xf metrics-web-x.y.z.tar /srv/metrics-web/ +$ cd /srv/metrics.torproject.org/ +$ git clone git://git.torproject.org/metrics-web metrics -... or clone the metrics-web Git repository: +Install OpenJDK 7, Ant 1.9.4, and PostgreSQL 9.4 that are necessary for +setting up the metrics database. -$ git clone git://git.torproject.org/metrics-web /srv/metrics-web/ - -Install Sun Java 6, Ant 1.8, and PostgreSQL 8.4 that are necessary for -setting up the metrics database (be sure to include Debian's non-free -repository in /etc/apt/sources.list). - -# apt-get install sun-java6-jdk ant postgresql-8.4 - -Make Sun's Java the default. - -# update-java-alternatives -s java-6-sun +# apt-get install openjdk-7-jdk ant postgresql-9.4 Check the versions of the newly installed tools. $ java -version -java version "1.6.0_24" -Java(TM) SE Runtime Environment (build 1.6.0_24-b07) -Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) +java version "1.7.0_101" +OpenJDK Runtime Environment (IcedTea 2.6.6) (7u101-2.6.6-2~deb8u1) +OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) $ ant -version -Apache Ant version 1.8.0 compiled on March 11 2010 +Apache Ant(TM) version 1.9.4 compiled on October 7 2014 $ psql --version -psql (PostgreSQL) 8.4.7 -contains support for command-line editing +psql (PostgreSQL) 9.4.8 1.2. Configuring the database @@ -97,7 +87,7 @@ Create a new database tordir owned by user metrics. Import the metrics database schema. -$ psql -f /srv/metrics-web/db/tordir.sql tordir +$ psql -f /srv/metrics.torproject.org/db/tordir.sql tordir Confirm that the database now contains tables to hold metrics data. In the following, => will be used as the database prompt. @@ -115,8 +105,8 @@ archived relay descriptors from the official metrics website. Download the relay descriptor tarballs from the metrics website at https://metrics.torproject.org/data.html#relaydesc and extract them to -/srv/metrics-web/archives/ . The database importer can process v3 votes, -v3 consensuses, server descriptors, and extra-infos. +/srv/metrics.torproject.org/archives/ . The database importer can process +v3 votes, v3 consensuses, server descriptors, and extra-infos. Edit the config file ~/metrics-web/config (or create it if it's not there) to contain the following five lines (be sure to remove the linebreak in @@ -131,7 +121,7 @@ RelayDescriptorDatabaseJDBC Compile and run the Java database importer. -$ cd /srv/metrics-web/ +$ cd /srv/metrics.torproject.org/ $ ./run.sh The database import will take a while. Once it's complete, check that the @@ -147,8 +137,8 @@ once they are imported. An alternative to importing relay descriptor tarballs directly into the database is to convert them into a data format that psql's \copy command can process. Look for the config option WriteRelayDescriptorsRawFiles in -/srv/metrics-web/config.template for more information on this experimental -feature. +/srv/metrics.torproject.org/config.template for more information on this +experimental feature. In a future version of metrics-web it may also be possible to update local relay descriptor tarballs from the official metrics server via rsync and @@ -174,15 +164,15 @@ FetchDirInfoExtraEarly 1 Tell the metrics database importer where to find the cached descriptor files. One way to achieve this is to add symbolic links to -/srv/metrics-web/archives/ like this. Tor's data directory is assumed to -be /srv/tor/ here. +/srv/metrics.torproject.org/archives/ like this. Tor's data directory is +assumed to be /srv/tor/ here. -$ cd /srv/metrics-web/archives/ +$ cd /srv/metrics.torproject.org/archives/ $ ln -s /srv/tor/cached-* . Add a crontab entry for the database importer to run once per hour: -15 * * * * cd /srv/metrics-web/ && ./run.sh +15 * * * * cd /srv/metrics.torproject.org/ && ./run.sh 1.5. Pre-calculating relay statistics @@ -201,24 +191,7 @@ If the metrics database gets updated automatically, write a script and add a crontab entry for pre-calculating statistics every 6 or 12 hours. -1.6. Generating network status information -========================================== - -The metrics database importer can analyze the most recently parsed network -status consensus for irregularities indicating problems with the directory -authorities. There are two possible outputs: the consensus-health page -that can be found at https://metrics.torproject.org/consensus-health.html -and a local file that can be parsed by Nagios that will be written to -/srv/metrics-web/website/consensus-health . - -Edit /srv/metrics-web/config to contain either or both of the following -options: - -WriteConsensusHealth 1 -WriteNagiosStatusFile 1 - - -1.7. Importing sanitized bridge descriptors +1.6. Importing sanitized bridge descriptors =========================================== The metrics database can store aggregate statistics about running bridges @@ -227,9 +200,9 @@ descriptors available on the official metrics website. Download a sanitized bridge descriptor tarball from the metrics website at https://metrics.torproject.org/data.html#bridgedesc and extract it to, -e.g., /srv/metrics-web/bridges/bridge-descriptors-2011-05/ . +e.g., /srv/metrics.torproject.org/bridges/bridge-descriptors-2011-05/ . -Edit /srv/metrics-web/config to contain the following options: +Edit /srv/metrics.torproject.org/config to contain the following options: ImportSanitizedBridges 1 SanitizedBridgesDirectory bridges/ @@ -240,15 +213,15 @@ Note that the bridge usage statistics require parsing relay descriptors of the same time period in order to filter bridges that have been running as relays from the results. When parsing sanitized bridge descriptors for the first time it may be necessary to delete the relay descriptor import -history in /srv/metrics-web/stats/archives-import-history and import all -relay descriptors once again. +history in /srv/metrics.torproject.org/stats/archives-import-history and +import all relay descriptors once again. Run the database import: $ ./run.sh -1.8. Importing Torperf performance data +1.7. Importing Torperf performance data ======================================= Torperf measures the performance of the Tor network as users experience @@ -257,9 +230,9 @@ can be imported into the metrics database, too. Download the Torperf measurement files from the metrics website at https://metrics.torproject.org/data.html#performance and put them in a -subdirectory, e.g., /srv/metrics-web/torperf/ . +subdirectory, e.g., /srv/metrics.torproject.org/torperf/ . -Edit /srv/metrics-web/config to contain the following options: +Edit /srv/metrics.torproject.org/config to contain the following options: ImportWriteTorperfStats 1 TorperfDirectory torperf/ @@ -287,64 +260,30 @@ Setting up the graphing engine requires installing PostgreSQL's header files and R 2.8 or higher. R 2.8 or higher is required for the ggplot2 library. -# apt-get install libpq-dev r-base-dev - -Run R as user metrics and install required packages to ~/R/. In the -following, R commands will be prefixed with >. - -$ R -> install.packages("Rserve") -> install.packages("ggplot2") -> install.packages("RPostgreSQL") -> q() - -Start the Rserve daemon (the exact path of Rserve-bin.so may vary), check -that it's working by connecting via telnet, and shut it down: - -$ R CMD ~/R/x86_64-pc-linux-gnu-library/2.11/Rserve/libs/Rserve-bin.so -$ telnet 127.0.0.1 6311 -$ echo "library(Rserve); RSshutdown(RSconnect())" | R --slave - -Also check that a database connection can be established from within R -(using the actual password instead of "password"): - -$ R -> library(RPostgreSQL) -> drv <- dbDriver("PostgreSQL") -> con <- dbConnect(drv, user = "metrics", password = "password", - dbname = "tordir") -> dbDisconnect(con) -> dbUnloadDriver(drv) -> q() - -Insert the database password in the Rserve initialization script in -/srv/metrics-web/rserve/rserve-init.R. - -Update the workdir path in /srv/metrics-web/rserve/Rserv.conf . +# apt-get install r-base r-cran-rserve r-cran-ggplot2 r-cran-reshape \ + r-cran-scales Start Rserve, this time with the metrics-web-specific configuration that includes pre-loading the graph code: -$ cd /srv/metrics-web/rserve/ && ./start.sh +$ cd /srv/metrics.torproject.org/rserve/ && ./start.sh Add a crontab entry to start Rserve on reboot: -@reboot cd /srv/metrics-web/rserve/ && ./start.sh +@reboot cd /srv/metrics.torproject.org/metrics/website/rserve/ && ./start.sh Rserve will pre-load the graph code at startup. If changes are made to the graph code, Rserve must be restarted: -$ cd /srv/metrics-web/rserve/ -$ ./shutdown.sh && ./start.sh +$ cd /srv/metrics.torproject.org/metrics/website/rserve/ +$ killall Rserve; ./start.sh 3. Installing the metrics website ================================= The metrics website lets web users search parts of the metrics database -and visualizes custom graphs. Both the metrics database and the graphing -engine are required to set up the metrics website as described in this -section. +and visualizes custom graphs. Note that the description here has a few specific parts that only apply to the official metrics website. These parts should be changed when setting @@ -364,7 +303,7 @@ Start by installing Apache 2: Disable Apache's default site. -# a2dissite default +# a2dissite 000-default Enable mod_rewrite to tell Apache where to find static resources on disk. Also enable mod_proxy to forward requests to Tomcat. @@ -381,30 +320,17 @@ content: ErrorLog /var/log/apache2/error.log CustomLog /var/log/apache2/access.log combined ServerSignature On - <IfModule mod_rewrite.c> - RewriteEngine On - RewriteRule /(data|dist|papers)/(.*) /srv/metrics-web/$1/$2 [L] - RewriteRule /(consensus-health.html) /srv/metrics-web/website/$1 [L] - </IfModule> <IfModule mod_proxy.c> <Proxy *> Order deny,allow Allow from all </Proxy> - ProxyPass / http://127.0.0.1:8080/ernie/ retry=15 - ProxyPassReverse / http://127.0.0.1:8080/ernie/ + ProxyPass / http://127.0.0.1:8080/metrics/ retry=15 + ProxyPassReverse / http://127.0.0.1:8080/metrics/ ProxyPreserveHost on </IfModule> </VirtualHost> -Create the directories containing static resources: /srv/metrics-web/data/ -contains the tarballs and other metrics data linked from data.html. -/srv/metrics-web/dist/ contains the software packages linked from -tools.html. /srv/metrics-web/papers/ contains the papers and technical -reports linked from papers.html. Note that there is no option not to -serve these files other than manually removing the links from the .html -pages. - Enable the new virtual host. # a2ensite metrics.torproject.org @@ -420,11 +346,11 @@ Restart Apache just to be sure that all changes are effective. Apache Tomcat will process requests for dynamic resources, including web pages and graphs. -Install Tomcat 6: +Install Tomcat 8: -# apt-get install tomcat6 +# apt-get install tomcat8 -Replace Tomcat's default configuration in /etc/tomcat6/server.xml with the +Replace Tomcat's default configuration in /etc/tomcat8/server.xml with the following configuration: <Server port="8005" shutdown="SHUTDOWN"> @@ -436,11 +362,11 @@ following configuration: compression="off" compressionMinSize="2048" noCompressionUserAgents="gozilla, traviata" compressableMimeType="text/html,text/xml,text/plain" /> - <Engine name="Catalina" defaultHost="yatei.torproject.org"> + <Engine name="Catalina" defaultHost="metrics.torproject.org"> <Host name="metrics.torproject.org" appBase="webapps" unpackWARs="true" autoDeploy="true" xmlValidation="false" xmlNamespaceAware="false"> - <Alias>yatei.torproject.org</Alias> + <Alias>metrics.torproject.org</Alias> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="metrics_access_log." suffix=".txt" pattern="%l %u %t %r %s %b" resolveHosts="false"/> @@ -452,12 +378,11 @@ following configuration: Be sure to replace *.torproject.org with something else, unless this is a re-installation of the official metrics website. -Update the database password in /srv/metrics-web/etc/context.xml. - Update the paths starting with /srv/metrics.torproject.org/ in -/srv/metrics-web/etc/web.xml to the correct paths in /srv/metrics-web/. -The default paths in that file are correct for the official metrics -website setup which is slightly different than the one described here. +/srv/metrics.torproject.org/metrics/website/etc/web.xml to the correct +paths in /srv/metrics.torproject.org/. The default paths in that file are +correct for the official metrics website setup which is slightly different +than the one described here. Now generate the web application. @@ -465,7 +390,7 @@ $ ant war Create a symbolic link to the ernie.war file: -# ln -s /srv/metrics-web/ernie.war /var/lib/tomcat6/webapps/ +# ln -s /srv/metrics.torproject.org/metrics/website/metrics.war /var/lib/tomcat8/webapps/ Tomcat will now attempt to deploy the web application automatically. @@ -474,7 +399,7 @@ file and Tomcat will reload the web application automatically. Restart Tomcat to make all configuration changes effective: -# /etc/init.d/tomcat6 restart +# service tomcat8 restart The metrics website should now work. diff --git a/deps/metrics-lib b/deps/metrics-lib deleted file mode 160000 index 1960762..0000000 --- a/deps/metrics-lib +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 1960762ca3ad515a22b90fdd3a1f4c68a3fa9588 diff --git a/modules/advbwdist/build.xml b/modules/advbwdist/build.xml index 16845ee..05dea0b 100644 --- a/modules/advbwdist/build.xml +++ b/modules/advbwdist/build.xml @@ -1,24 +1,19 @@ <project default="run" name="advbwdist" basedir="."> <property name="sources" value="src"/> + <property name="libs" value="../../shared/lib"/> <property name="classes" value="classes"/> <path id="classpath"> <pathelement path="${classes}"/> - <fileset dir="/usr/share/java"> - <include name="commons-codec.jar"/> - <include name="commons-compress.jar"/> - <include name="commons-lang.jar"/> - </fileset> - <fileset dir="../../deps/metrics-lib"> - <include name="descriptor.jar"/> + <fileset dir="${libs}"> + <include name="commons-codec-1.9.jar"/> + <include name="commons-compress-1.9.jar"/> + <include name="commons-lang-2.6.jar"/> + <include name="descriptor-1.2.0.jar"/> </fileset> </path> - <target name="metrics-lib"> - <ant dir="../../deps/metrics-lib"/> - </target> - - <target name="compile" depends="metrics-lib"> + <target name="compile"> <mkdir dir="${classes}"/> <javac destdir="${classes}" srcdir="${sources}" diff --git a/modules/clients/build.xml b/modules/clients/build.xml index f90e138..a5090b1 100644 --- a/modules/clients/build.xml +++ b/modules/clients/build.xml @@ -1,24 +1,19 @@ <project default="run" name="clients" basedir="."> <property name="sources" value="src"/> + <property name="libs" value="../../shared/lib"/> <property name="classes" value="classes"/> <path id="classpath"> <pathelement path="${classes}"/> - <fileset dir="/usr/share/java"> - <include name="commons-codec-1.6.jar"/> - <include name="commons-compress-1.4.1.jar"/> + <fileset dir="${libs}"> + <include name="commons-codec-1.9.jar"/> + <include name="commons-compress-1.9.jar"/> <include name="commons-lang-2.6.jar"/> - </fileset> - <fileset dir="../../deps/metrics-lib"> - <include name="descriptor.jar"/> + <include name="descriptor-1.2.0.jar"/> </fileset> </path> - <target name="metrics-lib"> - <ant dir="../../deps/metrics-lib"/> - </target> - - <target name="compile" depends="metrics-lib"> + <target name="compile"> <mkdir dir="${classes}"/> <javac destdir="${classes}" srcdir="${sources}" diff --git a/modules/collectdescs/build.xml b/modules/collectdescs/build.xml index bcbea76..3e7fbf0 100644 --- a/modules/collectdescs/build.xml +++ b/modules/collectdescs/build.xml @@ -1,23 +1,18 @@ <project default="run" name="collectdescs" basedir="."> <property name="sources" value="src"/> + <property name="libs" value="../../shared/lib"/> <property name="classes" value="classes"/> <path id="classpath"> <pathelement path="${classes}"/> - <fileset dir="/usr/share/java"> - <include name="commons-compress-1.4.1.jar"/> + <fileset dir="${libs}"> + <include name="commons-compress-1.9.jar"/> <include name="commons-lang-2.6.jar"/> - </fileset> - <fileset dir="../../deps/metrics-lib"> - <include name="descriptor.jar"/> + <include name="descriptor-1.2.0.jar"/> </fileset> </path> - <target name="metrics-lib"> - <ant dir="../../deps/metrics-lib"/> - </target> - - <target name="compile" depends="metrics-lib"> + <target name="compile"> <mkdir dir="${classes}"/> <javac destdir="${classes}" srcdir="${sources}" diff --git a/modules/connbidirect/build.xml b/modules/connbidirect/build.xml index 29629e8..2a1d138 100644 --- a/modules/connbidirect/build.xml +++ b/modules/connbidirect/build.xml @@ -2,25 +2,20 @@ <property name="connbidirect-sources" value="src/main/java"/> <property name="connbidirect-tests" value="src/test/java"/> + <property name="connbidirect-libs" value="../../shared/lib"/> <property name="connbidirect-classes" value="classes"/> <path id="classpath"> <pathelement path="${connbidirect-classes}"/> - <fileset dir="/usr/share/java"> + <fileset dir="${connbidirect-libs}"> <include name="commons-codec-1.6.jar"/> - <include name="commons-compress-1.4.1.jar"/> + <include name="commons-compress-1.9.jar"/> <include name="commons-lang-2.6.jar"/> - <include name="junit4.jar"/> - </fileset> - <fileset dir="../../deps/metrics-lib"> - <include name="descriptor.jar"/> + <include name="junit4-4.11.jar"/> + <include name="descriptor-1.2.0.jar"/> </fileset> </path> - <target name="metrics-lib"> - <ant dir="../../deps/metrics-lib"/> - </target> - - <target name="compile" depends="metrics-lib"> + <target name="compile"> <mkdir dir="${connbidirect-classes}"/> <javac destdir="${connbidirect-classes}" srcdir="${connbidirect-sources}" diff --git a/modules/disagreement/build.xml b/modules/disagreement/build.xml index ce1c022..aed679b 100644 --- a/modules/disagreement/build.xml +++ b/modules/disagreement/build.xml @@ -1,12 +1,13 @@ <project default="run" name="disagreement" basedir="."> <property name="disagreement-sources" value="src/main/java"/> + <property name="disagreement-libs" value="../../shared/lib"/> <property name="disagreement-classes" value="classes"/> <path id="classpath"> <pathelement path="${disagreement-classes}"/> - <fileset dir="../../shared/lib"> - <include name="descriptor-1.1.0.jar"/> - <include name="commons-compress-1.4.1.jar"/> + <fileset dir="${disagreement-libs}"> + <include name="descriptor-1.2.0.jar"/> + <include name="commons-compress-1.9.jar"/> <include name="xz-1.0.jar"/> </fileset> </path> diff --git a/modules/hidserv/build.xml b/modules/hidserv/build.xml index 7480b8c..50cb3b0 100644 --- a/modules/hidserv/build.xml +++ b/modules/hidserv/build.xml @@ -1,24 +1,19 @@ <project default="run" name="hidserv" basedir="."> <property name="sources" value="src"/> + <property name="libs" value="../../shared/lib"/> <property name="classes" value="classes"/> <path id="classpath"> <pathelement path="${classes}"/> - <fileset dir="/usr/share/java"> - <include name="commons-codec-1.6.jar"/> - <include name="commons-compress-1.4.1.jar"/> + <fileset dir="${libs}"> + <include name="commons-codec-1.9.jar"/> + <include name="commons-compress-1.9.jar"/> <include name="commons-lang-2.6.jar"/> - </fileset> - <fileset dir="../../deps/metrics-lib"> - <include name="descriptor.jar"/> + <include name="descriptor-1.2.0.jar"/> </fileset> </path> - <target name="metrics-lib"> - <ant dir="../../deps/metrics-lib"/> - </target> - - <target name="compile" depends="metrics-lib"> + <target name="compile"> <mkdir dir="${classes}"/> <javac destdir="${classes}" srcdir="${sources}" diff --git a/modules/legacy/build.xml b/modules/legacy/build.xml index 3ecf147..6cacf50 100644 --- a/modules/legacy/build.xml +++ b/modules/legacy/build.xml @@ -2,18 +2,17 @@  <property name="sources" value="src"/> + <property name="libs" value="../../shared/lib"/> <property name="classes" value="classes"/> <property name="config" value="etc"/> <path id="classpath"> <pathelement path="${classes}"/> - <fileset dir="/usr/share/java"> - <include name="commons-codec.jar"/> - <include name="commons-compress.jar"/> - <include name="postgresql-jdbc3.jar"/> - <include name="commons-lang.jar"/> - </fileset> - <fileset dir="../../deps/metrics-lib"> - <include name="descriptor.jar"/> + <fileset dir="${libs}"> + <include name="commons-codec-1.9.jar"/> + <include name="commons-compress-1.9.jar"/> + <include name="postgresql-jdbc3-9.2.jar"/> + <include name="commons-lang-2.6.jar"/> + <include name="descriptor-1.2.0.jar"/> </fileset> </path> @@ -23,13 +22,8 @@ <mkdir dir="${classes}"/> </target> -  - <target name="metrics-lib"> - <ant dir="../../deps/metrics-lib"/> - </target> -  - <target name="compile" depends="metrics-lib,init"> + <target name="compile" depends="init"> <javac destdir="${classes}" srcdir="${sources}" source="1.5" diff --git a/website/build.xml b/website/build.xml index a0ec33e..9f37d13 100644 --- a/website/build.xml +++ b/website/build.xml @@ -1,5 +1,7 @@ <project default="war" name="metrics-web" basedir="."> + <property name="libs" value="../shared/lib"/> +  <target name="compile"> <mkdir dir="classes"/> @@ -13,16 +15,14 @@ failonerror="true" includeantruntime="false"> <classpath> - <fileset dir="/usr/share/java"> - <include name="commons-codec.jar"/> - <include name="commons-lang.jar"/> - <include name="postgresql-jdbc3.jar"/> - <include name="servlet-api-2.5.jar"/> - <include name="gson-2.1.jar"/> - </fileset> - <fileset dir="lib"> + <fileset dir="${libs}"> + <include name="commons-codec-1.9.jar"/> + <include name="commons-lang-2.6.jar"/> + <include name="postgresql-jdbc3-9.2.jar"/> + <include name="servlet-api-3.0.jar"/> + <include name="gson-2.2.4.jar"/> <include name="REngine.jar"/> - <include name="RserveEngine.jar"/> + <include name="Rserve.jar"/> </fileset> </classpath> </javac> @@ -34,17 +34,15 @@ <war destfile="metrics.war" webxml="etc/web.xml"> <fileset dir="web"/> - <lib dir="lib"> - <include name="jstl.jar"/> + <lib dir="${libs}"> + <include name="jstl1.1-1.1.2.jar"/> <include name="REngine.jar"/> - <include name="RserveEngine.jar"/> - <include name="standard.jar"/> - </lib> - <lib dir="/usr/share/java"> - <include name="commons-codec.jar"/> - <include name="commons-lang.jar"/> - <include name="postgresql-jdbc3.jar"/> - <include name="gson-2.1.jar"/> + <include name="Rserve.jar"/> + <include name="standard-1.1.2.jar"/> + <include name="commons-codec-1.9.jar"/> + <include name="commons-lang-2.6.jar"/> + <include name="postgresql-jdbc3-9.2.jar"/> + <include name="gson-2.2.4.jar"/> </lib> <classes dir="classes"/> <zipfileset dir="etc" diff --git a/website/lib/REngine.jar b/website/lib/REngine.jar deleted file mode 100644 index ddf4059..0000000 Binary files a/website/lib/REngine.jar and /dev/null differ diff --git a/website/lib/RserveEngine.jar b/website/lib/RserveEngine.jar deleted file mode 100644 index 77b22ce..0000000 Binary files a/website/lib/RserveEngine.jar and /dev/null differ diff --git a/website/lib/jstl.jar b/website/lib/jstl.jar deleted file mode 100644 index 6b41358..0000000 Binary files a/website/lib/jstl.jar and /dev/null differ diff --git a/website/lib/standard.jar b/website/lib/standard.jar deleted file mode 100644 index 258daae..0000000 Binary files a/website/lib/standard.jar and /dev/null differ diff --git a/website/rserve/shutdown.sh b/website/rserve/shutdown.sh deleted file mode 100755 index dd5ecd8..0000000 --- a/website/rserve/shutdown.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/sh - -echo "library(Rserve) -c <- RSconnect() -RSshutdown(c)" | R --slave

1 0

[metrics-web/master] Upgrade to ggplot2 1.0.0.
by karsten＠torproject.org 05 Jul '16

05 Jul '16

commit 86c76738a5a3405d11c68d94063d1ce6a9fa656d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Jun 14 16:49:17 2016 +0200 Upgrade to ggplot2 1.0.0. --- website/rserve/graphs.R | 163 +++++++++++++++++++++++++------------------ website/rserve/rserve-init.R | 1 + 2 files changed, 95 insertions(+), 69 deletions(-) diff --git a/website/rserve/graphs.R b/website/rserve/graphs.R index d304395..9d1db7b 100644 --- a/website/rserve/graphs.R +++ b/website/rserve/graphs.R @@ -293,13 +293,14 @@ plot_networksize <- function(start, end, path) { colour = variable)) + geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "", limits = c(0, max(networksize$value, na.rm = TRUE))) + scale_colour_hue("", breaks = c("relays", "bridges"), labels = c("Relays", "Bridges")) + - opts(title = "Number of relays\n") + ggtitle("Number of relays\n") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -329,11 +330,12 @@ plot_relaycountries <- function(start, end, country, path) { geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "", limits = c(0, max(s$relays, na.rm = TRUE)), formatter = formatter) + - opts(title = title) + ggtitle(title) ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -364,14 +366,15 @@ plot_versions <- function(start, end, path) { geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "", limits = c(0, max(versions$relays, na.rm = TRUE))) + scale_colour_manual(name = "Tor version", values = colours[colours$breaks %in% visible_versions, 2], breaks = visible_versions) + - opts(title = "Relay versions\n") + ggtitle("Relay versions\n") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -392,14 +395,15 @@ plot_platforms <- function(start, end, path) { geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "", limits = c(0, max(platforms$value, na.rm = TRUE))) + scale_colour_manual(name = "Platform", breaks = c("Linux", "Darwin", "BSD", "Windows", "Other"), values = c("#E69F00", "#56B4E9", "#009E73", "#0072B2", "#333333")) + - opts(title = "Relay platforms\n") + ggtitle("Relay platforms\n") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -421,14 +425,16 @@ plot_bandwidth <- function(start, end, path) { geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "Bandwidth (Gbit/s)", limits = c(0, max(bandwidth$value, na.rm = TRUE) * 8 / 1e9)) + scale_colour_hue(name = "", h.start = 90, breaks = c("bwadv", "bwhist"), labels = c("Advertised bandwidth", "Bandwidth history")) + - opts(title = "Total relay bandwidth", legend.position = "top") + ggtitle("Total relay bandwidth") + + theme(legend.position = "top") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -466,14 +472,15 @@ plot_bwhist_flags <- function(start, end, path) { geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name="Bandwidth (Gbit/s)", limits = c(0, max(bw$value, na.rm = TRUE) * 8 / 1e9)) + scale_colour_manual(name = "", values = c("#E69F00", "#56B4E9", "#009E73", "#0072B2")) + - opts(title = "Bandwidth history by relay flags", - legend.position = "top") + ggtitle("Bandwidth history by relay flags") + + theme(legend.position = "top") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -494,15 +501,16 @@ plot_dirbytes <- function(start, end, path) { geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name="Bandwidth (Gbit/s)", limits = c(0, max(dir$value, na.rm = TRUE) * 8 / 1e9)) + scale_colour_hue(name = "", breaks = c("dirwrite", "dirread"), labels = c("Written dir bytes", "Read dir bytes")) + - opts(title = "Number of bytes spent on answering directory requests", - legend.position = "top") + ggtitle("Number of bytes spent on answering directory requests") + + theme(legend.position = "top") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -535,14 +543,15 @@ plot_relayflags <- function(start, end, flags, path) { colour = as.factor(variable))) + geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor, limits = as.Date(c(start, end))) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor, limits = as.Date(c(start, end))) + scale_y_continuous(name = "", limits = c(0, max(networksize$value, na.rm = TRUE))) + scale_colour_manual(name = "Relay flags", values = c("#E69F00", "#56B4E9", "#009E73", "#EE6A50", "#000000", "#0072B2"), breaks = flags, labels = flags) + - opts(title = "Number of relays with relay flags assigned\n") + ggtitle("Number of relays with relay flags assigned\n") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -581,16 +590,18 @@ plot_torperf <- function(start, end, source, filesize, path) { ymax = q3/1e3, fill = "ribbon")) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "", limits = c(0, maxY) / 1e3) + scale_fill_manual(name = paste("Measured times on", ifelse(source == "all", "all sources", source), "per day"), breaks = c("line", "ribbon"), labels = c("Median", "1st to 3rd quartile"), values = paste(colour, c("", "66"), sep = "")) + - opts(title = paste("Time in seconds to complete", filesizeStr, - "request"), legend.position = "top") + ggtitle(paste("Time in seconds to complete", filesizeStr, + "request")) + + theme(legend.position = "top") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -635,15 +646,17 @@ plot_torperf_failures <- function(start, end, source, filesize, path) { geom_point(size = 2) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + - scale_y_continuous(name = "", formatter = "percent") + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + + scale_y_continuous(name = "", labels = percent) + scale_colour_hue(name = paste("Problems encountered on", ifelse(source == "all", "all sources", source)), h.start = 45, breaks = c("timeouts", "failures"), labels = c("Timeouts", "Failures")) + - opts(title = paste("Timeouts and failures of", filesizeStr, - "requests"), legend.position = "top") + ggtitle(paste("Timeouts and failures of", filesizeStr, + "requests")) + + theme(legend.position = "top") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -664,12 +677,13 @@ plot_connbidirect <- function(start, end, path) { ggplot(c, aes(x = date, y = X0.5, colour = direction)) + geom_line(size = 0.75) + geom_ribbon(aes(x = date, ymin = X0.25, ymax = X0.75, - fill = direction), alpha = 0.5, legend = FALSE) + + fill = direction), alpha = 0.5, show_guide = FALSE) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + - scale_y_continuous(name = "", formatter = "percent") + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + + scale_y_continuous(name = "", labels = percent) + scale_colour_hue(name = "Medians and interquartile ranges", breaks = c("both", "write", "read"), labels = c("Both reading and writing", "Mostly writing", @@ -678,8 +692,8 @@ plot_connbidirect <- function(start, end, path) { breaks = c("both", "write", "read"), labels = c("Both reading and writing", "Mostly writing", "Mostly reading")) + - opts(title = "Fraction of connections used uni-/bidirectionally\n", - legend.position = "top") + ggtitle("Fraction of connections used uni-/bidirectionally\n") + + theme(legend.position = "top") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -735,14 +749,16 @@ plot_bandwidth_flags <- function(start, end, path) { geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name="Bandwidth (Gbit/s)", limits = c(0, max(bandwidth$value, na.rm = TRUE) * 8 / 1e9)) + scale_colour_manual(name = "", values = c("#E69F00", "#D6C827", "#009E73", "#00C34F")) + - opts(title = paste("Advertised bandwidth and bandwidth history by", - "relay flags"), legend.position = "top") + ggtitle(paste("Advertised bandwidth and bandwidth history by", + "relay flags")) + + theme(legend.position = "top") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -867,11 +883,12 @@ plot_userstats <- function(start, end, node, variable, value, events, geom_line(size = 1) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "", limits = c(0, max_y), - formatter = formatter) + - opts(title = title) + labels = formatter) + + ggtitle(title) if (length(value) > 1) { plot <- plot + scale_colour_hue(name = "", breaks = value, @@ -921,13 +938,14 @@ plot_advbwdist_perc <- function(start, end, p, path) { geom_line(size = 0.75) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "Advertised bandwidth in Gbit/s\n", limits = c(0, max(t$advbw, na.rm = TRUE))) + scale_colour_hue(name = "Percentile", breaks = rev(levels(t$percentile))) + - opts(title = "Advertised bandwidth distribution\n") + ggtitle("Advertised bandwidth distribution\n") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -948,12 +966,13 @@ plot_advbwdist_relay <- function(start, end, n, path) { geom_line(size = 0.75) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "Advertised bandwidth in Gbit/s\n", limits = c(0, max(t$advbw, na.rm = TRUE))) + scale_colour_hue(name = "n", breaks = levels(t$relay)) + - opts(title = "Advertised bandwidth of n-th fastest relays\n") + ggtitle("Advertised bandwidth of n-th fastest relays\n") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -967,14 +986,15 @@ plot_hidserv_dir_onions_seen <- function(start, end, path) { wiqm = ifelse(h$frac >= 0.01, h$wiqm, NA))) date_breaks <- date_breaks(as.numeric(max(h$date, na.rm = TRUE) - min(h$date, na.rm = TRUE))) - ggplot(h, aes(x = date, y = wiqm)) + + ggplot(h, aes(x = as.Date(date, origin = "1970-01-01"), y = wiqm)) + geom_line(size = 0.75) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "") + - opts(title = "Unique .onion addresses\n") + ggtitle("Unique .onion addresses\n") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -989,14 +1009,16 @@ plot_hidserv_rend_relayed_cells <- function(start, end, path) { wiqm = ifelse(h$frac >= 0.01, h$wiqm, NA))) date_breaks <- date_breaks(as.numeric(max(h$date, na.rm = TRUE) - min(h$date, na.rm = TRUE))) - ggplot(h, aes(x = date, y = wiqm * 8 * 512 / (86400 * 1e6))) + + ggplot(h, aes(x = as.Date(date, origin = "1970-01-01"), + y = wiqm * 8 * 512 / (86400 * 1e6))) + geom_line(size = 0.75) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + scale_y_continuous(name = "") + - opts(title = "Hidden-service traffic in Mbit/s\n") + ggtitle("Hidden-service traffic in Mbit/s\n") ggsave(filename = path, width = 8, height = 5, dpi = 72) } @@ -1012,20 +1034,23 @@ plot_hidserv_frac_reporting <- function(start, end, path) { frac = h$frac, type = h$type)) date_breaks <- date_breaks(as.numeric(max(h$date, na.rm = TRUE) - min(h$date, na.rm = TRUE))) - ggplot(h, aes(x = date, y = frac, colour = type)) + + ggplot(h, aes(x = as.Date(date, origin = "1970-01-01"), y = frac, + colour = type)) + geom_line(size = 0.75) + geom_hline(yintercept = 0.01, linetype = 2) + scale_x_date(name = paste("\nThe Tor Project - ", "https://metrics.torproject.org/", sep = ""), - format = date_breaks$format, major = date_breaks$major, - minor = date_breaks$minor) + - scale_y_continuous(name = "", formatter = "percent") + + labels = date_format(date_breaks$format), + breaks = date_breaks$major, + minor_breaks = date_breaks$minor) + + scale_y_continuous(name = "", labels = percent) + scale_colour_hue(name = "", breaks = c("rend-relayed-cells", "dir-onions-seen"), labels = c("Hidden-service traffic", "Unique .onion addresses")) + - opts(title = paste("Fraction of relays reporting hidden-service", - "statistics"), legend.position = "top") + ggtitle(paste("Fraction of relays reporting hidden-service", + "statistics")) + + theme(legend.position = "top") ggsave(filename = path, width = 8, height = 5, dpi = 72) } diff --git a/website/rserve/rserve-init.R b/website/rserve/rserve-init.R index 4c6fa10..004d870 100644 --- a/website/rserve/rserve-init.R +++ b/website/rserve/rserve-init.R @@ -7,6 +7,7 @@ library("reshape") library("plyr") library("digest") library("RColorBrewer") +library("scales") source('graphs.R') source('tables.R')

1 0

[metrics-web/master] Additions to task-19540.
by karsten＠torproject.org 05 Jul '16

05 Jul '16

commit 840a889e3c8ed919d589d0e3eb4edc7999b391b6 Author: iwakeh <iwakeh(a)torproject.org> Date: Mon Jul 4 17:37:11 2016 +0000 Additions to task-19540. --- HACKING | 13 ++----- LICENSE | 2 +- README | 126 +++++++++++++++++++++++++++++++++++++++++----------------------- 3 files changed, 86 insertions(+), 55 deletions(-) diff --git a/HACKING b/HACKING index 1a05f41..f981750 100644 --- a/HACKING +++ b/HACKING @@ -5,16 +5,9 @@ General: - Files end with two newlines. Java code: -- Indentation is two spaces in general and four spaces for continued lines. -- Imports are ordered alphabetically. Imports with different top-level - package names or libraries (java, javax, org) are separated by an - additional newline. -- Don't catch Exception, but the specific exceptions that can be thrown - and that you can handle. Don't catch RuntimeException at all, but fix - the code so that it doesn't throw RuntimeExceptions. -- Write comments as full sentences as in /* This is a comment. */ -- Don't write your own util classes. Make use of Apache Commons instead. -- Qualify instance attributes and instance methods using this. +Please refer to the Metrics Team Java Style Guide: +https://trac.torproject.org/projects/tor/wiki/org/teams/MetricsTeam/MetricsJavaStyleGuide#CodingStyle + SQL code: - All SQL keywords and PostgreSQL functions are written in capital diff --git a/LICENSE b/LICENSE index e1c60eb..a7dc2b4 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright 2010 The Tor Project +Copyright 2010--2016 The Tor Project Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are diff --git a/README b/README index 06204bd..1bd3d8f 100644 --- a/README +++ b/README @@ -29,21 +29,19 @@ This README describes the steps for installing metrics-web on a Debian GNU/Linux Jessie server. Instructions for other operating systems may vary. -In the following it is assumed that root privileges are available. -Commands requiring root privileges will be prefixed with # below. +In the following it is assumed that sudo (or root) privileges are available. Start by adding a metrics user that will be used to execute all commands -that do not require root privileges. These commands will be prefixed with -$ below. +that do not require root privileges. -# adduser metrics +$ sudo adduser metrics The database importer and website sources will be installed in /srv/metrics.torproject.org/ that is created as follows: -# mkdir /srv/metrics.torproject.org/ -# chmod g+ws /srv/metrics.torproject.org/ -# chown metrics:metrics /srv/metrics.torproject.org/ +$ sudo mkdir /srv/metrics.torproject.org/ +$ sudo chmod g+ws /srv/metrics.torproject.org/ +$ sudo chown metrics:metrics /srv/metrics.torproject.org/ Clone the metrics-web Git repository: @@ -53,7 +51,13 @@ $ git clone git://git.torproject.org/metrics-web metrics Install OpenJDK 7, Ant 1.9.4, and PostgreSQL 9.4 that are necessary for setting up the metrics database. -# apt-get install openjdk-7-jdk ant postgresql-9.4 +$ sudo apt-get install openjdk-7-jdk ant postgresql-9.4 + +Setting up the graphing engine (cf. 2.) requires installing R 2.8 or +higher as well as the ggplot2 library. + +$ sudo apt-get install r-base r-cran-rserve r-cran-ggplot2 r-cran-reshape \ + r-cran-scales r-cran-java Check the versions of the newly installed tools. @@ -68,6 +72,47 @@ Apache Ant(TM) version 1.9.4 compiled on October 7 2014 $ psql --version psql (PostgreSQL) 9.4.8 +Now prepare the library folder for all ant projects. + +$ cd /srv/metrics.torproject.org/metrics/ +$ mkdir shared/lib + +Download .jar files listed below. Metrics usually uses Debian stable +provided libraries, but you can also just download them elsewhere. + +Copy or link the following jars, annotated with file names in Debian +stable packages, to /srv/metrics.torproject.org/metrics/shared/lib: +commons-codec-1.9.jar + [/usr/share/java/commons-codec-1.9.jar in libcommons-codec-java] +commons-compress-1.9.jar + [/usr/share/java/commons-compress-1.9.jar in libcommons-compress-java] +commons-lang-2.6.jar + [/usr/share/java/commons-lang-2.6.jar in libcommons-lang-java] +gson-2.2.4.jar + [/usr/share/java/gson.jar in libgoogle-gson-java] +jstl1.1-1.1.2.jar + [/usr/share/java/jstl1.1-1.1.2.jar in libjstl1.1-java] +junit4-4.11.jar + [/usr/share/java/junit4-4.11.jar in junit4] +postgresql-jdbc3-9.2.jar + [/usr/share/java/postgresql-jdbc3-9.2.jar in libpostgresql-jdbc-java] +REngine.jar + [/usr/lib/R/site-library/Rserve/java/REngine.jar in r-cran-rserve] +Rserve.jar + [/usr/lib/R/site-library/Rserve/java/Rserve.jar in r-cran-rserve] +servlet-api-3.0.jar + [/usr/share/java/servlet-api-3.0.jar in libservlet3.0-java] +standard-1.1.2.jar + [/usr/share/java/standard-1.1.2.jar in libjakarta-taglibs-standard-java] +xz-1.5.jar + [/usr/share/java/xz-1.5.jar in libxz-java] + +DescripTor is provided by The Tor Project and can be found here: + https://dist.torproject.org/descriptor/ +Download the tar.gz file with the version number listed in build.xml. +The README inside the tar.gz file has all the information about DescripTor +and explains how to verify the downloaded files. +Copy descriptor-<version>.jar to /srv/metrics.torproject.org/shared/lib 1.2. Configuring the database ============================= @@ -77,22 +122,22 @@ PostgreSQL database and import a database schema. Start by creating a new metrics database user. There is no need to give the metrics user superuser privileges or allow it to create databases or -new roles. +new roles. You will be prompted for the password. -# sudo -u postgres createuser -P metrics +$ sudo -u postgres createuser -P metrics Create a new database tordir owned by user metrics. -# sudo -u postgres createdb -O metrics tordir +$ sudo -u postgres createdb -O metrics tordir Import the metrics database schema. -$ psql -f /srv/metrics.torproject.org/db/tordir.sql tordir +$ sudo -u metrics psql -f /srv/metrics.torproject.org/metrics/modules/legacy/tordir.sql tordir Confirm that the database now contains tables to hold metrics data. In the following, => will be used as the database prompt. -$ psql tordir +$ sudo -u metrics psql tordir => \dt+ => \q @@ -103,10 +148,10 @@ $ psql tordir In most cases it makes sense to populate the metrics database with archived relay descriptors from the official metrics website. -Download the relay descriptor tarballs from the metrics website at -https://metrics.torproject.org/data.html#relaydesc and extract them to -/srv/metrics.torproject.org/archives/ . The database importer can process -v3 votes, v3 consensuses, server descriptors, and extra-infos. +Download the relay descriptor tarballs from the CollecTor website at +https://collector.torproject.org/archive/relay-descriptors/ +and extract them to /srv/metrics.torproject.org/archives/ . The database +importer can process v3 votes, v3 consensuses, server descriptors, and extra-infos. Edit the config file ~/metrics-web/config (or create it if it's not there) to contain the following five lines (be sure to remove the linebreak in @@ -121,13 +166,13 @@ RelayDescriptorDatabaseJDBC Compile and run the Java database importer. -$ cd /srv/metrics.torproject.org/ -$ ./run.sh +$ cd /srv/metrics.torproject.org/metrics/ +$ ./run-web.sh The database import will take a while. Once it's complete, check that the database tables now contain metrics data: -$ psql tordir +$ sudo -u metrics psql tordir => \dt+ => \q @@ -143,7 +188,7 @@ experimental feature. In a future version of metrics-web it may also be possible to update local relay descriptor tarballs from the official metrics server via rsync and import only the changes into the metrics database. The idea is to simply -rsync the data/ directory from the metrics server and have all information +rsync the data/ directory from the CollecTor server and have all information available. However, this feature is not implemented yet. @@ -172,7 +217,7 @@ $ ln -s /srv/tor/cached-* . Add a crontab entry for the database importer to run once per hour: -15 * * * * cd /srv/metrics.torproject.org/ && ./run.sh +15 * * * * cd /srv/metrics.torproject.org/metrics/ && ./run-web.sh 1.5. Pre-calculating relay statistics @@ -185,7 +230,7 @@ sufficient to pre-calculate statistics 2 or 4 times a day. Calculate statistics manually after large imports (this may take a while): -$ psql tordir -c 'SELECT * FROM refresh_all();' +$ sudo -u metrics psql tordir -c 'SELECT * FROM refresh_all();' If the metrics database gets updated automatically, write a script and add a crontab entry for pre-calculating statistics every 6 or 12 hours. @@ -199,8 +244,8 @@ and bridge usage. These statistics are added by parsing sanitized bridge descriptors available on the official metrics website. Download a sanitized bridge descriptor tarball from the metrics website at -https://metrics.torproject.org/data.html#bridgedesc and extract it to, -e.g., /srv/metrics.torproject.org/bridges/bridge-descriptors-2011-05/ . +https://collector.torproject.org/archive/bridge-descriptors/ and extract +it to, e.g., /srv/metrics.torproject.org/bridges/bridge-descriptors-2011-05 Edit /srv/metrics.torproject.org/config to contain the following options: @@ -218,7 +263,7 @@ import all relay descriptors once again. Run the database import: -$ ./run.sh +$ ./run-web.sh 1.7. Importing Torperf performance data @@ -229,7 +274,7 @@ it. Torperf's measurement data are available on the metrics website and can be imported into the metrics database, too. Download the Torperf measurement files from the metrics website at -https://metrics.torproject.org/data.html#performance and put them in a +https://collector.torproject.org/archive/torperf/ and put them in a subdirectory, e.g., /srv/metrics.torproject.org/torperf/ . Edit /srv/metrics.torproject.org/config to contain the following options: @@ -239,7 +284,7 @@ TorperfDirectory torperf/ Run the database import: -$ ./run.sh +$ ./run-web.sh 2. Installing the graphing engine @@ -256,13 +301,6 @@ saves time when processing user requests. In this configuration, Rserve will run in the context of the metrics user. -Setting up the graphing engine requires installing PostgreSQL's header -files and R 2.8 or higher. R 2.8 or higher is required for the ggplot2 -library. - -# apt-get install r-base r-cran-rserve r-cran-ggplot2 r-cran-reshape \ - r-cran-scales - Start Rserve, this time with the metrics-web-specific configuration that includes pre-loading the graph code: @@ -299,16 +337,16 @@ Apache Tomcat. Start by installing Apache 2: -# apt-get install apache2 +$ sudo apt-get install apache2 Disable Apache's default site. -# a2dissite 000-default +$ sudo a2dissite 000-default Enable mod_rewrite to tell Apache where to find static resources on disk. Also enable mod_proxy to forward requests to Tomcat. -# a2enmod rewrite proxy_http +$ sudo a2enmod rewrite proxy_http Create a new virtual host configuration and store it in a new file /etc/apache2/sites-available/metrics.torproject.org with the following @@ -333,11 +371,11 @@ content: Enable the new virtual host. -# a2ensite metrics.torproject.org +$ sudo a2ensite metrics.torproject.org Restart Apache just to be sure that all changes are effective. -# /etc/init.d/apache2 restart +$ sudo service apache2 restart 3.2. Configuring Apache Tomcat @@ -348,7 +386,7 @@ pages and graphs. Install Tomcat 8: -# apt-get install tomcat8 +$ sudo apt-get install tomcat8 Replace Tomcat's default configuration in /etc/tomcat8/server.xml with the following configuration: @@ -390,7 +428,7 @@ $ ant war Create a symbolic link to the ernie.war file: -# ln -s /srv/metrics.torproject.org/metrics/website/metrics.war /var/lib/tomcat8/webapps/ +$ sudo ln -s /srv/metrics.torproject.org/metrics/website/metrics.war /var/lib/tomcat8/webapps/ Tomcat will now attempt to deploy the web application automatically. @@ -399,7 +437,7 @@ file and Tomcat will reload the web application automatically. Restart Tomcat to make all configuration changes effective: -# service tomcat8 restart +$ sudo service tomcat8 restart The metrics website should now work.

1 0

[webwml/master] Direct RPM users to Fedora/EPEL packages
by hiviah＠torproject.org 05 Jul '16

05 Jul '16

commit 5134510306dddc923a792e7e26d896564433ae2c Author: Ondrej Mikle <ondrej.mikle(a)gmail.com> Date: Tue Jul 5 22:09:31 2016 +0200 Direct RPM users to Fedora/EPEL packages --- docs/en/rpms.wml | 100 ++----------------------------------------------------- 1 file changed, 3 insertions(+), 97 deletions(-) diff --git a/docs/en/rpms.wml b/docs/en/rpms.wml index 158f9fa..7d9c3f3 100644 --- a/docs/en/rpms.wml +++ b/docs/en/rpms.wml @@ -15,106 +15,12 @@ linux distributions.</a></h2> <br> - <p><strong>Do not</strong> use the packages in the native repositories. They are - frequently out of date. That means you'll be missing stability and - security fixes. - </p> - - <p> - You'll need to set up our package repository before you can fetch - Tor. Repositories contain i686 and x86_64 builds. Assuming yum, in - /etc/yum.repos.d/, create a file called - torproject.repo. Edit this file with the following information: - </p> - - <h3>Fedora 22/23 and EL6/7 packages</h3> + <h3>Fedora, RHEL, CentOS, Scientific Linux packages</h3> - <p>For Fedora 22, Fedora 23, RHEL 6, RHEL 7 (and clones), use following - repo file - substitute DISTRIBUTION with one of the following: fc/22, fc/23, - el/6, el/7 according to your distribution. - </p> - -<pre>[tor] -name=Tor repo -enabled=1 -baseurl=https://deb.torproject.org/torproject.org/rpm/DISTRIBUTION/$basearch/ -gpgcheck=1 -gpgkey=https://deb.torproject.org/torproject.org/rpm/RPM-GPG-KEY-torproject.org.asc -repo_gpgcheck=1 - -[tor-source] -name=Tor source repo -enabled=1 -autorefresh=0 -baseurl=https://deb.torproject.org/torproject.org/rpm/DISTRIBUTION/SRPMS -gpgcheck=1 -gpgkey=https://deb.torproject.org/torproject.org/rpm/RPM-GPG-KEY-torproject.org.asc -repo_gpgcheck=1 -</pre> - <p> - The key's fingerprint should be: + <p>Use native Fedora packages for Fedora distribution or <a href="https://fedoraproject.org/wiki/EPEL">EPEL</a> + packages for distribitons derived from RHEL. </p> -<pre>E273 44C4 BD24 BEDF E4F4 C741 803F EFB7 F4B8 5E0F</pre> - <h3>Name clash warning</h3> - - <p>There is identically named 'tor' package in the EPEL and Fedora - repositories. Due to this unfortunate name clash, the packages might - "update over one another", depending on patchversion. Solution is to use - <tt>Exclude=tor</tt> line in the relevant <tt>/etc/yum.repos.d/</tt> - file. - </p> - - <p>E.g. you may exclude EPEL's Tor package by putting the mentioned exclude - line under [epel] section in <tt>/etc/yum.repos.d/epel.repo</tt>. - For Fedora, you usually need that exclude in fedora.repo and - fedora-updates.repo file. - </p> - - <h3>Package installation and running</h3> - - <p> - Once you have the repo file, install Tor using (yum will ask about the - key's fingerprint the first time): - </p> -<pre>yum install tor</pre> - - <p> - Start Tor using: - </p> -<pre>service tor start</pre> - - <p> - Now Tor is installed and running. Move on to <a href="<page - docs/tor-doc-unix>#using">step two</a> of the "Tor on Linux/Unix" - instructions. - </p> - - <p style="font-size: small"> - The DNS name <code>deb.torproject.org</code> is actually a set of independent - servers in a DNS round-robin configuration. If for some reason you cannot - use it, you might be able to access one of the individual servers - instead. Try - <code>deb-master.torproject.org</code>, - <code>mirror.netcologne.de</code> or - <code>tor.mirror.youam.de</code>. - </p> - - <hr> - - <hr> - - <h2>Alpha/development packages</a></h2> - <br> - - <p> - There is a repository that contains packages for latest alpha branch of Tor. - If you'd like to experiment with them, use one of the above repo files, just - change the baseurl to (similarly to previous section, substitute - DISTRIBUTION based on your distribution; for source rpms use "SRPMS" instead - of $basearch): - </p> -<pre>baseurl=https://deb.torproject.org/torproject.org/rpm/tor-testing/DISTRIBUT…</pre> <a id="source"></a> <h2><a class="anchor" href="#source">Building from source</a></h2>

1 0

[translation/https_everywhere_completed] Update translations for https_everywhere_completed
by translation＠torproject.org 05 Jul '16

05 Jul '16

commit 4cb016e8b3851af30c8fdcb2441252c9e3c80668 Author: Translation commit bot <translation(a)torproject.org> Date: Tue Jul 5 19:15:37 2016 +0000 Update translations for https_everywhere_completed --- fr_CA/ssl-observatory.dtd | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fr_CA/ssl-observatory.dtd b/fr_CA/ssl-observatory.dtd index 1b4656a..3b27b3f 100644 --- a/fr_CA/ssl-observatory.dtd +++ b/fr_CA/ssl-observatory.dtd @@ -13,7 +13,7 @@ and auditing the certificates being presented to your browser. Would you like to turn it on?">--> <!ENTITY ssl-observatory.popup.title -"HTTPS partout devrait-il utiliser l'Observatoire SSL?"> +"HTTPS partout devrait-il utiliser l'observatoire SSL?"> <!ENTITY ssl-observatory.popup.yes "Oui"> @@ -27,10 +27,10 @@ un réseau d'entreprise très intrusif:"> "Sécuritaire, sauf si vous utilisez un réseau d'entreprise avec noms de serveurs Intranet secrets:"> <!ENTITY ssl-observatory.prefs.alt_roots -"Envoyer et vérifier les certificats signés par des autorités racines non-standard"> +"Envoyer et vérifier les certificats signés par des autorités racines non standards"> <!ENTITY ssl-observatory.prefs.alt_roots_tooltip -" Il est sécuritaire (et une bonne idée) d'activer cette option, sauf si vous utilisez un réseau d'entreprise intrusif ou le logiciel antivirus Kaspersky qui surveille votre navigation avec un mandataire TLS et une autorité de certificat racine privée. Si elle est activée sur un tel réseau, cette option pourrait publier des preuves de quels domaines https:// ont été visités à travers ce mandataire, de par les certificats uniques qu'elle produirait. Alors nous la désactivons par défaut"> +" Il est sécuritaire (et une bonne idée) d'activer cette option, sauf si vous utilisez un réseau d'entreprise intrusif ou le logiciel antivirus Kaspersky qui surveille votre navigation avec un mandataire TLS et une autorité de certificat racine privée. Si elle était activée sur un tel réseau, cette option pourrait publier des preuves sur les domaines https:// ont été visités avec ce mandataire en raison certificats uniques qu'elle produirait. Alors nous la désactivons par défaut"> <!ENTITY ssl-observatory.prefs.anonymous "Vérifier les certificats en utilisant Tor pour l'anonymat"> <!ENTITY ssl-observatory.prefs.anonymous_unavailable @@ -76,7 +76,7 @@ Mouseover the options for further details:">--> "Nous essaierons encore de garder les données anonymes, mais cette option est moins sécuritaire"> <!ENTITY ssl-observatory.prefs.priv_dns -"Envoyer et vérifier les certificats des noms de DNS non-publics "> +"Envoyer et vérifier les certificats des noms de DNS non publics "> <!ENTITY ssl-observatory.prefs.priv_dns_tooltip "Au moins que cette option est activée, l'observatoire n'enregistrera pas les certificats pour les noms qu'il ne peut pas résoudre par le système DNS."> @@ -93,6 +93,6 @@ Mouseover the options for further details:">--> <!ENTITY ssl-observatory.warning.defense "Si vous êtes connecté à ce site, il serait conseillé de changer votre mot de passe dès que vous avez une connexion sécuritaire. (Ces avertissements peuvent être désactivés dans l'onglet « Observatoire SSL » de la fenêtre de préférences de HTTPS partout)."> <!ENTITY ssl-observatory.prefs.self_signed -"Envoyer et vérifier les certificats auto-signés"> +"Envoyer et vérifier les certificats autosignés"> <!ENTITY ssl-observatory.prefs.self_signed_tooltip -"Ceci est recommandé; les problèmes de chiffrement sont particulièrement communs dans les dispositifs embarqués auto-signés"> +"Ceci est recommandé; les problèmes de chiffrement sont particulièrement communs dans les dispositifs embarqués autosignés">

1 0