[tor-commits] [tech-reports/master] Added some ideas on extensibility

[karsten at torproject.org]

commit 0628a193d3b33b37b8fe44a03c0d39d4d5f071a7
Author: Kevin Butler <haqkrs at gmail.com>
Date:   Wed Sep 18 02:40:06 2013 +0100

    Added some ideas on extensibility
---
 2013/torperf2/torperf2.tex |   51 +++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 48 insertions(+), 3 deletions(-)

diff --git a/2013/torperf2/torperf2.tex b/2013/torperf2/torperf2.tex
index 029d8a1..ceb5677 100644
--- a/2013/torperf2/torperf2.tex
+++ b/2013/torperf2/torperf2.tex
@@ -120,6 +120,24 @@ What's not easily possible with this approach is to configure client and
 server to run on different hosts.  Is there an experiment where this
 might be useful?
 
+\subsubsection{Experiment extensibility considerations}
+
+It should be easy for a user to implement or install an experiment that isn't
+bundled with the core distribution. Installing an experiment should be as
+simple as unzipping a folder or config file into an experiments folder.
+Optimally, the Torperf service would detect the new experiment and schedule
+it as soon as practical without additional user intervention.
+
+For implementing, this will require a clearly defined interface for developers
+to work with.
+\footnote{Ideally, this interface would allow experiment implementations in
+any language, not only the language of choice for the core Torperf service.}
+This could be as simple as conforming to a common data format or a tight
+integration with the core service components which could allow lighter weight
+experiment implementations. If the experiment reuses the services
+file server implementation, the service should post-process the result
+file(s) to add any additional serverside timing information available.
+
 \subsubsection{User-defined tor version or binary}
 
 A key part of measurements is the tor software version or binary used to
@@ -128,8 +146,22 @@ It should be easy to specify a tor version that is not the current tor
 version shipped with the OS.
 Similarly, it should be easy to point to a custom tor binary to use for
 measurements.
+
 It should be possible to run different experiments with different tor
 versions or binaries in the same Torperf service instance.
+An experiment should be able to be configured to run for multiple
+versions of tor, potentially simultaneously, to spot performance change
+across tor releases. Note that the tor version should be contained in the
+results.
+\footnote{Loosely specified tor versions, e.g. 'latest-stable, git:master,
+\textasciitilde>0.2.4',
+would be a boon in reducing error prone configuration updates everytime
+there are new tor versions released.}
+
+It might be beneficial to provide a mechanism to download and verify the
+signature of new tor versions as they are released. The user could speficy
+if they plan to test stable, beta or alpha versions of tor with their
+torperf instance.
 
 \subsubsection{User-defined third-party software version or binary}
 
@@ -137,7 +169,7 @@ Similar to the previous requirement, it should be easy to specify custom
 versions or binaries of third-party software other than the version
 currently shipped with the OS.
 This applies, for example, to Firefox when attempting to make measurements
-more reaslistic.
+more realistic.
 
 \subsection{Results requirements}
 
@@ -175,10 +207,14 @@ plausible, too.
 \subsubsection{Results web interface}
 
 Torperf should provide all its results via a public web interface.
+\footnote{It is a requirement to also provide a machine readable representation
+to enable easier development of custom visualisations of the result data.}
 This includes measured timestamps as well as any meta data collected in
 an experiment.
 Results should be available for all measurements as well as for a subset
-specified by measurement time or filtered by other criteria.
+specified by measurement time or filtered by other criteria. The interface
+should highlight experiment failures and provide a convenient mechanism to
+run a set of experiments on demand.
 
 \subsubsection{Results accumulator}
 
@@ -216,7 +252,8 @@ codebases, which is suboptimal.}
 \subsubsection{Results parsing library}
 
 The new Torperf should come with an easy-to-use library to process its
-results.
+results. The library should also make it simple to create Torperf compatible
+results or clearly highlight areas of non conformance with existing results.
 Alternatively, this library could be provided and maintained as part of
 stem or another parsing library.
 
@@ -374,6 +411,14 @@ client processes, using a previously configured tor version or binary.
 \item[tor controller] Connect to tor's control port, force-set guards if
 required by the experiment, register for and handle asynchronous events,
 set up and tear down hidden services.
+\item[experiment scheduler] Run experiments following a schedule. Must also
+handle immediate scheduling requests by users.
+\item[experiment runner] Handle the execution of a scheduled experiment. It
+should take care of the entire experiments lifecycle and reserve an
+exclusive tor instance for its lifetime. Finally, collect the results
+and post-process them (if applicable) before saving the data to the
+results database.If the experiment fails it should save as much information
+as possible for future inspection.
 \item[request scheduler] Start new requests following a previously
 configured schedule.
 \item[request runner] Handle a single request from creation over various