commit 0628a193d3b33b37b8fe44a03c0d39d4d5f071a7 Author: Kevin Butler haqkrs@gmail.com Date: Wed Sep 18 02:40:06 2013 +0100

Added some ideas on extensibility --- 2013/torperf2/torperf2.tex | 51 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 3 deletions(-)

diff --git a/2013/torperf2/torperf2.tex b/2013/torperf2/torperf2.tex index 029d8a1..ceb5677 100644 --- a/2013/torperf2/torperf2.tex +++ b/2013/torperf2/torperf2.tex @@ -120,6 +120,24 @@ What's not easily possible with this approach is to configure client and server to run on different hosts. Is there an experiment where this might be useful?

+\subsubsection{Experiment extensibility considerations} + +It should be easy for a user to implement or install an experiment that isn't +bundled with the core distribution. Installing an experiment should be as +simple as unzipping a folder or config file into an experiments folder. +Optimally, the Torperf service would detect the new experiment and schedule +it as soon as practical without additional user intervention. + +For implementing, this will require a clearly defined interface for developers +to work with. +\footnote{Ideally, this interface would allow experiment implementations in +any language, not only the language of choice for the core Torperf service.} +This could be as simple as conforming to a common data format or a tight +integration with the core service components which could allow lighter weight +experiment implementations. If the experiment reuses the services +file server implementation, the service should post-process the result +file(s) to add any additional serverside timing information available. + \subsubsection{User-defined tor version or binary}

A key part of measurements is the tor software version or binary used to @@ -128,8 +146,22 @@ It should be easy to specify a tor version that is not the current tor version shipped with the OS. Similarly, it should be easy to point to a custom tor binary to use for measurements. + It should be possible to run different experiments with different tor versions or binaries in the same Torperf service instance. +An experiment should be able to be configured to run for multiple +versions of tor, potentially simultaneously, to spot performance change +across tor releases. Note that the tor version should be contained in the +results. +\footnote{Loosely specified tor versions, e.g. 'latest-stable, git:master, +\textasciitilde>0.2.4', +would be a boon in reducing error prone configuration updates everytime +there are new tor versions released.} + +It might be beneficial to provide a mechanism to download and verify the +signature of new tor versions as they are released. The user could speficy +if they plan to test stable, beta or alpha versions of tor with their +torperf instance.

\subsubsection{User-defined third-party software version or binary}

@@ -137,7 +169,7 @@ Similar to the previous requirement, it should be easy to specify custom versions or binaries of third-party software other than the version currently shipped with the OS. This applies, for example, to Firefox when attempting to make measurements -more reaslistic. +more realistic.

\subsection{Results requirements}

@@ -175,10 +207,14 @@ plausible, too. \subsubsection{Results web interface}

Torperf should provide all its results via a public web interface. +\footnote{It is a requirement to also provide a machine readable representation +to enable easier development of custom visualisations of the result data.} This includes measured timestamps as well as any meta data collected in an experiment. Results should be available for all measurements as well as for a subset -specified by measurement time or filtered by other criteria. +specified by measurement time or filtered by other criteria. The interface +should highlight experiment failures and provide a convenient mechanism to +run a set of experiments on demand.

\subsubsection{Results accumulator}

@@ -216,7 +252,8 @@ codebases, which is suboptimal.} \subsubsection{Results parsing library}

The new Torperf should come with an easy-to-use library to process its -results. +results. The library should also make it simple to create Torperf compatible +results or clearly highlight areas of non conformance with existing results. Alternatively, this library could be provided and maintained as part of stem or another parsing library.

@@ -374,6 +411,14 @@ client processes, using a previously configured tor version or binary. \item[tor controller] Connect to tor's control port, force-set guards if required by the experiment, register for and handle asynchronous events, set up and tear down hidden services. +\item[experiment scheduler] Run experiments following a schedule. Must also +handle immediate scheduling requests by users. +\item[experiment runner] Handle the execution of a scheduled experiment. It +should take care of the entire experiments lifecycle and reserve an +exclusive tor instance for its lifetime. Finally, collect the results +and post-process them (if applicable) before saving the data to the +results database.If the experiment fails it should save as much information +as possible for future inspection. \item[request scheduler] Start new requests following a previously configured schedule. \item[request runner] Handle a single request from creation over various