commit 8142b3e91685e679d4edc4da0a3097ed6ffba125 Author: Karsten Loesing karsten.loesing@gmx.net Date: Wed Sep 18 12:04:02 2013 +0200
Tweak Kevin's suggestions, comment out some of them. --- 2013/torperf2/torperf2.tex | 63 ++++++++++++++++++++++++++++---------------- 1 file changed, 40 insertions(+), 23 deletions(-)
diff --git a/2013/torperf2/torperf2.tex b/2013/torperf2/torperf2.tex index ceb5677..abc9c39 100644 --- a/2013/torperf2/torperf2.tex +++ b/2013/torperf2/torperf2.tex @@ -123,20 +123,29 @@ might be useful? \subsubsection{Experiment extensibility considerations}
It should be easy for a user to implement or install an experiment that isn't -bundled with the core distribution. Installing an experiment should be as +bundled with the core distribution. Ideally, installing an experiment should be as simple as unzipping a folder or config file into an experiments folder. Optimally, the Torperf service would detect the new experiment and schedule it as soon as practical without additional user intervention.
-For implementing, this will require a clearly defined interface for developers -to work with. -\footnote{Ideally, this interface would allow experiment implementations in -any language, not only the language of choice for the core Torperf service.} -This could be as simple as conforming to a common data format or a tight -integration with the core service components which could allow lighter weight -experiment implementations. If the experiment reuses the services -file server implementation, the service should post-process the result -file(s) to add any additional serverside timing information available. +% Commented out Kevin's text by Karsten on 2013-09-18: +% For implementing, this will require a clearly defined interface for developers +% to work with. +% \footnote{Ideally, this interface would allow experiment implementations in +% any language, not only the language of choice for the core Torperf service.} +% This could be as simple as conforming to a common data format or a tight +% integration with the core service components which could allow lighter weight +% experiment implementations. If the experiment reuses the service's +% file server implementation, the service should post-process the result +% file(s) to add any additional server-side timing information available. +% +% Reason: We're not trying to build a generic experiment runner, but a +% specific Tor performance measurement tool. Most experiments will depend +% on various Torperf components to start a tor process and parse tor +% events to storing results in the database. There's no (reasonable) way +% for those experiments to interface with Torperf components if they don't +% speak Torperf's programming language. If people want to add a simple +% experiment, they can write a wrapper for using their tool in Torperf.
\subsubsection{User-defined tor version or binary}
@@ -148,20 +157,20 @@ Similarly, it should be easy to point to a custom tor binary to use for measurements.
It should be possible to run different experiments with different tor -versions or binaries in the same Torperf service instance. +versions or binaries in the same Torperf service instance.% +\footnote{Loosely specified tor versions, e.g. 'latest-stable, git:master, +$\geq$ 0.2.4' (following a well-defined schema), +would be a boon in reducing error prone configuration updates everytime +there are new tor versions released.} An experiment should be able to be configured to run for multiple versions of tor, potentially simultaneously, to spot performance change across tor releases. Note that the tor version should be contained in the results. -\footnote{Loosely specified tor versions, e.g. 'latest-stable, git:master, -\textasciitilde>0.2.4', -would be a boon in reducing error prone configuration updates everytime -there are new tor versions released.}
It might be beneficial to provide a mechanism to download and verify the signature of new tor versions as they are released. The user could speficy if they plan to test stable, beta or alpha versions of tor with their -torperf instance. +Torperf instance.
\subsubsection{User-defined third-party software version or binary}
@@ -206,15 +215,18 @@ plausible, too.
\subsubsection{Results web interface}
-Torperf should provide all its results via a public web interface. +Torperf should provide all its results via a public web interface.% \footnote{It is a requirement to also provide a machine readable representation to enable easier development of custom visualisations of the result data.} This includes measured timestamps as well as any meta data collected in an experiment. Results should be available for all measurements as well as for a subset specified by measurement time or filtered by other criteria. The interface -should highlight experiment failures and provide a convenient mechanism to -run a set of experiments on demand. +should highlight experiment failures. +% Commented out Kevin's text by Karsten on 2013-09-18: +% and provide a convenient mechanism to +% run a set of experiments on demand. +% Reason: The web interface has to be read-only for security reasons.
\subsubsection{Results accumulator}
@@ -411,14 +423,19 @@ client processes, using a previously configured tor version or binary. \item[tor controller] Connect to tor's control port, force-set guards if required by the experiment, register for and handle asynchronous events, set up and tear down hidden services. -\item[experiment scheduler] Run experiments following a schedule. Must also -handle immediate scheduling requests by users. +\item[experiment scheduler] Run experiments following a schedule. +% Commented out Kevin's text by Karsten on 2013-09-18: +% Must also +% handle immediate scheduling requests by users. +% Reason: Following a pre-defined schedule should be sufficient. See also +% above why users shouldn't trigger new measurements from the web +% interface. \item[experiment runner] Handle the execution of a scheduled experiment. It should take care of the entire experiments lifecycle and reserve an exclusive tor instance for its lifetime. Finally, collect the results and post-process them (if applicable) before saving the data to the -results database.If the experiment fails it should save as much information -as possible for future inspection. +results database. If the experiment fails it should be possible to track +down the reason for failure from looking at the experiment results. \item[request scheduler] Start new requests following a previously configured schedule. \item[request runner] Handle a single request from creation over various