[tor-dev] Torperf implementation considerations (was: Torperf)

[Sathyanarayanan Gunasekaran]

Hi,

I have some comments on the updated pdf -

> It should be easy for a user to implement or install an experiment that isn’t bundled with the core distribution. Ideally, installing an experiment should be as simple as unzipping a folder or config file into an experiments folder.

I don't understand how this will work when users just apt-get install
torperf. Ideally if someone writes a good experiment, they should send
the patches upstream and get it merged, and then we update torperf to
include those tests and then the users just update torperf with their
package managers.

> It should be possible to run different experiments with different tor versions or binaries in the same Torperf service instance.

I don't think we need this now. I'm totally ok with having users run
different torperf instances for different tor versions.

>It might be beneficial to provide a mechanism to download and verify the signature of new tor versions as they are released. The user could speficy if they plan to test stable, beta or alpha versions of tor with their Torperf instance.

IMHO, torperf should just measure performance, not download Tor or
verify signatures. We have good package managers that do that already.

> A Torperf service instance should be able to accumulate results from its own experiments and remote Torperf service instances

Torperf should not accumulate results from remote Torperf service
instances. If by "accumulate", you mean read another file from
/results which the *user* has downloaded, then yes. Torperf shouldn't
*download* result files from remote instances.

> The new Torperf should come with an easy-to-use library to process its results

Torperf results should just be JSON(or similar) files that already
have libraries and we should invent a new result format and write a
library for it.

> request scheduler Start new requests following a previously configured schedule.
> request runner Handle a single request from creation over various possible sub states to timeout, failure, or completion.

These are experiment specific. Some tests may not even need to do
requests. No need for these to be a part of torperf.

> results database Store request details, retrieve results, periodically delete old results if configured.

Not sure if we really need a database. These tests look pretty simple to me.

Thanks,
--Sathya


On Wed, Sep 18, 2013 at 6:21 AM, Karsten Loesing <karsten at torproject.org> wrote:
> On 9/18/13 3:49 AM, Kevin Butler wrote:
>>> Executing scripts and reading stdout/stderr is probably too low-level.
>>> I think we need a Python/Twisted (or whatever language Torperf will be
>>> written in) interface for running an experiment and retrieving results.
>>
>> You're probably right on stdout/err being too low level, but most
>> experiments would be reusing the provided implementation, just wrapping
>> them in a simple manner. Anyway I think there's a language agnostic way to
>> get this working, without forcing any extensibility on matching Torperfs
>> language of choice. I've tried to be pretty generic in my attached changes.
>> :)
>
> I don't see how we could make new experiments language agnostic.  These
> new experiments will want to re-use Torperf classes/components, which
> they can't do if they're just "shell scripts".  They need to implement
> an interface provided by Torperf, and Torperf needs to provide
> functionality via an interface that the experiments understand.  If an
> experiment is in fact just a shell script, it should be trivial to write
> a wrapper class to call that.  But I think that's the exception, not the
> rule.
>
> Or maybe we have a different understanding of an "experiment".  Can you
> give an example for an experiment that is not listed in Section 3 of the
> design document and state how you'd want to integrate that into Torperf
> without touching its sources?
>
>>> Well, thanks for your input!  As I said above, it would help a lot if
>>> you added these ideas to the appropriate sections of the design document.
>>
>> Please see attached.
>
> Awesome!  I applied your patch, though I tweaked some parts and
> commented out other parts, explaining my reasons in the LaTeX sources.
> Happy to discuss these things further if you want!
>
> I also added an Appendix A with suggested data formats.  Maybe these
> data formats make it clearer what I'd expect as output from an experiment.
>
> https://people.torproject.org/~karsten/volatile/torperf2.pdf
>
> If you have further suggestions how to improve the requirements or
> design, please send me a new patch, and I'll apply it and comment on it.
>  Thanks!
>
> All the best,
> Karsten
>