[tor-project] Sponsor 38 update (Shadow simulator)

[Jim Newsome]

Sponsor 38:
https://github.com/shadow/shadow/blob/master/docs/nsf-sponsorship.md

Previous updates:

  * 2020-02-10
    https://lists.torproject.org/pipermail/tor-project/2020-February/002718.html
  * 2020-03-19
    https://lists.torproject.org/pipermail/tor-project/2020-March/002785.html
  * 2020-06-01
    https://lists.torproject.org/pipermail/tor-project/2020-June/002859.html

We've moved our NSF sponsor page from Tor's now-deprecated Trac site
into our source repository on GitHub, next to our code.
https://github.com/shadow/shadow/blob/master/docs/nsf-sponsorship.md.
(We also have a place-holder in Tor's new GitLab instance, but it just
directs readers to the GH page).

We've continued to make progress on the new process-based architecture
(aka Phantom) https://github.com/shadow/shadow/milestone/16. In particular:

  * We've migrated this work from a private repository to a new |dev|
    branch in the public Shadow repo:
    https://github.com/shadow/shadow/tree/dev.
  * This branch can now run Shadow's |phold| simulator performance
    benchmark, which we've been using to iterate on performance
    measurements and improvements before continuing on to add additional
    syscall support to run |tor|.
    https://github.com/shadow/shadow/issues/825
  *

    Preliminary results were as much as 10x slower than the
    single-process architecture in a small-scale, ad-hoc test. These
    results led us to think more carefully about our design and we have
    begun to implement optimizations that we expect will help close this
    performance gap.

      o

        We've modified the |ptrace| interposition mechanism to use
        |PTRACE_SYSEMU| instead of |PTRACE_SYSCALL|
        (https://github.com/shadow/shadow/commit/bdf9fe529ed8090bb07aa5f2683b21bc55018a32),
        giving about a 20% speedup. Adding a caching layer to address
        lookups also gave a substantial speedup. Together these got us
        down to roughly 3.5x slower than the single-process approach.

      o

        We have work in progress to further improve performance by
        sharing memory between Shadow and its plugins
        (https://github.com/shadow/shadow/issues/888) and by using
        spin-locks for faster control-transfers
        (https://github.com/shadow/shadow/issues/894).

  *

    Since the |phold| benchmark essentially does nothing but send and
    receive UDP packets, it spends most of the benchmark crossing the
    new process boundaries between the Shadow process and the plugins.
    We think this is a worst case scenario in which to compare the new
    architecture to the current single-process architecture. We've added
    a tunable CPU-load parameter to |phold| to let us evaluate how this
    gets amortized, and plan on doing analysis with it soon.
    https://github.com/shadow/shadow/pull/897

We've integrated Rust into our build system such that Shadow code can be
written in Rust, and Shadow's C and Rust code can call each-other. New
code is now being being written in Rust.
https://github.com/shadow/shadow/milestone/17

We've continued to improve Shadow's test suite and continuous integration.

  * We've added CI for all of the Linux distros we intend to support.
    https://github.com/shadow/shadow/milestone/19
  * We've started collecting test coverage measurements
    https://github.com/shadow/shadow/milestone/20, and generating
    differential coverage reports on PRs.
    https://codecov.io/gh/shadow/shadow/branch/dev
  * We've moved core CI logic into bash scripts, and added an
    alternative driver script to run some or all of it in local Docker
    containers (vs GitHub's) for debugging.
    https://github.com/shadow/shadow/pull/875
  * We've made progress on more-thoroughly testing the libc socket API.
    https://github.com/shadow/shadow/milestone/21

The Shadow team
(https://github.com/shadow/shadow/blob/master/docs/nsf-sponsorship.md#people)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200724/3c67fbfe/attachment.htm>