Project management:

We’ve filled in a more granular plan in the Sponsor 38 page.

We’ve made some progress on Shadow’s milestone Improve development assurance and velocity. Notably:

Fixed remaining compiler warnings in our continuous-integration environments, and enabled -Werror there. [#726]
Added continuous integration for shadow-plugin-tor, testing each pull request of shadow-plugin-tor against stable versions of shadow and Tor [#85, #86, #90], and also run it for pull requests in the Shadow plugin against stable versions of shadow-plugin-tor and Tor [#727]

v1.14.0 of Shadow was released. In addition to the changes above, it modularizes the router queue management algorithm and makes CoDel the default algorithm.

Our current focus is on prototyping Phantom: a new architecture for Shadow that will run applications in their own processes rather than using a custom ELF loader to load them directly into Shadow’s process. The new architecture will work with unmodified program binaries (rather than having to recompile them with -fPIC), should be stabler and easier to maintain, and may have performance benefits.

Because Rob Jansen and Ryan Wails at NRL are actively involved in this development work, NRL regulations require that the active development happen in a private branch. When it’s ready to be upstreamed, they’ll go through an NRL release process and merge it into the public repo. In the meantime there is a place-holder issue [#738] in the public repo.

Recent progress on Shadow-Phantom:

Added ability to simultaneously support alternative “thread” strategies for controlling and communicating with plugins.
Working proof-of-concept of “Shim-Pipe” threads, which use LD_PRELOAD to interpose the libc API to call a version of the “syscall” function that communicates via Shadow using a pipe.
Nearly done with a shared-memory-based IPC mechanism for the Shim-Pipe, which will be used for syscall pointer arguments.
Working proof-of-concept of “Ptrace” threads, which use ptrace to attach to plugin threads, and intercept and service syscalls. This approach is expected to be a bit simpler (the tracing process can directly read and write the tracee’s memory, making marshalling easier) and more reliable (e.g. handle direct usage of the syscall instruction), but be less performant (likely more context switches between Shadow, the OS, and the plugin).

In the coming month our main focus will continue to be on Shadow-Phantom. We intend to flesh out the functionality enough to start doing simple networking benchmarks, and if results look promising will continue to flesh it out enough to run Tor.