[tor-project] Sponsor 38 update (Shadow simulator)
jnewsome at torproject.org
Mon Jun 1 18:59:27 UTC 2020
We've been primarily focused on prototyping the new process-based
simulation architecture https://github.com/shadow/shadow/milestone/16:
- General libc + syscall interposition plumbing is fairly stable now,
including marshaling pointer arguments across the Shadow/Plugin
process boundary, in both the ptrace and the LD_PRELOAD
- Settled on a logging design and implemented it. In particular we
needed to address the problem of logging from our "shim" code that's
LD_PRELOADed into the plugin's process. Since it's no longer in
Shadow's process, we can't directly use Shadow's logger, which
references internal Shadow data structures and in-memory queues. To
further complicate things, some code runs both inside Shadow and
inside the LD_PRELOADed library, and that code should use Shadow's
logger when and only when running inside Shadow. We developed a
simple plugabble logger that writes to stdout by default, uses
Shadow's logging when running inside Shadow, and a "shim logger"
when running inside the LD_PRELOADed shim. This also allows us to
use the same logging API inside tests and support libraries.
- We've gotten more syscalls and parts of libc implemented, including
syscalls from unistd.h and most of the socket.h syscalls. These are
the basic interfaces used by almost every network program to handle
communication and file descriptor i/o. We are nearly ready to start
running some multi-process networking benchmarks.
We've also made some general improvements on the public Shadow branch
- Added both debug and release builds to our continuous integration
test matrix, helping us catch more bugs before they get merged.
- Got rid of the monolithic shadow.h header, substantially speeding up
incremental builds, and improving compatibility with code analysis
- Documentation cleanup and additions, including adding contributor
- Reduced flakiness in multi-process networking tests by moving away
from hard-coded server ports.
- Added additional integration tests to cover functionality being
implemented in the new process-based model.
- Migrated from Python 2 to 3.
We also spent some time strategizing about how to use Github
"milestones" and "issues" to help us plan and prioritize development
work and to keep the community updated on our progress. We are still in
the early stages and hope to make more effective use of these planning
tools over time.
Finally, we're thrilled to welcome Steve Engler to the project. Steve is
starting today: Monday, June 1.
More information about the tor-project