We've shared the latest project update at https://github.com/shadow/shadow/discussions/1485; copied below:
This is part of a series of periodic updates of development in Shadow. This work is sponsored by the NSF https://github.com/shadow/shadow/blob/main/docs/nsf_sponsorship.md. Previous update: 2021-04 https://github.com/shadow/shadow/discussions/1274.
Shadow 2.0.0-pre release
Shadow 2.0, which features the new process-based architecture https://github.com/shadow/shadow/projects/1 is now in pre-release! We've merged https://github.com/shadow/shadow/pull/1441 the |dev| branch into |main|, and tagged v2.0.0-pre.1 https://github.com/shadow/shadow/releases/tag/v2.0.0-pre.1. For users, this means:
* Users cloning the repository at head will get Shadow 2.0 (pre). * We've /mostly/ stabilized some major changes to command-line options, config files, and topology files; enough so that users preparing to set up new simulations, or perhaps even heavily modify existing simulations or scripts, are better off targeting 2.0 than the Shadow 1.x series. * Because this is a pre-release, we're still reserving the right to make breaking changes without further bumping the version number, but we expect such changes to be minimal.
Users migrating existing simulations and scripts should have a look at our migration guide https://github.com/shadow/shadow/blob/main/docs/migrating_from_1x.md.
Progress since the last update
We're still experimenting a bit with how to use GitHub's various project management tools. We've started using "projects" to represent Shadow releases. For example, you can see the issues for the v2.0.0 release here https://github.com/shadow/shadow/projects/3.
Notable changes since the April update: (By milestone; #'s are GitHub PR or issue #'s):
* Minimum viable prototype of process-based simulation architecture https://github.com/shadow/shadow/milestone/16: Closed, now that Shadow 2.x runs Tor simulations, and we've cross-validated simulation results with Shadow 1.x. o Added new internal APIs for accessing managed process memory, and used them to reduce the number of data copies. (#1264, #1280, #1290) * Fully support process-based simulation architecture https://github.com/shadow/shadow/milestone/22 o This milestone is a bit of a catch-all, and includes a number of stability and performance improvements https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-04-06..2021-07-28+sort%3Acreated-asc+milestone%3A%22Fully+support+process-based+simulation+architecture%22 that we wanted to finish before the full 2.0 release. o We're in the process of adding seccomp-based syscall interposition in combination with preload-based interposition. More on that in "What's next", below. So far we've already implemented and enabled it (#1452, #1471). * Convert existing modules to Rust https://github.com/shadow/shadow/milestone/28 o We've ported several of Shadow's core internal objects into Rust (#1325, #1336, #1367, #1376, #1434). o We've reduced our reliance on globals to fit better in the Rust ownership model (#1386, #1393, #1396) * Updates to and new formats for Shadow configuration https://github.com/shadow/shadow/milestone/27 o We've changed the configuration file format from XML to YAML (#1278). o We've changed the topology format from graphml (XML) to gml (YAML-like) (#1369). o Lots of updates to configuration options and defaults, making things clearer, giving more sensible defaults, and fixing parsing bugs. (#1263, #1266, #1270, #1304, #1305). o Additionally, we've /unified/ much of the configuration and command-line parsing to reduce code duplication and inconsistencies, and migrated it to Rust. (#1258, #1286) o Most numbers in our configuration and command-line options can now include explicit units. (#1300) o Made updates to tornettools to stay consistent with Shadow's new config option and network graph formats, and other 2.x changes (tornettools#7 https://github.com/shadow/tornettools/pull/7, tornettools#8 https://github.com/shadow/tornettools/pull/8, tornettools#17 https://github.com/shadow/tornettools/pull/17) * Code health and maintenance https://github.com/shadow/shadow/milestone/24: o Fixed some sources of nondeterminism in simulation results (#1460). o Improve and automate Tor simulation test (#1411, #1458). o Updated documentation for Shadow 2.0. (#1412, #1417, #1406, #1403) o Changed default install prefix from |~/.shadow| to |~/.local|. (#1404) o More https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-04-06..2021-07-28+milestone%3A%22Code+health+and+maintenance%22+sort%3Acreated-asc+
What's next
We'll be continuing to gain experience running Shadow experiments with the 2.0.0 pre-release. We will be working to fix bugs, tie up some loose ends and smooth out some rough edges, and generally stabilize the code as we continue to prepare for a 2.0.0 stable release.
We're currently tracking outstanding work on the 2.0.0 release page https://github.com/shadow/shadow/projects/3.
Seccomp
In the course of our work on the 2.x architecture we've largely been exploring two system call interposition mechanisms that allow us to co-opt running processes into a simulation - |LD_PRELOAD| (similar to what Shadow 1.x uses), and |ptrace|. While preload-based interposition has better performance, and plays better with debuggers, we've largely been focusing on the ptrace-based approach because it can reliably catch all syscalls without having to re-implement large parts of libc and/or rely on undocumented glibc implementation details. This has allowed us to work out all of the Shadow internals that needed to change to support interposition at the Linux system call interface.
Recently, though, we added seccomp-based interposition to the preload-based interposition mode. This reliably catches any syscalls that would otherwise be missed by our |LD_PRELOAD| wrappers. We also have a working prototype of this mode of operation with /thread support/, which was previously the biggest missing functionality in preload-mode. Assuming thread support gets merged as expected and no other problems crop up, this is likely to become the new default mode of operation; it's faster than ptrace, and it allows us to trace the processes running in Shadow using debugging tools such as |gdb|.
Happy simulating!
The Shadow team https://github.com/shadow/shadow/blob/master/docs/nsf-sponsorship.md#people