<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>We've shared the latest project update at
<a class="moz-txt-link-freetext" href="https://github.com/shadow/shadow/discussions/1485">https://github.com/shadow/shadow/discussions/1485</a>; copied below:<br>
</p>
<p>This is part of a series of periodic updates of development in
Shadow. This work is sponsored by the <a
href="https://github.com/shadow/shadow/blob/main/docs/nsf_sponsorship.md">NSF</a>.
Previous update: <a
href="https://github.com/shadow/shadow/discussions/1274">2021-04</a>.</p>
<h2 id="shadow-2.0.0-pre-release">Shadow 2.0.0-pre release</h2>
<p>Shadow 2.0, which features the new <a
href="https://github.com/shadow/shadow/projects/1">process-based
architecture</a> is now in pre-release! We've <a
href="https://github.com/shadow/shadow/pull/1441">merged</a> the
<code>dev</code> branch into <code>main</code>, and tagged <a
href="https://github.com/shadow/shadow/releases/tag/v2.0.0-pre.1">v2.0.0-pre.1</a>.
For users, this means:</p>
<ul>
<li>Users cloning the repository at head will get Shadow 2.0
(pre).</li>
<li>We've <em>mostly</em> stabilized some major changes to
command-line options, config files, and topology files; enough
so that users preparing to set up new simulations, or perhaps
even heavily modify existing simulations or scripts, are better
off targeting 2.0 than the Shadow 1.x series.</li>
<li>Because this is a pre-release, we're still reserving the right
to make breaking changes without further bumping the version
number, but we expect such changes to be minimal.</li>
</ul>
<p>Users migrating existing simulations and scripts should have a
look at our <a
href="https://github.com/shadow/shadow/blob/main/docs/migrating_from_1x.md">migration
guide</a>.</p>
<h2 id="progress-since-the-last-update">Progress since the last
update</h2>
<p>We're still experimenting a bit with how to use GitHub's various
project management tools. We've started using "projects" to
represent Shadow releases. For example, you can see the issues for
the v2.0.0 release <a
href="https://github.com/shadow/shadow/projects/3">here</a>.</p>
<p>Notable changes since the April update: (By milestone; #'s are
GitHub PR or issue #'s):</p>
<ul>
<li><a href="https://github.com/shadow/shadow/milestone/16">Minimum
viable prototype of process-based simulation architecture</a>:
Closed, now that Shadow 2.x runs Tor simulations, and we've
cross-validated simulation results with Shadow 1.x.
<ul>
<li>Added new internal APIs for accessing managed process
memory, and used them to reduce the number of data copies.
(#1264, #1280, #1290)</li>
</ul>
</li>
<li><a href="https://github.com/shadow/shadow/milestone/22">Fully
support process-based simulation architecture</a>
<ul>
<li>This milestone is a bit of a catch-all, and includes a
number of <a
href="https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-04-06..2021-07-28+sort%3Acreated-asc+milestone%3A%22Fully+support+process-based+simulation+architecture%22">stability
and performance improvements</a> that we wanted to finish
before the full 2.0 release.</li>
<li>We're in the process of adding seccomp-based syscall
interposition in<br>
combination with preload-based interposition. More on that
in "What's<br>
next", below. So far we've already implemented and enabled
it (#1452, #1471).</li>
</ul>
</li>
<li><a href="https://github.com/shadow/shadow/milestone/28">Convert
existing modules to Rust</a>
<ul>
<li>We've ported several of Shadow's core internal objects
into Rust (#1325, #1336, #1367, #1376, #1434).</li>
<li>We've reduced our reliance on globals to fit better in the
Rust ownership model (#1386, #1393, #1396)</li>
</ul>
</li>
<li><a href="https://github.com/shadow/shadow/milestone/27">Updates
to and new formats for Shadow configuration</a>
<ul>
<li>We've changed the configuration file format from XML to
YAML (#1278).</li>
<li>We've changed the topology format from graphml (XML) to
gml (YAML-like)<br>
(#1369).</li>
<li>Lots of updates to configuration options and defaults,
making things clearer, giving more sensible defaults, and
fixing parsing bugs. (#1263, #1266, #1270, #1304, #1305).</li>
<li>Additionally, we've <em>unified</em> much of the
configuration and command-line parsing to reduce code
duplication and inconsistencies, and migrated it to Rust.
(#1258, #1286)</li>
<li>Most numbers in our configuration and command-line options
can now include explicit units. (#1300)</li>
<li>Made updates to tornettools to stay consistent with
Shadow's new config option and network graph formats, and
other 2.x changes (<a
href="https://github.com/shadow/tornettools/pull/7">tornettools#7</a>,
<a href="https://github.com/shadow/tornettools/pull/8">tornettools#8</a>,
<a href="https://github.com/shadow/tornettools/pull/17">tornettools#17</a>)</li>
</ul>
</li>
<li><a href="https://github.com/shadow/shadow/milestone/24">Code
health and maintenance</a>:
<ul>
<li>Fixed some sources of nondeterminism in simulation results
(#1460).</li>
<li>Improve and automate Tor simulation test (#1411, #1458).</li>
<li>Updated documentation for Shadow 2.0. (#1412, #1417,
#1406, #1403)</li>
<li>Changed default install prefix from <code>~/.shadow</code>
to <code>~/.local</code>. (#1404)</li>
<li><a
href="https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-04-06..2021-07-28+milestone%3A%22Code+health+and+maintenance%22+sort%3Acreated-asc+">More</a></li>
</ul>
</li>
</ul>
<h2 id="whats-next">What's next</h2>
<p>We'll be continuing to gain experience running Shadow experiments
with the 2.0.0 pre-release. We will be working to fix bugs, tie up
some loose ends and smooth out some rough edges, and generally
stabilize the code as we continue to prepare for a 2.0.0 stable
release.</p>
<p>We're currently tracking outstanding work on the <a
href="https://github.com/shadow/shadow/projects/3">2.0.0 release
page</a>.</p>
<h3 id="seccomp">Seccomp</h3>
<p>In the course of our work on the 2.x architecture we've largely
been exploring two system call interposition mechanisms that allow
us to co-opt running processes into a simulation - <code>LD_PRELOAD</code>
(similar to what Shadow 1.x uses), and <code>ptrace</code>. While
preload-based interposition has better performance, and plays
better with debuggers, we've largely been focusing on the
ptrace-based approach because it can reliably catch all syscalls
without having to re-implement large parts of libc and/or rely on
undocumented glibc implementation details. This has allowed us to
work out all of the Shadow internals that needed to change to
support interposition at the Linux system call interface.</p>
<p>Recently, though, we added seccomp-based interposition to the
preload-based interposition mode. This reliably catches any
syscalls that would otherwise be missed by our <code>LD_PRELOAD</code>
wrappers. We also have a working prototype of this mode of
operation with <em>thread support</em>, which was previously the
biggest missing functionality in preload-mode. Assuming thread
support gets merged as expected and no other problems crop up,
this is likely to become the new default mode of operation; it's
faster than ptrace, and it allows us to trace the processes
running in Shadow using debugging tools such as <code>gdb</code>.</p>
<p>Happy simulating!</p>
<p><a
href="https://github.com/shadow/shadow/blob/master/docs/nsf-sponsorship.md#people">The
Shadow team</a></p>
</body>
</html>