[tor-project] What's happening in Shadow 2022-09

Fri Sep 30 21:32:58 UTC 2022

Mirrored from https://github.com/shadow/shadow/discussions/2428 :

This is part of a series of periodic updates of development in Shadow. 
This work is sponsored by the 
[NSF](https://shadow.github.io/docs/guide/nsf_sponsorship.html). 
Previous update: 
[2022-06](https://github.com/shadow/shadow/discussions/2243).

We've merged [103 non-dependabot pull 
requests](https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2022-06-27..2022-09-28+-author%3Aapp%2Fdependabot+) 
and closed [12 
issues](https://github.com/shadow/shadow/issues?q=closed%3A2022-06-27..2022-09-28+is%3Aissue+) 
since our previous update.

Release status
==============

We've released  [Shadow 
2.2.0](https://github.com/shadow/shadow/releases/tag/v2.2.0)! Notable 
user-facing changes, from the release notes:

* We have removed ptrace-mode, and the associated experimental options 
`use-o-n-waitpid-workaround` and `--interpose-method`. ptrace-mode was 
an alternative to Shadow's current interposition mechanism that uses 
`LD_PRELOAD` and `seccomp`. This change should be transparent to most 
users, since it hasn't been the default for several releases, and was 
only accessible via experimental options. See 
https://github.com/shadow/shadow/issues/1945

* `dup()` and related syscalls are now supported for all file descriptors

* Fixed behavior when multiple threads are blocked in `epoll_wait` on 
the same epoll file description. 
https://github.com/shadow/shadow/issues/2260

* Fixed bugs causing `timerfd_settime` to not reset the internal timer's 
expiration count (https://github.com/shadow/shadow/pull/2279), and not 
cancel previously scheduled timer-fire events 
(https://github.com/shadow/shadow/pull/2282).

* Fixed a panic when patching the VDSO in newer kernels, such as those 
in Ubuntu 22.04. https://github.com/shadow/shadow/issues/2273

* Fixed the errno returned from calling `connect()` on a unix socket. 
This fixes a `getaddrinfo()` test failure on some systems. 
https://github.com/shadow/shadow/issues/2286

* Fixed minor memory leaks. https://github.com/shadow/shadow/pull/2249

Other user-facing improvements since last update
================================================

We also continue to make compatibility improvements:

* Added support for escaping additional "busy loops" - ones that only 
use `rdtsc` without making any actual syscalls. This affected some 
versions of 
[libopenblas](https://github.com/shadow/shadow/discussions/2299#discussioncomment-3198368). 
https://github.com/shadow/shadow/pull/2314

* Replaced custom cmake code for locating glib with pkg-config. This 
makes shadow easier to compile in environments with non-standard layouts 
such as guix or nix. https://github.com/shadow/shadow/pull/2331

* Fixed a bug handling files opened with `O_NOFOLLOW`. 
https://github.com/shadow/shadow/pull/2353

* Fixed our documentation and CI to support mapping pages with 
`PROT_EXEC` when run under Docker, fixing 
[2400](https://github.com/shadow/shadow/issues/2400). When using `docker 
run`, you should also pass the `--tmpfs 
/dev/shm:rw,nosuid,nodev,exec,size=1024g`. 
https://github.com/shadow/shadow/pull/2402

* Made the shadow installation directory relocatable. e.g. installing to 
`$HOME/opt/shadow` and later moving to `$HOME/opt/shadow-old` now 
behaves as expected. This also makes it easier to share and reuse 
pre-built shadow binaries, e.g. for CI, though we still do not yet 
publish pre-compiled binaries. https://github.com/shadow/shadow/pull/2391

* Added support for emulating `PR_SET_DUMPABLE`, which fixes 
compatibility with managed programs that "harden" themselves against 
debuggers, including 
[arti](https://gitlab.torproject.org/tpo/core/arti). 
https://github.com/shadow/shadow/pull/2370

* Fixed a bug affecting getaddrinfo on some systems. 
https://github.com/shadow/shadow/pull/2292

* Partly implemented fcntl. https://github.com/shadow/shadow/pull/2259

* Made failures due to running out of tmpfs space easier to understand. 
https://github.com/shadow/shadow/pull/2267

Other notable changes since last update
=======================================

Since the 2.2.0 release, our current focus is primarily on migrating 
Shadow's C code to Rust. These changes should be mostly invisible to 
Shadow's end users, but may be of interest to folks interested in 
hacking on Shadow itself. Along those lines we've:

* Migrated Shadow's scheduler to Rust. This is a fairly complex and 
performance-sensitive component, and we're taking care to prevent 
performance regressions during the migration:
   * https://github.com/shadow/shadow/pull/2340
   * https://github.com/shadow/shadow/pull/2343
   * https://github.com/shadow/shadow/pull/2344
   * https://github.com/shadow/shadow/pull/2352
   * https://github.com/shadow/shadow/pull/2412
   * https://github.com/shadow/shadow/pull/2414
   * https://github.com/shadow/shadow/pull/2416

* Migrated other modules to Rust.
   * Event: https://github.com/shadow/shadow/pull/2327
   * TokenBucket: https://github.com/shadow/shadow/pull/2306
   * Manager: https://github.com/shadow/shadow/pull/2277

* Implemented a Rust mutex suitable for use in shared memory, and 
compatible with [rkyv](https://rkyv.org/). 
https://github.com/shadow/shadow/pull/2386

* Moved most of the compilation and linking of Shadow's remaining C code 
from cmake into Cargo build scripts. This fixes some dependency issues 
and will make it easier to start migrating some of our additional 
support libraries from C to Rust. https://github.com/shadow/shadow/pull/2404

* Improved organization of global state in Rust.
   * https://github.com/shadow/shadow/pull/2361
   * https://github.com/shadow/shadow/pull/2363
   * https://github.com/shadow/shadow/pull/2367
   * https://github.com/shadow/shadow/pull/2372
   * https://github.com/shadow/shadow/pull/2373

* Refactored other components to simplify migration to Rust.
   * Controller, Manager, Scheduler: 
https://github.com/shadow/shadow/pull/2255
   * Thread: https://github.com/shadow/shadow/pull/2328

Shadow in Tor
=============

We've been working on using Shadow to help test and evaluate 
[arti](https://gitlab.torproject.org/tpo/core/arti) - The Tor Projects's 
new Rust-based implementation of tor. This has driven many of the 
compatibility fixes above, and we are happy to report that arti now runs 
under Shadow! We plan to merge a shadow-based continuous integration 
test into arti itself within the next few days: 
https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/634

Happy simulating!
The Shadow team