I've just shared the 2022-04 update in the Shadow forum: https://github.com/shadow/shadow/discussions/2007 Also it looks like I forgot to share the previous 2021-12 update here: https://github.com/shadow/shadow/discussions/1824 Both mirrored below: 2022-04 update This is part of a series of periodic updates of development in Shadow. This work is sponsored by the NSF <https://github.com/shadow/shadow/blob/main/docs/nsf_sponsorship.md>. Previous update: 2021-12 <https://github.com/shadow/shadow/discussions/1824>. We've merged 82 non-dependabot pull requests <https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-12-17..2022-04-05+-author%3Aapp%2Fdependabot> and closed 18 issues <https://github.com/shadow/shadow/issues?q=closed%3A2021-12-17..2022-04-05+is%3Aissue> since our previous update. Release status We are continuing to work on Shadow 2.1 <https://github.com/shadow/shadow/projects/5>. The biggest user-facing goal for this release is to support running golang programs in Shadow <https://github.com/shadow/shadow/milestone/30>, especially tor simulations using the snowflake pluggable transport <https://github.com/shadow/shadow/issues/1549>. We've also begun planning the Shadow 2.2 <https://github.com/shadow/shadow/projects/6> release, which will largely be a push to refactor and migrate more of the core Shadow code to Rust. Notable change since last update Emulation accuracy * Optionally move time forward in non-blocking syscalls <https://github.com/shadow/shadow/pull/1995>. Historically Shadow doesn't move time forward except when explicitly waiting for an event, such as for a deadline to pass or for data to arrive over the network. Conceptually, this emulates a system with an infinite number of infinitely fast CPUs. Normally this is sufficient for modeling networks where CPUs aren't expected to be a bottleneck. Unfortunately as we've explored running more software under Shadow, we've found a growing number of examples of code with "busy loops" <https://github.com/shadow/shadow/issues/1792>, which deadlock in this model. This new feature optionally models every syscall taking some small amount of time (e.g. a microsecond), which allows the simulation to escape most such loops. We are still testing and improving this feature, and expect some version of it to be enabled by default in the next release. * Fixed a corner case in getaddrinfo <https://github.com/shadow/shadow/pull/1998>. * Fixed several bugs in handling file descriptors: o Fixed pipe writable state when reader is closed <https://github.com/shadow/shadow/pull/1985>. o Have pipe writers return EPIPE if there are no readers <https://github.com/shadow/shadow/pull/1983>. o Fixed errno for socket with SOCK_SEQPACKET type <https://github.com/shadow/shadow/pull/1972>. o Don't fail on TCP_NODELAY <https://github.com/shadow/shadow/pull/1936>. * Reliably intercept time via vdso <https://github.com/shadow/shadow/pull/1951>. Previously we relied on intercepting calls to VDSO functions (such as |gettimeofday|) to be intercepted at the libc level via |LD_PRELOAD|. However, this doesn't work when the VDSO is used more directly, such as in golang, which would cause the program to get the real-world time instead of the simulated time. We now patch the VDSO itself at program start to reliably intercept these functions in such cases. * Implemented basic signal emulation <https://github.com/shadow/shadow/pull/1881>. This allows managed code to install signal handlers and send and receive signals within the simulation. (Sending a signal from outside the simulation to a managed process is still not supported). Notably this support is required to handle golang programs, and allows simulated processes to be shut down cleanly by scheduling |kill| processes to send appropriate signals. * Implemented the |select| system call <https://github.com/shadow/shadow/pull/1910>. Usability * Added an experimental option for strace-style logging <https://github.com/shadow/shadow/pull/1886> of syscalls made by managed code. * Stabilized the |--progress| <https://github.com/shadow/shadow/pull/1937> option, which periodically updates stderr with the simulation progress. Performance * Nightly shadow benchmarks are now being run to help detect performance regressions. We also have the ability to run the benchmark on our own development branches to investigate performance changes before merging these branches into Shadow. Benchmark results for a 5% Tor network are published publicly <https://github.com/shadow/benchmark-results/tree/master/tor>, but are only intended to be useful for Shadow developers. * Added a library for overriding crypto functions <https://github.com/shadow/shadow/pull/1853>. When enabled, this option overrides some openssl APIs with "no-op" implementations. This is a reimplementation of a feature previously available in Shadow's "tor plugin". It is intended primarily to improve the performance of tor simulations on hardware without accelerated AES operations, and is only supported on Debian 11. Happy simulating! The Shadow team 2021-12 update This is part of a series of periodic updates of development in Shadow. This work is sponsored by the NSF <https://github.com/shadow/shadow/blob/main/docs/nsf_sponsorship.md>. Previous update: 2021-10 <https://github.com/shadow/shadow/discussions/1695>. We've merged 78 pull requests <https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A%3E%3D2021-09-30> and closed 17 issues <https://github.com/shadow/shadow/issues?q=closed%3A%3E%3D2021-09-30+is%3Aissue+> since our previous update. Release status We have released Shadow 2.0 <https://github.com/shadow/shadow/releases/tag/v2.0.0>! Give it a try, and let us know if you run into any issues! We have begun work on Shadow 2.1 <https://github.com/shadow/shadow/projects/5>. Notable planned features are support for signals (which is needed <https://github.com/shadow/shadow/issues/1549> to reliably run golang programs under Shadow), and improved Unix socket support. Notable change since last update Emulation accuracy * Add unnamed unix sockets created with socketpair() <https://github.com/shadow/shadow/pull/1787> * Add support for the 'O_DIRECT' flag to pipes (packet mode) <https://github.com/shadow/shadow/pull/1746> * poll and epoll: wait for timeout even with no fd's <https://github.com/shadow/shadow/pull/1697> Performance improvements * Set default min runahead to 1ms <https://github.com/shadow/shadow/pull/1711>. This fixes a performance regression <https://github.com/shadow/shadow/issues/1699> in many-host simulations using multiple CPUs. Stability * Detect and handle death by signals <https://github.com/shadow/shadow/pull/1722> * Handle process exit while it has events scheduled <https://github.com/shadow/shadow/pull/1704> UI improvements * Move host heartbeat options to the experimental options, and disable by default <https://github.com/shadow/shadow/pull/1703> Shadow at Tor We have been collaborating with The Tor Project to use Shadow in its development and testing. We have been running Shadow simulations inside a Gitlab CI pipeline <https://gitlab.torproject.org/jnewsome/sponsor-61-sims> to help develop and tune improved congestion control algorithms in the upcoming 0.4.7 <https://lists.torproject.org/pipermail/tor-project/2021-December/003241.html> release. This is the first major application of Shadow inside the Tor Project itself, and we plan to use the pipeline we've developed for further testing and profiling. We have also been making progress on running Arti (the experimental new Rust implemenation of tor) under Shadow: https://gitlab.torproject.org/tpo/core/arti/-/issues/174. In the course of this work we have fixed some bugs in Arti, fixed an upstream bug in the async-io <https://github.com/smol-rs/async-io/pull/73> crate, and found and identified and fixed several emulation accuracy bugs in Shadow. Happy simulating! The Shadow team