We've shared the latest project update at
https://github.com/shadow/shadow/discussions/1485; copied below:
This is part of a series of periodic updates of development in Shadow.
This work is sponsored by the NSF
<https://github.com/shadow/shadow/blob/main/docs/nsf_sponsorship.md>.
Previous update: 2021-04
<https://github.com/shadow/shadow/discussions/1274>.
Shadow 2.0.0-pre release
Shadow 2.0, which features the new process-based architecture
<https://github.com/shadow/shadow/projects/1> is now in pre-release!
We've merged <https://github.com/shadow/shadow/pull/1441> the |dev|
branch into |main|, and tagged v2.0.0-pre.1
<https://github.com/shadow/shadow/releases/tag/v2.0.0-pre.1>. For users,
this means:
* Users cloning the repository at head will get Shadow 2.0 (pre).
* We've /mostly/ stabilized some major changes to command-line
options, config files, and topology files; enough so that users
preparing to set up new simulations, or perhaps even heavily modify
existing simulations or scripts, are better off targeting 2.0 than
the Shadow 1.x series.
* Because this is a pre-release, we're still reserving the right to
make breaking changes without further bumping the version number,
but we expect such changes to be minimal.
Users migrating existing simulations and scripts should have a look at
our migration guide
<https://github.com/shadow/shadow/blob/main/docs/migrating_from_1x.md>.
Progress since the last update
We're still experimenting a bit with how to use GitHub's various project
management tools. We've started using "projects" to represent Shadow
releases. For example, you can see the issues for the v2.0.0 release
here <https://github.com/shadow/shadow/projects/3>.
Notable changes since the April update: (By milestone; #'s are GitHub PR
or issue #'s):
* Minimum viable prototype of process-based simulation architecture
<https://github.com/shadow/shadow/milestone/16>: Closed, now that
Shadow 2.x runs Tor simulations, and we've cross-validated
simulation results with Shadow 1.x.
o Added new internal APIs for accessing managed process memory,
and used them to reduce the number of data copies. (#1264,
#1280, #1290)
* Fully support process-based simulation architecture
<https://github.com/shadow/shadow/milestone/22>
o This milestone is a bit of a catch-all, and includes a number of
stability and performance improvements
<https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-04-06..2021-…>
that we wanted to finish before the full 2.0 release.
o We're in the process of adding seccomp-based syscall
interposition in
combination with preload-based interposition. More on that in
"What's
next", below. So far we've already implemented and enabled it
(#1452, #1471).
* Convert existing modules to Rust
<https://github.com/shadow/shadow/milestone/28>
o We've ported several of Shadow's core internal objects into Rust
(#1325, #1336, #1367, #1376, #1434).
o We've reduced our reliance on globals to fit better in the Rust
ownership model (#1386, #1393, #1396)
* Updates to and new formats for Shadow configuration
<https://github.com/shadow/shadow/milestone/27>
o We've changed the configuration file format from XML to YAML
(#1278).
o We've changed the topology format from graphml (XML) to gml
(YAML-like)
(#1369).
o Lots of updates to configuration options and defaults, making
things clearer, giving more sensible defaults, and fixing
parsing bugs. (#1263, #1266, #1270, #1304, #1305).
o Additionally, we've /unified/ much of the configuration and
command-line parsing to reduce code duplication and
inconsistencies, and migrated it to Rust. (#1258, #1286)
o Most numbers in our configuration and command-line options can
now include explicit units. (#1300)
o Made updates to tornettools to stay consistent with Shadow's new
config option and network graph formats, and other 2.x changes
(tornettools#7 <https://github.com/shadow/tornettools/pull/7>,
tornettools#8 <https://github.com/shadow/tornettools/pull/8>,
tornettools#17 <https://github.com/shadow/tornettools/pull/17>)
* Code health and maintenance
<https://github.com/shadow/shadow/milestone/24>:
o Fixed some sources of nondeterminism in simulation results (#1460).
o Improve and automate Tor simulation test (#1411, #1458).
o Updated documentation for Shadow 2.0. (#1412, #1417, #1406, #1403)
o Changed default install prefix from |~/.shadow| to |~/.local|.
(#1404)
o More
<https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-04-06..2021-…>
What's next
We'll be continuing to gain experience running Shadow experiments with
the 2.0.0 pre-release. We will be working to fix bugs, tie up some loose
ends and smooth out some rough edges, and generally stabilize the code
as we continue to prepare for a 2.0.0 stable release.
We're currently tracking outstanding work on the 2.0.0 release page
<https://github.com/shadow/shadow/projects/3>.
Seccomp
In the course of our work on the 2.x architecture we've largely been
exploring two system call interposition mechanisms that allow us to
co-opt running processes into a simulation - |LD_PRELOAD| (similar to
what Shadow 1.x uses), and |ptrace|. While preload-based interposition
has better performance, and plays better with debuggers, we've largely
been focusing on the ptrace-based approach because it can reliably catch
all syscalls without having to re-implement large parts of libc and/or
rely on undocumented glibc implementation details. This has allowed us
to work out all of the Shadow internals that needed to change to support
interposition at the Linux system call interface.
Recently, though, we added seccomp-based interposition to the
preload-based interposition mode. This reliably catches any syscalls
that would otherwise be missed by our |LD_PRELOAD| wrappers. We also
have a working prototype of this mode of operation with /thread
support/, which was previously the biggest missing functionality in
preload-mode. Assuming thread support gets merged as expected and no
other problems crop up, this is likely to become the new default mode of
operation; it's faster than ptrace, and it allows us to trace the
processes running in Shadow using debugging tools such as |gdb|.
Happy simulating!
The Shadow team
<https://github.com/shadow/shadow/blob/master/docs/nsf-sponsorship.md#people>
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2021/tor-meeting.2021-06-24-16.00.html
and our meeting pad:
Anti-censorship work meeting pad
--------------------------------
Next meeting: Thursday June 24th 16:00 UTC
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC
(channel is logged while meetings are in progress)
== Goal of this meeting ==
Weekly checkin about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at Tor.
== Announcements ==
No announcements this week
== Discussion ==
- do we have previous info on the requirements to run a snowflake proxy?
- (re
https://lists.torproject.org/pipermail/anti-censorship-team/2021-June/00017…)
- we currently have
https://community.torproject.org/relay/setup/snowflake/standalone/
- we have an issue to improve documentation for a standalone proxy
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- we could add Jacobo's hardware observations to it
== Actions ==
No actions this week
== Interesting links ==
Jacobo provided some details on the snowflake proxies they are running
https://lists.torproject.org/pipermail/anti-censorship-team/2021-June/00017…
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
cecylia (cohosh): last updated 2021-06-24
Last week:
- wrote, deployed snowflake!43 to solve a stats bug
- fixed up snowflake!42 and found a few more race conditions
- commented on bridgestrap#15
- worked on debugging snowflake memory leak (snowflake#40052)
- deployed some memory profiling tools
- wrote a fix snowflake!45
- started reviewing rdsys!11
- s28 deadline work
- planning for new s96
- reviewed bridgedb!20
This week:
- finish reviewing rdsys!11
- review bridgstrap#15 when ready
- work on bridgedb#32276 deployment
- follow up on arlo's snowflake broker work
- afk next week
Needs help with:
- review of snowflake!45
arlolra: 2021-06-10
Last week:
- More snowflake!39
Next week:
- Maybe get back to snowflake-webext #10
Help with:
- snowflake !39
dcf: 2021-06-24
Last week:
- did a security upgrade of tor on the snowflake bridge
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- helped debug snowflake server memory leak
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- answered a core tor question about managed transports
https://gitlab.torproject.org/tpo/core/tor/-/issues/40414
Next week:
- review fix for snowflake server resource leak
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- begin AMP cache rendezvous for snowflake
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Help with:
maxb: 2021-06-03
Last week:
- Made some improvements to github.com/max-b/nat-testing for
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- Now have separate containers for each piece along w/ a reasonably
simulated network topology
- Got really good feedback from cohosh as always
Next week:
- Make config fixes cohosh suggested
- Implement alternative NAT types
- Add testing wrapper w/ "pass/fail" conditions
meskio: 2021-06-24
Last week:
- proof of concept of gettor in rdsys (rdsys!11 rdsys#32)
- add bridgestrap CollecTor exporter (bridgestrap!7 bridgestrap#15)
- code reviews: snowflake!44 snowflake!43 snowflake!42
Next week:
- modify snowflake proxy to assign proxies based on self-reported
client load (snowflake#40048)
- act on snowflake!11 review
- merge access keys translators help brdigedb!20
Help with:
-
Hi,
I have released Onionoo 8.0-1.26.1.
Onionoo is a web-based protocol to learn about currently
running Tor relays and bridges. Onionoo itself was not designed
as a service for human beings—at least not directly. Onionoo
provides the data for other applications and websites which in
turn present Tor network status information to humans.
The latest changes:
* Minor changes
- Restore the "AS" prefix string to the as field in documents.
- Fix search by AS number.
You can find the release at:
https://dist.torproject.org/onionoo/8.0-1.26.1/
You can find the sources at:
https://gitlab.torproject.org/tpo/metrics/onionoo/-/tree/onionoo-8.0-1.26.1
This is already deployed to the Onionoo backends, although it may
take a few hours for the updater to correct all relays AS numbers.
Thanks,
Iain.
Hi everyone!
Two new sysadmins were hired, so we're holding meetings again! Welcome
again to kez and lavamind, who both joined us last week.
Here are the minutes from the meeting we held on June 14 and 16.
# Roll call: who's there and emergencies
* anarcat
* gaba
* kez
* lavamind
No emergencies.
# Triage & schedules
* Introduce the triage system
* the "triage star of the weeks" rotates every two weeks
* the star triages the boards regularly, making sure there are no
"Open" tickets, and assigning tickets or dealing with small
tickets
* see also [TPA-RFC-5][] for the labeling nomenclature
* we were doing weekly checkins with hiro during the handover on
wednesday, since we were both part time
* work schedules:
* j: monday, tuesday - full day; wednesday - partially
* kez: flexible - TBD
* anarcat: monday to thursday - full day
[TPA-RFC-5]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-5-gitlab#…
# Communication in the team
What is expected? When to bring it in IRC versus email versus ticket?
Acknowledgements.
* we expect people to update tickets when they work on them
* we expect acknowledgements when people see their names mentions on
IRC
# Short term planning: anarcat going AFK
This basically involves making sure our new hires have enough work
while they are gone.
We reviewed the Doing/Next columns and assign issues in the [TPA
board][] and [web board][].
[web board]: https://gitlab.torproject.org/groups/tpo/web/-/boards
[TPA board]: https://gitlab.torproject.org/groups/tpo/tpa/-/boards
We also reviewed the letter anarcat later sent to `tor-internal@`
(private, not linked here).
Then the meeting paused after one hour.
When we returned on wednesday, we jumped to the roadmap review
(below), and then returned here to briefly review the Backlog.
We reviewed anarcat's queue to make sure things would be okay after he
left, and also made sure kez and lavamind had enough work. gaba will
make sure they are assigned work from the Backlog as well.
# Roadmap review
Review and prioritize:
* [yearly TPA roadmap][]
* web
* [board][]
* [priorities][]
[yearly TPA roadmap]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2021
[board]: board
[priorities]: https://gitlab.torproject.org/tpo/web/team
## Web priorities allocations (sort out by priorities)
We reviewed the priorities page and made sure we had most of the stuff
covered. We don't assign tasks directly in the wiki page, but we did a
tentative assignation pass here:
* Donations page redesign (support to Openflows) - kez
* Onion Services v2 deprecation support - lavamind
* Improves bridges.torproject.org - kez
* Remove outdated documentation from the header - kez & gus
* Migrate blog.torproject.org from Drupal To Lektor: it needs a
milestone and planning - lavamind
* Support forum - lavamind
* Developer portal - lavamind & kez
* Get website build from jenkins into to gitlabCI for the static
mirror pool (before December) - kez
* Get up to speed on maintenance tasks:
* Bootstrap upgrade - lavamind
* browser documentation update (this is content and mostly is on
gus's plate) gus
* get translation stats available - kez
* rename 'master' branch as 'main' - lavamind
* fix wiki for documentation - gaba
* get onion service tooling into tpo gitlab namespace - lavamind
## TPA roadmap review
We reviewed the TPA roadmap for the first time since the beginning of
the year, which involved going through the first two quarters to
identify what was done and missed. We also established the priorities
for Q3 and Q4. Those changes are mostly contained in [this commit on
the wiki][].
[this commit on the wiki]: https://gitlab.torproject.org/tpo/tpa/wiki-replica/-/commit/5968075c75d4249…
# Other discussions
No new item came up in the meeting, which already was extended an
extra hour to cover for the extra roadmap work.
# Next meeting
* we do quick checkin on monday 14 UTC 10 eastern, at the beginning
of the office hours
* we do monthly meetings instead of checkins on the first monday of
the month
# Metrics of the month
Those were sent on June 2nd, it would be silly to send them again.
--
Antoine Beaupré
torproject.org system administration
Hi!
In the last couple of weeks we reviewed teams priorities for the next
few months. We will do this again at the end of September. I'm sending
this mail for transparency on what the Tor Project is focus on right now.
# General priorities for the Tor Project
performance
network health
moving tor from C to Arti
improving censorship circumvention
usability
Work board: https://gitlab.torproject.org/groups/tpo/-/boards
# Applications
1. V2 Deprecation UI/UX
2. about:torconnect
3. Desktop ESR Transition - starts in July
4. Enable HTTPS-Only Mode
5. Integrate libhttps-everywhere-core (replacing https-everywhere
webextension) - tor-browser#40458
6. Go/Rust/Java dependency resolution: how to resolve dependencies ahead
of time.
7. improve bridge selection in tor browser
Work board: https://gitlab.torproject.org/groups/tpo/applications/-/boards
# Core
1. security fixes
2. tor network performance
- shadow experiments
- congestion control
3. arti
- margot things for network health team
- shadow simulations should be possible with arti
4. CI with follow up with sysadmin teams
5. HS v2 deprecation
- Onion Balance for V3: Multi-service support. Possibly to reduce
memory consumption to avoid having to run multiple instances.
6. Shadow v2 will come out in this period.
7. Update the fallbackdirs list
Work board: https://gitlab.torproject.org/groups/tpo/core/-/boards
# Network Health
1. Run the "bad hsdir" hunter scripts and other bad-relay detection
scripts.
2. Deploy sbws on all bandwidth authorities replacing Torflow and fix
critical issues
3. Various work within our performance and scalability project
4. Support for researchers for network experiments
5. Integrate metrics into network-health
6. bad-relay tooling improvements
7. Improve user support for relays operators
8. Handle EOL relays
9. sbws2
10. Start accumulating a list of metrics we want to have from arti-based
relays
11. Map out possible plans for quantifying and improving our trust in
relays/operators
12. Surprise 'anomaly analysis' on the network as needed
Work board: https://gitlab.torproject.org/groups/tpo/network-health/-/boards
# Anti-Censorship
1. Better bridge distribution strategies (e.g. Conjure, Salmon)
2. Make GetTor more reliable and reduce maintenance burden
3. Rewrite gettor in Go to include it in rdsys
4. Complete integration of bridgedb into the more general rdsys
5. Improve the performance of Snowflake for users in Asia
6. Provide more feedback to users that run Snowflake proxies
7. Tor reachability from various countries
Work board:
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
# TPI Websites
1. Donations page redesign
2. Improve bridges.torproject.org
3. Remove outdated documentation from the header
4. Migrate blog.torproject.org from Drupal To Lektor
5. Support forum
6. Developer portal
7. Get website build from jenkins into to gitlabCI for the static mirror
pool (before December)
8. Maintenance tasks:
- Bootstrap upgrade
- browser documentation update
- get translation stats available
- rename 'master' branch as 'main'
- fix wiki for documentation
- get onion service tooling into tpo gitlab namespace
Work board: https://gitlab.torproject.org/groups/tpo/web/-/boards
# Tor Project Administrators
Priorities: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2021
Work board: https://gitlab.torproject.org/tpo/tpa/team/-/boards/117
# User Experience
1. Tor Browser 10.5 > V2 Onion Services Deprecation.
2. Tor Browser 10.5 > Improving connecting to Tor.
3. Tor Browser 10.5 release, 11.0 planning.
4. User Research > Reporting Tor Browser survey from Q1/Q2 2021.
5. Working with net on arti DX.
6. Working with applications on enabling HTTPS-Only Mode.
7. Working with circumvention on improving bridges.torproject.org.
8. Working with community on publishing peers material in
community.torproject.org.
9. Working with community on Tor Documentation Hackathon.
10. Working with community on support.torproject.org.
11. Working with fundraising on donate.torproject.org.
12. Working with web on dev.torproject.org.
13. Working with comms on social media suppport.
Work board:
https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&…
# Community Team
1. Organize Tor facilitators meetup with human rights defenders and
partners in Latam and East Africa.
2. Release the new Community section with Tor training resources.
3. Work with Outreachy intern to help Tor users and delivery user
monthly feedback report to Tor Browser developers and other teams.
4. Organize the 3rd #DocsHackathon to update Tor websites and user
support documentation.
5. Work on the new support forum and blog comments platform with UX,
Web, and TPA team.
6. Contributing with Comms team on contents and campaigns related to our
community, mission, and values.
7. Contact and build partnerships with new human rights defenders
organizations in the Global South.
8. Localize the new donate.torproject.org website.
9. Follow up #KeepItOn and Stop Stalkerware coalitions work and campaigns.
10. Monthly Tor localization hangout with translators.
11. Meet with Tor relay associations and relay operators for Network
Health team & research.
12. Build stats and automation for localization and Tor websites.
13. Publish reviewed translations to the websites and Tor Browser.
Work board: https://gitlab.torproject.org/groups/tpo/community/-/boards
cheers,
gaba
Hello everybody:
Prepare yourself for this Friday June 18th Localization Hangout!
We are going to go through the priorities of translation for the
Tor Project, solve doubts, share localization tricks and tools,
favorite translation music, funny anecdotes, long held questions...
We are starting at 12 UTC on the chat, and we will have a BBB call
as well, at 13 UTC.
More information and how to join:
https://gitlab.torproject.org/tpo/community/l10n/-/wikis/Monthly-Tor-Locali…
emmapeel
Localization Coordinator
Tor Project
Hi,
I have released Onionoo 8.0-1.26.0.
Onionoo is a web-based protocol to learn about currently
running Tor relays and bridges. Onionoo itself was not designed
as a service for human beings—at least not directly. Onionoo
provides the data for other applications and websites which in
turn present Tor network status information to humans.
The latest changes:
Medium changes
* Use IPFire geolocation database instead of MaxMind.
* Fix inconsistencies between ASN/GeoIP/rDNS lookups in
summary and details documents.
* Decode percent-encoded characters in the search parameter in the
same way as in the other parameters.
* Stop decoding a + sign to a space character in any of the
parameters.
Minor changes
* Simplify logging configuration.
* Set default locale US and default time zone UTC at the
beginning of the execution.
* Update to metrics-lib 2.17.0.
You can find the release at:
https://dist.torproject.org/onionoo/8.0-1.26.0/
You can find the sources at:
https://gitlab.torproject.org/tpo/metrics/onionoo/-/tree/onionoo-8.0-1.26.0
Thanks,
Iain.