[tor-relays] How to reduce tor CPU load on a single bridge?
David Fifield
david at bamsoftware.com
Sat Jan 29 02:58:49 UTC 2022
> On the matter of onion key rotation, I had the idea of making the onion key files read-only. Roger did some source code investigation and said that it might work to prevent onion key rotation, with some minor side effects. I plan to give the idea a try on a different bridge. The possible side effects are that tor will continue trying and failing to rotate the onion key every hour, and "force a router descriptor rebuild, so it will try to publish a new descriptor each hour."
Making secret_onion_key and secret_onion_key_ntor read-only does not quite work, because tor first renames them to secret_onion_key.old and secret_onion_key_ntor.old before writing new files. (Making the *.old files read-only does not work either, because the `tor_rename` function first unlinks the destination.)
https://gitweb.torproject.org/tor.git/tree/src/feature/relay/router.c?h=tor-0.4.6.9#n497
But a slight variation does work: make secret_onion_key.old and secret_onion_key_ntor.old *directories*, so that tor_rename cannot rename a file over them. It does result in an hourly `BUG` stack trace, but otherwise it seems effective.
I did a test with two tor instances. The rot1 instance had the directory hack to prevent onion key rotation. The rot2 had nothing to prevent onion key rotation.
```plain
# tor-instance-create rot1
# tor-instance-create rot2
```
/etc/tor/instances/rot1/torrc:
```plain
Log info file /var/lib/tor-instances/rot1/onionrotate.info.log
BridgeRelay 1
AssumeReachable 1
BridgeDistribution none
ORPort 127.0.0.1:auto
ExtORPort auto
SocksPort 0
Nickname onionrotate1
```
/etc/tor/instances/rot2/torrc:
```plain
Log info file /var/lib/tor-instances/rot2/onionrotate.info.log
BridgeRelay 1
AssumeReachable 1
BridgeDistribution none
ORPort 127.0.0.1:auto
ExtORPort auto
SocksPort 0
Nickname onionrotate2
```
Start rot1, copy its keys to rot2, then start rot2:
```plain
# service tor at rot1 start
# cp -r /var/lib/tor-instances/rot1/keys /var/lib/tor-instances/rot2/
# chown -R _tor-rot2:_tor-rot2 /var/lib/tor-instances/rot2/keys
# service tor at rot2 start
```
Stop the two instances, check that the onion keys are the same, and that `LastRotatedOnionKey` is set in both state files:
```plain
# service tor at rot1 stop
# service tor at rot2 stop
# ls -l /var/lib/tor-instances/rot*/keys/secret_onion_key*
-rw------- 1 _tor-rot1 _tor-rot1 888 Jan 28 22:57 /var/lib/tor-instances/rot1/keys/secret_onion_key
-rw------- 1 _tor-rot1 _tor-rot1 96 Jan 28 22:57 /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor
-rw------- 1 _tor-rot2 _tor-rot2 888 Jan 28 23:05 /var/lib/tor-instances/rot2/keys/secret_onion_key
-rw------- 1 _tor-rot2 _tor-rot2 96 Jan 28 23:05 /var/lib/tor-instances/rot2/keys/secret_onion_key_ntor
# md5sum /var/lib/tor-instances/rot*/keys/secret_onion_key*
fb2a8a8f9de56f061eccbb3fedd700c4 /var/lib/tor-instances/rot1/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5 /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor
fb2a8a8f9de56f061eccbb3fedd700c4 /var/lib/tor-instances/rot2/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5 /var/lib/tor-instances/rot2/keys/secret_onion_key_ntor
# grep LastRotatedOnionKey /var/lib/tor-instances/rot*/state
/var/lib/tor-instances/rot1/state:LastRotatedOnionKey 2022-01-28 22:57:14
/var/lib/tor-instances/rot2/state:LastRotatedOnionKey 2022-01-28 23:11:04
```
Set `LastRotatedOnionKey` 6 weeks into the past to force an attempt to rotate the keys the next time tor is restarted:
```plain
# sed -i -e 's/^LastRotatedOnionKey .*/LastRotatedOnionKey 2021-12-15 00:00:00/' /var/lib/tor-instances/rot*/state
# grep LastRotatedOnionKey /var/lib/tor-instances/rot*/state
/var/lib/tor-instances/rot1/state:LastRotatedOnionKey 2021-12-15 00:00:00
/var/lib/tor-instances/rot2/state:LastRotatedOnionKey 2021-12-15 00:00:00
```
Create the secret_onion_key.old and secret_onion_key_ntor.old directories in the rot1 instance.
```plain
# mkdir -m 700 /var/lib/tor-instances/rot1/keys/secret_onion_key{,_ntor}.old
```
Check the identity of keys before starting:
```plain
# md5sum /var/lib/tor-instances/rot*/keys/secret_onion_key*
fb2a8a8f9de56f061eccbb3fedd700c4 /var/lib/tor-instances/rot1/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5 /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor
md5sum: /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor.old: Is a directory
md5sum: /var/lib/tor-instances/rot1/keys/secret_onion_key.old: Is a directory
fb2a8a8f9de56f061eccbb3fedd700c4 /var/lib/tor-instances/rot2/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5 /var/lib/tor-instances/rot2/keys/secret_onion_key_ntor
```
Start both instances:
```plain
# service tor at rot1 start
# service tor at rot2 start
```
Verify that the rot1 instance is still using the same onion keys, while rot2 has rotated them:
```plain
# ls -ld /var/lib/tor-instances/rot*/keys/secret_onion_key*
-rw------- 1 _tor-rot1 _tor-rot1 888 Jan 28 23:45 /var/lib/tor-instances/rot1/keys/secret_onion_key
-rw------- 1 _tor-rot1 _tor-rot1 96 Jan 28 23:45 /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor
drwx--S--- 2 root _tor-rot1 4096 Jan 28 23:44 /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor.old
drwx--S--- 2 root _tor-rot1 4096 Jan 28 23:44 /var/lib/tor-instances/rot1/keys/secret_onion_key.old
-rw------- 1 _tor-rot2 _tor-rot2 888 Jan 28 23:47 /var/lib/tor-instances/rot2/keys/secret_onion_key
-rw------- 1 _tor-rot2 _tor-rot2 96 Jan 28 23:47 /var/lib/tor-instances/rot2/keys/secret_onion_key_ntor
-rw------- 1 _tor-rot2 _tor-rot2 96 Jan 28 23:05 /var/lib/tor-instances/rot2/keys/secret_onion_key_ntor.old
-rw------- 1 _tor-rot2 _tor-rot2 888 Jan 28 23:05 /var/lib/tor-instances/rot2/keys/secret_onion_key.old
# md5sum /var/lib/tor-instances/rot*/keys/secret_onion_key*
fb2a8a8f9de56f061eccbb3fedd700c4 /var/lib/tor-instances/rot1/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5 /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor
md5sum: /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor.old: Is a directory
md5sum: /var/lib/tor-instances/rot1/keys/secret_onion_key.old: Is a directory
fb8a5e8787141dba4e935267f818cc2a /var/lib/tor-instances/rot2/keys/secret_onion_key
2c3f7d81e96641e2c04fb9c452296337 /var/lib/tor-instances/rot2/keys/secret_onion_key_ntor
2066ab7e01595adf42fc791ad36e1fc5 /var/lib/tor-instances/rot2/keys/secret_onion_key_ntor.old
fb2a8a8f9de56f061eccbb3fedd700c4 /var/lib/tor-instances/rot2/keys/secret_onion_key.old
```
The rot1 instance's `LastRotatedOnionKey` remains the same, while rot2's is updated:
```plain
# grep LastRotatedOnionKey /var/lib/tor-instances/rot*/state
/var/lib/tor-instances/rot1/state:LastRotatedOnionKey 2021-12-15 00:00:00
/var/lib/tor-instances/rot2/state:LastRotatedOnionKey 2022-01-28 23:47:02
```
The rot1 instance's log shows the failure to rotate the keys:
/var/lib/tor-instances/rot1/onionrotate.info.log
```plain
Jan 28 23:46:59.000 [info] rotate_onion_key_callback(): Rotating onion key.
Jan 28 23:46:59.000 [warn] Couldn't rotate onion key.
Jan 28 23:46:59.000 [info] router_rebuild_descriptor(): Rebuilding relay descriptor (forced)
...
Jan 28 23:46:59.000 [info] check_onion_keys_expiry_time_callback(): Expiring old onion keys.
```
While the rot2 rotation was successful:
/var/lib/tor-instances/rot2/onionrotate.info.log
```plain
Jan 28 23:47:02.000 [info] rotate_onion_key_callback(): Rotating onion key.
Jan 28 23:47:02.000 [info] rotate_onion_key(): Rotating onion key
Jan 28 23:47:02.000 [info] mark_my_descriptor_dirty(): Decided to publish new relay descriptor: rotated onion key
```
After 1 hour, the rot1 instance tries to rebuild its relay descriptor, and triggers a `BUG` non-fatal assertion failure in [`router_rebuild_descriptor`](https://gitweb.torproject.org/tor.git/tree/src/feature/relay/router.c?h=tor-0.4.6.9#n2475). I let it run for 1 more hour after that, and it happened again.
/var/lib/tor-instances/rot1/onionrotate.info.log
```plain
Jan 29 00:46:59.000 [info] router_rebuild_descriptor(): Rebuilding relay descriptor (forced)
Jan 29 00:46:59.000 [warn] The IPv4 ORPort address 127.0.0.1 does not match the descriptor address 172.105.3.197. If you have a static public IPv4 address, use 'Address <IPv4>' and 'OutboundBindAddress <IPv4>'. If you are behind a NAT, use two ORPort lines: 'ORPort <PublicPort> NoListen' and 'ORPort <InternalPort> NoAdvertise'.
Jan 29 00:46:59.000 [info] extrainfo_dump_to_string_stats_helper(): Adding stats to extra-info descriptor.
Jan 29 00:46:59.000 [info] read_file_to_str(): Could not open "/var/lib/tor-instances/rot1/stats/bridge-stats": No such file or directory
Jan 29 00:46:59.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/relay/router.c:2452: router_rebuild_descriptor: Non-fatal assertion !(desc_gen_reason == NULL) failed. (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: Tor 0.4.5.10: Non-fatal assertion !(desc_gen_reason == NULL) failed in router_rebuild_descriptor at ../src/feature/relay/router.c:2452. Stack trace: (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(log_backtrace_impl+0x57) [0x5638b9538047] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(tor_bug_occurred_+0x16b) [0x5638b954327b] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(router_rebuild_descriptor+0x13d) [0x5638b94f4e1d] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(+0x21f163) [0x5638b9665163] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(+0x83577) [0x5638b94c9577] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.7(+0x239ef) [0x7f701bae49ef] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.7(event_base_loop+0x52f) [0x7f701bae528f] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(do_main_loop+0x101) [0x5638b94b1321] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(tor_run_main+0x1d5) [0x5638b94acdd5] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(tor_main+0x49) [0x5638b94a92e9] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(main+0x19) [0x5638b94a8ec9] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f701b391d0a] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [warn] Bug: /usr/bin/tor(_start+0x2a) [0x5638b94a8f1a] (on Tor 0.4.5.10 )
Jan 29 00:46:59.000 [info] router_upload_dir_desc_to_dirservers(): Uploading relay descriptor to directory authorities
Jan 29 00:46:59.000 [info] directory_post_to_dirservers(): Uploading an extrainfo too (length 822)
Jan 29 00:46:59.000 [info] rep_hist_note_used_internal(): New port prediction added. Will continue predictive circ building for 3332 more seconds.
Jan 29 00:46:59.000 [info] connection_ap_make_link(): Making internal anonymized tunnel to [scrubbed]:9001 ...
Jan 29 00:46:59.000 [info] connection_ap_make_link(): ... application connection created and linked.
Jan 29 00:46:59.000 [info] check_onion_keys_expiry_time_callback(): Expiring old onion keys.
```
Stopping and restarting the tor1 instance keeps the same onion keys, and the first rotation does not hit the assertion failure:
```plain
# service tor at rot1 stop
# service tor at rot1 start
# md5sum /var/lib/tor-instances/rot*/keys/secret_onion_key*
fb2a8a8f9de56f061eccbb3fedd700c4 /var/lib/tor-instances/rot1/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5 /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor
md5sum: /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor.old: Is a directory
md5sum: /var/lib/tor-instances/rot1/keys/secret_onion_key.old: Is a directory
fb8a5e8787141dba4e935267f818cc2a /var/lib/tor-instances/rot2/keys/secret_onion_key
2c3f7d81e96641e2c04fb9c452296337 /var/lib/tor-instances/rot2/keys/secret_onion_key_ntor
2066ab7e01595adf42fc791ad36e1fc5 /var/lib/tor-instances/rot2/keys/secret_onion_key_ntor.old
fb2a8a8f9de56f061eccbb3fedd700c4 /var/lib/tor-instances/rot2/keys/secret_onion_key.old
```
/var/lib/tor-instances/rot1/onionrotate.info.log
```plain
Jan 29 02:06:13.000 [info] rotate_onion_key_callback(): Rotating onion key.
Jan 29 02:06:13.000 [warn] Couldn't rotate onion key.
Jan 29 02:06:13.000 [info] router_rebuild_descriptor(): Rebuilding relay descriptor (forced)
...
Jan 29 02:06:13.000 [info] check_onion_keys_expiry_time_callback(): Expiring old onion keys.
```
More information about the tor-relays
mailing list