[tor-bugs] #28877 [Core Tor/Tor]: Paginate large controller commands like 'GETINFO desc/all-recent'

Tor Bug Tracker & Wiki blackhole at torproject.org
Wed Dec 19 08:54:16 UTC 2018


#28877: Paginate large controller commands like 'GETINFO desc/all-recent'
--------------------------+--------------------------
 Reporter:  wagon         |          Owner:  (none)
     Type:  defect        |         Status:  assigned
 Priority:  Medium        |      Milestone:
Component:  Core Tor/Tor  |        Version:
 Severity:  Normal        |     Resolution:
 Keywords:                |  Actual Points:
Parent ID:                |         Points:
 Reviewer:                |        Sponsor:
--------------------------+--------------------------

Comment (by wagon):

 Hello, atagar. Thank you for the explanation, now it is more clear for me.

 > saturates the control connection, preventing further commands and events
 from being transmitted.
 What it actually saturates? Indeed, in some setups I did see this problem,
 but `tor-prompt` doesn't have  it. Is it some internal TCP or socket thing
 that is saturated? Can that limit be increased by setting some preference
 in the system?

 > but I've been cautioned that any direct use of tor's data directory is a
 bad idea.
 I agree. Furthermore, you cannot use tor's data directory if you are not
 running Nyx as `debian-tor` user. If your user is just a member of
 `debian-tor` group (this is recommended setup, see discussion in #25890),
 you still cannot read files in that directory because of #28356.

 > We need some way of breaking up these responses. Pagination probably
 isn't a good fit so ideas welcome.

 > The above is a long time problem I've had with tor's 'get all
 descriptor' commands
 Could you explain why it is a problem? Yes, it may block controller for a
 few seconds if user type this command in Nyx interpreter, but only in this
 case. This blocking would happen anyway because printing the output to a
 terminal always requires some time. If you need this output internally in
 some functions in Stem or Nyx, you can get it almost immediately. In
 addition you don't need to run such commands very often. Maybe you need to
 run them only once, when Nyx is started (then response can be cached for
 some time).

 To be clear, let us check exact numbers. To be sure we are not wrong here
 because of specifics in implementation of Nyx or Stem, we can use the
 following simple script `test.sh` for these tests:
 {{{
 #!/bin/bash -e

 cmd="$@"
 pass="ControlPortPassword"

 function test_tor() {
     echo "$1" >&3
     sed "/^250 OK\r$/q" <&3
     echo QUIT >&3
     exec 3<&-
 }

 exec 3<>/dev/tcp/127.0.0.1/9051
 echo AUTHENTICATE \"$pass\" >&3
 read -u 3
 test_tor "$cmd"
 }}}
 Now fun begins. Let us check some simple fast command:

 {{{
 $ time tor-prompt --run 'GETINFO version' 1>/dev/null
   0.14s user 0.04s system 77% cpu 0.227 total
 $ time ./test.sh 'GETINFO version' 1>/dev/null
   0.00s user 0.00s system 36% cpu 0.011 total
 }}}

 Compare it with more heavy output such as `ns/all`:
 {{{
 $ time tor-prompt --run 'GETINFO ns/all' 1>/dev/null
   0.28s user 0.07s system 58% cpu 0.591 total
 $ time ./test.sh 'GETINFO ns/all' 1>/dev/null
   0.02s user 0.00s system  8% cpu 0.230 total
 }}}

 Now check it with the most resource consuming command, `desc/all-recent`:
 {{{
 $ time tor-prompt --run 'GETINFO desc/all-recent' 1>/dev/null
 Traceback (most recent call last):
   File "/path/to/stem/tor-prompt", line 8, in <module>
     stem.interpreter.main()
   File "/path/to/stem/stem/interpreter/__init__.py", line 151, in main
     interpreter.run_command(args.run_cmd, print_response = True)
   File "/path/to/stem/stem/util/conf.py", line 289, in wrapped
     return func(*args, config = config, **kwargs)
   File "/path/to/stem/stem/interpreter/commands.py", line 381, in
 run_command
     print(output)
 UnicodeEncodeError: 'ascii' codec can't encode character u'\u021b' in
 position 1237805: ordinal not in range(128)

 $ time ./test.sh 'GETINFO desc/all-recent' 1>/dev/null
   0.21s user 0.03s system 60% cpu 0.391 total
 }}}

 People always say that shell is too slow and inconvenient, while python is
 really fast and convenient. However, as you see, simple shell is 20-30
 times faster than python on simple commands, about 2-3 times faster than
 python on heavy commands, and gets many megabytes output of `desc/all-
 recent` within less that half of second. Old UNIX style still rocks. Had
 you choose Bash or some other shell instead of python for writing Stem?

 If I don't redirect output to `/dev/null`, terminal becomes a bottleneck
 for these commands. It will take about 5-6 seconds to print `decs/all-
 recent` output to a terminal. Nyx interpreter is even more slow than `tor-
 prompt` (I think it is due to curses), it will take 15 seconds or like
 that for this printing. If I don't redirect output to `/dev/null`, the
 last `tor-prompt` command doesn't  crash.

 Thus, I still doubt we need to ask tor network people to do something with
 it. As you see, well written tools work fast with current `ControlPort`
 implementation.

 > Feel free to file a separate ticket with the 'nyx --debug' output when
 Nyx freezes so I can see what's up.
 I've created #28902. Now we have many tickets where I didn't get reply
 from you yet, and our discussion stopped, so I'm afraid I overload you
 with many tasks or questions.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/28877#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online


More information about the tor-bugs mailing list