[tor-bugs] #21206 [Core Tor/Tor]: Measure client up/down bandwidth for directory requests, split by type

[Tor Bug Tracker & Wiki]

#21206: Measure client up/down bandwidth for directory requests, split by type
--------------------------+------------------------------------
 Reporter:  nickm         |          Owner:  ahf
     Type:  defect        |         Status:  assigned
 Priority:  Medium        |      Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor  |        Version:
 Severity:  Normal        |     Resolution:
 Keywords:  sponsor4      |  Actual Points:
Parent ID:  #21205        |         Points:  3
 Reviewer:                |        Sponsor:
--------------------------+------------------------------------

Comment (by ahf):

 > We need a way to measure, over time, split up by type of directory
 > request, how much bandwidth a client uses for requests and for
 > responses.

 Right now my approach is to catch the size of the `inbuf` in
 `connection_dir_client_reached_eof` and log a message with the status
 code and the purpose of the request (`DIR_PURPOSE_*`).

 > We should include both directory lookups that are successful and those
 > that are not.

 For the case where we interact with a directory and the status-code is
 erroneous we log that using the above approach. For the case where we
 are simply unable to reach a given directory, we should log an error
 reason as well as how much data that have been fetched before the error
 was encountered. This should be possible in
 `connection_dir_request_failed`.

 > We should, if possible, measure this with and without circuit
 > overhead. But not if it's too hard.
 >
 > We should, if possible, count failed circuits that are opened only for
 > directory requests. But not if it's too hard.

 I haven't looked into the circuit/channel code yet. I'll do that once
 the two first bullets are solved, but hopefully this shouldn't be too
 difficult.

 How should we go on about this measuring code in general:

 - I don't think the log message approach is suitable to enter the
 codebase.

 - Elsewhere in the code we do collect metrics and save them to a set of
   "stats" files. Should we use the same approach for this measurement such
 that
   it can be reused at a later point in time, where someone might, say,
   look into using an entirely different compression algorithm than what
   is provided by zlib or a likewise task that requires knowledge of the
   currently used bandwidth used by directory requests/responses.

 - One of the requests I have for my "onboarding period" is to get a lot
   of feedback on timeframes: I currently have tor running locally that
   emits information on failed directory requests as well as bytes read
   and written for a request. If we go for using an additional stats-file
   I should be able to have some code working at noon Monday that could
   be reviewed. Once that code is in I'll continue with support for
   measurements that includes the overhead of the circuit connection.

   Once that is done we should have a talk about testing the various
   scenarios listed in the parent ticket (#21205).

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/21206#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online