[tor-bugs] #30125 [Obfuscation/Snowflake]: Port server's log sanitization to client, broker, and proxy-go

[Tor Bug Tracker & Wiki]

#30125: Port server's log sanitization to client, broker, and proxy-go
-----------------------------------+-----------------------------
 Reporter:  dcf                    |          Owner:  cohosh
     Type:  enhancement            |         Status:  merge_ready
 Priority:  Medium                 |      Milestone:
Component:  Obfuscation/Snowflake  |        Version:
 Severity:  Normal                 |     Resolution:
 Keywords:                         |  Actual Points:
Parent ID:                         |         Points:
 Reviewer:                         |        Sponsor:  Sponsor19
-----------------------------------+-----------------------------
Changes (by dcf):

 * status:  needs_review => merge_ready


Comment:

 Replying to [comment:6 cohosh]:
 > Replying to [comment:4 dcf]:
 > > The refactoring looks good. I have a few ideas about deployment to
 save us some trouble later. My main goal is that there should be a clean
 break between the old unsanitized logs and the new sanitized logs, so that
 we don't later have to trawl through a log file and figure out where the
 change happened. This is because I'd like us to extract what we need from
 the old logs and then delete them.
 >
 > Thanks! This looks reasonable to me. Do you have something in mind for
 extracting useful data from the unsanitized logs? I suppose we could write
 a separate scrubber to sanitize them retroactively.

 For myself, I just want to make graphs showing the number of client and
 proxy requests per second, like these from flash proxy:
  * https://people.torproject.org/~dcf/graphs/fp-facilitator-201601.png
  * https://people.torproject.org/~dcf/graphs/fp-num-proxies-201601.png

 I'm fine with keeping/publishing a scrubbed version of the old logs, or
 e.g. CSV files derived from them. I don't think we should keep the
 originals indefinitely. I'll add this topic to the agenda for the next
 check-in meeting.

 > > For proxy-go, it will be similar, except that there are several /home
 /snowflake-proxy/*.log.d log directories. Also /home/snowflake-proxy
 /snowflake-proxy-*.log{,.xz} are unsanitized logs from before we started
 using runit log directories (happened in #28390).
 > I've noticed that there are a lot of old logs from different proxy-go
 instances. I'll set up the tarball to keep the directory structure, but I
 guess my question is the same as above about what we're planning on using
 these logs for.

 Yeah, the log structure changed in the past in order to allow compression
 and rotation, because we ran out of disk space using single files :/ (That
 was #28390.) For me personally, I don't have any use in mind for the old
 proxy-go logs and would be fine with just deleting them.

 > > For the client, we'll need a Tor Browser ticket to pick up the
 upgrade. A sample ticket and patch that can serve as a template is #26795.
 I know you are interested in the reproducible build and this would be a
 good introduction to
 [[doc/TorBrowser/Hacking#BuildingOfficialTorBrowserReleaseBinaries|rbm]]
 if you haven't used it yet. Basically, you just need to edit
 projects/snowflake/config and update `git_hash`, then run `make testbuild`
 to make sure it still builds, then open a ticket in the Applications/Tor
 Browser component.
 > Cool! I also wanted to ask you about thoughts you have about when to
 make snowflake client releases. I'm assuming it's just whenever there are
 changes we think are important to have people start using. But I also
 don't want to overwhelm the applications team.

 Yes, so far it's whenever there's a change we want people to start using.
 Doing it once per alpha release is not too much. As long as you test that
 the build works across all platforms (that's what `make testbuild` does),
 it's not so much trouble for the Tor Browser devs--it's when something
 breaks the build and they have to start backing out changes that it's
 cumbersome. IMO it's justified in this case and also a good excuse to file
 a first Tor Browser ticket.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/30125#comment:7>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online