[tor-bugs] #24368 [Core Tor/Tor]: Tune zstd parameters to decrease memory usage during streaming

Tor Bug Tracker & Wiki blackhole at torproject.org
Tue Feb 6 16:15:55 UTC 2018


#24368: Tune zstd parameters to decrease memory usage during streaming
-------------------------------------------------+-------------------------
 Reporter:  teor                                 |          Owner:  ahf
     Type:  defect                               |         Status:
                                                 |  assigned
 Priority:  Medium                               |      Milestone:  Tor:
                                                 |  0.3.4.x-final
Component:  Core Tor/Tor                         |        Version:  Tor:
                                                 |  0.3.1.1-alpha
 Severity:  Normal                               |     Resolution:
 Keywords:  regression, compression, zstd, tor-  |  Actual Points:
  dir, 032-backport                              |
Parent ID:                                       |         Points:  1
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------
Changes (by nickm):

 * milestone:  Tor: 0.3.3.x-final => Tor: 0.3.4.x-final


Comment:

 I've run some initial experimentation, and here's what I found:

 Adjusting the pre-set values shouldn't be necessary if instead we tell
 zstd to build its own parameters (using `ZSTD_getCParams()` or
 `ZSTD_getParams`), with the `estimatedSrcSize` argument to tell zstd how
 big we expect the input to be.

 I also think that our current estimates are higher than zstd actually
 uses, which is a good thing.  I'm attaching a python script that I used
 for these tests; it requires the "zstandard" package.

 Now the catch here is that we can't actually adjust the parameters to
 anything besides the presets unless we use the "advanced" (a.k.a "static-
 only" zstd APIs).  I've opened ticket #25162 about doing that safely.  But
 the complexity is enough that I think we should call this an 0.3.4.x
 ticket: it is more than simply tweaking a couple of numbers.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/24368#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online


More information about the tor-bugs mailing list