[tbb-dev] Media Codecs and the canPlayType/Media Capabilities API

[Tom Ritter]

On Thu, 31 Jan 2019 at 10:51, Georg Koppen <gk at torproject.org> wrote:
> Skimming the spec a bit, following the advice of the spec in §4.1 in
>
> https://wicg.github.io/media-capabilities/#security-privacy-considerations
>
> seems not unreasonable to me at first glance. But I am still a bit
> unsure about the trade-offs here as I'd need to look closer at all the
> things besides the codec being exposed. Do we have a table somewhere
> showing the entropy those things add? CanPlayType and actually trying to
> play it give just a yes/no back per codec, right?

canPlayType can return yes, no, or maybe. The maybe case *appears* to
happen when the container format of the media allows the codec
parameter; but the parameter is not provided. Someone trying to
fingerprint could induce the maybe case but doing so would get them no
benefit when they could get a definitive answer; so it is effectively
'yes/no'.

> How many bits get
> exposed by the Media Capabilities API?

It exposes three bits of information about a request: support, smooth,
and power efficient. powerEfficient maps to whether the codec is
hardware accelerated. smooth is always true except for
non-hardware-accelerated VP9 on low power devices.

Effectively this means that Media Capabilities exposes:
a) If you don't have hardware-accelerated VP9, a fairly precise CPU benchmark
b) Whether certain codecs are hardware accelerated
c) The codecs supported

I tried running down (b).  It's pretty tough to get precise info.
I found one case in WMFVideoMFTManager relating to .wmf playback but I
couldn't figure out what the hardware feature involved was easily.
On mac it appears that all modern macs (2010 and newer) have hardware
acceleration of h264. HEVC/h265 is hardware accelerated on some
models. [0]

For the rest I couldn't figure out how we get that information from
Mozilla's code =/  Wikipedia says:
https://en.wikipedia.org/wiki/VP9#Hardware_implementations
There's also opus, vp8...

AFAIK no one's built a media codec fingerprinting test page.  It
requires a listing of codecs to test for and I don't know an
exhaustive list yet.

[0] * MacBookPro October 2016 and newer: yes
* iMac 5K, Late 2015: yes
* iMac 21.5, 4K and 5K, June 2017: yes
* iMac Pro: yes
* any Mac mini, MacBook Air, Mac Pro or earlier releases of iMac or
MacBook Pro are not

> When the spec is saying:
>
> """
> This information is expected to have a high correlation with other
> information already available to the web pages as a given class of
> device is expected to have very similar decoding/encoding capabilities.
> """
>
> what other information available to web pages is it talking about? Do we
> spoof/deal with that already?

I imagine it's referring to the number of CPU cores as well as a rough
CPU benchmark that could be done by WebGL/javascript...

We don't really intentionally try to protect against an attacker
benchmarking your computer's performance; aside from making WebGL
click to play, turning off the JIT optimizations....

> And if so, would it help here taking the
> decisions that got made for them into account while developing defense?
> And if not, would it help tackling those other vectors together with the
> codec one?

/shrug

Preventing benchmarking by someone who wants to do it against Tor
Browser seems very difficult...

> > I don't think we know what codecs we care about; or are installed by ffmpeg/WMF.
> > I (personally) don't know if Firefox will pass non-standard codec
> > requests out to the system to see "Hey do you happen to have a codec
> > for 'randomweirdthing'"
> > I don't think we know what the install rate for ffmpeg/WMF is.
> > We have some data about codec use; however I don't fully understand
> > it: https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2019-01-10&include_spill=0&keys=video%252Favc!audio%252Fmp4a-latm!audio%252Fopus!video%252Fvp9&max_channel_version=beta%252F65&measure=MEDIA_CODEC_USED&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2018-12-10&table=0&trim=1&use_submission_date=0
>
> I wonder how we could get the answer to all of those questions to
> actually make an informed decision as you suggest.

I suppose step one is to make a fingerprinting test that identifies
what codecs you can handle...

-tom