[tbb-dev] Media Codecs and the canPlayType/Media Capabilities API

[Tom Ritter]

This is an old concern I'm trying to revive to figure out a path
forward on. I'm going to do my best to summarize this issue.
The issue at hand is that codec support is not uniform across all
users.  The 'why' for this is that ffmpeg is not installed on all
Linux machines; and that certain versions of Windows (N and KN
versions in Europe) don't have the Windows Media Pack.

I don't know for certain of any other situations where unusual codecs
might be present and usable that allow fingerprinting or situations
where common codecs would be missing or disabled (excepting, of
course, about:config changes.)  But it seems likely? If a user has
installed a weird and unusual codec, the browser might still be able
to play the video because the codec's on the system?  Or perhaps a
user is on an old Linux that has an old ffmpeg that doesn't support a
newer codec?

I don't think we have data about how common/uncommon it is that Linux
users don't have ffmpeg or Windows users are missing WMF; although an
approximate answer was '90% or more people have it.'  Firefox detects
when a user tries to play a video with a codec and it doesn't work;
MediaWMFNeeded is the most relevant; it redirects people to a page
that tells them to install WMF:
https://support.mozilla.org/en-US/kb/fix-video-audio-problems-firefox-windows
https://searchfox.org/mozilla-central/source/dom/locales/en-US/chrome/dom/dom.properties#127
https://searchfox.org/mozilla-central/rev/bee8cf15c901b9f4b0c074c9977da4bbebc506e3/dom/media/doctor/DecoderDoctorDiagnostics.cpp#655

So the codec is the issue; but how it gets exposed through the
platform is three ways:
- canPlayType - and 'old' API that asks if a <video> element can play
a particular codec
- The Media Capabilities API - the new style API
- Just trying to play videos with different codecs and seeing if it works

In addition to exposing codec information; it also exposes things like
whether certain codecs are hardware accelerated or if a machine can
handle high resolution video or not.

My understanding is that the third one is essentially impossible to
prevent; absent requiring the user to manually click play for each
video.

As far as a path forward; there are a few ideas:
 - Maybe we can allowlist codecs so only certain codecs are usable?
 - In the short term; we can modify canPlayType and Media Capabilities
to accurately report (effectively) whether or not the user has
WMF/ffmpeg or not but lie about hardware acceleration and all other
aspects.  (We have a patch for this written.)
 - Alternately; we could lie and say no one has it installed (this
would be a bad experience for most users) or everyone has it installed
(bad for a small percentage of users)
 - Tor Browser could detect whether or not you have ffmpeg / WMF and
inform you in some capacity that installing this software will make
you less identifiable
 - Tor can probably bundle ffmpeg in some capacity; but it can't
bundle WMF due to licensing.

All of these have some drawbacks; and we probably want to use data to
make an informed decision.
I don't think we know what codecs we care about; or are installed by ffmpeg/WMF.
I (personally) don't know if Firefox will pass non-standard codec
requests out to the system to see "Hey do you happen to have a codec
for 'randomweirdthing'"
I don't think we know what the install rate for ffmpeg/WMF is.
We have some data about codec use; however I don't fully understand
it: https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2019-01-10&include_spill=0&keys=video%252Favc!audio%252Fmp4a-latm!audio%252Fopus!video%252Fvp9&max_channel_version=beta%252F65&measure=MEDIA_CODEC_USED&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2018-12-10&table=0&trim=1&use_submission_date=0

There is a bug on file here
https://bugzilla.mozilla.org/show_bug.cgi?id=1461454 ; however most
discussion over the months has taken place in an email thread.

-tom