MediaPlayerElement/MediaElement: Closed Caption and Subtitle Styling got Broken by Recent Update

[softworkz]

This is a tough one, because it spans wide and it's interdisciplinary - a kind of RACE condition between multiple

Teams/Areas Involved

• WinUI2, WinUI3 Media Elements
  Microsoft.UI.Xaml
• Windows.Media
  Windows.Media.ClosedCaptioning.ClosedCaptionProperties
• Chrome/Windows and Edge/Windows
• Windows Media Player
• Windows Settings
  Accessibility Section
• New Live Captions Feature
  (probably originated by)

I'm not sure whether this is the best place for this, but for Windows.Media and UWP, there's a cluttered forum only and over at MS.UI.Xaml, an earlier report has been ignored since (precisely) one year. The App Sdk seems to have the widest scope and - lastly - has the most welcoming text 😄
Feel free to move it, though.

Symptoms

WinUI3: Subtitles are not visible

There's no way for an application to change that through the media apis.

WinUI2: Subtitle Styles cannot be controlled

Certain properties of TimedTextStyle, most importantly the FontFamily.

Impact

This is showstopping for us. We are migrating a media player application to UWP (xbox) and WinUI3 and are blocked by this.
I guess the "invisible" subtitles effect doesn't need further explanation, and regarding the font setting, one might easily think that this is a minor problem - but it's not like that:

• Videos often use different fonts and styles for different situations, which aren't shown due to the current situation
• Videos often have special fonts embedded, especially in Asian titles where subtitles have even become a kind of an art form
  Not being able to properly display them won't be acceptable for many users
• We have our own settings for overriding subtitle styling and this can't be driven from two different places with conflicting settings
• Many subtitles have their own rich styling, which would never become visible when overriding settings exist which cannot be deactivated

But it's not just about us. It probably affects Millions of systems and not only custom WinUI apps - it also includes Media Player and Chrome + Edge browsers.

It's not invisible on the latter, but they're still affected. To understand the issue, we need to step back a little first.

Background

After intensive research, I have come to the following picture of what has happened:

Attempting to Reconstruct

In 2015, a new class was introduced in the Windows.Media namespace: ClosedCaptionProperties.
The static readonly properties are corresponding to these values in the registry:

(also mirrored under HKCU)

The purpose of this is obviously a system-wide (or rather "user-session-wide") override of caption/subtitle styles to provide an adjustable experience which fits everybody's needs. A very good thing without doubt!

The controls on Windows 10 looked like this:

Pattern: Default + Optional Override

Here we get to the core of the subject:

• When looking at the settings, all are set to default
• Looking at the registry values, all are set to zero
• All the enums of the properties are starting with a "Default" entry valued zero

The pattern is clear and makes sense: Those settings do nothing until the user deliberately changes a value.

Implementations followed

Probably there are many more cases where implementations were done to make the presentation of captions adhere to those settings. These are the ones I know so far:

• Windows Media Player
  (the new one art least)
• WinUI2 MediaElement
• WinUI3 MediaPlayerElement
  (in CueStyler.cpp)
• Chrome and Edge Browsers
  (caption_style_win.cc)

In all cases this has been implemented consequently and without further ado: When an override is set, then it will be applied.
Again, this makes sense, because nothing happens until a user makes such a setting change. And when he does that, he should be able to rely on getting the result as configured.

The Fatal Mistake

I can only speculate about why it happened - but at least it's quite plausible that during the process of bringing those captions settings into the new Win11 settings UI, someone (or multiple) started to wonder and doubt about all the settings having the value "Default" - which raises the question "what does 'default' even mean'?"
Clearly, such things as audio transcriptions or live captions do not have any default text style - but seemingly it wasn't considered or known that subtitles can very well have their own ("default") styles and even their own embedded fonts.
Even ATSC captions (EIA-708 and CEA-708)) can have some style information. Or maybe it wasn't known that it would affect all the media players and browsers.

No matter why, the new UI for caption styles was introduced:

You can choose between five presets and you can also create new ones - but three mistakes were made:

• There is no default 'empty'' preset like all values at 0)
  => You can't turn off all overrides
• The default entries are not included in the individual settings
  => It is impossible to reset individual setting so that the content defaults are used
• And the worst of all:

The shipment of the update has set override values on - probably - many, most or even all Windows machines in the world

Remember - these are override values, which were meant to be applied explicitly only by users.

And even worse: Users cannot get out of this situation, because it is impossible set all values back to zero (= defaults, no override).
People are forced to view subtitles with the Windows styles settings applied and are unable to set all back to default, to have captions being presented in the original way and style.

What you get from the update is this:

The preset is called "Default", but it's just a preset like all the others and it brings explicit values for all properties.

Individual Remedy

As an individual, you can fix this for your own machine, for example by merging a REG file like this: WindowsCaptionsUser.zip

It adds a custom preset named "Controlled by Application" which has all zeros and sets it as default.
But it's not something an application can do. For once, you can't mess around with a user's settings and as a store or UWP app, you don't even have the permissions to do so at runtime.

Point of Return?

For the MS side, it's a similar problem: It has been rolled out already. Can it be reverted in another update? Just adding a "real default" preset wouldn't be sufficient, it would also need to be set active. I'm not sure whether that's feasible...

If it cannot be done like that, then a different method would be needed. For example a way to prevent those styles from being applied. But all other video playback implementations that I mentioned plus those that I don't know about would all need to implement it on their side to get everything right again.

[softworkz]

Is there anybody in charge or do I need to escalate otherwise?
Thanks

[softworkz]

Thanks for transferring.

The offending element in this context is the ClosedCaptionProperties class which conveys the data from the registry key (mentioned above) to CueStyler.cpp (in case of WinUI3).
Unfortunately there is no mechanism implemented to prevent this from happening.

Besides changing the implementation (which only MS can do), I had identified the following ways as potential methods for working around this bug:

1 Unregister the WinRT Class ClosedCaptionProperties

WinUI3

CueStyler.cpp expects a CLASS_E_NOT_REGISTERED error, in which case it is just resorting to apply the settings from TimedTextStyle.

This might work with a WinUI3 app with runFullTrust, but it would require to manipulate the general registration of this class, which would be a really dirty way.

WinUI2

Yet, runFullTrust is not the goal and - most importantly - we're also targeting Xbox/UWP where permissions don't allow any such manipulation.

2 Ship a Custom Replacement for ClosedCaptionProperties

WinUI2

I made some attempts to include a custom implementation of the IClosedCaptionProperties interface in the app, exposed via CSWinRT, but at some point during debugging I ended up at a place where some guards were implemented to prevent an app from providing types in the Windows.x namespace.

What I haven't tried is to do the same but as a C++/WinRT component. I'm not sure whether this might have better chances to inject a runtime replacement for ClosedCaptionProperties...?

3 Manipulate at the Registry Level

WinUI3

Without runFullTrust, there's no way for the application to change those registry values, but eventually I discovered a (rarely documented) way to include certain registry settings in the app package for apps that are running containerized.
This is the only little success I had. It allowed me to override all those ClosedCaptioning values from that registry key, setting them all to 0. It's still not possible to change the values at runtime (like for setting the text effect for which no setting exists in TimedTextStyle), but at least it's an install-time solution to eliminate those caption style settings from Control Pane => accessibility => Caption Themes).

WinUI2

UWP uses a different containerization model and doesn't seem to support those install-time registry settings (no matter what I tried). Even though the tooling for viewing data/registry in the context of the running application was showing the override values, in the registry, the UWP app itself got the actual registry data instead of the overridden ones from the container.

Windows.Media.dll

I "analyzed" the core implementation of Windows.Media, specifically looking for some kind of mechanism to disable or intercept the behavior of ClosedCaptionProperties. I even found one. there's an undocumented way to redirect the registry key from which the configuration is read. But that's rather for achieving a system-wide override for caption settings and requires changing keys under HKLM.
So, it's again not doable for a UWP application package.

Summing Up

There's a half-way reasonable workaround for WinUI3 - but nothing for UWP.
So, the focus and major concern is on WinUI2/UWP for Windows and Xbox

What can be done to resolve or workaround this on UWP? We want to get this ready and ship it to the store rather sooner thatn later.

Thanks again
sw

[brabebhin]

I wouldn't call those workarounds. This is the kind of headache that disqualifies UWP/winui 3 for any project.

[softworkz]

I think in this case, it's been about an action taken on one side (caption styles in sysctl dropping defaults) after it had previously been mandated that all playback components have to obey to those settings (while relying on the premise that defaults will always be neutral).

But what's really not understandable is that it has been reported a year ago already (even when the full impact might not have been visible) and no action has been taken since then. A new framework will always have bugs, but what makes the difference between a solid and relatable platform on which want to build and rely on - or something whether rather like to turn around quickly is whether and how quickly such issues are getting addressed.

It appears that the subtitle rendering has been completely re-built for WinUI3 (presumably due to its disabilities in composition), and when looking at the code and all the (admittedly nice and funny) ASCII paintings there, it surely wasn't done quickly and under time pressure - easily a 6-figure total. With all the effort, time and money having been invested in just this part, it should actually be a red alert for every program manager, when it turns out that the result of that effort gets nullified by the fact that nobody can use it.

Hopefully ASCII art can be suspended for a moment for getting this resolved.

[brabebhin]

This kind of bug in this kind of endeavour shouldn't have seen the outsides of Microsoft offices. Makes you question how did they test the implementation in the first place.

[softworkz]

I believe that the captions settings change came afterwards.

It's also this change alone which is to blame for the mess that it has created.

What goes on this team, is "just" the fact that it hasn't been addressed.

[softworkz]

@codendone - Do you have any updates, what's the status?

Thanks

[softworkz]

Any news?