Unify content_type for Media player play_media

[marcelveldt]

Media player entities have a "media_content_type" parameter for the play_media service call.
That parameter is a bit of a mystery and total chaos across all integrations.

I'm developing Music Assistant that utilizes the media players present in Home Assistant for playback and I quickly realized this is a bit of a mess:

• Some players only play audio from URL if you send "music" as media_content_type
• Some players only play TTS audio and crash if you try to send something else
• Some players require a mime type to be sent (e.g. audio/flac)
• No doubt there are other variations
• Recently a new "resolve_media" was added since TTS moved to media source which resolves into valid mime type but the url is relative unless user sets external url in config.
• Media Source seems to use some half incarnation of this and introduced "media class" ?!

I think this should be consolidated into a more uniform solution.
My proposal (open for debate):

Parameters for play_media service call:

• media_content_id: [string] leave as-is (this can be an ID or URL/URI)
• media_type: Enum with media types (see below)
• content_type: [optional] string with mime type (e.g. audio/flac, video/mp4 etc)
• metadata: [optional] some optional metadata about the item like title etc ? Or better resolve that from the id ?

Enum MediaType:
ARTIST
ALBUM
TRACK
EPISODE
MOVIE
GENERIC_AUDIO
GENERIC_VIDEO
PLAYLIST_AUDIO
PLAYLIST_VIDEO
PLAYLIST_MIXED

BONUS: Additional attribute on media_player entity what content_types it supports. Based on that a play media request can be denied or in case of Music Assistant, transcoded to another format.

[parautenbach]

How would one distinguish between a playlist of audio vs video (and I can imagine a mix of these could be possible too)?

[marcelveldt]

That's a good one... Maybe add additional content types to the enum?
PLAYLIST_AUDIO
PLAYLIST_VIDEO
etc

[parautenbach]

That is one option.

I think I need to understand better how these enums are used.

Can only one enum value be used at a time? I can see that artist, album and track all be relevant in the case of audio, so would one need to choose are can they all be used at the same time?

[marcelveldt]

Yes, in this case one enum value of what your are going to stream in relation to the id.
For example content_id = 1 and content_type = playlist you want the player to play playlist with id 1
Or the more generic approach content_id = http://someaudio.wav and content_type 'audio'
etc.

[Mariusthvdb]

uniform solution sounds good, as as we speak I filed several Apple TV issues that underline the issue you seek to solve. see home-assistant/core#71569 just so this is tied and referenced. I do believe its not only the playing (content_type) that requires some love. The Controls are also not correctly applied, eg when playing a track on an Apple TV device (HomePod mini), the device is never released, so we can not click Next, or use the tts service... because 'Already streaming to that device' home-assistant/core#71571.

[marcelveldt]

The enum could be simplified a lot if we don't care exactly what the media type is. In that case it's either "audio" or "video" and "playlist_audio", "playlist_video", "playlist_mixed".

[jjlawren]

I'm not sure this is the best long-term path. Stepping back as we discuss a redesign, is the media_content_type truly required for media_player.play_media playback? Why should the user have to provide the media_content_id and specifically provide the media type in the call? This makes the service call (and automation of them) more complicated than it needs to be.

What if the convention was to (try to) include all necessary identifying information about the media in the media_content_id and optionally allow the use of media_content_type for integrations that absolutely need it? For example, if a MIME type needs to be passed to the device.

Are there going to be situations where the same ID is used with differing types? I can't think of a reasonable scenario where that should occur. If it does, it probably means the ID schema is not unique enough. For example a media_content_id of "1" shouldn't be allowed if that can possibly refer to multiple different items.

I guarantee there are exceptions that I haven't considered where the above couldn't apply cleanly, but the current media_player.play_media implementations are incredibly integration-specific and I only use a handful of them myself.

[marcelveldt]

imo media_content_type is NOT required but it is now partly in use and some integrations seem to rely on it. Also the new Media source implementation returns the mime/content type. That's why I made it optional in my proposal above.

I think in most cases you either play some internal (known to the player) uri/id OR you play some url or file.
In the last case the content type is 99% of the cases automatically determined by nowadays players/solutions, either by the filename or by headers.

I'm all in for dropping content_type as long as media_type is going to be consistent.
If you want to have some fun just search for play_media in the HA source code and see how each integration has different assumptions. It is created this way over time and because there was no common agreement so now look for something more generic.

[marcelveldt]

Thinking of this even further it could be really simplified if we only have an URI field and let the player sort out how to handle that. A Media Source uri can be parsed by the browse helper added recently and all other is player specific ?

[jjlawren]

I think in most cases you either play some internal (known to the player) uri/id OR you play some url or file.

Yes, that's most of them. Plex, for example, also allows a search query in the media_content_id field where you're not passing a specific media identifier/URL, but asking the integration or Plex server to attempt to find a match before playing.

I currently maintain Plex and Sonos and it's honestly hard to keep track of all the possible edge cases for play_media just within those integrations. I'd love to have more consistency. To make this a bit easier, I've recently added HA-specific URL support to Plex to make the media browser easier to work with. These URLs uniquely identify media and can be passed to other integrations to allow cross-integration playback (for example, to the Sonos or Cast integrations). They look something like plex://<plex_server_id>/<media_identifier>. Perhaps standardizing around URLs is a possibility for those internal identifiers which are not in a URL format already.

All the above being said, any changes are going to introduce significant breaking changes and efforts across many media integrations. Maybe it's best to start over with a clean design with a brand new service to play media. 😆

[nagyrobi]

One shameful situation is that we have to set media_content_type of the player entity to be MEDIA_TYPE_MUSIC in order for Lovelace to show both artist and title attributes on the card. This limits the usablility...