feat: Android Auto Support

[lovegaoshi]

This PR adds android auto support as discussed in here.

What I did:

• I copied HeadlessJsTaskService.java from the react-native repo and extended MediaBrowserServiceCompat instead of Service. No further changes other than adding a few imports.
• I added an if in onBind() that returns the MediaBrowserServiceCompat.onBind() when intent is not null, and the defaultbinderwhen intent is null (which is used by RNTP's serviceConnector)
• I added relevant declarations as outlined here.

This uses my kotlinAudio fork that exposes its mediaSession's token, I'm submitting a PR there as well.

testing on an android auto emulator is described here.

I achieved synced media playback control both in app and android auto as shown below.

relevant links:
https://developer.android.com/training/cars/media
https://developer.android.com/training/cars/testing

I also tried to put MediaBrowserServiceCompat inside MusicService here but I was having some errors. If this works then we don't have to mess with maintaining a copied HeadlessJsTaskService.java .

[lovegaoshi]

new update:
I added content hierarchy support that should complete this PR functionality-wise. For users they must implement event listeners RemotePlayId and RemotePlaySearch to properly handle playback calls from AA. Content hierarchy can be built as outlined here. It will look like below:

I will try to submit my app to the appstore and see if google complains any features not implemented for AA
my app passed google's open testing review with the declared android auto permissions

[dcvz]

Nice work @lovegaoshi ! I’ll try to go through this in the coming week

[czekFREE]

I am not part of RNTP team,

but I am interested AA in RNTP as well and I was did some preliminary investigation last month (and getting back to it now). I approached the problem in similar way (HeadlessJsMediaService extends MediaBrowserServiceCompat), but I remember that I faced some problems, so I wanted to ask you regarding these @lovegaoshi.

Main problem I faced (in the auto simulator) was the fact, that MediaBrowserServiceCompat can run without UI/without RN app actually started on mobile phone. When RN app is not opened on the phone, but user opens app in Android auto, MediaBrowserServiceCompat spins up its lifecycle methods (onGetRoot, onLoadChildren...), but data-fetching logic is actually implemented in RN app (but RN app did not started)... My idea was to open app programatically or show error message on the AA screen saying user needs to have mobile app opened (I am not sure it it will be possible/reliable to open app programatically, I thing that there are different behaviors when app is in recents etc...), but I had to postpone working on RNTP, so I did not yet come to conclusion how to approach that

It seems to me that browsing AA screens additionaly require emiting at least onLoadChildren "events" to RN app (and reacting to these) - I am probably not only one who loads data on demand for every "browsable screen" separately. My idea was to emit onLoadChildren event to RN app for every browsable screen -> RN app could do navigation/data fetching and when this is done, it would emit result event / call predefined callback with result back to RN. I have noticed that you implemented setBrowseTree which is expecting to have all data available regardless of navigation state.

Do you have your changes published on npm so that I could check your out by installing them @lovegaoshi?

Btw good work checking this out, I know that it is not easy stuff...

[lovegaoshi]

Hi there, Ill admit I didn't consider the scenario the app is not opened in the phone but called in AA. Though what you can do is to load a default browseTree in MusicService with some error msg as a node, then overwrite it when your app is opened. very not sure how to actually start the app from AA. I have 2 weeks of native experience :P As for the onloadchildren question any browsable nodes have their mediaid as the keys in the browseTree - I don't think you can wait for results in onLoadChildren from an emitted event to RN then somehow back to native? My approach is to set the entire browseTree structure upfront and modify as needed. eg in the example I set tab1, you can put down tab2 and tab3, or nest more tabs within tabs. I think AA browsing needs to be as simple as possible so transferring a lot of node data through the bridge isn't a good idea to begin with, so I'm sticking with a small browsable tree, and I can afford to update this entire tree with a hook when necessary without performance concerns. You can just use my RNTP git in your packages.json https://github.com/lovegaoshi/azusa-player-mobile/blob/master/package.json#L67

…

On Wed, Jul 19, 2023, 6:14 AM Jiří Bělecký ***@***.***> wrote: I am not part of RNTP team, but I am interested AA in RNTP as well and I was did some preliminary investigation last month (and getting back to it now). I approached the problem in similar way (HeadlessJsMediaService extends MediaBrowserServiceCompat), but I remember that I faced some problems, so I wanted to ask you regarding these @lovegaoshi <https://github.com/lovegaoshi>. Main problem I faced (in the auto simulator) was the fact, that MediaBrowserServiceCompat can run without UI/without RN app actually started on mobile phone. When RN app is not opened on the phone, but user opens app in Android auto, MediaBrowserServiceCompat spins up its lifecycle methods (onGetRoot, onLoadChildren...), but data-fetching logic is actually implemented in RN app (but RN app did not started)... My idea was to open app programatically or show error message on the AA screen saying user needs to have mobile app opened (I am not sure it it will be possible/reliable to open app programatically, I thing that there are different behaviors when app is in recents etc...), but I had to postpone working on RNTP, so I did not yet come to conclusion how to approach that It seems to me that browsing AA screens additionaly require emiting at least onLoadChildren "events" to RN app (and reacting to these) - I am probably not only one who loads data on demand for every "browsable screen" separately. My idea was to emit onLoadChildren event to RN app for every browsable screen -> RN app could do navigation/data fetching and when this is done, it would emit result event / call predefined callback with result back to RN. I have noticed that you implemented setBrowseTree which is expecting to have all data available regardless of navigation state. Do you have your changes published on npm so that I could check your out by installing them @lovegaoshi <https://github.com/lovegaoshi>? Btw good work checking this out, I know that it is not easy stuff... — Reply to this email directly, view it on GitHub <#2043 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZMOVVQPESL3MQXWQ7SOYJLXQ7MUHANCNFSM6AAAAAAZVXME7M> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[lovegaoshi]

i have some test ideas to fix

MediaBrowserServiceCompat can run without UI/without RN app actually started on mobile phone

at lovegaoshi#4 (comment) . wish someone with android experience can chime in

[lovegaoshi]

@czekFREE I actually encountered the first problem you described quite often this past week, guess i was just ignoring it until now; and after too many frustrating debug prints, I came up with something probably sketchy and definitely could use some advice.

The issue is exactly as you described, MusicService is created but MainActivity is not. its mentioned some years ago at https://stackoverflow.com/questions/44354261/android-auto-communication-between-background-service-and-activity . However I'm not understanding that RNTP already has its activity and service separated, and it just appears that MusicService isn't even created at all - and it doesnt! I put a timber.d inside override onCreate and it doesnt print when launched from AA. I don't understand that not even extending MusicService to MediaBrowserCompat gets it printed. While UAMP works just as expected, MusicService's onCreated gets called both from UI and from AA.

I did find out that overriding onCreate in HeadlessJsMediaService will get printed just fine. I'm then launching an intent to call up the UI via deeplinking at its onCreate, so UI will be brought up, RNTP will get properly set up and all listeners will be properly registered, rather than trying to be smart about setting up service without the activity like AIMP or UAMP does. I have this change below though I dont really understand what I'm doing with 3 weeks of android experience... Ill use it in my app for a while.

lovegaoshi@5971065

[czekFREE]

@lovegaoshi Just to be clear - I am not Android developer, I work purely with javascript, so my knowledge is limited

We have following problem:

• Recommended architecture for media apps from Google is to implement it as client-server interface - MediaBrowserServiceCompat is "server" that is either "started" by client - UI app - or "created" by other clients/devices that can connect to it (auto, watch...) without need to have UI app running.
• Unfortunately RN can be started/initialized only by starting UI app (MainActivity) - or I am not aware of other approaches - but in our case MediaBrowserServiceCompat can be created without UI.
• because in RN apps all the implementation logic lives in RN (code for UI app), we need either to have UI available/running (and comunicate with it to get results) or we would have to duplicate all the data-fetching/playback code that we have in RN in android as well (suggested in SO comment you mentioned).

How should we ensure that UI is actually running? As far as I know

1. we can spin up UI ourselves - for me this is probably be very unsafe approach as we could be opening app based on system requests in situations when user does not want that. That would have to be really bullet-proof and I do not have skills for that. Additionally I think that system may impose limits/not allow to start app based on some evaluation (app is not in "recents" or something like that). Additionaly there is this another problem that i realized recently - there is this "doze" mode and new "battery optimizations" in recent Android versions - simple example: when phone is locked (and screen is off), Android "suspends" the app and reloads it when user unlocks the screen. So the app is formally "running", but it is suspended by the system -> unavailable for communication with MediaBrowserServiceCompat

2. We can show (error) messages to users that UI is not available -> we will "force" user to open it. This is problematic as well as (a) it may piss off the user and also (b) this might actually violate android-auto safety instructions/requirements (because this might distract the driver), so there might be problems with approval of the app.

I am still cracking my head with that, but I am starting to think that any chosen solution will have significant downsides

[lovegaoshi]

@czekFREE welp, you will be surprised - my app with this capability passed google's open testing review just yesterday.

IMO this is very much dwelling into RN's limitations - MediaBrowserService will start independently from MainActivity, without MainActivity RN will not start, RNTP will not be set up, the whole RNTP part wont work, without bringing up the UI the entire RNTP package becomes nonsense, should have started with native to begin with. And this also means there is no way to customize this behavior (bringing up the UI from HeadlessJSMediaService's onCreate via deeplinking) from the RN side.

and imo an error message here is really bad - AA apps should never prompt the user to use their phone and open an app. Auto start MainActivity from MediaBrowserService is the necessary lesser evil.

I'll leave this change in my own branch until the devs weighs in...