Client-side sonification for datacubes

[james-trayford]

Description

CC @javerbukh @mariobuikhuizen @kecnry

Reimplementation of javerbukh#33 to avoid complex rebase, see original description below:

---

Initial implementation with playback handling fully in JS so we can but audio rendering client side.

With the JS-heavy implementation, some diagnostic logging has been added so turn on the JS console in the browser to see this.

Basic summary of behaviour: This creates the sonified cube as before, but flattens the array into a single 1D audio buffer of audio clips stacked end-to-end, representing the entire cube. Each (x,y) pixel then represents an exact audio sample range in the audio clip. To handle chnges between these we operate two audio players where:

1. initialise two players, A and B with two copies of the fullaudio clip
2. on entry of the viewer start players A on that pixel and fade up master audio, with crossfade 100% on player A
3. player A, say, is the active player current looping current pixel sound
4. On a change in pixel detected, change the loop points in player B, and cross fade audio from A=>B
5. switch B to be the active player, on the next change repeat 3->5 with A and B transposed.
6. on exit, fade down master audio and stop players

For this to work we need to hold two copies of the audio in memory at once. the size of the audio clip depends on the number of pixels. We can catch big cubes. Need to work out tolerable limits for transmitting audio data and whether to load staggered. would also need to generallise player A and B to list of players to handle simultaneously playing audio layers (rather than just most recent)

Outstanding issues / Questions:

• This uses Tone.js - an external library. Currently it pulls it when sonify plugin is opened, but could we package with install for offline mode?
• Some of the previous functionality is not yet integrated - the volume slider should be easy, but I don;t think choosing audio device wll work without something complicated with cookies etc. I think we ultimately just remove for client-side playback and just provide default audio.
• Is there a flag to know if we are operating on a server vs local instance? Currently this operates as before simultaneously - so the audio is still playing through the local available sound device if available. This has been useful for development/ debugging, but in operation need to either use one universal method or switch based on server vs local instance flag.
• I need to make sure I'm using the right viewer coordinates for the player at all times. I followed jdaviz/configs/imviz/plugins/coords_info and jdaviz/configs/default/plugins/markers for inspiration
• This should be generalised for audio layers: Can extend this to process lists of paired players to play multiple audio layers - mixing would be handled in JS too.

---

Some additional issues noticed since:

• Align viewer coords: I think there is a half-spaxel offset between the spectrum-at-spaxel viewer and the heard sonification, would be good to get advice on the best coordinate set to use, rounding pixels etc.
• Thread errors: I'm getting a lot of thread errors though things seem to still work as expected- I haven't tested if that is main problem or specific to the new features yet
• strauss requirement: currently strauss isn't required by default so will say sonification unavailable until it is installed manually - is there a reason not to add?

This pull request is to solve client-side sonification, making sonification via platforms viable

Fixes #

Change log entry

• Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
  list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
  should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
• Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
• Do the proposed changes follow the STScI Style Guides?
• Are tests added/updated as required? If so, do they follow the STScI Style Guides?
• Are docs added/updated as required? If so, do they follow the STScI Style Guides?
• If new remote data is added that uses MAST, is the URI added to the cache-download.yml workflow?
• Did the CI pass? If not, are the failures related?
• Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
• After merge, any internal documentations need updating (e.g., JIRA, Innerspace)?

[mariobuikhuizen]

Hi @james-trayford,

I could not find a potential memory leak in your code. What could happen though, is when running the cell with UI again, the old widget is kept around. This is inherent to the ipywidget system and in this case is keeping a reference to hundreds of MBs of wav data in front and back end. Could this be the cause?

A potential solution to one of the open questions:

This uses Tone.js - an external library. Currently it pulls it when sonify plugin is opened, but could we package with install for offline mode?

I think the native browser API supports everything you use, so Tone.js might not be needed. See:

• https://developer.mozilla.org/en-US/docs/Web/API/AudioContext
• https://developer.mozilla.org/en-US/docs/Web/API/AudioParam/linearRampToValueAtTime
  I can help with the conversion if you want.

I also think we could send only the audio data that is used for the target pixel when lindx changes. This value is already coming from the back end and adding 33kb wav data won't add much latency (30ms on 10Mbit/s line)

[mariobuikhuizen]

I was wondering how Tone.js did the scheduling you mentioned and found out they use requestAnimationFrame (see https://github.com/Tonejs/Tone.js/blob/feffad41cdcd8ec644d8b3e00e5ce7bebaefc0f6/Tone/core/util/Draw.ts#L108).

This runs at the refresh rate of the monitor, which is 60 Hz for a typical monitor. That gives a granularity of 16.7 ms for the scheduling. They also use a 0.008 s margin in which they call the scheduled task early (see https://github.com/Tonejs/Tone.js/blob/feffad41cdcd8ec644d8b3e00e5ce7bebaefc0f6/Tone/core/util/Draw.ts#L42-L47)

Maybe this had an impact on the issues you had with the 30ms crossfade time I saw in the comments.

I think it could be better for this use case to just use setTimeout(), it has a smaller granularity and doesn't run early.

Tone.Draw.schedule(() => this.handleFadeComplete(newVal), Tone.now() + fadeTime);

could be replaced with

setTimeout(() => this.handleFadeComplete(newVal), fadeTime * 1000);

[codecov]

Codecov Report

❌ Patch coverage is 32.67327% with 136 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.06%. Comparing base (40f6dbc) to head (b514e1a).

Files with missing lines	Patch %	Lines
...configs/cubeviz/plugins/sonify_data/sonify_data.py	40.00%	96 Missing ⚠️
jdaviz/configs/cubeviz/plugins/cube_listener.py	4.76%	40 Missing ⚠️

❌ Your patch check has failed because the patch coverage (32.67%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3843      +/-   ##
==========================================
- Coverage   88.28%   88.06%   -0.23%     
==========================================
  Files         198      198              
  Lines       27241    27366     +125     
==========================================
+ Hits        24049    24099      +50     
- Misses       3192     3267      +75     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[james-trayford]

Just to summarise some outstanding aspects:

There's a difficulty in terms of data volume for an arbitrary number of layers (representing a cube of any size can be >~ 10s-100s Mb per layer to transmit, so multiple layers will stack up fast) - also these need to be transmitted to the user before mixing to efficiently control things like relative volume. The layer implementation is great of course and will allow us to mix / match / export layers efficiently and allows us to follow the same data handling as other layer types. However, the use-case for playing multiple audio layers at once isn't really there yet as we explore the fundamental approach. What I thought would be best would be to start with an 'active layer' implementation with just the one layer at once loaded into the JS player. This simpler implementation will be feasible to operate on platforms sooner I think - and for our project we'd be really keen to prioritise that feature to be so we can broaden our testers for the approach (open a web url and listen to the data)

This 'single active layer' approach is basically how it's working at the moment, and is something we discussed as having precedent in the code already. If that sounds good, I would appreciate pointers to places in the code where a similar approach so I can make sure I'm conforming. I'm keen to continue working on multiple layers at the same time as we develop the approach.

A couple of other things I'm thinking about:

• Do we have instances of data used in jdaviz that have different spectral axes orders? I see this is adaptive in the code, but haven't been able to test with real data as I can't find any with a different spectral axis
• I need to make sure the testing ith the new approach conforms to the jdaviz standards

appreciate any thoughts on this!

[kecnry]

What I thought would be best would be to start with an 'active layer' implementation with just the one layer at once loaded into the JS player.

I think the way to get here and still keep the door open for the flexibility down the road would be to just enforce one "visible" audio layer per-viewer (when one gets turned on, all others get turned off). Is that something you want to add here or would you rather we add it to main and you pull it in?

[james-trayford]

@kecnry this sounds great - I think the '1 visible (audible?) layer' paradigm sounds like a good solution. I'd be happy to add it here if that's cleaner - I think it's closely tied to this functionality - but no very strong feelings either way. For me to implement this, I'd appreciate an example of this being done for visual layers in the existing code if that was available

[kecnry]

Ok, works for me! We don't have an example of anything terribly similar, but I suspect it would best belong in the data-menu _layers_changed - event is a dictionary with event.get('new') and event.get('old') that you can use to ensure only a single audible layer is enabled at a time, regardless of where the visibility was triggered.