docs: Add Sound Effects documentation. #753

microbit-carlos · 2022-06-20T10:23:28Z

Proposal

The current proposal adds a new class in the audio module named SoundEffect, which can be played using the audio.play() function.

The user can then create new instances, configuring them via constructor arguments, and after creation they can be modified via attributes.

A collection of a Sound Effects could be played by grouping them in a list (or something more advanced like a generator, so essentially an iterable).

Preview build of the docs: https://microbit-micropython--753.org.readthedocs.build/en/753/audio.html

In progress

The default values have not been considered yet, the current values are more or less random
- More info in this comment: docs: Add Sound Effects documentation. #753 (comment)
The descriptions and explanations could use some work, as the document is getting long a bit hard to digest, but we can update that later
The order of the constructor arguments could be modified if it makes more sense to group them differently
The built-in effects (if they are not dropped) are not defined yet. MakeCode has these examples, which we could reuse or adapt/modify
- https://github.com/microsoft/pxt/blob/v8.1.1/webapp/src/components/soundEffectEditor/soundUtil.ts#L52-L182
- We should reduce the total number of provided Sound Effects, as their usefulness is limited (compared with their availability in the editors) and take additional flash space
- We are currently planning to play with this feature at europython and attendees can then submit their Sounds Effects

Things Still Under Consideration

Effect() might not be the best name for this feature
- We might want to use the word "effect" for other future features, like filters than could be applied to recorded sound before it is played back
- SoundEffect() is better but also longer (and we prefer shorter names that are easier to type for young learners)
- This PR description has been updated already to reflect the SoundEffect name as described in comment docs: Add Sound Effects documentation. #753 (comment)
The parameter name interpolation is too long, we are still looking for alternatives.
- curve is being considered, but one of the options is actually INTER_CURVE, so we might have to rename that to something else
- ramp?
The idea to provide built-in pre-created Sound Effects was to let the user easily create and modify an Effect. In the MakeCode Editor and Python Editor the user can select one from a list, and the code/block is automatically generated showing all the values used to create that sound.

Alternative Proposals

A new function to play an effect instead of creating a class instance

We know that some beginner programmers struggle with the concept of instantiating classes in variables, and then using attributes to modify them. An example of this would be NeoPixels, although on the other hand things like Images where a new instance is immediately used in a function has generally been okay. So:

# This has generally been more difficult
np = neopixel.NeoPixel(pin0, 8)
np[0] = (255, 0, 128) 
np.show()

# But this has been okay, which is similar to audio.play(Effect(...))
display.show(Image('00300:'
                   '03630:'
                   '36963:'
                   '03630:'
                   '00300'))

So we are considering providing a function to play a single Sound Effect:

# Configure the effect on the accelerometer data
my_effect = audio.play_effect(
    freq_start=400,
    freq_end=2000,
    duration=500,
    vol_start=100,
    vol_end=255,
    wave=audio.WAVE_TRIANGLE,
    fx=audio.FX_VIBRATO,
    interpolation=audio.INTER_LOG
)

# And using sensor data
while True:
   audio.play_effect(
        freq_start=accelerometer.get_x(),
        freq_end=accelerometer.get_y(),
        duration=500,
        vol_start=100,
        vol_end=255,
        wave=audio.WAVE_TRIANGLE,
        fx=audio.FX_VIBRATO,
        interpolation=audio.INTER_LOG
    )

If we adopt this we might need a better name that play_effect(...) as it is not that different than play(Effect(...)).

Implementation details not shown in the docs

Classes normally have a longer "internal" name (e.g. Image-> MicroBitImage, so the Effect class could have the longer MicroBitSoundEffect internal name
Built-in pre-made Effects should be immutable
Serialisation of a Sound Effect, for example to be sent via radio, could be done in a string format that could be eval'ed directly
- The provided constants depend on the way the audio module is imported (import audio vs from audio import * vs from microbit import *), so their value would have to be used directly
  - For these values to be human readable we might need to use strings
- i.e. print(audio.Effect(vol_start=100) -> Effect(freq_start=400, freq_end=2000, duration=500, vol_start=100, vol_end=255, wave="triangle", fx="vibrato", interpolation="log")
- If the worse case scenario doesn't fit in the default radio packet size (or the max size), maybe we could use positional arguments for __repr__ and keyword arguments for __str__?
  - The example from before would become Effect(None, 400, 2000, 500, 100, 255, "triangle", "vibrato", "log")
Not all the CODAL effect parameters are exposed to the user, for example, the fx has parameters that can be tweaked.
- The hardcoded parameters can be found in the MakeCode implementation: https://github.com/microsoft/pxt-microbit/blob/v5.1.1/libs/core/soundexpressions.ts#L370
By default audio.play() sets wait=False so it's a blocking function. If the CODAL call async version of play is called, and multiple calls are done inmediately one after another, they will not overlap.
- So these three sound effects will still play one after another
```
audio.play(audio.Effect.CHIRP, wait=False)
audio.play(audio.Effect.CHIRP, wait=False)
audio.play(audio.Effect.CHIRP, wait=False)
```
  But this will show the image while playing the sound
```
audio.play(audio.Effect.CHIRP, wait=False)
display.show(Image.HAPPY
```
- We'll need to figure out how to document it, or if there is anything that can be done about this.

microbit-carlos · 2022-06-23T17:57:33Z

@dpgeorge @jaustin @microbit-giles This PR is ready for review.

As we'd like not to delay the implementation too much it'd be appreciated if any feedback could be provided soonish.

microbit-carlos · 2022-06-23T17:59:47Z

Oh, and @microbit-matt-hillsdon, I think you might be interested to have a look at these as well at some point for the stubs.

microbit-giles · 2022-06-24T08:29:40Z

My main concern is that the name Effect() may cause confusion and problems later on, especially if we implement sound 'filters' like reverb, amplification and reverse which are known as 'Effects' in widely-used sound editing software. We also have an 'fx' parameter within Effect(). Although longer to type, I think SoundEffect() may be clearer and less ambiguous in future.

docs/audio.rst

microbit-matt-hillsdon · 2022-06-24T10:11:36Z

Oh, and @microbit-matt-hillsdon, I think you might be interested to have a look at these as well at some point for the stubs.

Thanks, yes, I think I'd like to have a stubs branch for this so we can spot any issues typing the API early.

docs/audio.rst

jaustin

This is looking great. I think SoundEffect overall is better than Effect, but one minor counter argument amongst the comments (that it confuses it more with the built-in sounds).

Still open to other suggestions for the name. I think if it were a function to create these I'd expect it to be something like audio.generate(...) or something related to a synth. It's the act of having to name the object that's hard.

jaustin · 2022-06-30T20:54:26Z

docs/audio.rst

+====================
+
+.. py:class::
+    Effect(preset=None, freq_start=400, freq_end=200, duration=500, vol_start=100, vol_end=255, wave=WAVE_SQUARE, fx=None, interpolation=INTER_LINEAR)


I think it's worth giving some thought to how this relates to the MakeCode interface and JS function calls.

In those the wave type parameter comes first, and there's a slightly different ordering between the block presentation and the JS one. The ordering here matches the block display (aside from wave type).

I'd be in favour of moving wave= to the first parameter and move preset= to the end of the list

I agree that on first impressions is really attractive to duplicate the MakeCode parameter order, as that would technically offer us the advantage of copy/paste arguments from the output of the really nice MakeCode widget:

However, as the wave, fx, and interpolation parameters depend on constants defined in MicroPython this is not possible, so I think we should prioritise an order that makes more sense for MicroPython users.
Although it is true, that using keyword arguments we do have the flexibility on how tutorials or the users order them.

jaustin · 2022-06-30T21:02:26Z

docs/audio.rst

+* ``audio.Effect.CHIRP``
+* ``audio.Effect.CROAK``
+* ``audio.Effect.CLICK``
+


I was really happy with including these, but I think their relationship to the built-in sounds is curious to explain. I don't know that this means we shouldn't have them, but I'm struggling to work out what to call them.

audio.SoundEffect.CROAK for example is sort of weird to distinguish from Sound.TWINKLE - why can one of them be used as a preset and not the other? ("because one is a SoundEffect and one is a Sound"?)

This is one downside to "SoundEffect" over pure "Effect" - it ties the built-in sounds more closely to the SoundEffects - though @microbit-carlos points out that it does make sense that a 'Sound' is made out of SoundEffects...

The existing built-in sounds could be rewritten as a (constant) tuple of (constant) SoundEffect instances. This is similar to sequences of images, like all the clock faces.

Eg:

>>> audio.Sound.SPRING (SoundEffect(...), SoundEffect(...))

Then users could really see how Sound's are made and tweak them if they like.

Yes, this is something that was quite desirable, our only concern is all the extra flash usage, specially if the strings are already in CODAL and probably cannot be excluded as they are.
Would be good to get an estimation of how much extra flash space this would consume?

jaustin · 2022-06-30T21:05:06Z

docs/audio.rst

+        fx=audio.FX_VIBRATO,
+        interpolation=audio.LOG
+    ))
+


I like this, it's nice and clear, but also think we should give some thought to when to document the positional, not keyword options.

@microbit-matt-hillsdon when autocomplete works well with positional arguments, it can be a really smooth editing experience - are there ways we could document the code here so that we get good positional argument help during autocomplete as long as the user hasn't used any keyword arguments

(for example, if I start typing audio.play(Effect(400 will the autocomplete in the editor be showing the parameter help from freq_start (as long as we re-order the parameters so start is first))

jaustin · 2022-06-30T21:07:23Z

docs/audio.rst

+    while True:
+        my_effect.freq_start=accelerometer.get_x()
+        my_effect.freq_end=accelerometer.get_y()
+        audio.play(my_effect)


This is really nice and much clearer than also containing all the parameters and @microbit-giles I'd like us to make sure we include some nice lean examples in the edu content that encourage people to use this cleaner interface.

microbit-carlos · 2022-07-01T18:56:19Z

Okay, based on @microbit-giles and @jaustin suggestion I've renamed Effect() to SoundEffect() in b89aa5d

Also updated the default values in ce19f55 to produce this sound: https://makecode.microbit.org/_ajAW04hDL0MP
As these are the default values, that will be the sound played with audio.play(audio.SoundEffect())

microbit-carlos · 2022-07-01T19:21:20Z

@dpgeorge I think we are at a point were we can implement the current version of the docs (minus the addition of the pre-made built-in SoundEffects), so that we can start playing with the feature and get a feeling for it.

For that I think it might be good to keep the implementation in a feature branch, so that we can still tweak it before merging it.

Also we would like to of bringing the Python Editor beta with a MicroPython build with Sound Effects for workshops at EuroPython. I think it would be really cool if we could play with the SoundEffects with people at the conference and if they'd like to contribute their sounds to an example file, we could then pick some of those to use as the pre-build SoundEffects and credit them.
The conference starts on the 11th of July, do you think that is doable? Ideally with a bit of time for us to be able to make sure any workshop material we prepare works.

dpgeorge · 2022-07-10T02:23:12Z

The parameter name interpolation is too long, we are still looking for alternatives

How about shape? That matches the internal CODAL name.

docs/audio.rst

dpgeorge · 2022-07-10T03:43:42Z

docs/audio.rst

+
+    :param preset: An existing SoundEffect instance to use as a base, its values
+        are cloned in the new instance, and any additional arguments provided
+        overwrite the base values.


The interaction with a preset and default keyword arguments is problematic.

Consider SoundEffect(). Does that create a new instance with all default values?

Consider SoundEffect(vol_end=255). Does that create a new instance with all default values except vol_end which is overridden to 255?

Consider SoundEffect(audio.SoundEffect.CLICK). Does that copy CLICK and then override everything with the default values? If not, how do you explain the difference between that and SoundEffect() behaviour?

How about changing the constructor signature to:

SoundEffect(preset=SoundEffect.DEFAULT, *, freq_start, freq_end, duration, vol_start, vol_end, wave, fx, interpolation)

Then SoundEffect.DEFAULT has the default values for the keyword-only args as previously specified above (freq_start=500 etc).

Consider SoundEffect(). Does that create a new instance with all default values?

Yes, I think this one is straight forward.

Consider SoundEffect(vol_end=255). Does that create a new instance with all default values except vol_end which is overridden to 255?

Yes, as it would if the preset argument didn't exist.

Consider SoundEffect(audio.SoundEffect.CLICK). Does that copy CLICK and then override everything with the default values? If not, how do you explain the difference between that and SoundEffect() behaviour?

Okay, I could see how maybe that is a bit less obvious.
The way I would see it is that SoundEffect(audio.SoundEffect.CLICK) is like a copy constructor, so it ignores any of the __init__ default values.
I think SoundEffect(audio.SoundEffect.CLICK, duration=2000) could be easily read as "copy all audio.SoundEffect.CLICK values, but overwrite duration to 2000, but I could see how that might not be as obvious if written as SoundEffect(duration=2000, preset=audio.SoundEffect.CLICK).

Then SoundEffect.DEFAULT has the default values for the keyword-only args as previously specified above (freq_start=500 etc).

The disadvantage of that is we wouldn't be able to created shorter strings for serialisation with positional arguments.
So this shorter string wouldn't eval:

SoundEffect(988, 440, 190, 255, 255, 0, 1, 8)

If the confusion arising from having a preset argument in the constructor, what if the only way to duplicate and modify would be to use a copy() method?

my_effect = audio.SoundEffect.CLICK.copy() my.effect.duration = 2000

It's an extra line, but might be easier to understand?

SoundEffect(988, 440, 190, 255, 255, 0, 1, 8)

This won't work because 988 is not a valid preset. It would need to be SoundEffect(None, 988, 440, 190, 255, 255, 0, 1, 8).

Actually, the way it's implemented at the moment is that if you pass None as the preset argument then the preset is the default set of values. And any extra keyword args override that default /None preset value.

The signature for that is:

SoundEffect(preset=None, *, freq_start, freq_end, duration, vol_start, vol_end, wave, fx, interpolation)

SoundEffect(988, 440, 190, 255, 255, 0, 1, 8)

This won't work because 988 is not a valid preset. It would need to be SoundEffect(None, 988, 440, 190, 255, 255, 0, 1, 8).

Yes, sorry, that would only work like that if we moved preset to be the last parameter, as it was suggested in #753 (comment). I thought I had added a comment somewhere to indicate that having it as the last parameter would help with __repr__, but I couldn't find it, so I probably forgot to write it down.

The more I think about it, the better the copy() method sounds. Do you have any thoughts about it?

It's been agreed to remove the preset constructor parameter, and add a copy() method to be able to clone SoundEffects.

dpgeorge · 2022-07-10T04:13:58Z

The existing Sound class lives in microbit.Sound, and its microbit.audio.AudioFrame. So where should SoundEffect go, in microbit or microbit.audio?

docs/audio.rst

microbit-carlos · 2022-07-11T09:06:44Z

The existing Sound class lives in microbit.Sound, and its microbit.audio.AudioFrame. So where should SoundEffect go, in microbit or microbit.audio?

The original proposal was to have these inside the audio module, as it's related to things like the audio.AufioFrame. So it ends up looking like this: audio.play(audio.SoundEffect())

The reason we ended up putting Sound in the microbit module, instead of the audio module was:

We considered these sounds to be micro:bit specific, to be part of the micro:bit personality, i.e. playing the "micro:bit HAPPY sound".
Shorter (audio.Sound.HAPPY vs Sound.HAPPY) always helps

For now the test branch has it as audio.SoundEffect, but we should reconsider if there is a good use case for having it as microbit.SoundEffect.

…ect`.

…y()` method.

…file.

microbit-carlos · 2022-09-06T17:48:57Z

Merging this as it is the current state of SoundEffect in the v2.1.0-beta.1 release.
Will create a new PR to include proposals on predefined sounds.

* docs: Add Sound Effects documentation. * docs: Change proposed audio.Effect() to audio.SoundEffect() * docs: Update the proposed default values for audio.SoundEffect(). * docs: Move the SoundEffects constants from `audio` to `audio.SoundEffect`. * docs: Remove SoundEffect `preset` constructor parameter, and add `copy()` method. * docs: Change SoundEffect() `interporlation` parameter to `shape`. * docs: Remove mentions of SoundEffect presets, add ranges, example to file.

microbit-carlos · 2022-09-06T18:10:23Z

docs: Rename SoundEffect wave to waveform #765

* docs: Add Sound Effects documentation. * docs: Change proposed audio.Effect() to audio.SoundEffect() * docs: Update the proposed default values for audio.SoundEffect(). * docs: Move the SoundEffects constants from `audio` to `audio.SoundEffect`. * docs: Remove SoundEffect `preset` constructor parameter, and add `copy()` method. * docs: Change SoundEffect() `interporlation` parameter to `shape`. * docs: Remove mentions of SoundEffect presets, add ranges, example to file.

microbit-carlos marked this pull request as draft June 20, 2022 10:23

microbit-carlos force-pushed the docs-sound branch from d8a058d to b0edf5c Compare June 23, 2022 08:55

docs: Add Sound Effects documentation.

e7ff826

microbit-carlos force-pushed the docs-sound branch from b0edf5c to e7ff826 Compare June 23, 2022 17:56

carlosperate reviewed Jun 24, 2022

View reviewed changes

docs/audio.rst Show resolved Hide resolved

microbit-carlos commented Jun 30, 2022

View reviewed changes

docs/audio.rst Outdated Show resolved Hide resolved

jaustin reviewed Jun 30, 2022

View reviewed changes

docs: Change proposed audio.Effect() to audio.SoundEffect()

b89aa5d

docs: Update the proposed default values for audio.SoundEffect().

ce19f55

microbit-carlos mentioned this pull request Jul 4, 2022

New feature: Sound Effects microbit-foundation/micropython-microbit-v2#103

Closed

dpgeorge reviewed Jul 10, 2022

View reviewed changes

docs/audio.rst Outdated Show resolved Hide resolved

dpgeorge reviewed Jul 10, 2022

View reviewed changes

docs/audio.rst Outdated Show resolved Hide resolved

dpgeorge reviewed Jul 10, 2022

View reviewed changes

microbit-carlos commented Jul 10, 2022

View reviewed changes

docs/audio.rst Outdated Show resolved Hide resolved

docs: Move the SoundEffects constants from audio to `audio.SoundEff…

c0a2909

…ect`.

microbit-carlos force-pushed the docs-sound branch from e36b5d8 to 14860a7 Compare August 10, 2022 18:28

docs: Remove SoundEffect preset constructor parameter, and add `cop…

63fe6a0

…y()` method.

microbit-carlos force-pushed the docs-sound branch from 14860a7 to 63fe6a0 Compare August 15, 2022 09:33

docs: Change SoundEffect() interporlation parameter to shape.

eac0497

microbit-carlos marked this pull request as ready for review September 6, 2022 17:44

docs: Remove mentions of SoundEffect presets, add ranges, example to …

8cfea23

…file.

microbit-carlos merged commit cbc668c into bbcmicrobit:v2-docs Sep 6, 2022

microbit-carlos deleted the docs-sound branch September 6, 2022 17:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Add Sound Effects documentation. #753

docs: Add Sound Effects documentation. #753

microbit-carlos commented Jun 20, 2022 •

edited

Loading

microbit-carlos commented Jun 23, 2022

microbit-carlos commented Jun 23, 2022

microbit-giles commented Jun 24, 2022

microbit-matt-hillsdon commented Jun 24, 2022

jaustin left a comment

jaustin Jun 30, 2022

microbit-carlos Jul 1, 2022

jaustin Jun 30, 2022

dpgeorge Jul 10, 2022

microbit-carlos Jul 11, 2022

jaustin Jun 30, 2022

jaustin Jun 30, 2022

microbit-carlos commented Jul 1, 2022 •

edited

Loading

microbit-carlos commented Jul 1, 2022

dpgeorge commented Jul 10, 2022

dpgeorge Jul 10, 2022

microbit-carlos Jul 10, 2022

dpgeorge Jul 11, 2022

microbit-carlos Jul 11, 2022

microbit-carlos Aug 9, 2022

dpgeorge commented Jul 10, 2022

microbit-carlos commented Jul 11, 2022

microbit-carlos commented Sep 6, 2022

microbit-carlos commented Sep 6, 2022

docs: Add Sound Effects documentation. #753

docs: Add Sound Effects documentation. #753

Conversation

microbit-carlos commented Jun 20, 2022 • edited Loading

Proposal

In progress

Things Still Under Consideration

Alternative Proposals

A new function to play an effect instead of creating a class instance

Implementation details not shown in the docs

microbit-carlos commented Jun 23, 2022

microbit-carlos commented Jun 23, 2022

microbit-giles commented Jun 24, 2022

microbit-matt-hillsdon commented Jun 24, 2022

jaustin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

microbit-carlos commented Jul 1, 2022 • edited Loading

microbit-carlos commented Jul 1, 2022

dpgeorge commented Jul 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpgeorge commented Jul 10, 2022

microbit-carlos commented Jul 11, 2022

microbit-carlos commented Sep 6, 2022

microbit-carlos commented Sep 6, 2022

microbit-carlos commented Jun 20, 2022 •

edited

Loading

microbit-carlos commented Jul 1, 2022 •

edited

Loading