Global condition and Local conditioning

[thomasmurphycodes]

In the white paper, they mention conditioning to a particular speaker as an input they condition globally, and the TTS component as an up-sampled (deconvolution) conditioned locally. For the latter, they also mention that they tried just repeating the values, but found it worked less well than doing the deconvolutions.

Is there effort underway to implement either of these? Practically speaking, implementing the local conditioning would allow us to begin to have this implementation speak recognizable words.

[ibab]

Yeah, it's definitely a planned feature.
I'll get to it eventually, but I'd also accept contributions if someone is interested.
A solution to this should also integrate with the AudioReader interface.

[Zeta36]

Is somebody working on this already?

[alexbeloi]

I'm starting to work on it, I think I can get some basic implementation working over the next couple of days. Global part should be easy, and a dumb implementation (upsampling by repeating values) of local conditioning should be fast to implement as well.

This way, we can get to a stage where the net can produces some low-quality speech. Then work on improving the quality by adding more sophisticated upsampling methods.

The white paper also talks about using local conditioning features beyond just the text data, they do some preprocessing to compute phonetic features from the text. That would be nice to add later as well.

[thomasmurphycodes]

I agree global will be easier, should just be a one-hot vector representing
the speaker. Am I thinking about this wrong that the local conditioning
requires us to train on data sets that contain the phonetic data as a
feature vector in addition to the waveform feature? What dataset are you
thinking of using?

On Sat, Oct 8, 2016 at 1:56 PM, Alex Beloi notifications@github.com wrote:

I'm starting to work on it, I think I can get some basic implementation
working over the next couple of days. Global part should be easy, and a
dumb implementation (upsampling by repeating values) of local conditioning
should be fast to implement as well.

This way, we can get to a stage where the net can produces some
low-quality speech. Then work on improving the quality by adding more
sophisticated upsampling methods.

The white paper also talks about using local conditioning features beyond
just the text data, they do some preprocessing to compute phonetic features
from the text. That would be nice to add later as well.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#112 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEKqWYd6YWpFgmjPuTd12PWhrGHD5bdVks5qx9ligaJpZM4KKVOa
.

[alexbeloi]

I was thinking to just use the raw text from the corpus data for local conditioning to start, just encode each character into a vector and upsample (by repeats) it to the number of samples in the audio file, not ideal but it's a start. Characters should be able to act as a really rough proxy for phonetic features.

Ideally, the raw text should be processed (perhaps via some other model) into a sequence of phonetic features and then that would be upsampled to the size of the audio sample.

[thomasmurphycodes]

I mean let's give it a shot and see what happens. Google Research has a
bunch of papers over on there page about HMM-ing characters to phonemes, so
we could look into a subproject where we try to implement that.

On Sat, Oct 8, 2016 at 2:25 PM, Alex Beloi notifications@github.com wrote:

I was thinking to just use the raw text from the corpus data for local
conditioning to start, just encode each character into a vector and
upsample (by repeats) it to the number of samples in the audio file, not
ideal but it's a start. Characters should be able to act as a really rough
proxy for phonetic features.

Ideally, the raw text should be processed (perhaps via some other model)
into a sequence of phonetic features and then that would be upsampled
to the size of the audio sample.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#112 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEKqWZmBL3E8ObZmo44eoP_ht2lqW8Ufks5qx-AGgaJpZM4KKVOa
.

[nakosung]

@thomasmurphycodes Could you post the list of papers?

[thomasmurphycodes]

Yeah will tomorrow when in the office, they're on a box I have there.

On Sat, Oct 8, 2016 at 10:38 PM, Nako Sung notifications@github.com wrote:

@thomasmurphycodes https://github.com/thomasmurphycodes Could you post
the list of papers?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#112 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEKqWQBOgH-ifllSzpZuJVXOafNVXramks5qyFOTgaJpZM4KKVOa
.

[ibab]

I've also thought about just plugging in the raw text, but I'm pretty sure we would need at least some kind of attention mechanism if we want it to work properly (i.e. some way for the network to figure out which parts of the text correspond to which sections of the waveform).

[thomasmurphycodes]

I think that's the case for sure. They explicitly mention the convolution
up-sampling (zero-padding) in the paper.

On Mon, Oct 10, 2016 at 7:30 AM, Igor Babuschkin notifications@github.com
wrote:

I've also thought about just plugging in the raw text, but I'm pretty sure
we would need at least some kind of attention mechanism if we want it to
work properly (i.e. some way for the network to figure out which parts of
the text correspond to which sections of the waveform).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#112 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEKqWdC5vND-v17ykcajGuKc95ry3MD0ks5qyiHbgaJpZM4KKVOa
.

[wuaalb]

In #92 HMM-aligned phonetic features are already provided. The upsampling/repeating values step is for going from feature vector per HMM frame to feature vector per time-domain sample..