[API DESIGN REVIEW] Recurrent Attention Layer(s)

[andhus]

I've laid out an API suggestion for application of attention mechanisms in a recurrent setting.

Full design review doc: https://docs.google.com/document/d/1psFXnmMlSTg5JapgZKz26ag-zBu3ERrxkKoEzNpzl4w/edit?usp=sharing

There is a proof-of-concept implementation of the API in my (play-ground) add-on library extkeras as well as an example of its use in model training.

There is still a lot of work needed, but if you think it's a good general direction I'm happy to take lead in completing the implementation.

Happy for all feedback at any level!

[hubayirp]

+1

…

On Sun, Aug 13, 2017 at 1:29 PM, andhus ***@***.***> wrote: I've laid out an API suggestion for application of attention mechanisms in a *recurrent setting*. Full design review doc: https://docs.google.com/document/d/ 1psFXnmMlSTg5JapgZKz26ag-zBu3ERrxkKoEzNpzl4w/edit?usp=sharing There is a proof-of-concept implementation of the API in my (play-ground) add-on library extkeras <https://github.com/andhus/extkeras/blob/master/extkeras/layers/attention.py> as well as an example <https://github.com/andhus/extkeras/blob/master/examples/attention/graves_sequence_attention.py> of its use in model training. There is still a lot of work needed, but if you think it's a good general direction I'm happy to take lead in completing the implementation. Happy for all feedback at any level! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7633>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACWGWKeajqfYEYKV6BFYAYLzAPsVncwPks5sX1yYgaJpZM4O1yfi> .

[hamelsmu]

+1

[titu1994]

This is important. Attention mechanisms is often used in various capacities in new papers. It would be highly beneficial to have a standard way to apply attention to general RNNs / LSTM / GRU.

[fchollet]

I fully agree that we need built-in support for RNN attention in Keras, and I think this is a solid first proposal. What would be some alternative proposals? If none, we will go with this one.

[andhus]

As discussed in comments of the API Review doc, the introduction of standalone RNN Cells (#7943) will simplify and make addition of Recurrent Attention more modular. I'll adapt the API suggestion accordingly asap.

[andhus]

@fchollet The API doc is now updated, see the PR:s above which covers most crucial parts. Let me know if you think it's on the right track and I'll continue with step 3 and 4 according document. The concept of standalone cells really makes these things more modular and neat :D

[stabgan]

@andhus I need a help . I've researched a lot but I'm stuck on something . I need to extract the attention weights . But couldn't find an tutorial or blog on how to do that . Just consider IMDB dataset , and after training , I will input a test data . Then I want to know which words in the test data are most important . I only know keras well . Is there any way ? For a better understanding I want something like deepmoji.mit.edu but it's simpler than that . Any kind of help by anyone is appreciated

[gabrieldemarmiesse]

Closing this issue in favor of #11172 for organization purposes. This issue can be reopened later on if necessary.