New Paper: RETRO

[jacobbieker]

Detailed Description

https://medium.com/syncedreview/deepminds-retro-retrieval-enhanced-transformer-retrieves-from-trillions-of-tokens-achieving-c54313fec51d

Context

Paper is mostly about language models, but has an interesting idea for searching for similar text to help the model learn, and seems to work well. I believe we talked about that at some point, using similarity search, for satellite predictions, and this seems to say it works for nlp at least.

Possible Implementation

Have a database of similar historical PV data to search from, or satellite images, and use something like faiss for fast search. Then would be able to include historical past and future gsp data as well for those days to help with predictions

[jacobbieker]

Related to #65