My implementation of a very simple subword-level LSTM model for generating text using PyTorch. I trained it on the Tiny Shakespeare dataset, a 1 MB collection of various Shakespeare texts.
With 37.629201 M parameters and a vocab size of 50257, the network reached an approximate val loss of 5.86 after a bit more than 13 hours of training using an NVIDIA GTX 1060 3G GPU (for better interpretation of the loss, a completely random text generator would yield a loss of 10.82, or -ln(1/vocab_size)). The text generated sometimes can be nonsensical but definely looks "Shakespearian."
The text is converted into tokens using the tiktoken encoder (https://github.com/openai/tiktoken), with each token representing a subword piece in the text. Then, the tokens are embedded, reshaped, and fed into the LSTM layer. After that, the output goes into a linear layer to form the logits. Finally, the logits are either used to calculate the loss (during training, using cross-entropy loss) or used to predict the next character (during inference, via a softmax layer).
To run on your device, do py model.py or py model.py [PRETRAINED_MODEL] to load and use a pretrained model.
QUEEN:
And so lies few, I sent for my soul of death
And I am a soldier, as go in Ireland,
Our time did not prepared or rage.
NORTHUMBERLAND:
My lord, the Lord Northumberland?
DERBY:
We are transported my part, though light be a coward
Even through the secrets by the level or night
Virtue all and fix'd and smiles,
Comes no other dead, hearing'st remit,
First Lord Hastings, when full of beauty;
Under whose circuit is Elysicles
Nightly envious office shall be ingrate upon.
GREGORY:
You are but ever of your device.
YORK:
My lord, grown to be, ere you do thus far?
Speak enough, my liege. I would not Tybalt,
I'ld do poison my for ourselves;
Leave rich, and the matter an aspect George
Shall be buried,'twas a office of never;
Or sent by that knows you contend.
POLIXENES:
So thrive so look, pretty foot,
That have courage to feed my death.
Now, for our sunshine earth I name it unable,
And fright me from all Cawdy head to-night;
Whose sting is just in December snow on abuse,
Both stir: he may make gods cannot dewen,
Oxford, as ways, as custom, if
they must receive, and what he makes him that
Your jealously, her actions as blanks's;
For her sister Edward, like an oldables
Is look yourself to endure him down;
And stay, what first we do this hour, it would hither
Till you have spoke their mistaled foot, which say
You seem upon your pleasure, that expect him now
That more unfortunate.
Due to GitHub file size limit of 100 MB, I couldn't push the tiktoken pretrained model (the one that produced the above output) to this repository. You can still use a pretrained model just by cloning my second commit, but keep in mind that you'd be using the old character-level model. Clone it by typing on the terminal
git clone -n https://github.com/filip-fernandes/generative-lstm.git
cd generative-lstm
git checkout 1f23ec691fcfd7edb19e5abf1c7e067d87be94f2