How to generate or convert vocab.json, merges.txt, and config.json to match huggingface/transformers requirements ?

[ycat3]

Hi,
Great work !!
I am migrating to GPT2 japanese version for pytorch
Pretrained GPT2 japanese is working fine.
Thanks a lot.
However,
I miss vocab.json, merges.txt and config.json files in the run-root directory.
Let me know the suggestion how to do it.

Best regards.

[lopuhin]

Thanks @ycat3 !

I miss vocab.json, merges.txt and config.json files in the run-root directory.

Sorry I might be not familiar with these files. To clarify this, could you please tell from which repo and to which repo are you trying to migrate?

[ycat3]

Thank you for quick reply.
There is German GPT-2 repo in the huggingface/transformers.
https://huggingface.co/anonymous-german-nlp/german-gpt2
This repo might be your fork.

[lopuhin]

Thanks for the pointers - that would essentially mean converting the model to huggingface/transformers format, right? That would be a great feature to add and looks feasible to do, but it's not currently supported.

[ycat3]

I know huggingface/tokenizers generates merges.txt and vocab.json.
https://github.com/huggingface/tokenizers

However I doubt is this solution.
Anyway I feel your repo is active.
I will push pull request when I need.

Best regards.