Joined dictionary for more than 2 languages

[okgrammer]

Hello, in the multilingual translation example, a joined dictionary is created between de-en, then the resulting dictionary is used for fr-en. In this case I think it's fine because there are probably a lot of overlaps in the vocabulary among these languages. However, what if I have 3 really different languages with fewer overlaps, for instance English-Korean-Chinese, each having their own writing systems? For example, if I create a joined dictionary for English-Korean first, then a lot of Chinese subwords may be missing in the final dictionary.

One workaround that I did is to combine the training data from all languages, then call fairseq-preprocess once to generate a joined dictionary. After that, I run fairseq-preprocess separately on each language pair, re-using the joined dictionary in the first step.

[pipibjc]

Good catch. The example seems not ideal, and your workaround makes sense.
When training multilingual model, we usually learn the BPE (byte pair encoding) model from all languages and generate the shared BPE units. We then use the BPE units as predefined vocabulary when binarizing each language pairs. This is similar to your workaround.

[pipibjc]

To be more specific - if you use similar preprocess as prepare-iwslt17-multilingual.sh for English-Korean-Chinese, the sentence piece package will generate a vocabulary file [1]. We can convert it to fairseq vocabulary file format and assign it as predefined vocabulary when running fairseq-preprocess, e.g.

Adding something like this in prepare-iwslt17-multilingual.sh:

# strip the first three special tokens and append fake counts for each vocabulary
tail -n +4 $DATA/sentencepiece.bpe.vocab | cut -f1 | sed 's/$/ 100/g' > fairseq.vocab

and when binarization, we add --tgtdict fairseq.vocab option to assign predefined vocabulary file.

[1] https://github.com/pytorch/fairseq/blob/master/examples/translation/prepare-iwslt17-multilingual.sh#L105

[okgrammer]

@pipbjc Thanks for your reply! I'll try your method. A side question about this line in the prepare script:
https://github.com/pytorch/fairseq/blob/master/examples/translation/prepare-iwslt17-multilingual.sh#L113. Isn't this doing the call to spm_encode twice?

[feralvam]

@okgrammer Hi! Did you ever find out the purpose of that loop for LANG in "$SRC" "$TGT"; do? Thanks!

[munael]

@pipibjc This advice is out of date with the current version of the .sh :/
The current example doesn't use fairseq-preprocess.

[ever4244]

@pipibjc This advice is out of date with the current version of the .sh :/
The current example doesn't use fairseq-preprocess.

What do you mean?
I see that all multilingual translation examples still use fairseq-preprocess?

[sonalsannigrahi]

quick question, if I am using a different tokenisation apart from bpe.. can I still use the learnt joint bpe vocabulary to induce the fairseq vocab still? At the moment I am trying different tokenisations for multilingual translation and am having trouble with the joint dictionary for multiple languages.

[stale]

This issue has been automatically marked as stale. If this issue is still affecting you, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!

[stale]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue with up-to-date information. Thank you!