Add support for Python3.11

[majiayu000]

Before submitting

• [√] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
• [√] Did you read the contributor guideline?
• [√] Did you make sure to update the docs?
• [√] Did you write any new necessary tests?

What does this PR do?

Fixes #5012 (issue).
This PR updates supports for Python 3.11 and also add the adaptation of hydra by updating the version of hydra to 1.3.2.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Sure!

[majiayu000]

Fix a dependency conflicts.
Now anyone use Python 3.11 can install this lib by

git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./

[mdmalhou]

Switching the operator from < to > for omegaconf is not a good idea in general if it is not known why it was there in the first place. Anyways, there is still an issue with the dependencies. I get this error when running the preprocess command

fairseq-preprocess \
    --only-source \
    --trainpref $TEXT/wiki.train.tokens \
    --validpref $TEXT/wiki.valid.tokens \
    --testpref $TEXT/wiki.test.tokens \
    --destdir data-bin/wikitext-103 \
    --workers 5
common - <dataclasses._MISSING_TYPE object at 0x7f881a973490>
Traceback (most recent call last):
  File "/home/mohamed/miniconda3/envs/fairseq/bin/fairseq-preprocess", line 5, in <module>
    from fairseq_cli.preprocess import cli_main
  File "/mnt/c/Users/mohamed/phd/fairseq/fairseq_cli/preprocess.py", line 18, in <module>
    from fairseq import options, tasks, utils
  File "/mnt/c/Users/mohamed/phd/fairseq/fairseq/__init__.py", line 31, in <module>
    hydra_init()
  File "/mnt/c/Users/mohamed/phd/fairseq/fairseq/dataclass/initialize.py", line 24, in hydra_init
    cs.store(name=k, node=v)
  File "/home/mohamed/miniconda3/envs/fairseq/lib/python3.11/site-packages/hydra/core/config_store.py", line 85, in store
    cfg = OmegaConf.structured(node)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mohamed/miniconda3/envs/fairseq/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 125, in structured
    return OmegaConf.create(obj, parent, flags)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mohamed/miniconda3/envs/fairseq/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 178, in create
    return OmegaConf._create_impl(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mohamed/miniconda3/envs/fairseq/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 900, in _create_impl
    format_and_raise(node=None, key=None, value=None, msg=str(e), cause=e)
  File "/home/mohamed/miniconda3/envs/fairseq/lib/python3.11/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/home/mohamed/miniconda3/envs/fairseq/lib/python3.11/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mohamed/miniconda3/envs/fairseq/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 896, in _create_impl
    raise ValidationError(
omegaconf.errors.ValidationError: Object of unsupported type: '_MISSING_TYPE'
    full_key: 
    object_type=None

[majiayu000]

Sorry for I didn't go through a lot of tests. I am pretty new to this lib.

This seems an error for omegaconf(any version) which is not compatible for Python 3.11's dataclasses.

I fix this by change code in omegaconf/omegaconf.py

Just like below:

from dataclasses import _MISSING_TYPE
...
...



    @staticmethod
    def _create_impl(  # noqa F811
        obj: Any = _EMPTY_MARKER_, parent: Optional[BaseContainer] = None
    ) -> Union[DictConfig, ListConfig]:
        try:
           ...
           ...
                    if isinstance(obj, type):
                        raise ValidationError(
                            f"Input class '{obj.__name__}' is not a structured config. "
                            "did you forget to decorate it as a dataclass?"
                        )


                    # Add a judgement for missing type
                    elif isinstance(obj, _MISSING_TYPE):
                        return DictConfig(content={}, parent=parent)

                    else:
                        raise ValidationError(
                            f"Object of unsupported type: '{type(obj).__name__}'"
                        )

It looks like I need to improve compatibility with other libraries first.

[chenjiasheng]

fairseq/models/transformer/transformer_config.py:120 should be field(default_factory=DecoderConfig)

[ZeaMays14142]

excuse me, did this work, at least when in sync with the updates on the base branch?
and, how is this supposed to fix the conflict with fairseq in the first place?

[fgregg]

this is background for the omegaconf and hydra-cor pinned requirements:

#3722

[GitHub-20200408]

Sorry for I didn't go through a lot of tests. I am pretty new to this lib.

This seems an error for omegaconf(any version) which is not compatible for Python 3.11's dataclasses.

I fix this by change code in omegaconf/omegaconf.py

Just like below:

from dataclasses import _MISSING_TYPE
...
...



    @staticmethod
    def _create_impl(  # noqa F811
        obj: Any = _EMPTY_MARKER_, parent: Optional[BaseContainer] = None
    ) -> Union[DictConfig, ListConfig]:
        try:
           ...
           ...
                    if isinstance(obj, type):
                        raise ValidationError(
                            f"Input class '{obj.__name__}' is not a structured config. "
                            "did you forget to decorate it as a dataclass?"
                        )


                    # Add a judgement for missing type
                    elif isinstance(obj, _MISSING_TYPE):
                        return DictConfig(content={}, parent=parent)

                    else:
                        raise ValidationError(
                            f"Object of unsupported type: '{type(obj).__name__}'"
                        )

It looks like I need to improve compatibility with other libraries first.

do this face 'NameError: name "EMPTY_MARKER" is not defined'， why？