[Bug] Argument parsing fails for multirun with `range` if `PYTHONOPTIMIZE=1`

[EspenHa]

🐛 Bug

Description

Hydra will fail to parse a range statement in multi-run mode if PYTHONOPTIMIZE=1.

Checklist

• I checked on the latest version of Hydra
• I created a minimal repro (See this for tips).

To reproduce

Create a simple hydra app like this:

# run.py
from dataclasses import dataclass

import hydra
from hydra.core.config_store import ConfigStore

@dataclass
class MySQLConfig:
    host: str = "localhost"
    port: int = 3306

cs = ConfigStore.instance()
cs.store(name="config", node=MySQLConfig)

@hydra.main(version_base=None, config_name="config")
def my_app(cfg: MySQLConfig) -> None:
    if cfg.port == 80:
        print("Is this a webserver?!")

if __name__ == "__main__":
    my_app()

Now run PYTHONOPTIMIZE=1 python run.py -m port=80,81.
Hydra should handle this correctly, outputting something like:

[2023-02-02 10:05:07,993][HYDRA] Launching 2 jobs locally
[2023-02-02 10:05:07,993][HYDRA]        #0 : port=80
Is this a webserver?!
[2023-02-02 10:05:08,294][HYDRA]        #1 : port=81

If we instead run PYTHONOPTIMIZE=1 python run.py -m port="range(80,82)", which should be equivalent, we get an error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 219, in run_and_report
    return func()
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 466, in <lambda>
    lambda: hydra.multirun(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py", line 143, in multirun
    cfg = self.compose_config(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py", line 594, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
    parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
    parsed_overrides = parser.parse_overrides(overrides=overrides)
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_parser.py", line 82, in parse_overrides
    parsed = self.parse_rule(override, "override")
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_parser.py", line 67, in parse_rule
    ret = visitor.visit(tree)
  File "/usr/local/lib/python3.8/dist-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/usr/local/lib/python3.8/dist-packages/hydra/grammar/gen/OverrideParser.py", line 195, in accept
    return visitor.visitOverride(self)
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 174, in visitOverride
    value = self.visitValue(ctx.value())  # type: ignore[no-untyped-call]
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 132, in visitValue
    return self.visitElement(ctx.element())  # type: ignore[no-untyped-call]
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 118, in visitElement
    return self.visitFunction(ctx.function())  # type: ignore
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 233, in visitFunction
    value = self.visitElement(cur)
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 117, in visitElement
    if ctx.function():  # type: ignore[no-untyped-call]
AttributeError: 'TerminalNodeImpl' object has no attribute 'function'

Expected Behavior

Expected the output of PYTHONOPTIMIZE=1 python run.py -m port="range(80,82)" to be identical to the output of PYTHONOPTIMIZE=1 python run.py -m port=80,81.

System information

• Hydra Version : 1.3.1
• Python version : 3.8.10
• Virtual environment type and version : System install inside docker
• Operating system : Ubuntu 20.04 (docker) / Ubuntu 22.04 (host)

[Jasha10]

Thanks for the report, @EspenHa. I'm able to reproduce the error.

[Jasha10]

The traceback looks like it's related to antlr (which Hydra uses for parsing CLI overrides).

[timkpaine]

This is fixed in lerna