[Bug] Appending to Defaults List via command line doesn't work if config is in subfolder

[siferati]

🐛 Bug

Description

When overriding config groups in the terminal, hydra interprets them as being in the global package.
This has been stated in different issues:

• [Question] Override a config group of absolute path from command line #2434 (comment)
• [Bug] Absolute path for overrides group does not work like defaults list #2105 (comment)

However, this doesn't seem to work when appending to configs loaded via --config-name=subfolder/config_name.
Hydra takes the given override as a relative path and fails to append it (see below for minimal example)

Checklist

• I checked on the latest version of Hydra
• I created a minimal repro (See this for tips).

To reproduce

# see vanilla config
> python main.py --config-name=server/alpha
db:
  driver: mysql
  user: omry
  password: secret
name: alpha

# override mysql db with postgresql -- works as expected
> python main.py --config-name=server/alpha db=postgresql
db:
  driver: postgresql
  user: postgres_user
  password: drowssap
  timeout: 10
name: alpha

# append a new db configuration -- doesn't work because unlike override, append is treating postgresql as relative path
> python main.py --config-name=server/alpha +db@db_2=postgresql
hydra.errors.MissingConfigException: In 'server/alpha': Could not find 'server/db/postgresql'

Stack trace/error message:

Traceback (most recent call last):
  File "test/main.py", line 11, in <module>
    my_app()
  File "/virtual_env/lib/python3.10/site-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 222, in run_and_report
    raise ex
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 219, in run_and_report
    return func()
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 105, in run
    cfg = self.compose_config(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 594, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py", line 253, in _load_configuration_impl
    defaults_list = create_defaults_list(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 746, in create_defaults_list
    defaults, tree = _create_defaults_list(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 716, in _create_defaults_list
    defaults_tree = _create_defaults_tree(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 356, in _create_defaults_tree
    ret = _create_defaults_tree_impl(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 457, in _create_defaults_tree_impl
    return _expand_virtual_root(repo, root, overrides, skip_missing)
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 280, in _expand_virtual_root
    subtree = _create_defaults_tree_impl(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 573, in _create_defaults_tree_impl
    add_child(children, new_root)
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 520, in add_child
    subtree_ = _create_defaults_tree_impl(
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 488, in _create_defaults_tree_impl
    config_not_found_error(repo=repo, tree=root)
  File "/virtual_env/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 800, in config_not_found_error
    raise MissingConfigException(
hydra.errors.MissingConfigException: In 'server/alpha': Could not find 'server/db/postgresql'

Expected Behavior

Appending treats the given config as a global package, just like when overriding.
The following command should work instead of resulting in an error:

> python main.py --config-name=server/alpha +db@db_2=postgresql
db:
  driver: mysql
  user: omry
  password: secret
name: alpha
db_2:
  driver: postgresql
  user: postgres_user
  password: drowssap
  timeout: 10

System information

• Hydra Version : 1.3.0 (tested 1.3.2 too, same problem)
• Python version : 3.10.12
• Virtual environment type and version : virtualenv 20.16.5
• Operating system : Ubuntu 22.04

[ianxmason]

I agree that this behavior seems inconsistent and it has created some challenges for me when using the CLI or compose API.

Depending on the actual config structure, I have sometimes being able to workaround this by avoiding setting the config name. For example, in the minimal example provided, it is possible to get the desired output with this command:

python main.py +server=alpha +db@db_2=postgres

[timkpaine]

This is fixed in lerna