Package syntax in defaults list overrides (structured configs)

[addisonklinke]

In addition to other keys required in an ML application (like trainer, model, etc), my top-level structured config has a datamodule node which will contain all the data-related items (i.e. dataset/loader during training and preparing crops on disk ahead of time)

from dataclasses import dataclass, field
from typing import Any, List
import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import DictConfig, OmegaConf, MISSING


@dataclass
class MlConf:
    datamodule: Any = DictConfig({'dataset': MISSING, 'prepper': None})
    seed: int = 1234


@dataclass
class PrepConf:
    crop_size: int = 224
    

@dataclass
class DatasetConf:
    defaults: List[Any] = field(default_factory=lambda: [
        {'override datamodule/prepper@_global_': 'crop'},  # FIXME what's the correct package syntax here?
    ])
    batch_size: int = 2

    
cs = ConfigStore.instance()
cs.store('config', MlConf)
cs.store(group='datamodule/prepper', name='crop', node=PrepConf)
cs.store(group='datamodule/dataset', name='simple', node=DatasetConf)


@hydra.main(config_name='config', config_path=None)
def main(cfg):
    print(OmegaConf.to_yaml(cfg))
    
    
if __name__ == '__main__':
    
    main()

My desired behavior is

1. The prepper and dataset nested under datmodule are empty by default
2. Dataset selection is required, but configuring a prepper is optional
3. Datasets can override the defaults list to select their desired prepper (instead of the default prepper=None)

The above code runs as expected to accomplish goals 1 and 2. However, I cannot get the 3rd goal to work

$ python app.py +datamodule/dataset=simple 
In 'datamodule/dataset/simple': Could not override 'datamodule/dataset/datamodule/prepper'
No match in the defaults list.

The desired/expected output would be

datamodule:
  dataset:
    batch_size: 2
  prepper: 
    crop_size: 224  # As a result of the override contained in datamodule/dataset=simple
seed: 1234

Based on the terminology docs for packages in Hydra 1.1.1, I thought that @_global_ would force my override into the global (top-level) package rather than the default nesting under its node's package. However it appears to have no effect. What am I missing?

[Jasha10]

Hi @addisonklinke,
The following will do the trick:

@dataclass
class DatasetConf:
    defaults: List[Any] = field(default_factory=lambda: [
        {'/datamodule/prepper@_group_': 'crop'},
    ])
    batch_size: int = 2

There are three changes I've to DatasetConf's defaults list:

• Removed the "override" keyword
• Changed datamodule/prepper to /datamodule/prepper
• Changed _global_ to _group_

Why not use the "override" keyword?

The error message you pasted above says "Could not find override ..."
The "override" keyword only applies you are using a defaults list to select an option from a group that is already present in some other config's defaults list. I'll circle back to give an example at the end of this comment.

A good rule of thumb is to never use "override" unless Hydra gives you an error saying specifically that "override" it's necessary.

Why use a leading slash in /datamodule/prepper?

Having removed the override keyword, our defaults list in DatasetConf now looks like this:

[{"datamodule/prepper@_global_": "crop"}]

Re-running the application, we get the following error:

$ python app.py +datamodule/dataset=simple
In 'datamodule/dataset/simple': Could not find 'datamodule/dataset/datamodule/prepper/crop'

...

This error signals that, while reading the defaults list for the simple config stored in the datamodule/dataset group, Hydra tried and failed to find a group called datamodule/dataset/datamodule/prepper.
The desired behavior, of course, would be for Hydra to look up a group called datamodule/prepper, not a group called datamodule/dataset/datamodule/prepper. The incorrect group name comes from Hydra's interpretation of the "datamodule/prepper" key in your simple config's defaults list as being relative to the group of the simple config. Adding a leading slash, /datamodule/prepper, makes the address of the group absolute.
The analogy here is to a file system; if you are in the directory /foo/bar and you type cd dir at your command line, you'll find yourself in directory /foo/bar/dir. Meanwhile, if you type cd /dir at the command line, the pwd will be /dir. This is documented here in the Hydra docs.

Why use _group_ instead of _global_?

After changing datamodule/prepper@... to /datamodule/prepper@..., the application's output looks like this:

$ python app.py +datamodule/dataset=simple
In 'datamodule/prepper/crop': ValidationError raised while composing config:
Merge error: PrepConf is not a subclass of MlConf. value: {'crop_size': 224}
    full_key:
    object_type=MlConf

...

Looking at this error, one wonders: "why should it matter whether PrepConf is a subclass of MlConf?
The answer is that the _global_ keyword signals to Hydra that the selected config (which is, in this case, datamodule/prepper/crop) should be merged at the top-level. The same error message can be reproduced by running the following minimal python file:

# tmp.py
from app import MlConf, PrepConf
from omegaconf import OmegaConf

OmegaConf.merge(MlConf, PrepConf)

$ python tmp.py
Traceback (most recent call last):
...
omegaconf.errors.ValidationError: Merge error: PrepConf is not a subclass of MlConf. value: {'crop_size': 224}
    full_key:
    object_type=MlConf

Of course, merging the PrepConf config at the top-level is not desired. What want instead is this:

# tmp2.py
from app import MlConf, PrepConf
from omegaconf import OmegaConf

cfg = OmegaConf.merge(MlConf, {"datamodule": {"prepper": PrepConf}})
print(OmegaConf.to_yaml(cfg))

$ python tmp2.py
datamodule:
  dataset: ???
  prepper:
    crop_size: 224
seed: 1234

This is achieved by replacing _global_ with _group_. Whereas _global_ signals that the selected option should be merged at the top level, _group_ signals that it should be merged at the package corresponding to the path of the group.

After changing _group_ to _global_, the application has the desired behavior.

Circling back to the "override" keyword:

Now, for expository purposes, let me give an example of when the "override" keyword would be necessary.
Suppose we modify MlConf, adding a defaults list as follows:

@dataclass
class MlConf:
    defaults: List[Any] = field(default_factory=lambda: ["_self_", {"datamodule/prepper": None}])
    datamodule: Any = DictConfig({"dataset": MISSING, "prepper": None})
    seed: int = 1234

Let's stick with our new-and-improved defaults list for DatasetConf:

[{'/datamodule/prepper@_group_': 'crop'}]

Running the app now (without any command-line overrides), you'll see the same behavior as before:

$ python app.py
datamodule:
  dataset: ???
  prepper: null
seed: 1234

If you try to use the override +datamodule/dataset=simple, however, you'll get the following error:

$ python app.py +datamodule/dataset=simple
Multiple values for datamodule/prepper. To override a value use 'override datamodule/prepper: crop'

...

Note Hydra's explicit message saying that the "override" keyword should be used.
The key issue here is that the datamodule/prepper group appears in the defaults lists of two different configs; it shows up both in the defaults list of MlConf and in the defaults list of DatasetConf. Hydra requires that, if a group shows up more than once, all but one of those occurrences be marked with the "override" keyword. Requiring this keyword is intended to help prevent mistakes; the most common use-case for defaults lists will see each group appearing in the defaults list only once, so explicit use of "override" is required when multiple appearances are intended.

When you add the "override" keyword to the defaults list of DatasetConf:

[{'override /datamodule/prepper@_group_': 'crop'}]

$ python app.py +datamodule/dataset=simple
datamodule:
  dataset:
    batch_size: 2
  prepper:
    crop_size: 224
seed: 1234

[addisonklinke]

Thanks @Jasha10! That makes a lot of sense regarding the leading slash for absolute groups and @_group_ vs. @_global_. I think I had conditioned myself to add override to my defaults lists when migrating to Hydra 1.1, but I see now why it should be a response to exceptions rather than the starting point

One follow-up - when using {'/datamodule/prepper': 'crop'} (absolute group, but no package directive), the app outputs

datamodule:
  dataset:
    datamodule:  # What actually got added
      prepper:
        crop_size: 224
    batch_size: 2
  prepper: null  # What we want to change via defaults list
seed: 1234

I'm confused why the absolute group /datamodule/prepper received a relative placement under datamodule/dataset? I would think that the absolute syntax would be sufficient without having to add the @_group_ package directive

[Jasha10]

I would think that the absolute syntax would be sufficient without having to add the @_group_ package directive

One would think so 😆
The behavior does seem lacking in consistency... I guess this is where the analogy with the filesystem breaks down.
Not sure what to say besides that the behavior it is what it is, and changing it would require working around backwards-compatibility concerns with a careful deprecation-and-migration plan.