Skip to content

Fixes for compat='no_conflicts' and open_mfdataset #1007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 23, 2016

Conversation

shoyer
Copy link
Member

@shoyer shoyer commented Sep 19, 2016

  • xarray.merge now uses compat='no_conflicts' by default.
  • compat='no_conflicts' preserves attributes.
  • Variable.no_conflicts broadcasts arguments, making it a strictly more
    relaxed check than broadcast_equals.
  • Allow concat_dim=None in open_mfdataset and auto_combine to disable
    trying to infer a dimension for concatenating.
  • Add compat as an argument to open_mfdataset to allow more control over
    merging.

CC @jcmgray

- `xarray.merge` now uses `compat='no_conflicts'` by default.
- `compat='no_conflicts'` preserves attributes.
- `Variable.no_conflicts` broadcasts arguments, making it a strictly more
  relaxed check than `broadcast_equals`.
- Allow `concat_dim=None` in `open_mfdataset` and `auto_combine` to disable
  trying to infer a dimension for concatenating.
- Add `compat` as an argument to `open_mfdataset` to allow more control over
  merging.
merges will now succeed in cases that previously raised
``xarray.MergeError``. Set ``compat='broadcast_arrays'`` to restore the
previous default.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be 'no_conflicts' and 'broadcast_equals' here I think.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, thanks

@jcmgray
Copy link
Contributor

jcmgray commented Sep 20, 2016

This all looks great to me. Might the docstring for auto_combine need to be updated? Currently it seems to suggest that sorting, alignment and the addition of new data variables are not supported, whereas I think all these can occur with compat='no_conflicts'.

@shoyer
Copy link
Member Author

shoyer commented Sep 22, 2016

Might the docstring for auto_combine need to be updated? Currently it seems to suggest that sorting, alignment and the addition of new data variables are not supported, whereas I think all these can occur with compat='no_conflicts'.

Yep, with compat='no_conflicts' and recent align updates to concat, we are most of the way there on alignment.

It still can fail if there are different variables on datasets across the concatenated dimension, but that will be fixed when we get concat to handle different variables (#508). At that point, we might want to switch the logic from the current "concatenate by variable group, then merge" to "merge by unique coordinates, then concatenate".

@shoyer shoyer merged commit fbb4f06 into pydata:master Sep 23, 2016
@shoyer shoyer deleted the merge-combine-update branch September 23, 2016 03:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants