Skip to content

TypeError when merging masked scalar coord #3584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
trexfeathers opened this issue Dec 6, 2019 · 7 comments · Fixed by #6468
Closed

TypeError when merging masked scalar coord #3584

trexfeathers opened this issue Dec 6, 2019 · 7 comments · Fixed by #6468

Comments

@trexfeathers
Copy link
Contributor

The handling of scalar coordinates in _merge.py includes the build_indexes step. This cannot handle the scalar point being a masked value since it attempts to look the value up in the dictionary name_index_by_scalar, resulting in TypeError: unhashable type: 'MaskedConstant'.

This was discovered when attempting to load a NetCDF file that had previously been converted from NIMROD to NetCDF format. NetCDF had recognized a fill value for a scalar coordinate and therefore saved it as masked. load_cube includes a merge step, resulting in the error. This is demonstrated in the code below.

The presence of a masked point for a scalar coordinate is a possible scenario and should therefore be handled rather than producing an error.

import numpy as np
import iris
from iris import coords

my_cube = iris.cube.Cube(data=[1, 2])

# Generate a single-point array with a value NetCDF will recognise as a fill
# value and convert to a mask.
# Use the array as a scalar coordinate on my_cube.
my_array = np.array([-32767], dtype=np.int16)
my_scalar_coord = coords.DimCoord(points=my_array, standard_name="realization")
my_cube.add_aux_coord(my_scalar_coord)

# Save my_cube in NetCDF format.
iris.save(my_cube, "masked_scalar.nc")
# Attempt to load back the saved file. Results in the TypeError.
iris.load_cube("masked_scalar.nc")
@github-actions
Copy link
Contributor

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

@github-actions github-actions bot added the Stale A stale issue/pull-request label Jan 18, 2022
@schlunma
Copy link
Contributor

schlunma commented Feb 4, 2022

I just discovered a very similar problem: if the input cubes for merging have a scalar coordinate that uses a masked array, the scalar coordinate of the merged cube is not a masked array anymore.

MWE:

import numpy as np
import iris
import iris.coords
import iris.cube

# Setup cubes
scalar_coord = iris.coords.AuxCoord(np.ma.masked_array([0]), var_name='x')
merge_coord_1 = iris.coords.AuxCoord([1], var_name='merge_coord')
merge_coord_2 = iris.coords.AuxCoord([2], var_name='merge_coord')
cube_1 = iris.cube.Cube(1, var_name='cube',
                        aux_coords_and_dims=[(scalar_coord, ()), (merge_coord_1, ())])
cube_2 = iris.cube.Cube(2, var_name='cube',
                        aux_coords_and_dims=[(scalar_coord, ()), (merge_coord_2, ())])

# Create merged cube
cubes = iris.cube.CubeList([cube_1, cube_2])
merged_cube = cubes.merge_cube()

# Print scalar coordinates
print("Scalar coordinate of input cubes:")
print(cube_1.coord('x'))
print(cube_2.coord('x'))
print("")
print("Scalar coordinate of merged cube:")
print(merged_cube.coord('x'))

gives

Scalar coordinate of input cubes:
AuxCoord(masked_array(data=[0],
             mask=False,
       fill_value=999999), standard_name=None, units=Unit('unknown'), var_name='x')
AuxCoord(masked_array(data=[0],
             mask=False,
       fill_value=999999), standard_name=None, units=Unit('unknown'), var_name='x')

Scalar coordinate of merged cube:
DimCoord(array([0]), standard_name=None, units=Unit('unknown'), var_name='x')

As you can see, the scalar coordinate of the merged cube is not a masked_array anymore.

@github-actions
Copy link
Contributor

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

@github-actions github-actions bot added the Stale A stale issue/pull-request label Jun 21, 2023
@schlunma
Copy link
Contributor

Still relevant

@github-actions github-actions bot removed the Stale A stale issue/pull-request label Jun 22, 2023
@scitools-ci scitools-ci bot removed this from 🚴 Peloton Dec 15, 2023
Copy link
Contributor

github-actions bot commented Nov 3, 2024

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

@github-actions github-actions bot added the Stale A stale issue/pull-request label Nov 3, 2024
@schlunma
Copy link
Contributor

schlunma commented Nov 4, 2024

Still relevant

@github-actions github-actions bot removed the Stale A stale issue/pull-request label Nov 5, 2024
@bjlittle bjlittle moved this to 🆕 Candidate in 🦔 v3.12.0 Feb 6, 2025
@bjlittle bjlittle removed this from 🦔 v3.12.0 Apr 16, 2025
@bjlittle bjlittle moved this to 🆕 Candidate in 🦋 Iris 3.13.0 Apr 23, 2025
@bjlittle bjlittle moved this from 🆕 Candidate to 🔖 Assigned in 🦋 Iris 3.13.0 Apr 30, 2025
@ukmo-ccbunney ukmo-ccbunney moved this from 🔖 Assigned to 🚀 In Progress in 🦋 Iris 3.13.0 May 14, 2025
@ukmo-ccbunney
Copy link
Contributor

I just discovered a very similar problem: if the input cubes for merging have a scalar coordinate that uses a masked array, the scalar coordinate of the merged cube is not a masked array anymore.
(...snip...)
As you can see, the scalar coordinate of the merged cube is not a masked_array anymore.

Hi @schlunma

I've looked into this and it's a slightly tricky edge-case.

When aux coords are built from their coordinate data, Iris first attempts to construct them as a DimCoord, then if that fails it will try to construct them as an AuxCoord:

iris/lib/iris/_merge.py

Lines 1611 to 1631 in 16c6a33

# Build the auxiliary coordinates.
for template in self._aux_templates:
# Attempt to build a DimCoord and add it to the cube. If this
# fails e.g it's non-monontic or multi-dimensional or non-numeric,
# then build an AuxCoord.
try:
coord = iris.coords.DimCoord(
template.points, bounds=template.bounds, **template.kwargs
)
if len(template.dims) == 1 and template.dims[0] not in covered_dims:
dim_coords_and_dims.append(_CoordAndDims(coord, template.dims))
covered_dims.append(template.dims[0])
else:
aux_coords_and_dims.append(_CoordAndDims(coord, template.dims))
except ValueError:
# kwarg not applicable to AuxCoord.
template.kwargs.pop("circular", None)
coord = iris.coords.AuxCoord(
template.points, bounds=template.bounds, **template.kwargs
)
aux_coords_and_dims.append(_CoordAndDims(coord, template.dims))

Your x scalar coord in cube_1 and cube_2 can successfully be constructed with as a DimCoord, so that's what Iris does (note that it is a DimCoord in cube_merged, but an AuxCoord in cube_1 and cube_2).

The problem is that masked data is NOT ALLOWED in a DimCoord, so if any masked values exist it will fail. However, if your points are a masked array instance, but do not have any masked values, then the mask is simply dropped as part of the DimCoord._values setter:

iris/lib/iris/coords.py

Lines 2808 to 2809 in 16c6a33

# Cast to a numpy array for masked arrays with no mask.
points = np.array(points)

This is what is happening here and why your mask is lost.

The fix I am working on for the original problem mentioned in this issue will mitigate this slightly (although not fully) by allowing you to create AuxCoords with masked data. However, as Iris tries to make a DimCoord first, you will still loose the mask in your example; at lease one element of your array will need to be actually masked to make it an AuxCoord (and retain the mask).

@ukmo-ccbunney ukmo-ccbunney moved this from 🚀 In Progress to 👀 In Review in 🦋 Iris 3.13.0 May 23, 2025
@github-project-automation github-project-automation bot moved this from 👀 In Review to 🏁 Done in 🦋 Iris 3.13.0 May 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Status: 🏁 Done
Development

Successfully merging a pull request may close this issue.

5 participants