TypeError: object of type 'numpy.float64' has no len()

[hgrecco]

After merging the al pending PRs and linking to Pint 0.10.1 I am getting this error in many test
TypeError: object of type 'numpy.float64' has no len()

ping @andrewgsavage

[andrewgsavage]

I believe this is because pandas interprets a scalar quantity as list like.

..\..\anaconda3\envs\py37_pd251\lib\site-packages\pandas\core\indexers.py:94: in check_setitem_lengths
    if is_list_like(value) and len(indexer) != len(value):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Quantity(3.0, 'nanometer')>

    def __len__(self):
>       return len(self._magnitude)
E       TypeError: object of type 'numpy.float64' has no len()

pandas' is_list_like function checks if a object is an Iterable, which returns True for a quantity.

        isinstance(obj, abc.Iterable)

https://github.com/pandas-dev/pandas/blob/4f9bc8aee828ebfed72773fa0c9cf73a2c48200a/pandas/_libs/lib.pyx#L986

An object is an Iterable if it has an __iter__ method, which scalar quantities have. https://docs.python.org/3/library/collections.abc.html

Before deleting __iter__ in pint/quantity.py
==================== 17 failed, 282 passed, 28 xfailed, 2 xpassed, 7 warnings, 2 errors in 23.68s =====================
When I delete the __iter__ in pint/quantity.py
==================== 4 failed, 295 passed, 27 xfailed, 3 xpassed, 12 warnings, 2 errors in 20.70s =====================

[MichaelTiemannOSC]

import unittest
import pandas as pd
import numpy as np
from numpy.testing import assert_array_equal

from pint import set_application_registry
from pint_pandas import PintArray, PintType
from openscm_units import unit_registry
PintType.ureg = unit_registry
ureg = unit_registry
set_application_registry(ureg)
Q_ = ureg.Quantity

ureg.define("CO2e = CO2 = CO2eq = CO2_eq")

def pandas_mult_acc(a, b):
df = a.multiply(b)
return df.sum(axis=1)

def pint_mult_acc(a, b):
df = a.multiply(b)
return df.sum(axis=1).astype('pint[Mt CO2]')

class TestBaseProvider(unittest.TestCase):
"""
Test the Base provider
"""

def setUp(self) -> None:
    pass

def test_pandas_series_equality(self):
    projected_ei = pd.DataFrame([[1.0, 2.0], [3.0, 4.0]])
    projected_production = pd.DataFrame([[2.0, 4.0], [6.0, 8.0]])
    expected_data = pd.Series([10.0, 50.0],
                                index=[0, 1])
    print(pandas_mult_acc(projected_ei,projected_production))
    print(f"expected_data = {expected_data}")
    pd.testing.assert_series_equal(pandas_mult_acc(projected_ei,projected_production),
                                   expected_data)

def test_pint_series_equality(self):
    projected_ei = pd.DataFrame([[Q_(1.0, 't CO2/MWh'), Q_(2.0, 't CO2/MWh')], [Q_(3.0, 't CO2/MWh'), Q_(4.0, 't CO2/MWh')]], dtype='pint[t CO2/MWh]')
    projected_production = pd.DataFrame([[Q_(2.0, 'TWh'), Q_(4.0, 'TWh')], [Q_(6.0, 'TWh'), Q_(8.0, 'TWh')]], dtype='pint[TWh]')
    expected_data = pd.Series([10.0, 50.0],
                                index=[0, 1],
                                dtype='pint[Mt CO2]')
    print(pint_mult_acc(projected_ei,projected_production))
    print(f"expected_data = {expected_data}")
    pd.testing.assert_series_equal(pint_mult_acc(projected_ei,projected_production), expected_data)

x = TestBaseProvider('test_pint_series_equality')
TestBaseProvider.test_pandas_series_equality(x)
TestBaseProvider.test_pint_series_equality(x)

[andrewgsavage]

I think I ended up xfailing tests which gave that error in #40. Some pandas operations don't work with pint-pandas objects so if you're seeing that error then try doing things another way.

[MichaelTiemannOSC]

I've look at #40, and several other issues that it branches to. It seems like there's two problems, one with the infrastructure of pint_pandas own testing framework (and GitHub actions), and another related to pint_pandas functionality. In combination, the two seem to have stalemated a clear way forward. I've adjusted the test case above to show a clear pandas pass and a clear pint_pandas fail. I'll look at whether I or somebody on our team have the skills needed to walk down the path of creating a pint-aware assert_series_equal method.

[MichaelTiemannOSC]

I've looked harder, and it's not fixed in pandas-dev. But it's also not clear that it's broken in pandas-dev.

pandas is getting tripped up in _libs/testing.pyx. The function assert_almost_equal asks "isiterable(a)" and later asserts that the has_length(a) is also True. These are the definitions of those functions in that module:

cdef bint isiterable(obj):
    return hasattr(obj, '__iter__')

cdef bint has_length(obj):
    return hasattr(obj, '__len__')

It then proceeds to believe that because of yes to both, it can access it's length thusly:

        else:
            na, nb = len(a), len(b)

But this does not work for a Quantity:

>>> ureg = UnitRegistry()
>>> x = ureg('meter')
>>> x.__len__
<bound method Quantity.__len__ of <Quantity(1, 'meter')>>
>>> x.__len__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/michael/opt/miniconda3/envs/spyder-env/lib/python3.9/site-packages/pint/quantity.py", line 1864, in __len__
    return len(self._magnitude)
TypeError: object of type 'int' has no len()
>>> x
<Quantity(1, 'meter')>

We got there because assert_extension_array_equal in pandas/_testing/asserters.py is trying to compare arrays of Quantities and it calls down to the _lib function at line 858:

        _testing.assert_almost_equal(
            left_valid,
            right_valid,
            check_dtype=check_dtype,
            rtol=rtol,
            atol=atol,
            obj="ExtensionArray",
            index_values=index_values,
        )

and that call decides that the best way to grovel through the ExtensionArray is element by element (i.e., comparing the Quantities directly). Is there some more specific test of iterability or length that would convince pandas to avoid those paths and instead get on with the business of determining (1) neither are NA, or so it can (2) compare them for ==?

Here's the Issue I filed with pandas-dev: pandas-dev/pandas#45240

[andrewgsavage]

closed by #133