BUG: Resample.aggregate raising TypeError instead of SpecificationError with missing keys dtypes #39028

phofl · 2021-01-07T21:58:05Z

closes BUG: Missing keys using aggregation dictionary that are unsortable raise TypeError instead of SpecificationError #39025
tests added / passed
Ensure all linting tests pass, see here for how to run them
whatsnew entry

jreback · 2021-01-08T01:15:31Z

pandas/core/aggregation.py

@@ -738,7 +738,10 @@ def agg_dict_like(
        if isinstance(selected_obj, ABCDataFrame) and len(
            selected_obj.columns.intersection(keys)
        ) != len(keys):
-            cols = sorted(set(keys) - set(selected_obj.columns.intersection(keys)))
+            cols = sorted(


can you just use safe_sort?

Yes, of course. Thx. Forgot about safe sort...

Hm had to add list calls

simonjayhawkins · 2021-01-08T09:40:50Z

Thanks @phofl for the PR. There is a very similar issue pandas/core/aggregation.py:485: error: Value of type variable "_LT" of "sorted" cannot be "Optional[Hashable]" [type-var] that I didn't yet add an issue for.

There are many tests that go through that path that could be used as the basis for a test for a fix.

=============================================== short test summary info ================================================
FAILED pandas/tests/base/test_misc.py::test_memory_usage[series-with-empty-index] - assert 0 == 108
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 0-ops0-names0] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 0-ops1-names1] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 0-ops2-names2] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 0-ops3-names3] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 1-ops0-names0] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 1-ops1-names1] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 1-ops2-names2] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 1-ops3-names3] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 'index'-ops0-names0] - Assertio...
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 'index'-ops1-names1] - Assertio...
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 'index'-ops2-names2] - Assertio...
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 'index'-ops3-names3] - Assertio...
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 'columns'-ops0-names0] - Assert...
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 'columns'-ops1-names1] - Assert...
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 'columns'-ops2-names2] - Assert...
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_listlike[axis 'columns'-ops3-names3] - Assert...
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_empty_listlike[DataFrame-ops0] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_empty_listlike[DataFrame-ops1] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_dictlike[axis 0-dict] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_dictlike[axis 0-Series] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_dictlike[axis 1-dict] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_dictlike[axis 1-Series] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_dictlike[axis 'index'-dict] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_dictlike[axis 'index'-Series] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_dictlike[axis 'columns'-dict] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_dictlike[axis 'columns'-Series] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_empty_dictlike[DataFrame-ops1] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_empty_dictlike[DataFrame-ops2] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_empty_dictlike[DataFrame-ops3] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_empty_dictlike[DataFrame-ops4] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_empty_dictlike[DataFrame-ops5] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_and_agg_err[axis 0] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_and_agg_err[axis 1] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_and_agg_err[axis 'index'] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_and_agg_err[axis 'columns'] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_agg_dict_nested_renaming_depr - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[sum-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[max-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[min-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[mean-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[prod-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[std-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[var-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[median-DataFrame] - AssertionE...
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[kurt-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[skew-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[all-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_reducer_raises[any-DataFrame] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_bad_dtype[DataFrame-cummax] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_bad_dtype[DataFrame-cummin] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_bad_dtype[DataFrame-cumprod] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_bad_dtype[DataFrame-cumsum] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_bad_dtype[DataFrame-diff] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_bad_dtype[DataFrame-pct_change] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_bad_dtype[DataFrame-rank] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_bad_dtype[DataFrame-<lambda>] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_partial_failure[cummax] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_partial_failure[cummin] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_partial_failure[cumprod] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_partial_failure[cumsum] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_partial_failure[diff] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_partial_failure[pct_change] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_partial_failure[rank] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_missing_columns[axis 0] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_missing_columns[axis 1] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_missing_columns[axis 'index'] - AssertionError
FAILED pandas/tests/frame/apply/test_frame_transform.py::test_transform_missing_columns[axis 'columns'] - AssertionError
FAILED pandas/tests/series/apply/test_series_transform.py::test_transform_none_to_type - AssertionError

AssertionError due to assert False added to code

so could either fix here, or shall I create a dedicated bug report?

update: ignore FAILED pandas/tests/base/test_misc.py::test_memory_usage[series-with-empty-index] - assert 0 == 108 as unrelated. (maybe WSL issue, not yet investigated)

phofl · 2021-01-08T10:10:05Z

I could look into this in the evening, when this pr is not merged until then we could do this here

phofl · 2021-01-08T20:49:05Z

@simonjayhawkins Fixed it.

I am not quite sure why the test was where it was, so I moved it into the frame folder.

jreback · 2021-01-08T23:04:50Z

i know we marked the issues as a regression but that was from 1.0.x, so 1.3 is fine.

phofl · 2021-01-08T23:08:19Z

Thought the same, since only the message is the problem

jreback · 2021-01-08T23:18:03Z

thanks @phofl

…or with missing keys dtypes (pandas-dev#39028)

phofl added 3 commits January 7, 2021 22:45

Fix resample missing columns bug

ee16f26

Add whatsnew

4f9c282

Change whatsnew

2c7e58e

phofl added the Resample resample method label Jan 7, 2021

Fix bug

d5c3a66

jreback requested changes Jan 8, 2021

View reviewed changes

Add safe sort

f612899

Add list calls

7ab627d

phofl added 3 commits January 8, 2021 21:45

Fix other bug

e40a9d2

Move tests

ea32dba

Add whatsnew

21c4986

Add hash

d1ae40e

jreback approved these changes Jan 8, 2021

View reviewed changes

Fix mypy problem.

d67bc42

jreback added this to the 1.3 milestone Jan 8, 2021

jreback added the Error Reporting Incorrect or improved errors from pandas label Jan 8, 2021

jreback merged commit 8fe3dc6 into pandas-dev:master Jan 8, 2021

phofl deleted the 39025 branch January 8, 2021 23:18

simonjayhawkins mentioned this pull request Jan 9, 2021

TYP: fix typing errors for mypy==0.790, bump mypy version #37108

Merged

3 tasks

luckyvs1 pushed a commit to luckyvs1/pandas that referenced this pull request Jan 20, 2021

BUG: Resample.aggregate raising TypeError instead of SpecificationErr…

07174e8

…or with missing keys dtypes (pandas-dev#39028)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Resample.aggregate raising TypeError instead of SpecificationError with missing keys dtypes #39028

BUG: Resample.aggregate raising TypeError instead of SpecificationError with missing keys dtypes #39028

phofl commented Jan 7, 2021

jreback Jan 8, 2021

phofl Jan 8, 2021

phofl Jan 8, 2021

simonjayhawkins commented Jan 8, 2021 •

edited

Loading

phofl commented Jan 8, 2021

phofl commented Jan 8, 2021

jreback commented Jan 8, 2021

phofl commented Jan 8, 2021

jreback commented Jan 8, 2021

BUG: Resample.aggregate raising TypeError instead of SpecificationError with missing keys dtypes #39028

BUG: Resample.aggregate raising TypeError instead of SpecificationError with missing keys dtypes #39028

Conversation

phofl commented Jan 7, 2021

jreback Jan 8, 2021

Choose a reason for hiding this comment

phofl Jan 8, 2021

Choose a reason for hiding this comment

phofl Jan 8, 2021

Choose a reason for hiding this comment

simonjayhawkins commented Jan 8, 2021 • edited Loading

phofl commented Jan 8, 2021

phofl commented Jan 8, 2021

jreback commented Jan 8, 2021

phofl commented Jan 8, 2021

jreback commented Jan 8, 2021

simonjayhawkins commented Jan 8, 2021 •

edited

Loading