ENH: Optimize CoW for fillna with ea dtypes #51411
Conversation
pandas/core/internals/blocks.py
Outdated
| new_values = self.values.fillna(value=value, method=None, limit=limit) | ||
| nb = self.make_block_same_class(new_values) | ||
| return nb._maybe_downcast([nb], downcast) | ||
| if using_cow and self._can_hold_na and not self.values.isna().any(): |
There was a problem hiding this comment.
| if using_cow and self._can_hold_na and not self.values.isna().any(): | |
| if using_cow and self._can_hold_na and not self.values._hasna: |
(for some EAs that can be optimized)
| @@ -240,23 +240,26 @@ def test_factorize_empty(self, data): | |||
| tm.assert_numpy_array_equal(codes, expected_codes) | |||
| self.assert_extension_array_equal(uniques, expected_uniques) | |||
|
|
|||
| def test_fillna_copy_frame(self, data_missing): | |||
| def test_fillna_copy_frame(self, data_missing, using_copy_on_write): | |||
There was a problem hiding this comment.
If we need to use the fixture here (in the /extension) tests, then ideally we redefine it in extension/conftest.py, to make it easier for downstream authors.
It might become unavoidable to use it at some point, but for this specific case we could maybe also verify the "copy" by modifying the result and ensuring the parent didn't update. In the end, that's the behaviour we care about for users, the values being identical is more an implementation detail.
There was a problem hiding this comment.
I restructured the tests, but we need using_copy_on_write for the sparse test, so I copied it to conftest.py in extension
There was a problem hiding this comment.
but we need using_copy_on_write for the sparse test, so I copied it to conftest.py in extension
Sidenote: if it's only for the sparse test override, and not in the base class definitions, then we don't need it in the conftest.py in extension/ (because it's only needed if it is relevant for external packages subclassing those base test classes as well)
# Conflicts: # pandas/tests/copy_view/test_interp_fillna.py # pandas/tests/extension/json/test_json.py
|
Merging, happy to address potential comments in a follow up |
… ea dtypes) (#51639) Backport PR #51411: ENH: Optimize CoW for fillna with ea dtypes Co-authored-by: Patrick Hoefler <[email protected]>
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.