BUG: Preserve StringDtype in where with list-like other by sanrishi · Pull Request #64040 · pandas-dev/pandas

sanrishi · 2026-02-05T17:21:30Z

closes BUG: DataFrame[StringDtype].where(DataFrame[bool], list[str]) returns object type instead of StringDtype. #63842 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.
I have reviewed and followed all the contribution guidelines
If I used AI to develop this pull request, I prompted it to follow AGENTS.md.

Description:

DataFrame.where fell back to object dtype when operating on StringDtype columns with list-like other values, instead of preserving StringDtype.

Implementation:

Implemented ArrowStringArray._where using pyarrow.compute.if_else to avoid fragile assignment paths, and added length‑1 list broadcasting support for both string backends to maintain dtype preservation.

Verification:

Added regression coverage in test_where_string_listlike_other and updated v3.0.1 release notes.

(pandas-dev) C:\Users\My\Documents\GitHub\pandas>python -m pytest pandas/tests/frame/indexing/test_where.py -k "string_listlike_other
C:\Users\My\.conda\envs\pandas-dev\Lib\site-packages\pytest_cython\__init__.py:2: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import get_distribution
=========================================================================================================== test session starts ============================================================================================================
platform win32 -- Python 3.11.14, pytest-9.0.2, pluggy-1.6.0
PySide6 6.9.3 -- Qt runtime 6.9.3 -- Qt compiled 6.9.3
rootdir: C:\Users\My\Documents\GitHub\pandas
configfile: pyproject.toml
plugins: anyio-4.12.0, hypothesis-6.148.7, cov-7.0.0, cython-0.3.1, localserver-0.0.0, qt-4.5.0, xdist-3.8.0
collected 148 items / 142 deselected / 6 selected

pandas\tests\frame\indexing\test_where.py ......

---------------------------------------------------------------------------------- generated xml file: C:\Users\My\Documents\GitHub\pandas\test-data.xml -----------------------------------------------------------------------------------
=========================================================================================================== slowest 30 durations ===========================================================================================================
0.01s call     pandas/tests/frame/indexing/test_where.py::test_where_string_listlike_other[other0-expected0-string]
0.01s setup    pandas/tests/frame/indexing/test_where.py::test_where_string_listlike_other[other0-expected0-string]

(16 durations < 0.005s hidden.  Use -vv to show these durations.)
==================================================================================================== 6 passed, 142 deselected in 0.26s =====================================================================================================

I am CLI 🤖 : Code is changed by me Codex, I read agents.md, ensuring that every changed line is reviewed 😉.

…d regression with fillna

sanrishi · 2026-02-09T15:26:06Z

Pre-commit.ci autofix

sanrishi · 2026-02-09T17:54:46Z

@mroeschke It's been couple of days
could you please review this PR that will be appreciated

jorisvandenbossche · 2026-02-11T10:54:29Z

pandas/core/arrays/string_.py

+        if lib.is_list_like(value) and not lib.is_scalar(value) and len(value) == 1:
+            value = value[0]


Why the special case for a list of one element?

This is to preserve scalar broadcasting semantics for length‑1 list‑likes. ExtensionArray._where does val = value[~mask] for list‑likes, so a Python list fails boolean indexing outright and a length‑1 ndarray raises a boolean index length mismatch.

Unwrapping ['a'] to 'a' routes it through the scalar path, which broadcasts correctly to the masked slots.

Ah, that's a good point. Can you add a small comment about that?

Also, I suppose this could be moved into the base class implementation that we are calling here?

Also, I suppose this could be moved into the base class implementation that we are calling here?

Yea that's good idea!

You added it to the base class, but so then it can be removed here from this method?

Oh! Yeah i forget that

pandas/tests/frame/indexing/test_where.py

jorisvandenbossche · 2026-02-13T13:47:16Z

pandas/core/arrays/string_.py

+        if lib.is_list_like(value) and not isinstance(
+            value, (np.ndarray, ExtensionArray)
+        ):
+            value = self._from_sequence(value, dtype=self.dtype)


Also this could be moved to the base class?

jorisvandenbossche · 2026-02-13T13:49:06Z

You have failing CI due to some formatting issues (I recommend using pre-commit locally, see https://pandas.pydata.org/docs/dev/development/contributing_codebase.html#pre-commit

…ubclass overrides

…remove subclass overrides" This reverts commit 565dde5.

sanrishi · 2026-02-14T13:32:23Z

Pre-commit.ci autofix

for more information, see https://pre-commit.ci

jorisvandenbossche · 2026-02-17T10:05:58Z

pandas/core/arrays/string_arrow.py

+    def _where(self, mask, value) -> Self:
+        return super()._where(mask, value)


I suppose this override is no longer needed?

I tested removing the override, but it causes the result to lose its DType and fall back to object because the base class implementation doesn't correctly handle the Arrow-to-sequence coercion. Keeping the override is necessary to preserve the string[pyarrow] DType.

I don't understand how an override which only calls the parent method through super() does actually change anything.

And can you show the output of the test failure when removing those two lines of code?

You’re totally right the previous _where override was a no‑op (super() only), so removing it didn’t change behavior. Sorry for the confusion.

However, this does reveal that the base _where path is broken for Arrow strings when other is list‑like: it doesn’t coerce to string[pyarrow] and falls back to object dtype.

Minimal repro:

import numpy as np import pandas as pd s = pd.Series(["a", "b", "c"], dtype="string[pyarrow]") mask = np.array([True, False, True]) s.where(mask, ["x", "y", "z"])

Output dtype becomes object instead of string[pyarrow].

Failure output:

FAILURE: Attributes of Series are different Attribute "dtype" are different [left]: object [right]: <StringDtype(na_value=<NA>)>

I’ve pushed a new commit that adds a real _where override which coerces list‑like other to string[pyarrow] (via _from_sequence / astype) before calling super(), preserving dtype.

Are the above approach right path forward? @jorisvandenbossche

sanrishi · 2026-02-18T13:57:57Z

By the way @jorisvandenbossche
are we participating in gsoc this year ?
Any project idea which is good for pandas and should be in gsoc? Let me know!
I am so willing to contribute for pandas in that reputable environment

Should I post project idea proposal for pandas when registration portal opens for gsoc?

jorisvandenbossche · 2026-02-18T14:31:10Z

doc/source/whatsnew/v3.0.1.rst

 - Fixed a bug for comparison operators between :py:class:`range` and objects with :class:`StringDtype` with ``storage="pyarrow"`` (:issue:`63429`)
 - Fixed a bug in the :class:`DataFrame` constructor when passed a :class:`Series` or
  :class:`Index` correctly handling Copy-on-Write (:issue:`63899`)
+- Fixed a bug in :meth:`DataFrame.where` where a ``StringDtype`` DataFrame and


Can you move this to the v3.0.2 file?

jorisvandenbossche · 2026-02-18T14:31:55Z

By the way @jorisvandenbossche
are we participating in gsoc this year ?

As far as I know, we are not participating ..

sanrishi · 2026-02-18T14:50:20Z

Is that means if I propose any idea for a project in proposal portal is that wastage of time?

…-where-final

sanrishi · 2026-02-22T13:35:57Z

@jorisvandenbossche Its been couple of days its ready for review!
Never mind if you are busy!

sanrishi · 2026-03-01T05:04:29Z

@jorisvandenbossche is this looking good?

BUG: Preserve StringDtype in where with list-like other

1e4892e

sanrishi mentioned this pull request Feb 5, 2026

BUG: DataFrame[StringDtype].where(DataFrame[bool], list[str]) returns object type instead of StringDtype. #63842

Open

sanrishi added 2 commits February 5, 2026 23:10

CLN: Fix import order in where test

dd9ebbb

BUG: Fix GH#63842 - ArrowStringArray.where failure with list-likes an…

13d432d

…d regression with fillna

sanrishi force-pushed the fix-63842-stringdtype-where-final branch from 8f600e9 to 13d432d Compare February 9, 2026 15:21

jorisvandenbossche reviewed Feb 11, 2026

View reviewed changes

TST: use any_string_dtype in where test

fdb483f

sanrishi force-pushed the fix-63842-stringdtype-where-final branch from 9d4c97b to fdb483f Compare February 11, 2026 12:38

sanrishi requested a review from jorisvandenbossche February 11, 2026 12:43

REF: move length-1 unboxing to Base with comment

b043f5a

sanrishi force-pushed the fix-63842-stringdtype-where-final branch from 7b17d2d to b043f5a Compare February 11, 2026 14:12

jorisvandenbossche reviewed Feb 13, 2026

View reviewed changes

REF: move list conversion logic to ExtensionArray._where and remove s…

565dde5

…ubclass overrides

sanrishi force-pushed the fix-63842-stringdtype-where-final branch from 7a6feab to 565dde5 Compare February 14, 2026 09:18

sanrishi added 3 commits February 14, 2026 14:51

Revert "REF: move list conversion logic to ExtensionArray._where and …

a8fe6ac

…remove subclass overrides" This reverts commit 565dde5.

BUG: preserve StringDtype in where with listlike

c70ea91

CLN: apply ruff fixes for where changes

38a0203

sanrishi requested a review from jorisvandenbossche February 14, 2026 10:47

[pre-commit.ci] auto fixes from pre-commit.com hooks

59aabf6

for more information, see https://pre-commit.ci

jorisvandenbossche added Bug Strings String extension data type and string data Conditionals E.g. where, mask, case_when labels Feb 17, 2026

jorisvandenbossche added this to the 3.0.1 milestone Feb 17, 2026

jorisvandenbossche reviewed Feb 17, 2026

View reviewed changes

jorisvandenbossche modified the milestones: 3.0.1, 3.0.2 Feb 17, 2026

jorisvandenbossche reviewed Feb 18, 2026

View reviewed changes

sanrishi added 4 commits February 19, 2026 18:27

Merge remote-tracking branch 'origin/main' into fix-63842-stringdtype…

cda6d10

…-where-final

Merge upstream/main

0e12d62

DOC: Move changelog entry to v3.0.2

df9a0bb

BUG: Preserve pyarrow string dtype in where

35b8f73

sanrishi requested a review from jorisvandenbossche February 19, 2026 16:25

		if lib.is_list_like(value) and not lib.is_scalar(value) and len(value) == 1:
		value = value[0]

		def _where(self, mask, value) -> Self:
		return super()._where(mask, value)

Uh oh!

Conversation

sanrishi commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Implementation:

Verification:

I am CLI 🤖 : Code is changed by me Codex, I read agents.md, ensuring that every changed line is reviewed 😉.

Uh oh!

sanrishi commented Feb 9, 2026

Uh oh!

sanrishi commented Feb 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanrishi Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche commented Feb 13, 2026

Uh oh!

sanrishi commented Feb 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanrishi Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanrishi Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanrishi commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche commented Feb 18, 2026

Uh oh!

sanrishi commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sanrishi commented Feb 22, 2026

Uh oh!

sanrishi commented Mar 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sanrishi commented Feb 5, 2026 •

edited

Loading

sanrishi Feb 11, 2026 •

edited

Loading

sanrishi Feb 19, 2026 •

edited

Loading

sanrishi Feb 20, 2026 •

edited

Loading

sanrishi commented Feb 18, 2026 •

edited

Loading

sanrishi commented Feb 18, 2026 •

edited

Loading