Skip to content

TST: [ArrowStringArray] more parameterised testing - part 3 #40755

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 9, 2021
Merged

TST: [ArrowStringArray] more parameterised testing - part 3 #40755

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ccc6a70
TST: [ArrowStringArray] more parameterised testing - part 3
simonjayhawkins Apr 2, 2021
6d99d4f
Merge remote-tracking branch 'upstream/master' into nullable-string-t…
simonjayhawkins Apr 8, 2021
f68c7c8
separate, dedicated test for ArrowStringArray
simonjayhawkins Apr 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions pandas/tests/io/formats/test_to_csv.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,12 +204,17 @@ def test_to_csv_na_rep(self):
assert df.set_index("a").to_csv(na_rep="_") == expected
assert df.set_index(["a", "b"]).to_csv(na_rep="_") == expected

# GH 29975
# Make sure full na_rep shows up when a dtype is provided
csv = pd.Series(["a", pd.NA, "c"]).to_csv(na_rep="ZZZZZ")
expected = tm.convert_rows_list_to_csv_str([",0", "0,a", "1,ZZZZZ", "2,c"])
assert expected == csv
csv = pd.Series(["a", pd.NA, "c"], dtype="string").to_csv(na_rep="ZZZZZ")

def test_to_csv_na_rep_nullable_string(self, nullable_string_dtype):
# GH 29975
# Make sure full na_rep shows up when a dtype is provided
expected = tm.convert_rows_list_to_csv_str([",0", "0,a", "1,ZZZZZ", "2,c"])
csv = pd.Series(["a", pd.NA, "c"], dtype=nullable_string_dtype).to_csv(
na_rep="ZZZZZ"
)
assert expected == csv

def test_to_csv_date_format(self):
Expand Down
4 changes: 4 additions & 0 deletions pandas/tests/io/test_parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -812,11 +812,15 @@ def test_write_with_schema(self, pa):
def test_additional_extension_arrays(self, pa):
# test additional ExtensionArrays that are supported through the
# __arrow_array__ protocol

from pandas.core.arrays.string_arrow import ArrowStringDtype # noqa: F401
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

force registration of "arrow_string", not strictly necessary but won't be missed as ArrowStringDtype will be removed.


df = pd.DataFrame(
{
"a": pd.Series([1, 2, 3], dtype="Int64"),
"b": pd.Series([1, 2, 3], dtype="UInt32"),
"c": pd.Series(["a", None, "c"], dtype="string"),
"d": pd.Series(["a", None, "c"], dtype="arrow_string"),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test is @td.skip_if_no("pyarrow", min_version="0.15.0") so safe to add here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe not ImportError: pyarrow>=1.0.0 is required for PyArrow backed StringArray.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this test has a minimum of 0.15. I would make a separate, dedicated test for ArrowStringArray

}
)
if LooseVersion(pyarrow.__version__) >= LooseVersion("0.16.0"):
Expand Down
4 changes: 2 additions & 2 deletions pandas/tests/series/methods/test_convert_dtypes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,11 +212,11 @@ def test_convert_dtypes(
# Make sure original not changed
tm.assert_series_equal(series, copy)

def test_convert_string_dtype(self):
def test_convert_string_dtype(self, nullable_string_dtype):
# https://github.com/pandas-dev/pandas/issues/31731 -> converting columns
# that are already string dtype
df = pd.DataFrame(
{"A": ["a", "b", pd.NA], "B": ["ä", "ö", "ü"]}, dtype="string"
{"A": ["a", "b", pd.NA], "B": ["ä", "ö", "ü"]}, dtype=nullable_string_dtype
)
result = df.convert_dtypes()
tm.assert_frame_equal(df, result)
Expand Down
4 changes: 2 additions & 2 deletions pandas/tests/series/methods/test_replace.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,9 +254,9 @@ def test_replace2(self):
assert (ser[6:10] == -1).all()
assert (ser[20:30] == -1).all()

def test_replace_with_dictlike_and_string_dtype(self):
def test_replace_with_dictlike_and_string_dtype(self, nullable_string_dtype):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

arrow_string behaves the same as string

>>> pd.__version__
'1.3.0.dev0+1219.gccc6a707e2'
>>> 
>>> s = pd.Series(["one", "two", np.nan], dtype="string")
>>> 
>>> s
0     one
1     two
2    <NA>
dtype: string
>>> 
>>> s.replace({"one": "1", "two": "2"})
0       1
1       2
2    <NA>
dtype: object
>>>

is this correct?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say that's a bug, but that's in any case unrelated for this PR (there are a few open issues related to replace with nullable dtypes, so it might already be covered)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See eg #40732 (for other nullable dtypes)

# GH 32621
s = pd.Series(["one", "two", np.nan], dtype="string")
s = pd.Series(["one", "two", np.nan], dtype=nullable_string_dtype)
expected = pd.Series(["1", "2", np.nan])
result = s.replace({"one": "1", "two": "2"})
tm.assert_series_equal(expected, result)
Expand Down
8 changes: 5 additions & 3 deletions pandas/tests/series/test_constructors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1549,10 +1549,12 @@ def test_constructor_datetime64(self):
series = Series(dates)
assert np.issubdtype(series.dtype, np.dtype("M8[ns]"))

def test_constructor_datetimelike_scalar_to_string_dtype(self):
def test_constructor_datetimelike_scalar_to_string_dtype(
self, nullable_string_dtype
):
# https://github.com/pandas-dev/pandas/pull/33846
result = Series("M", index=[1, 2, 3], dtype="string")
expected = Series(["M", "M", "M"], index=[1, 2, 3], dtype="string")
result = Series("M", index=[1, 2, 3], dtype=nullable_string_dtype)
expected = Series(["M", "M", "M"], index=[1, 2, 3], dtype=nullable_string_dtype)
tm.assert_series_equal(result, expected)

@pytest.mark.parametrize(
Expand Down