Merge remote-tracking branch 'upstream/main' into share-datetime-parsing-format-paths

Author: MarcoGorelli

Dockerfile

@@ -8,6 +8,6 @@ RUN apt-get install -y build-essential
 RUN apt-get install -y libhdf5-dev
 
 RUN python -m pip install --upgrade pip
-RUN python -m pip install --use-deprecated=legacy-resolver \
+RUN python -m pip install \
     -r https://raw.githubusercontent.com/pandas-dev/pandas/main/requirements-dev.txt
 CMD ["/bin/bash"]

doc/source/whatsnew/v2.0.0.rst

@@ -93,6 +93,7 @@ Other enhancements
 - :func:`timedelta_range` now supports a ``unit`` keyword ("s", "ms", "us", or "ns") to specify the desired resolution of the output index (:issue:`49824`)
 - :meth:`DataFrame.to_json` now supports a ``mode`` keyword with supported inputs 'w' and 'a'. Defaulting to 'w', 'a' can be used when lines=True and orient='records' to append record oriented json lines to an existing json file. (:issue:`35849`)
 - Added ``name`` parameter to :meth:`IntervalIndex.from_breaks`, :meth:`IntervalIndex.from_arrays` and :meth:`IntervalIndex.from_tuples` (:issue:`48911`)
+- Improve exception message when using :func:`assert_frame_equal` on a :class:`DataFrame` to include the column that is compared (:issue:`50323`)
 - Improved error message for :func:`merge_asof` when join-columns were duplicated (:issue:`50102`)
 - Added :meth:`Index.infer_objects` analogous to :meth:`Series.infer_objects` (:issue:`50034`)
 - Added ``copy`` parameter to :meth:`Series.infer_objects` and :meth:`DataFrame.infer_objects`, passing ``False`` will avoid making copies for series or columns that are already non-object or where no better dtype can be inferred (:issue:`50096`)
@@ -721,6 +722,7 @@ Removal of prior version deprecations/changes
 - When providing a list of columns of length one to :meth:`DataFrame.groupby`, the keys that are returned by iterating over the resulting :class:`DataFrameGroupBy` object will now be tuples of length one (:issue:`47761`)
 - Removed deprecated methods :meth:`ExcelWriter.write_cells`, :meth:`ExcelWriter.save`, :meth:`ExcelWriter.cur_sheet`, :meth:`ExcelWriter.handles`, :meth:`ExcelWriter.path` (:issue:`45795`)
 - The :class:`ExcelWriter` attribute ``book`` can no longer be set; it is still available to be accessed and mutated (:issue:`48943`)
+- Removed unused ``*args`` and ``**kwargs`` in :class:`Rolling`, :class:`Expanding`, and :class:`ExponentialMovingWindow` ops (:issue:`47851`)
 -
 
 .. ---------------------------------------------------------------------------
@@ -740,6 +742,7 @@ Performance improvements
 - Performance improvement in :meth:`Index.union` and :meth:`MultiIndex.union` when index contains duplicates (:issue:`48900`)
 - Performance improvement in :meth:`Series.rank` for pyarrow-backed dtypes (:issue:`50264`)
 - Performance improvement in :meth:`Series.fillna` for extension array dtypes (:issue:`49722`, :issue:`50078`)
+- Performance improvement in :meth:`Index.join`, :meth:`Index.intersection` and :meth:`Index.union` for masked dtypes when :class:`Index` is monotonic (:issue:`50310`)
 - Performance improvement for :meth:`Series.value_counts` with nullable dtype (:issue:`48338`)
 - Performance improvement for :class:`Series` constructor passing integer numpy array with nullable dtype (:issue:`48338`)
 - Performance improvement for :class:`DatetimeIndex` constructor passing a list (:issue:`48609`)
@@ -790,8 +793,7 @@ Datetimelike
 - Bug in :func:`to_datetime` was raising ``ValueError`` when parsing :class:`Timestamp`, ``datetime.datetime``, ``datetime.date``, or ``np.datetime64`` objects when non-ISO8601 ``format`` was passed (:issue:`49298`, :issue:`50036`)
 - Bug in :func:`to_datetime` was raising ``ValueError`` when parsing empty string and non-ISO8601 format was passed. Now, empty strings will be parsed as :class:`NaT`, for compatibility with how is done for ISO8601 formats (:issue:`50251`)
 - Bug in :class:`Timestamp` was showing ``UserWarning``, which was not actionable by users, when parsing non-ISO8601 delimited date strings (:issue:`50232`)
-- Bug in :class:`Timestamp` was showing ``UserWarning`` which was not actionable by users (:issue:`50232`)
-- Bug in :func:`to_datetime` was raising ``ValueError`` when parsing empty string and non-ISO8601 format was passed. Now, empty strings will be parsed as :class:`NaT`, for compatibility with how is done for ISO8601 formats (:issue:`50251`)
+- Bug in :func:`to_datetime` was showing misleading ``ValueError`` when parsing dates with format containing ISO week directive and ISO weekday directive (:issue:`50308`)
 -
 
 Timedelta
@@ -810,12 +812,14 @@ Timezones
 Numeric
 ^^^^^^^
 - Bug in :meth:`DataFrame.add` cannot apply ufunc when inputs contain mixed DataFrame type and Series type (:issue:`39853`)
+- Bug in arithmetic operations on :class:`Series` not propagating mask when combining masked dtypes and numpy dtypes (:issue:`45810`, :issue:`42630`)
 - Bug in DataFrame reduction methods (e.g. :meth:`DataFrame.sum`) with object dtype, ``axis=1`` and ``numeric_only=False`` would not be coerced to float (:issue:`49551`)
 - Bug in :meth:`DataFrame.sem` and :meth:`Series.sem` where an erroneous ``TypeError`` would always raise when using data backed by an :class:`ArrowDtype` (:issue:`49759`)
 
 Conversion
 ^^^^^^^^^^
 - Bug in constructing :class:`Series` with ``int64`` dtype from a string list raising instead of casting (:issue:`44923`)
+- Bug in constructing :class:`Series` with masked dtype and boolean values with ``NA`` raising (:issue:`42137`)
 - Bug in :meth:`DataFrame.eval` incorrectly raising an ``AttributeError`` when there are negative values in function call (:issue:`46471`)
 - Bug in :meth:`Series.convert_dtypes` not converting dtype to nullable dtype when :class:`Series` contains ``NA`` and has dtype ``object`` (:issue:`48791`)
 - Bug where any :class:`ExtensionDtype` subclass with ``kind="M"`` would be interpreted as a timezone type (:issue:`34986`)
@@ -880,15 +884,18 @@ I/O
 - Bug when a pickling a subset PyArrow-backed data that would serialize the entire data instead of the subset (:issue:`42600`)
 - Bug in :func:`read_sql_query` ignoring ``dtype`` argument when ``chunksize`` is specified and result is empty (:issue:`50245`)
 - Bug in :func:`read_csv` for a single-line csv with fewer columns than ``names`` raised :class:`.errors.ParserError` with ``engine="c"`` (:issue:`47566`)
+- Bug in :func:`read_json` raising with ``orient="table"`` and ``NA`` value (:issue:`40255`)
 - Bug in displaying ``string`` dtypes not showing storage option (:issue:`50099`)
-- Bug in :func:`DataFrame.to_string` with ``header=False`` that printed the index name on the same line as the first row of the data (:issue:`49230`)
+- Bug in :meth:`DataFrame.to_string` with ``header=False`` that printed the index name on the same line as the first row of the data (:issue:`49230`)
+- Bug in :meth:`DataFrame.to_string` ignoring float formatter for extension arrays (:issue:`39336`)
 - Fixed memory leak which stemmed from the initialization of the internal JSON module (:issue:`49222`)
 - Fixed issue where :func:`json_normalize` would incorrectly remove leading characters from column names that matched the ``sep`` argument (:issue:`49861`)
 - Bug in :meth:`DataFrame.to_json` where it would segfault when failing to encode a string (:issue:`50307`)
 
 Period
 ^^^^^^
 - Bug in :meth:`Period.strftime` and :meth:`PeriodIndex.strftime`, raising ``UnicodeDecodeError`` when a locale-specific directive was passed (:issue:`46319`)
+- Bug in adding a :class:`Period` object to an array of :class:`DateOffset` objects incorrectly raising ``TypeError`` (:issue:`50162`)
 -
 
 Plotting
@@ -935,7 +942,7 @@ ExtensionArray
 - Bug in :meth:`Series.mean` overflowing unnecessarily with nullable integers (:issue:`48378`)
 - Bug in :meth:`Series.tolist` for nullable dtypes returning numpy scalars instead of python scalars (:issue:`49890`)
 - Bug when concatenating an empty DataFrame with an ExtensionDtype to another DataFrame with the same ExtensionDtype, the resulting dtype turned into object (:issue:`48510`)
--
+- Bug in :meth:`array.PandasArray.to_numpy` raising with ``NA`` value when ``na_value`` is specified (:issue:`40638`)
 
 Styler
 ^^^^^^

pandas/_libs/tslibs/period.pyx

@@ -1741,6 +1741,11 @@ cdef class _Period(PeriodMixin):
             raise TypeError(f"unsupported operand type(s) for +: '{sname}' "
                             f"and '{oname}'")
 
+        elif util.is_array(other):
+            if other.dtype == object:
+                # GH#50162
+                return np.array([self + x for x in other], dtype=object)
+
         return NotImplemented
 
     def __radd__(self, other):
@@ -1767,11 +1772,22 @@ cdef class _Period(PeriodMixin):
         elif other is NaT:
             return NaT
 
+        elif util.is_array(other):
+            if other.dtype == object:
+                # GH#50162
+                return np.array([self - x for x in other], dtype=object)
+
         return NotImplemented
 
     def __rsub__(self, other):
         if other is NaT:
             return NaT
+
+        elif util.is_array(other):
+            if other.dtype == object:
+                # GH#50162
+                return np.array([x - self for x in other], dtype=object)
+
         return NotImplemented
 
     def asfreq(self, freq, how="E") -> "Period":

pandas/_libs/tslibs/strptime.pyx

@@ -127,6 +127,38 @@ def array_strptime(
                 raise ValueError("Cannot use '%W' or '%U' without day and year")
         elif "%Z" in fmt and "%z" in fmt:
             raise ValueError("Cannot parse both %Z and %z")
+        elif "%j" in fmt and "%G" in fmt:
+            raise ValueError("Day of the year directive '%j' is not "
+                             "compatible with ISO year directive '%G'. "
+                             "Use '%Y' instead.")
+        elif "%G" in fmt and (
+            "%V" not in fmt
+            or not (
+                "%A" in fmt
+                or "%a" in fmt
+                or "%w" in fmt
+                or "%u" in fmt
+            )
+        ):
+            raise ValueError("ISO year directive '%G' must be used with "
+                             "the ISO week directive '%V' and a weekday "
+                             "directive '%A', '%a', '%w', or '%u'.")
+        elif "%V" in fmt and "%Y" in fmt:
+            raise ValueError("ISO week directive '%V' is incompatible with "
+                             "the year directive '%Y'. Use the ISO year "
+                             "'%G' instead.")
+        elif "%V" in fmt and (
+            "%G" not in fmt
+            or not (
+                "%A" in fmt
+                or "%a" in fmt
+                or "%w" in fmt
+                or "%u" in fmt
+            )
+        ):
+            raise ValueError("ISO week directive '%V' must be used with "
+                             "the ISO year directive '%G' and a weekday "
+                             "directive '%A', '%a', '%w', or '%u'.")
 
     global _TimeRE_cache, _regex_cache
     with _cache_lock:
@@ -365,26 +397,6 @@ def array_strptime(
                 weekday = int(found_dict["u"])
                 weekday -= 1
 
-        # don't assume default values for ISO week/year
-        if iso_year != -1:
-            if iso_week == -1 or weekday == -1:
-                raise ValueError("ISO year directive '%G' must be used with "
-                                 "the ISO week directive '%V' and a weekday "
-                                 "directive '%A', '%a', '%w', or '%u'.")
-            if julian != -1:
-                raise ValueError("Day of the year directive '%j' is not "
-                                 "compatible with ISO year directive '%G'. "
-                                 "Use '%Y' instead.")
-        elif year != -1 and week_of_year == -1 and iso_week != -1:
-            if weekday == -1:
-                raise ValueError("ISO week directive '%V' must be used with "
-                                 "the ISO year directive '%G' and a weekday "
-                                 "directive '%A', '%a', '%w', or '%u'.")
-            else:
-                raise ValueError("ISO week directive '%V' is incompatible with "
-                                 "the year directive '%Y'. Use the ISO year "
-                                 "'%G' instead.")
-
         # If we know the wk of the year and what day of that wk, we can figure
         # out the Julian day of the year.
         if julian == -1 and weekday != -1:

pandas/_testing/asserters.py

@@ -680,6 +680,7 @@ def assert_extension_array_equal(
     check_exact: bool = False,
     rtol: float = 1.0e-5,
     atol: float = 1.0e-8,
+    obj: str = "ExtensionArray",
 ) -> None:
     """
     Check that left and right ExtensionArrays are equal.
@@ -702,6 +703,11 @@ def assert_extension_array_equal(
         Absolute tolerance. Only used when check_exact is False.
 
         .. versionadded:: 1.1.0
+    obj : str, default 'ExtensionArray'
+        Specify object name being compared, internally used to show appropriate
+        assertion message.
+
+        .. versionadded:: 2.0.0
 
     Notes
     -----
@@ -719,7 +725,7 @@ def assert_extension_array_equal(
     assert isinstance(left, ExtensionArray), "left is not an ExtensionArray"
     assert isinstance(right, ExtensionArray), "right is not an ExtensionArray"
     if check_dtype:
-        assert_attr_equal("dtype", left, right, obj="ExtensionArray")
+        assert_attr_equal("dtype", left, right, obj=f"Attributes of {obj}")
 
     if (
         isinstance(left, DatetimeLikeArrayMixin)
@@ -729,21 +735,24 @@ def assert_extension_array_equal(
         # Avoid slow object-dtype comparisons
         # np.asarray for case where we have a np.MaskedArray
         assert_numpy_array_equal(
-            np.asarray(left.asi8), np.asarray(right.asi8), index_values=index_values
+            np.asarray(left.asi8),
+            np.asarray(right.asi8),
+            index_values=index_values,
+            obj=obj,
         )
         return
 
     left_na = np.asarray(left.isna())
     right_na = np.asarray(right.isna())
     assert_numpy_array_equal(
-        left_na, right_na, obj="ExtensionArray NA mask", index_values=index_values
+        left_na, right_na, obj=f"{obj} NA mask", index_values=index_values
     )
 
     left_valid = left[~left_na].to_numpy(dtype=object)
     right_valid = right[~right_na].to_numpy(dtype=object)
     if check_exact:
         assert_numpy_array_equal(
-            left_valid, right_valid, obj="ExtensionArray", index_values=index_values
+            left_valid, right_valid, obj=obj, index_values=index_values
         )
     else:
         _testing.assert_almost_equal(
@@ -752,7 +761,7 @@ def assert_extension_array_equal(
             check_dtype=bool(check_dtype),
             rtol=rtol,
             atol=atol,
-            obj="ExtensionArray",
+            obj=obj,
             index_values=index_values,
         )
 
@@ -909,6 +918,7 @@ def assert_series_equal(
                 right_values,
                 check_dtype=check_dtype,
                 index_values=np.asarray(left.index),
+                obj=str(obj),
             )
         else:
             assert_numpy_array_equal(
@@ -955,6 +965,7 @@ def assert_series_equal(
             atol=atol,
             check_dtype=check_dtype,
             index_values=np.asarray(left.index),
+            obj=str(obj),
         )
     elif is_extension_array_dtype_and_needs_i8_conversion(
         left.dtype, right.dtype
@@ -964,6 +975,7 @@ def assert_series_equal(
             right._values,
             check_dtype=check_dtype,
             index_values=np.asarray(left.index),
+            obj=str(obj),
         )
     elif needs_i8_conversion(left.dtype) and needs_i8_conversion(right.dtype):
         # DatetimeArray or TimedeltaArray
@@ -972,6 +984,7 @@ def assert_series_equal(
             right._values,
             check_dtype=check_dtype,
             index_values=np.asarray(left.index),
+            obj=str(obj),
         )
     else:
         _testing.assert_almost_equal(

pandas/compat/numpy/function.py

@@ -335,51 +335,6 @@ def validate_take_with_convert(convert: ndarray | bool | None, args, kwargs) ->
 )
 
 
-def validate_window_func(name, args, kwargs) -> None:
-    numpy_args = ("axis", "dtype", "out")
-    msg = (
-        f"numpy operations are not valid with window objects. "
-        f"Use .{name}() directly instead "
-    )
-
-    if len(args) > 0:
-        raise UnsupportedFunctionCall(msg)
-
-    for arg in numpy_args:
-        if arg in kwargs:
-            raise UnsupportedFunctionCall(msg)
-
-
-def validate_rolling_func(name, args, kwargs) -> None:
-    numpy_args = ("axis", "dtype", "out")
-    msg = (
-        f"numpy operations are not valid with window objects. "
-        f"Use .rolling(...).{name}() instead "
-    )
-
-    if len(args) > 0:
-        raise UnsupportedFunctionCall(msg)
-
-    for arg in numpy_args:
-        if arg in kwargs:
-            raise UnsupportedFunctionCall(msg)
-
-
-def validate_expanding_func(name, args, kwargs) -> None:
-    numpy_args = ("axis", "dtype", "out")
-    msg = (
-        f"numpy operations are not valid with window objects. "
-        f"Use .expanding(...).{name}() instead "
-    )
-
-    if len(args) > 0:
-        raise UnsupportedFunctionCall(msg)
-
-    for arg in numpy_args:
-        if arg in kwargs:
-            raise UnsupportedFunctionCall(msg)
-
-
 def validate_groupby_func(name, args, kwargs, allowed=None) -> None:
     """
     'args' and 'kwargs' should be empty, except for allowed kwargs because all

pandas/core/arrays/masked.py

@@ -609,9 +609,13 @@ def _propagate_mask(
             if other is libmissing.NA:
                 # GH#45421 don't alter inplace
                 mask = mask | True
+            elif is_list_like(other) and len(other) == len(mask):
+                mask = mask | isna(other)
         else:
             mask = self._mask | mask
-        return mask
+        # Incompatible return value type (got "Optional[ndarray[Any, dtype[bool_]]]",
+        # expected "ndarray[Any, dtype[bool_]]")
+        return mask  # type: ignore[return-value]
 
     def _arith_method(self, other, op):
         op_name = op.__name__

pandas/core/arrays/numeric.py

@@ -172,9 +172,7 @@ def _coerce_to_data_and_mask(values, mask, dtype, copy, dtype_cls, default_dtype
     inferred_type = None
     if is_object_dtype(values.dtype) or is_string_dtype(values.dtype):
         inferred_type = lib.infer_dtype(values, skipna=True)
-        if inferred_type == "empty":
-            pass
-        elif inferred_type == "boolean":
+        if inferred_type == "boolean" and dtype is None:
             name = dtype_cls.__name__.strip("_")
             raise TypeError(f"{values.dtype} cannot be converted to {name}")
 

pandas/core/arrays/numpy_.py

@@ -389,13 +389,17 @@ def to_numpy(
         copy: bool = False,
         na_value: object = lib.no_default,
     ) -> np.ndarray:
-        result = np.asarray(self._ndarray, dtype=dtype)
+        mask = self.isna()
+        if na_value is not lib.no_default and mask.any():
+            result = self._ndarray.copy()
+            result[mask] = na_value
+        else:
+            result = self._ndarray
 
-        if (copy or na_value is not lib.no_default) and result is self._ndarray:
-            result = result.copy()
+        result = np.asarray(result, dtype=dtype)
 
-        if na_value is not lib.no_default:
-            result[self.isna()] = na_value
+        if copy and result is self._ndarray:
+            result = result.copy()
 
         return result
 

pandas/core/arrays/string_.py

@@ -266,7 +266,7 @@ class StringArray(BaseStringArray, PandasArray):
 
     See Also
     --------
-    array
+    :func:`pandas.array`
         The recommended function for creating a StringArray.
     Series.str
         The string methods are available on Series backed by