BUG: Converting pd.Timestamp.min to equivalent pydatetime object and back raises OutOfBoundsDatetime

[milotoor]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

min_timestamp = pd.Timestamp.min
min_pydatetime = min_timestamp.to_pydatetime()
min_datetime = pd.to_datetime(min_pydatetime)
print(min_datetime)  # just to provide script output

Issue Description

Going from pd.Timestamp.min to the equivalent datetime.datetime instance and back works with pandas v1.2.5 but fails in v1.3.0 (and all subsequent versions) due to an out of bounds error. This behavior is unchanged in the current main branch:

See traceback for version 1.5.0.dev0+396.geaefc5c416

sys:1: UserWarning: Discarding nonzero nanoseconds in conversion
Traceback (most recent call last):
  File "/Users/milotoor/.pyenv/versions/pandas_1_3_0/lib/python3.9/site-packages/pandas/core/arrays/datetimes.py", line 2187, in objects_to_datetime64ns
    values, tz_parsed = conversion.datetime_to_datetime64(data.ravel("K"))
  File "pandas/_libs/tslibs/conversion.pyx", line 357, in pandas._libs.tslibs.conversion.datetime_to_datetime64
  File "pandas/_libs/tslibs/np_datetime.pyx", line 120, in pandas._libs.tslibs.np_datetime.check_dts_bounds
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1677-09-21 00:12:43

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/milotoor/tslib_bug_demo.py", line 5, in <module>
    min_datetime = pd.to_datetime(min_pydatetime)
  File "/Users/milotoor/.pyenv/versions/pandas_1_3_0/lib/python3.9/site-packages/pandas/core/tools/datetimes.py", line 914, in to_datetime
    result = convert_listlike(np.array([arg]), format)[0]
  File "/Users/milotoor/.pyenv/versions/pandas_1_3_0/lib/python3.9/site-packages/pandas/core/tools/datetimes.py", line 401, in _convert_listlike_datetimes
    result, tz_parsed = objects_to_datetime64ns(
  File "/Users/milotoor/.pyenv/versions/pandas_1_3_0/lib/python3.9/site-packages/pandas/core/arrays/datetimes.py", line 2193, in objects_to_datetime64ns
    raise err
  File "/Users/milotoor/.pyenv/versions/pandas_1_3_0/lib/python3.9/site-packages/pandas/core/arrays/datetimes.py", line 2175, in objects_to_datetime64ns
    result, tz_parsed = tslib.array_to_datetime(
  File "pandas/_libs/tslib.pyx", line 379, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 606, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 602, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 474, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslibs/np_datetime.pyx", line 120, in pandas._libs.tslibs.np_datetime.check_dts_bounds
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1677-09-21 00:12:43

I'm not sure what to make of the "Discarding nonzero nanoseconds in conversion" warning above the traceback.

I believe this regression was introduced in #39245, which modifies the _NS_MIN_DTS const value:

➜  pandas git:(main) git df v1.2.5 v1.3.0 -- pandas/_libs/tslibs/src

diff --git a/pandas/_libs/tslibs/src/datetime/np_datetime.c b/pandas/_libs/tslibs/src/datetime/np_datetime.c
index 8eb995dee6..9ad2ead5f9 100644
--- a/pandas/_libs/tslibs/src/datetime/np_datetime.c
+++ b/pandas/_libs/tslibs/src/datetime/np_datetime.c
@@ -32,7 +32,7 @@ This file is derived from NumPy 1.7. See NUMPY_LICENSE.txt
 #endif  // PyInt_AsLong

 const npy_datetimestruct _NS_MIN_DTS = {
-    1677, 9, 21, 0, 12, 43, 145225, 0, 0};
+    1677, 9, 21, 0, 12, 43, 145224, 193000, 0};
 const npy_datetimestruct _NS_MAX_DTS = {
     2262, 4, 11, 23, 47, 16, 854775, 807000, 0};

Expected Behavior

The script above should print out the datetime as a string. Pandas v1.2.5 does the following:

➜  ~ cat tslib_bug_demo.py
import pandas as pd

min_timestamp = pd.Timestamp.min
min_pydatetime = min_timestamp.to_pydatetime()
min_datetime = pd.to_datetime(min_pydatetime)
print(min_datetime)

➜  ~ pyenv activate pandas_1_2_5
(pandas_1_2_5) ➜  ~ python --version
Python 3.9.10
(pandas_1_2_5) ➜  ~ pip list | grep pandas
pandas          1.2.5
(pandas_1_2_5) ➜  ~ python tslib_bug_demo.py
1677-09-21 00:12:43.145225

Installed Versions

Versions for v1.2.5

commit           : 7c48ff4409c622c582c56a5702373f726de08e96
python           : 3.9.10.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 21.3.0
Version          : Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000
machine          : arm64
processor        : arm
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.2.5
numpy            : 1.22.2
pytz             : 2021.3
dateutil         : 2.8.2
pip              : 22.0.3
setuptools       : 58.1.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

Versions for v1.3.0

commit           : f00ed8f47020034e752baf0250483053340971b0
python           : 3.9.10.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 21.3.0
Version          : Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000
machine          : arm64
processor        : arm
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.3.0
numpy            : 1.22.2
pytz             : 2021.3
dateutil         : 2.8.2
pip              : 21.2.4
setuptools       : 58.1.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

Versions for current main branch

commit           : eaefc5c4160b73521f36dd04a43ea65423772b11
python           : 3.9.10.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 21.3.0
Version          : Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000
machine          : arm64
processor        : arm
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.5.0.dev0+396.geaefc5c416
numpy            : 1.22.2
pytz             : 2021.3
dateutil         : 2.8.2
pip              : 21.2.4
setuptools       : 58.1.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
fsspec           : None
gcsfs            : None
markupsafe       : None
matplotlib       : None
numba            : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
zstandard        : None

[jbrockmendel]

The pd.Timestamp.min.to_pydatetime() call issues a warning that nanoseconds are being dropped. The pydatetime object is not equivalent. This behavior is expected.

[milotoor]

Ok, thank you @jbrockmendel, I will close the issue.

However, I do find this behavior somewhat confusing. As I understand it, pd.Timestamp has a greater precision than the datetime.datetime class: the former has nanosecond precision while the latter only has microsecond precision. The tradeoff is that the range of possible dates is greater for datetime.datetime (year 0-9999) than it is for pd.Timestamp (year 1677-2262). That's all well and good, my particular use case requires neither nanosecond precision nor a millennia-long domain so this isn't an issue.

What I find peculiar is that it's so easy to trigger the OutOfBoundsDatetime error. It's a somewhat unfriendly API decision to have relatively benign, simple code throw an exception. Another instance of the problem:

import pandas as pd
pd.Timestamp.min.replace(nanosecond=0)

This fails because rounding down the nanoseconds takes us outside the pd.Timestamp range. What I find even stranger is that passing values greater than the minimum timestamp's number of nanoseconds also throw the error:

assert pd.Timestamp.min.nanosecond == 193
pd.Timestamp.min.replace(nanosecond=pd.Timestamp.min.nanosecond + 1)  # OutOfBoundsDatetime error!

Maybe this particular observation is a separate issue having to do with the implementation of replace, I don't know. But regardless, it seems to me like the API would be simpler and better aligned with user expectations if the minimum (and maximum) timestamp fell on a moment in time which could also be represented by datetime.datetime objects. I guess 807 nanoseconds on the bottom of the range and 193 nanoseconds on the upper end of the range (because pd.Timestamp.max.nanosecond == 807) would be lost. That seems like a tolerable sacrifice to me. this is how it worked prior to the pull request linked above, after all.

Is there a rationale for having the range set up this way?