API: Series([pd.NaT, None]) vs Index([pd.NaT, None])

[jbrockmendel]

pd.Series([pd.NaT, None])  # <- infers to dt64[ns]
pd.Index([pd.Nat, None])  # <- infers to object

The Series inference is done in maybe_infer_to_datetimelike, where infer_datetimelike_array returns "NaT". The Index inference is done in lib.maybe_convert_objects, which does not treat None as NaT.

Ideally this would match.

[jorisvandenbossche]

As comparison, for other data types Index([..]) will regard None generally as missing value and not take it into account for dtype inference (not requiring object dtype). For example:

In [20]: pd.Index([1, None])
Out[20]: Float64Index([1.0, nan], dtype='float64')

In [21]: pd.Index([np.nan, None])
Out[21]: Float64Index([nan, nan], dtype='float64')

So I think also those should be consistent with the above example? Also when passing an actual Timestamp scalar and not NaT, the None gets converted:

In [22]: pd.Index([pd.Timestamp("2012-01-01"), None])
Out[22]: DatetimeIndex(['2012-01-01', 'NaT'], dtype='datetime64[ns]', freq=None)

That all seems to hint towards choosing the Series behaviour?

I suppose the only reason why pd.NaT in combination with None would behaviour differently than a number, NaN or Timestamp in combination with None is that it is ambiguous which data type it should result in? (datetime64 vs timedelta64) But we have to make that choice for NaT only anyway:

In [23]: pd.Index([pd.NaT, pd.NaT])
Out[23]: DatetimeIndex(['NaT', 'NaT'], dtype='datetime64[ns]', freq=None)

[jorisvandenbossche]

Maybe to summarize my comment above: I think we should choose a consistent behaviour for how None interferes in dtype inference, regardless of which other values are present.

One option could be that None being present always means we end up with object dtype, but that would be a much bigger breaking change (and probably also not desirable on the short term). In most cases the logic seems to be "ignore None for dtype inference (except for indicating missing values are present), and convert it to the missing value appropriate for the resulting dtype inferred based on the other values".
If we follow that rule, then pd.Index([pd.Nat, None]) should result in datetime64[ns]

[jbrockmendel]

I generally agree with everything Joris said.

I'd also have some sympathy to the argument: If a user explicitly chose to use two different NA-likes, they probably don't expect to get just one back. (a little bit like #49421).

Either way, I care more that the behaviors be consistent than about which one we choose.

[jbrockmendel]

@mroeschke thoughts?

[mroeschke]

I guess I would be okay with the Series behavior of coercing None since the docstrings for Index & Series suggests that dtype=None essentially means "infer" the dtype

If a user explicitly chose to use two different NA-likes, they probably don't expect to get just one back

True, but given the coerce-y nature of pandas, I think we should definitely encourage users to astype(object)/dtype=object more often if they want two different NA-likes.

[jbrockmendel]

Good news! Both the Series and Index behaviors are mostly lib.maybe_convert_objects. Changing maybe_convert_objects to have the current Series behavior doesn't break anything!