Inconsistency in array_to_datetime "utc_convert"

[jbrockmendel]

If tslib.array_to_datetime sees a tz-aware datetime object and utc_convert isn't True, it raises. But later a tz-aware datetime may be parsed from dateutil and it is ignored:

            elif PyDateTime_Check(val):
                seen_datetime = 1
                if val.tzinfo is not None:
                    if utc_convert:
                        _ts = convert_datetime_to_tsobject(val, None)
                        iresult[i] = _ts.value
                        try:
                            check_dts_bounds(&_ts.dts)
                        except ValueError:
                            if is_coerce:
                                iresult[i] = NPY_NAT
                                continue
                            raise
                    else:
                        raise ValueError('Tz-aware datetime.datetime cannot '
                                         'be converted to datetime64 unless '
                                         'utc=True')
[...]
                    try:
                        py_dt = parse_datetime_string(val, dayfirst=dayfirst,
                                                      yearfirst=yearfirst)
                    except Exception:
                        if _parse_today_now(val, &iresult[i]):
                            continue
                        if is_coerce:
                            iresult[i] = NPY_NAT
                            continue
                        raise TypeError("invalid string coercion to datetime")

                    try:
                        _ts = convert_datetime_to_tsobject(py_dt, None)
                        iresult[i] = _ts.value
                    except ValueError:
                        if is_coerce:
                            iresult[i] = NPY_NAT
                            continue
                        raise

There are several ways (some of which are hit in the tests) by which parse_datetime_string could return a tzaware datetime. Should these raise iff utc_convert is not True? Or behave like the string_to_dts parsing above and allow FixedOffset through? What about if it is aware and already in UTC?

[jreback]

is there a user facing effect to this?

[jbrockmendel]

tstr = '2018-02-10 10:16:06 PST'
ts = pd.Timestamp(tstr)   # comes back with tzlocal()
>>> pd.to_datetime([ts])
DatetimeIndex(['2018-02-10 10:16:06-08:00'], dtype='datetime64[ns, tzlocal()]', freq=None)
>>> pd.to_datetime([tstr])
DatetimeIndex(['2018-02-10 18:16:06'], dtype='datetime64[ns]', freq=None)

[jreback]

ok, pls check outstanding issues, I think have seen this before.

[jbrockmendel]

@mroeschke did the recent arraytodatetime change handle this?

[mroeschke]

In [7]: pd.to_datetime([ts])
Out[7]: DatetimeIndex(['2018-02-10 10:16:06-08:00'], dtype='datetime64[ns, tzlocal()]', freq=None)

In [8]: pd.to_datetime([tstr])
Out[8]: DatetimeIndex(['2018-02-10 10:16:06-08:00'], dtype='datetime64[ns, pytz.FixedOffset(-480)]', freq=None)

So a little better. Is the ultimate goal is for PST to parse to 'US/Pacific'? (xref #18702 (comment))?

[jbrockmendel]

Is the ultimate goal is for PST to parse to 'US/Pacific'?

Yes, but I think #18702 is orthogonal. Closing.