CI: Fastparquet release broke ci

[phofl]

It looks like the fastparquet release broke our ci

Successful build:
https://github.com/pandas-dev/pandas/runs/2522988276

Erroneous build:
https://github.com/pandas-dev/pandas/runs/2524444950

This is the full version diff:

        packages                      version_error_build                version_successful_build
38         boto3                      1.17.68                      1.17.67
59       cramjam                        2.3.0                          NaN
79   fastparquet                        0.6.0                        0.5.0
228       pandas  1.3.0.dev0+1562.g6ea277f241  1.3.0.dev0+1561.gebf3b98596
316   sqlalchemy                       1.4.14                       1.4.13

[TomAugspurger]

cc @martindurant.

Is it just the doc build that failed? Can you point to a specific example that's causing the failure?

[phofl]

It looks like

df = pd.DataFrame(
    {
        "a": list("abc"),
        "b": list(range(1, 4)),
        "c": np.arange(3, 6).astype("u1"),
        "d": np.arange(4.0, 7.0, dtype="float64"),
        "e": [True, False, True],
        "f": pd.date_range("20130101", periods=3),
        "g": pd.date_range("20130101", periods=3, tz="US/Eastern"),
        "h": pd.Categorical(list("abc")),
        "i": pd.Categorical(list("abc"), ordered=True),
    }
)

df.to_parquet("example_fp.parquet", engine="fastparquet")
result = pd.read_parquet("example_fp.parquet", engine="fastparquet")

this is causing the failure, but can not reproduce locally and the ipython errors are ugly to read.

Additionally a few tests are failing in our windows ci with np18

FAILED pandas/tests/io/test_fsspec.py::test_s3_parquet - TypeError: Cannot co...
FAILED pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp - PermissionE...
FAILED pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_basic - ...
FAILED pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_timezone_aware_index[timezone_aware_date_list1]
FAILED pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_timezone_aware_index[timezone_aware_date_list2]
FAILED pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_timezone_aware_index[timezone_aware_date_list3]
FAILED pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_timezone_aware_index[timezone_aware_date_list4]
FAILED pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_timezone_aware_index[timezone_aware_date_list5]
FAILED pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_timezone_aware_index[timezone_aware_date_list6]

here for example https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=59379&view=logs&j=404760ec-14d3-5d48-e580-13034792878f&t=f81e4cc8-d61a-5fb8-36be-36768e5c561a&l=59

[martindurant]

The error seems to be in fastparquert.df.empty, where the main (only?) change was dask/fastparquet#571 (cc @jbrockmendel ). fastparquet is passing against released pandas (1.2.4).

[jbrockmendel]

The .empty thing i'll address. some of the other failures look pytz-related?

[martindurant]

Can you post the traceback of a remaining error, please - I find the log hard to parse.

[phofl]

Traceback:

    def test_timezone_aware_index(self, fp, timezone_aware_date_list):
        idx = 5 * [timezone_aware_date_list]
    
        df = pd.DataFrame(index=idx, data={"index_as_col": idx})
    
        expected = df.copy()
        expected.index.name = "index"
>       check_round_trip(df, fp, expected=expected)

pandas/tests/io/test_parquet.py:1065: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/tests/io/test_parquet.py:219: in check_round_trip
    compare(repeat)
pandas/tests/io/test_parquet.py:207: in compare
    actual = read_parquet(path, **read_kwargs)
pandas/io/parquet.py:497: in read_parquet
    return impl.read(
pandas/io/parquet.py:341: in read
    parquet_file = self.api.ParquetFile(path, **parquet_kwargs)
../../anaconda3/envs/omi_reports/pandas-dev-test/lib/python3.8/site-packages/fastparquet/api.py:97: in __init__
    self._parse_header(fn, verify)
../../anaconda3/envs/omi_reports/pandas-dev-test/lib/python3.8/site-packages/fastparquet/api.py:142: in _parse_header
    self._set_attrs()
../../anaconda3/envs/omi_reports/pandas-dev-test/lib/python3.8/site-packages/fastparquet/api.py:155: in _set_attrs
    self._dtypes()
../../anaconda3/envs/omi_reports/pandas-dev-test/lib/python3.8/site-packages/fastparquet/api.py:530: in _dtypes
    z = pytz.timezone(f"Etc/GMT{z}")

>               raise UnknownTimeZoneError(zone)
E               pytz.exceptions.UnknownTimeZoneError: 'Etc/GMTUTC'

Is this helpful?

Edit:

Here are the fixtures:

@pytest.fixture(
    params=[
        datetime.datetime.now(datetime.timezone.utc),
        datetime.datetime.now(datetime.timezone.min),
        datetime.datetime.now(datetime.timezone.max),
        datetime.datetime.strptime("2019-01-04T16:41:24+0200", "%Y-%m-%dT%H:%M:%S%z"),
        datetime.datetime.strptime("2019-01-04T16:41:24+0215", "%Y-%m-%dT%H:%M:%S%z"),
        datetime.datetime.strptime("2019-01-04T16:41:24-0200", "%Y-%m-%dT%H:%M:%S%z"),
        datetime.datetime.strptime("2019-01-04T16:41:24-0215", "%Y-%m-%dT%H:%M:%S%z"),
    ]
)
def timezone_aware_date_list(request):
    return request.param

It looks like the test fails for 6/7 (utc passes)

[martindurant]

That's interesting, yes. I wonder, what string would you give to pytz.timezone() to get the right timezone for these? Apparently I found a formalism that works for some subset (fractional hours, in particular, ought to be hard!).

[martindurant]

i.e., .tz_localize('-23:59') doesn't work.

[phofl]

Reopening until Pin is reverted