Skip to content

New Fsspec breaks local file serialization #635

@aulemahal

Description

@aulemahal

Here's a quick checklist in what to include:

  • Include a detailed description of the bug or suggestion

  • Output of intake_esm.show_versions()

  • Minimal, self-contained copy-pastable example that generates the issue if possible. Please be concise with code posted. See guidelines below on how to provide a good bug report:

Description

Serializing a catalog to a local filesystem is broken with fsspec 2023.10.0.

What I Did

import intake
url = "https://storage.googleapis.com/cmip6/pangeo-cmip6.json"
cat = intake.open_esm_datastore(url)

cat.serialize('mycopy', catalog_type='file')

and I got:

FileNotFoundError: [Errno 2] No such file or directory: "/home/meself/example/path/('file', 'local'):///home/meself/example/path/mycopy.csv"

The culprit is here :

csv_file_name = f'{mapper.fs.protocol}://{mapper.root}/{name}.csv'

In the recent fsspec release, Local filesystem now accepts both "file" and "local" as protocols. This, mapper.fs.protocol is a tuple, and not a string anymore. I'm not sure if this was always the case, but I see in the git blame that typing annotation showing this were added 5 months ago (see https://github.com/fsspec/filesystem_spec/blame/e20f626b87b5bb87d223495a56aefd768272a7ca/fsspec/spec.py#L107)

I see a mapper.fs.unstrip_protocol method that does what our f-string tries too. I can push a PR with this fix.

Version information: output of intake_esm.show_versions()

INSTALLED VERSIONS

cftime: 1.6.2
dask: 2023.3.0
fastprogress: 1.0.3
fsspec: 2023.10.0
gcsfs: 2023.10.0
intake: 0.7.0
intake_esm: 2022.9.18.post7
netCDF4: 1.6.0
pandas: 2.1.1
requests: 2.31.0
s3fs: 2023.10.0
xarray: 2023.10.1
zarr: 2.16.1


</details>

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions