Skip to content

[WIP] Add handling for chunk buffer on timeouts#251

Draft
bnlawrence with Copilot wants to merge 1 commit into
mainfrom
copilot/chunk-buffer-timeouts-handling
Draft

[WIP] Add handling for chunk buffer on timeouts#251
bnlawrence with Copilot wants to merge 1 commit into
mainfrom
copilot/chunk-buffer-timeouts-handling

Conversation

Copilot AI commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.


This section details on the original issue you should resolve

<issue_title>Chunk_buffer handling on timeouts</issue_title>
<issue_description>Currently there's no handling for where a timeout occurs within aiohttp/fsspec, so the response object is interpreted as a bytes-like object on decompression. Ideally this can be caught further up the chain of events.

Traceback (most recent call last):
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/file_access.py", line 87, in <module>
    raise exc
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/file_access.py", line 84, in <module>
    s = future.result()
  File "/opt/homebrew/Cellar/python@3.13/3.13.9_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.9_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/homebrew/Cellar/python@3.13/3.13.9_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/file_access.py", line 25, in _iterate_range
    return ds[i*multiplier:(i+1)*multiplier]  # np.min(ds[0:i])
           ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/high_level.py", line 445, in __getitem__
    data = self.id.get_data(args, self.fillvalue)
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 425, in get_data
    return self._get_selection_via_chunks(args)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 932, in _get_selection_via_chunks
    self._select_chunks(indexer, out, dtype)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 131, in _select_chunks
    self._read_bulk_fsspec(fh, chunks, out, dtype)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 218, in _read_bulk_fsspec
    out[out_sel] = self._decode_chunk(
                   ~~~~~~~~~~~~~~~~~~^
        chunk_buffer, storeinfo.filter_mask, dtype
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )[chunk_sel]
    ^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 104, in _decode_chunk
    chunk_buffer = BTreeV1RawDataChunks._filter_chunk(
        chunk_buffer,
    ...<2 lines>...
        self.dtype.itemsize,
    )
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/btree.py", line 282, in _filter_chunk
    chunk_buffer = zlib.decompress(chunk_buffer)
TypeError: a bytes-like object is required, not 'ConnectionTimeoutError'

Wherever the chunk_buffer object originates, a check to determine if the object is bytes-like would be good. There can then be a correct timeout error raised, or alternatively a second request to retry?</issue_description>

Comments on the Issue (you are @copilot in this section)

@bnlawrence Ah, it turns out that this isn't strictly pyfive's problem, fsspec defaults to not raising errors ... and returning them, which is odd, but we can at least ask them to do so and then raise any errors found. I am not sure we should handle retries though ... incoming code to at least raise them.

Copilot AI linked an issue Jul 10, 2026 that may be closed by this pull request
Copilot stopped work on behalf of bnlawrence due to an error July 10, 2026 11:24
Copilot AI requested a review from bnlawrence July 10, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chunk_buffer handling on timeouts

2 participants