Closed
Description
Hey @horta I would love to get your thoughts on this line of code from https://github.com/pystatgen/sgkit/pull/314:
@cached(cache, lock=lock) # type: ignore[misc]
def _read_metafile_partition(path: Path, partition: int) -> pd.DataFrame:
Did you find that dask was loading the same partition multiple times? Also, is the lock necessary because of any kind of thread-safety issue in the metafile reader code?
TODO:
- Remove cachetools dependency if this isn't necessary
Metadata
Metadata
Assignees
Labels
No labels