Skip to content

Determine if caching/lock is necessary in bgen reader #315

Closed
@eric-czech

Description

@eric-czech

Hey @horta I would love to get your thoughts on this line of code from https://github.com/pystatgen/sgkit/pull/314:

@cached(cache, lock=lock)  # type: ignore[misc]
def _read_metafile_partition(path: Path, partition: int) -> pd.DataFrame:

Did you find that dask was loading the same partition multiple times? Also, is the lock necessary because of any kind of thread-safety issue in the metafile reader code?

TODO:

  • Remove cachetools dependency if this isn't necessary

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions