Repository navigation
Fall back to nogds at runtime when cuFile handle registration fails - #87
Merged
takeshi-yoshimura merged 1 commit intoJul 7, 2026
Conversation
Collaborator
|
Thanks for the contribution. The change looks good to me, but please add signed-off-by line in your commit. Then, I will merge this after the change #81. |
gitbisector
force-pushed
the
gds-runtime-fallback
branch
2 times, most recently
from
July 3, 2026 05:32
96ff25f to
7775b5e
Compare
Collaborator
|
@gitbisector |
cuFile can probe as available yet fail cuFileHandleRegister at I/O time (compat-mode hosts without nvidia-fs, checkpoints on overlayfs, CI runners). Catch the failure in submit_io, warn once, and transparently delegate the copier to the nogds bounce path -- so every consumer stops carrying its own gds->nogds retry wrapper. The fallback (and its bounce-buffer reader) lives only for the file's submit/wait cycle. Signed-off-by: git bisector <gitbisector@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
gitbisector
force-pushed
the
gds-runtime-fallback
branch
from
July 6, 2026 22:04
7775b5e to
a82f67e
Compare
Contributor
Author
|
Rebased onto main and resolved the conflict — both |
takeshi-yoshimura
merged commit Jul 7, 2026
02dd37f
into
foundation-model-stack:main
13 checks passed
Collaborator
|
Merged. thanks! |
takeshi-yoshimura
added a commit
to takeshi-yoshimura/fastsafetensors
that referenced
this pull request
Jul 7, 2026
- README: ROCm is no longer nogds-only, drop "without GDS". - overview.md: document hipFile-based direct loading on ROCm >= 7.2 (added in foundation-model-stack#85) instead of claiming ROCm has no GDS equivalent, and reflect that the gds copier now falls back to the nogds path at runtime with a warning (foundation-model-stack#87) rather than failing to open files. - development.md: the Makefile target is test-vllm, not vllm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
2 of 5 tasks
gitbisector
added a commit
to gitbisector/fastsafetensors
that referenced
this pull request
Oct 8, 2026
Since foundation-model-stack#128 the nogds copier opens checkpoints with O_DIRECT by default. The filesystem policy in is_odirect_enabled cannot tell whether a given file accepts direct I/O: tmpfs before Linux 6.6 and some FUSE filesystems refuse it at open with EINVAL, and the gds copier's foundation-model-stack#87 downgrade to nogds landed on the same rejection. The gds and unified copiers already fall back when their O_DIRECT open fails; make nogds do the same, and also cover a rejection at read time. - NoGdsFileCopier retries the open without O_DIRECT on EINVAL and warns once. Other errors propagate unchanged. - nogds_file_reader clears O_DIRECT with fcntl(F_SETFL) and retries the exact range buffered when a direct pread fails with EINVAL, so a filesystem that accepts the flag at open but rejects direct reads still loads. The retry is on the shared descriptor, so every worker and the Python side see the change. - Read failures on an O_DIRECT descriptor name the flag and FASTSAFETENSORS_ODIRECT=0, and include strerror for failed preads. - The nogds copier reads tmpfs and ramfs buffered: its bounce buffers gain nothing from bypassing a page cache that is the storage, and the foundation-model-stack#128 table shows direct I/O a wash or slower there. The unified copier is not changed; its multithreaded direct reader is unmeasured on tmpfs. - O_DIRECT is defined to 0 on every platform that lacks it, not only MSVC. - docs: FASTSAFETENSORS_NOGDS_ODIRECT is new in foundation-model-stack#128, not "existing"; describe the fallback. Fixes foundation-model-stack#129 Signed-off-by: git bisector <gitbisector@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cuFile can probe as available (
is_gds_supported()/is_cufile_found()pass) yet failcuFileHandleRegisterat I/O time — compat-mode hosts without the nvidia-fs kernel module, checkpoints on filesystems cuFile can't register (e.g. overlayfs on CI runners), etc. Today that surfaces as a hardRuntimeError: raw_gds_file_handle: cuFileHandleRegister returned an error = 5027fromGdsFileCopier.submit_io, and every consumer ends up carrying its own gds→nogds retry wrapper — e.g. vllm-project/vllm#40183 needed exactly that fix when its CI runner hit 5027 (weights on overlayfs, probe says GDS is fine).This catches the registration failure inside
GdsFileCopier.submit_io, warns once, and transparently delegates the copier to the nogds bounce path. The fallback copier (and its bounce-buffer reader) lives only for that file's submit/wait cycle, so no pinned memory outlives the load.Test: monkeypatched
gds_file_handleraising the 5027 error → load completes via the fallback, tensors byte-identical tosafetensors.load_file, and the bounce buffer is released afterwards.make lintclean; unit suite green on CPU and on a CUDA host where cuFile genuinely fails registration (the fallback turns that previously-failing environment green).