Repository navigation
Conversation
With activation checkpointing, EdgeDegreeEmbedding.forward passed the full node tensor into every checkpointed edge chunk and received a new copy back. Checkpointing keeps each chunk's inputs until backward, so one node-feature tensor was retained per chunk. The number of chunks grows with the number of edges, so this memory grows roughly quadratically with system size. Scatter each chunk onto a shared zero tensor and add its contribution to x outside the checkpoint, so a single node-sized tensor is kept, as in Edgewise. Outputs and gradients are unchanged. Add a regression test that counts the node-feature copies kept for backward.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #2211 (this PR is based on its branch, I'll retarget to main once it's merged).
This was a problem Claude noticed when I was added the unit tests for
EdgeDegreeEmbedding. It is fairly niche, but has a significant impact on memory usage when activation checkpointing is turned on and running on large systems, so it seems worth fixing (the fix is also only 1 or 2 lines). This PR also adds a test based on the number of copies of the input tensor, this fails on the old code and passes with the fix in place. Below is the summary given by Claude.Problem
With activation checkpointing,
EdgeDegreeEmbedding.forwardpasses the full node tensor into every checkpointed edge chunk and gets a new copy back. Checkpointing keeps each chunk's inputs until backward, so one node-feature tensor is kept per chunk (131,072 edges each). Chunks grow with edges, so this memory grows roughly quadratically with system size.Edgewiseavoids this by passing one shared tensor and summing per-chunk contributions.Who hits it
Conservative force/stress inference or training on the general backend with activation checkpointing on, for systems with more than one chunk. Checkpointing is on with
InferenceSettings()defaults (used byMLIPPredictUnitwhen no settings are passed), thebatchpreset, and configs that enable it.Measured impact (one H100 80GB,
uma-s-1p2, defaultInferenceSettings(), energy + forces on rattled Cu fcc; peak GPU memory of the call above the pre-call baseline)Each difference matches (chunks − 1) × one node-feature tensor. At 79k atoms main peaks at 72 GB on the first call. Extrapolating, main exceeds 80 GB at around 90k atoms for this system, while this PR fits up to roughly 125k.
Fix
Each chunk scatters onto a shared zero tensor and its contribution is added outside the checkpoint, so one node-sized tensor is kept, as in
Edgewise.Correctness and cost
chunks)