Repository navigation
engram: host-side n-gram lookup for DeepSeek-V4.1, prefetched and overlapped - #2185
ZhangLirong-amd wants to merge 22 commits into
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
There was a problem hiding this comment.
🟡 Changes recommended
It contains correctness/operational issues in the compressed-vocab cache key and prefetch lifecycle (drop/shutdown behavior) that can lead to wrong caches or racy/stale cache entries.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds the host-side components of DeepSeek Engram support to ATOM: configuration parsing, reference-compatible n‑gram hashing, and a host embedding-table gather path with an async prefetcher to overlap CPU table work with GPU execution. This lays the groundwork for later wiring into the DeepSeek-V4.1 model runner.
Changes:
- Introduces
EngramConfig, compressed-tokenizer table build/cache, andNgramHashMappingto reproduce the reference hashing and validate table layout. - Adds host-resident embedding table gather plus an LRU prefetch cache and single-worker prefetcher API.
- Adds unit tests covering hashing/layout invariants, gather bounds, cache eviction/drop behavior, and prefetch-vs-inline equivalence.
File summaries
| File | Description |
|---|---|
atom/model_ops/engram.py |
Engram config + compressed vocab caching + reference-compatible n‑gram hashing and layout validation. |
atom/model_ops/engram_host.py |
Host embedding gather, LRU cache, and async prefetch worker for overlap. |
tests/model_ops/test_engram.py |
Unit tests validating hashing/layout, gather correctness, cache behavior, and prefetch behavior. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 7
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
810123f to
a164e26
Compare
9bcf5d1 to
ce8a871
Compare
…rlapped DeepSeek-V4.1-Flash puts Engram modules on layers 1 and 14, backed by tables that cannot live in HBM: 384,006,168 and 384,016,682 rows of 256 fp8 values, about 98 GB each, 196B parameters in total. The lookup has to run on the host and it has to be off the critical path, so this adds the host path, the device modules, and the runner hooks that hide the work behind the previous step's GPU time. Everything here is written against the shipped checkpoint and the V4.1 tech report, not against the January prototype's toy configuration. Hashing -- atom/model_ops/engram.py EngramConfig reads the engram_* block of a HF text config and returns None for models without one. NgramHashMapping reproduces the reference hashing (github.com/deepseek-ai/Engram): per-layer odd multipliers from a seeded generator, an XOR mix over the n-gram window, one distinct prime modulus per (layer, n-gram order, head) from a search whose seen set is shared across layers. Any other hash indexes the wrong rows of a trained table. The layout is checkable rather than assumed. Summing each layer's 24 primes has to reproduce engram_num_embeddings, and does -- 384,006,168 and 384,016,682 exactly -- and the compressed vocabulary built from the shipped tokenizer comes out at 99092, the declared value. A mismatched tokenizer or a wrong n-gram order fails at startup instead of quietly reading the wrong table. The compressed vocabulary is cached on disk rather than rebuilt by decoding 129,280 tokens on every start. Host tables and prefetch -- atom/model_ops/engram_host.py HostEmbeddingTable memory-maps a table, gathers only the rows a step asks for, and applies the block scales the checkpoint stores beside it: these tables are quantized per 32 values, and ignoring the scale does not fail, it returns values about 200x too large. EngramPrefetchCache is an LRU with an explicit drop for finished requests. EngramPrefetcher runs hash and gather on one worker and returns a Future. EngramRuntime owns the pinned staging buffers, issues the H2D on a side stream and orders the compute stream behind it; a sequence whose prefetch has not landed is computed inline, so the race decides latency and never the answer. Device modules -- atom/model_ops/engram_layer.py EngramOp matches the checkpoint's six tensors per layer: one fused wkv holding hc_mult key projections followed by a single value projection, and stacked RMSNorm gains k_weight / q_weight. There is no short causal convolution -- the tech report states V4.1 omits it, and the checkpoint carries no weights for one. The module takes embeddings as an argument instead of looking them up; a layer that indexed a 98 GB table inline would stall the step it belongs to. Token dimensions are flat, [num_tokens, hc_mult, hidden], matching what DeepseekV4ForCausalLM already returns. Attachment -- atom/model_ops/engram_module.py EngramModules.from_checkpoint builds config, tokenizer, hash layout, mapped tables and loaded device modules in one call, so nothing about n-gram hashing or 98 GB tables appears in a model definition. A model attaches it and exposes build_engram_runtime; the staging buffers take their dtype from the loaded modules, since staging in float32 against bf16 weights is a dtype error at the first matmul. Runner -- atom/model_engine/model_runner.py Three hooks, each a no-op for models without engram: build the runtime once the weights land, stage before run_model, and queue the next step's prefetch where the previous step's sampled ids are already on the host and already synchronized, so it adds no synchronization of its own. Model -- atom/models/deepseek_v41.py, plus registration DeepSeek-V4.1 with only Engram implemented. Attention and the MoE are stubs that preserve the residual and compute nothing, so the output is not meaningful and this must not be used for accuracy -- but the model loads and runs through ATOM's ordinary path, which is the only way the engram host path gets exercised at all. Registration is three pieces: the architecture in the model registry, "deepseek_v41" in the multimodal text_config hoist, and its text half mapped onto the V3 schema the way deepseek_v4 already is. V4.1 replaced MLA with CSA2, so its config has head_dim / o_lora_rank / o_groups / index_* and no kv_lora_rank, qk_nope_head_dim or v_head_dim. ATOM sizes the KV cache and builds attention metadata from those names on the config, not from the model, so a V4.1 checkpoint cannot be loaded until they exist. _fill_v41_attention_shims synthesizes self-consistent values for that sizing only; they are explicitly not V4.1's attention geometry, and a model that actually runs CSA2 has to stop relying on them. The runtime is held on the model rather than passed in: run_model calls forward with a fixed (input_ids, positions) signature, so there is no argument to thread it through. Loading is verified end to end on the shipped checkpoint: atom.config.Config resolves deepseek_v41 through the text_config hoist, the registry resolves the architecture, the engram modules load from the checkpoint at layers 1 and 14, and forward returns [num_tokens, hc_mult, hidden] with logits over the real vocabulary. Verified against deepseek-ai/DeepSeek-V4.1-Flash rather than only in unit tests: tables map instantly, a batched gather runs about 0.9 us/row against 8 us/row row-at-a-time, a 32-sequence decode step costs about 1.9 ms per layer, and both layers forward at the right shapes from the real wkv / k_weight / q_weight. The two tables are 197 GB against 2.7 TB of host memory, so the page cache holds them without the process ever allocating them. Loading and startup are exercised on the shipped checkpoint through the real server path: ModelRunner builds the runtime and logs "engram host path active on layers [1, 14]", and the engram layers run inside forward. Getting there needed four pieces of registration beyond the model itself -- the multimodal text_config hoist, the V3 schema mapping, the synthesized attention fields, and deepseek_v41_text in the per-request-cache set -- each marked as scaffolding a real CSA2 model will replace. Known gaps: prefill and dummy passes stage zeros, which propagate to exactly zero through EngramOp, so engram contributes nothing during prefill. That is wrong for a real model -- the prompt's n-grams are precisely what should be looked up, and the addressing is deterministic so they could be -- and it has to be replaced by a per-prompt-token gather before this serves anything for accuracy. Decode uses the real path.The tables are also mapped whole on every rank, where the tech report describes sharding them; and the overlap here is cross-step, where the report describes prefetching within a forward, overlapping the first Transformer block. Refs #168, which prototyped the overlap on a standalone toy model. Tests: 40 cases in tests/model_ops/test_engram.py.
ce8a871 to
00afbe3
Compare
Route DeepseekV41* to the MLA attention family instead of V4 (a bare "DeepseekV4" substring caught "DeepseekV41"), so the KV pool sizes off the real layer count rather than dividing by zero; fall back to num_hidden_layers when the stub walks no attention modules; drop deepseek_v41_text from the per-request state-cache types (it is pure MLA, no per-req state pool). Key the prefetch gather on the just-sampled token (next step's model input), matching the deferred-output pipeline, so a cache hit equals an on-the-spot recompute. Stage then reads a miss token off the scheduler's host scheduled_tokens anchor rather than a device->host copy of input_ids: a hit is keyed by seq id and needs no token, and the only misses are just-admitted rows whose committed anchor already sits on the host -- so the hot path has no D2H. Add an env-gated (ATOM_ENGRAM_DEBUG_VERIFY) check that recomputes a reference from the ground-truth input and compares it to the async-prefetched rows; it takes the ground-truth token separately, which is what lets it also catch a bad host anchor feeding a miss.
The engram op was split across engram.py (hash primitives), engram_host.py (tables + prefetch + runtime), engram_layer.py (the device op) and engram_module.py (the model attachment) -- four files for one feature, with only engram_module importing the other three. Fold it along the host/device line: engram.py now holds the whole host pipeline (config, tokenizer, hash, HostEmbeddingTable, prefetcher, cache, EngramRuntime), and engram_layer.py holds the device side (EngramOp) plus the EngramModules that builds the ops and their runtime. Behavior is unchanged -- classes moved verbatim, only imports rewired; deepseek_v41 and the tests now import from the two surviving modules.
The host-side object read "runtime" (generic) and its methods "stage" and "build_engram_runtime" did not say what they act on. Rename for what they are: EngramRuntime -> EngramHost (it is engram's host half) EngramHost.stage -> stage_embeddings (stages this step's rows) *.build_engram_runtime -> build_engram_host (factory + getattr contract) DeepseekV41*.engram_runtime -> engram_host `stage_dummy` keeps its name. Pure rename across the host path, the two model build sites, the ModelRunner getattr string, and the tests; no behavior change.
Prefetching the just-sampled token used a blocking `.to("cpu").numpy()` in
postprocess, which synchronizes the compute thread on the sampling kernel every
decode step -- exactly the sampled-token sync deferred output exists to avoid.
Hand the raw sampled ids to EngramHost.prefetch_next instead: it slices the last
id per row and copies it to a pinned host buffer ASYNC on a side stream, records
an event, and the prefetch worker waits on that event before reading (a new
`token_source` hook on the prefetcher). The compute thread only launches the copy
and returns. `stage_embeddings`' existing wait on the worker keeps the single
reused buffer/event safe. The CPU test path submits the ids directly.
Verified with ATOM_ENGRAM_DEBUG_VERIFY: 0/N mismatch, 0 wrong-token, no hang.
Clearer names on the host path and its callers: EngramPrefetcher.submit -> submit_compute (queues an async compute) EngramHost.wait_for_copy -> wait_for_embeddings ModelRunner local `make_host` -> `build_engram_host` (matches the method) Trim the verbose engram comments in ModelRunner down to the load-bearing why. No behavior change.
The DeepSeek-V4.1 stub only ever existed to exercise and verify the Engram host path end-to-end (done: ATOM_ENGRAM_DEBUG_VERIFY shows byte-exact prefetch, and it serves under load). With correctness proven, drop the demo scaffolding and leave Engram as a self-contained, tested library plus ModelRunner's generic drive hooks. Removed: the stub model atom/models/deepseek_v41.py and its registration; and the V4.1-arch-specific bring-up that only routed it -- the DeepseekV4(?!\\d) regexes in selector/register/config (back to the plain "DeepseekV4" substring, correct again with no V4.1 arch to mis-match), config's deepseek_v41_text registry entry + _fill_v41_attention_shims + multimodal entry, aiter_mla's zero-module KV-pool fallback, and llm_engine's per-request-cache note. Kept: atom/model_ops/engram*.py and the (getattr-gated, inert-without-a-model) ModelRunner hooks. recipes/Engram.md documents the model-side contract, using the removed stub's wiring as the worked example. History keeps the bring-up commits (00afbe3 onward) as the record of how it was hooked up and verified.
The compressed-vocab normalizer used a bare U+E000 (Private Use Area) character as its lone-space sentinel, which renders as mojibake and reads as corruption. Write it as chr(0xE000) with a comment; behavior is unchanged.
The verify check was bring-up scaffolding to prove the async prefetch equals an inline recompute (done: byte-exact, 0 wrong-token). With correctness established, remove it: the `verify_token_ids` parameter and the whole compare/log block in stage_embeddings, the per-step counters, the prefetcher's _debug_prefetch_token, the _stage_engram input_ids D2H it required, and the recipe's verify section.
EngramHost sorted out of place after the EngramRuntime->EngramHost rename; CI's ruff runs the isort rule that the default local select does not.
There was a problem hiding this comment.
🟡 Changes recommended
Multiple unresolved critical correctness, integration, checkpoint-decoding, and staging issues remain.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (12)
atom/model_engine/model_runner.py:879
run_modelcan executerunning_tokensrows rather thanbatch.total_tokens_num(piecewise graph buckets and DP padding do this; capture already stages the settled padded width). This stages only the shorter local prefix, soEngramOpcan see a row-count mismatch or graph replay can consume stale tail embeddings. Stage the settled running width and initialize the padded tail consistently.
self.engram.stage_embeddings(seq_ids, tokens)
atom/model_engine/model_runner.py:879
- This fallback source is not always the current model input. In deferred decode,
prepare_input_idsdocuments thatscheduled_tokensis a placeholder for carried-over requests, while the actual input comes fromprev_sampled_token_ids. If such a request misses the cache (for example by LRU eviction), this recomputation hashes the placeholder and silently uses the wrong Engram embedding. Pass the actual per-request input ids to the fallback.
# `tokens` is read only to recompute a prefetch miss (a just-admitted
# row), whose anchor is already in the host `scheduled_tokens` -- so the
# hot path needs no D2H.
tokens = batch.scheduled_tokens[: len(seq_ids)].astype(np.int64).reshape(-1, 1)
self.engram.stage_embeddings(seq_ids, tokens)
atom/model_engine/model_runner.py:849
- The runner says it owns
EngramHost's lifecycle, butexit()never callsself.engram.shutdown()before deleting the model. TheThreadPoolExecutorworker is non-daemon and may still be scanning a 98 GB table, so server shutdown can block at interpreter exit; a running worker can also outlive the model's table handles. Close/join the prefetcher during runner exit before destroying model state.
The model owns its tables (it is what loaded them) and hands back the
EngramHost; the runner owns the lifecycle and the two hooks below. Models
without engram leave this None and pay nothing but the attribute.
"""
self.engram = None
atom/model_engine/model_runner.py:840
- The new runner integration is not exercised by the added tests: they call the helper classes directly but never run
_init_engram_host,_stage_engram,postprocess, or the capture paths through aModelRunner. A regression in prefill/speculative row alignment or side-stream/cudagraph staging can therefore pass the entire new test file while producing wrong output or capture failures. Author must add a mocked runner-level integration test covering normal decode and the non-decode paths.
self._init_engram_host()
atom/model_ops/engram.py:145
- The cache key only hashes the first 1024 token strings and omits their token IDs. Two tokenizer revisions can therefore reuse the same
.npzwhile mapping later IDs differently (or while reassigning IDs), producing silently wrong compressed IDs and table rows. Hash the complete token-to-ID mapping, along with the normalization/cache version, or use a tokenizer/checkpoint-specific cache key.
for tok in sorted(vocab)[:1024]:
h.update(tok.encode("utf-8", "replace"))
atom/model_ops/engram.py:663
torch.cuda.current_stream(self.device)is evaluated inside thecopy_streamcontext, so it returnsself.copy_streamrather than the compute stream. Thewait_streamis therefore a self-wait and does not protect the previous forward from this buffer's next H2D overwrite; concurrent decode/PP forwards can read torn Engram embeddings. Capture the compute stream before entering the context and wait on that object.
with torch.cuda.stream(self.copy_stream):
self.copy_stream.wait_stream(torch.cuda.current_stream(self.device))
atom/model_ops/engram.py:686
- The same stream-ordering bug is duplicated for dummy/capture staging:
current_stream()is queried after enteringcopy_stream, making this wait a no-op. A subsequent dummy or real stage can overwrite the single GPU buffer while the preceding capture/forward still reads it. Capture the compute stream before the context here as well.
with torch.cuda.stream(self.copy_stream):
self.copy_stream.wait_stream(torch.cuda.current_stream(self.device))
atom/model_ops/engram.py:681
- Clamping an oversized dummy request makes
_staged_rowssmaller than thenum_tokenspassed to the model, soEngramOplater sees a shape mismatch (or a future caller reads truncated rows) instead of a capacity violation.stage_embeddingsalready raises for this condition;stage_dummyshould reject it too rather than silently changing the requested shape.
num_rows = min(int(num_rows), self.max_num_tokens)
atom/model_ops/engram.py:738
- This cleanup API is never called by
ModelRunner(the only references are its definition and wrapper), so completed requests remain in the cache until LRU eviction. With two layers per request, high request churn can evict active prefetched rows and forcestage_embeddingsonto synchronous host gathers, defeating the overlap and making latency scale with the 98-GB tables. Author must invoke request cleanup from the request-finished/preempted lifecycle.
def drop_requests(self, seq_ids: list[int]) -> None:
self.prefetcher.drop_requests(seq_ids)
recipes/Engram.md:99
- This recipe documents
_stage_engramwith aninput_idsparameter, but the added runner API isdef _stage_engram(self, batch). The example is the model-integration contract, so this mismatch can lead implementers to call a nonexistent argument; document the actual signature.
- `_stage_engram(batch, input_ids)` — before each forward: `stage_dummy` for
recipes/Engram.md:48
- The prescribed model-side contract requires adding the Engram module/forward hook to the DeepSeek-V4 model, but
DeepseekV4Modelis@support_torch_compile-decorated; editing that compiled model path breaks Dynamo tracing. Integrate through an outer wrapper or another call site outside the compile boundary, and update this recipe so it does not direct users to modify the decorated model.
## Model-side contract (3 hooks)
A model opts in by doing three things. ModelRunner does the rest automatically,
gated on `getattr(model, "build_engram_host", None)` — a model without it pays
nothing.
tests/model_ops/test_engram.py:147
- The hash tests only check shape and repeatability; they never compare a known input against reference hash outputs. A wrong multiplier RNG, padding convention, or XOR order would still satisfy every assertion here while indexing the wrong trained rows. Add golden vectors from the reference for each layer/order, including the left-padded prefix and compressed IDs.
def test_hash_shape_and_determinism():
mapping = build_mapping()
ids = np.array([[5, 9, 13, 21], [1, 2, 3, 4]], dtype=np.int64)
first = mapping.hash_layer(ids, layer_id=0)
assert first.shape == (2, 4, mapping.config.num_hash_heads)
np.testing.assert_array_equal(first, mapping.hash_layer(ids, layer_id=0))
- Files reviewed: 5/5 changed files
- Comments generated: 10
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Critical and moderate issues remain in FP8 decoding, configuration handling, batching, and request cleanup.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (7)
atom/model_engine/model_runner.py:984
- On DP decode,
ForwardMode.decidesetscontext.running_bs/running_tokensfrom the DP-wide maximum, notlen(seq_ids); this rank can have one request while replaying the graph selected for a peer's larger batch. Choosinggbsfrom this local count can stage only two rows whilerun_modelexecutesEngramOpat eight rows, causing a shape mismatch or stale embeddings. Derive the staging height from the settled forward shape (context.running_tokens) instead.
gbs = next(
(g for g in reversed(self.capture_sizes) if g >= padded_rows), None
)
if gbs is not None:
padded_rows = max(padded_rows, int(gbs))
atom/model_engine/scheduler.py:1821
- This drains the pending IDs even when
scheduled_seqsis empty.EngineCoreskipsrunner.forward()for such a batch (atom/model_engine/engine_core.py:412-433), so_stage_engram()never callsdrop_requests()and the cache/window cleanup is lost. Only drain these IDs when a real batch will reach the runner, as the prefill construction already does.
engram_dropped=self._drain_engram_dropped(),
atom/model_engine/scheduler.py:524
- This extraction repeats the dictionary-handling bug:
getattrleavestcat the outer dict. Once nested-dict detection is corrected (or when the declaration is otherwise recognized), line 527 indexes the outer dict and raisesKeyErrorinstead of readingtext_config["engram_max_ngram_size"]. Select the nested key explicitly for dictionary configs here as well.
tc = getattr(config.hf_config, "text_config", None) or config.hf_config
atom/model_ops/engram.py:734
- A queued worker can pass
drop_requests()after the cache/window removal but still reachadvance()here and recreate_window[seq_id]. Unlikecache.put, window updates have no tombstone or generation check, so a canceled request leaves stale rolling context behind and can associate it with a later reused sequence id. Coordinate the drop with the worker or reject updates from an older request generation.
windows = self.advance(seq_ids, toks)
atom/model_ops/engram.py:572
- The rolling-window cap is fixed at 4096, independently of the scheduler's configurable
max_num_seqs, and the Engram host factory never passes the live-request limit. With more than 4096 active requests,_evict_windowsdrops still-live windows; a later cache miss then falls back to only the current token, so multi-token Engram hashes are wrong. Size this storage for the scheduler capacity or retain windows for live requests.
self._window_capacity = cache_capacity
atom/model_ops/engram_layer.py:134
- No test asserts the numerical result of this new forward path; the current
EngramOptests only check shapes, errors, and checkpoint loading. A wrong key/value split, RMSNorm/gate formula, or branch broadcasting would pass while changing logits. Add a deterministic forward case with independently calculated expected output, including multiple branches.
gate = (key * query).sum(dim=-1) / math.sqrt(self.hidden_size)
tests/model_ops/test_engram.py:150
- This test invokes
hash_layertwice and compares the implementation with itself, so it cannot catch a wrong multiplier, padding/shift, XOR order, or layer hash that is still deterministic and in range. Add fixed vectors generated independently from the reference for both layers and representative padding/context windows; otherwise compatibility with the trained table row layout is not tested.
np.testing.assert_array_equal(first, mapping.hash_layer(ids, layer_id=0))
- Files reviewed: 6/6 changed files
- Comments generated: 3
- Review effort level: Lite
… h2d_done Extract the slot rotation + reuse barrier into _SlotRing, used by both the staging H2D ring and the token D2H ring -- they differ only in the per-slot barrier (a CUDA event synchronize vs a threading.Event wait). Rename copy_done to h2d_done to name what the event marks (the host-to-device staging copy).
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate findings affect correctness, integration, lifecycle cleanup, and the claimed overlap.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (5)
atom/model_engine/model_runner.py:903
- For the architecture registered as
DeepseekV4ForCausalLM, no in-tree model implementsbuild_engram_hostor injectsEngramOpinto the decoder, so a V4.1 config reaches this fail-closed branch and startup aborts. As written, the host/device implementation is unreachable for the model named in the PR; either wire the model (including its checkpoint prefixes and forward path) in this change or scope/retitle this as infrastructure-only.
build_engram_host = getattr(self.model, "build_engram_host", None)
if build_engram_host is None:
# Fail closed rather than silently serve the base model: a config that
# declares engram layers but resolves to a model that does not wire
# build_engram_host would otherwise ignore the engram weights and
atom/model_engine/model_runner.py:3220
- This hook is invoked only after
run_modeland sampling, whileprefetch_next's CUDA path explicitly makes its D2H stream wait on the current compute stream. Therefore the token cannot reach the worker until the preceding GPU step has completed;stage_embeddingsthen waits for the worker before launching the next forward. The host gather is overlapped with CPU postprocessing/scheduling, not the previous GPU work described by this PR, so the claimed critical-path hiding is not achieved for a host lookup that exceeds that CPU gap.
# Prefetch on the just-sampled token (the NEXT step's input), not the
# deferred one, which would prefetch a step stale. `prefetch_next` gathers
# its embedding on the worker overlapping postprocess, and copies the
# token to the host async, so this thread pays no D2H sync.
if getattr(self, "engram", None) is not None and not batch.is_dummy_run:
atom/model_engine/scheduler.py:1821
- When decode scheduling produces an empty batch,
EngineCoreskips the forward and postprocess entirely, so this drains the pending IDs without ever calling_stage_engram. The finished request's cache/window is therefore left alive (and a reused ID can observe stale state). Keep the pending IDs until a real batch is built, as the prefill branch already does.
engram_dropped=self._drain_engram_dropped(),
atom/model_ops/engram.py:48
- When
hf_configis a rawdictwith the usual nestedtext_config,getattralways returnsNone, sotcbecomes the outer dict and this function missestext_config.engram_layer_ids. The dict branch suggests raw configs are supported, but such a config can bypass the runner's Engram setup; read thetext_configmapping before falling back to the root.
tc = getattr(hf_config, "text_config", None) or hf_config
return (
tc.get("engram_layer_ids") is not None
if isinstance(tc, dict)
else getattr(tc, "engram_layer_ids", None) is not None
atom/model_ops/engram.py:734
drop_requestscan remove a request's window while this queued worker is still running, butadvance()recreates it unconditionally beforecache.putchecks the drop mark. A normal finish during an in-flight prefetch therefore leaks a stale rolling window and can contaminate a reused request id; the worker needs to observe cancellation/generation before advancing, not only before cache insertion.
def _run() -> None:
toks = token_source() if token_source is not None else token_ids
windows = self.advance(seq_ids, toks)
for key, value in self.compute(seq_ids, windows).items():
self.cache.put(key[0], key[1], value)
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
cef87c7 to
7a564ca
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Critical Engram quantization/cache issues and moderate integration, DP-shape, and lifecycle issues remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (5)
atom/model_engine/model_runner.py:905
- For the in-tree
DeepseekV4ForCausalLM, nobuild_engram_hosthook or residual Engram injection is present, so a V4.1 config takes this branch and raises after model loading instead of serving Engram. That leaves the advertised DeepSeek-V4.1 path unusable; either add the model-side integration (including skipped host-table weights and layer injection) in this change or scope the PR as library-only.
if self._config_declares_engram(self.config):
atom/model_engine/model_runner.py:947
- A DP dummy also goes through
ForwardMode.decide:dummy_execution()creates a one-token batch, but graph replay uses the DP-widecontext.running_tokenssettled from the peer. Staging onlybatch.total_tokens_numtherefore leaves the Engram buffer at one row while the dummy hidden states have the running height, causing a shape mismatch or stale padded rows. Stage the context's settled running height here as well.
if batch.is_dummy_run or not seq_ids:
self.engram.stage_dummy(batch.total_tokens_num)
self.engram.wait_for_embeddings()
atom/model_engine/model_runner.py:984
- The decode shape can be widened to the DP group's
running_bs/running_tokens, whilelen(seq_ids)is only this rank's local batch. On a rank with fewer requests, choosing the bucket from the local count stages too few rows; the graph then executes the larger padded rectangle andEngramOpreceives hidden states and embeddings with different leading dimensions. Use the already-set forward mode's actual running row count for graph-backed decode (or explicitly reject this DP mode).
gbs = next(
(g for g in reversed(self.capture_sizes) if g >= padded_rows), None
)
if gbs is not None:
padded_rows = max(padded_rows, int(gbs))
atom/model_engine/scheduler.py:1821
- This decode path drains
_engram_drop_pendingeven whenscheduled_seqsis empty.EngineCoreskipsrunner.forwardfor such a batch, so the IDs are cleared from the scheduler without ever reachingEngramHost.drop_requests; their cache/window (and any in-flight stale state) then remains indefinitely and can consume LRU capacity or contaminate a later request. Keep the IDs pending until a non-empty batch is forwarded, as the prefill path already does.
engram_dropped=self._drain_engram_dropped(),
atom/model_ops/engram.py:1057
stage_prefillcan run while the previous decode's prefetch future is still in flight. On preemption/resume,drop_requestsremoves the current window but does not prevent that worker'sadvance()from recreating and mutating it; the worker can therefore append its stale sampled token after this method seeds the new prompt tail, making the next decode lookup use the wrong n-gram (and cache rows may also be stale). Synchronize or generation-guard the in-flight worker before reseeding and staging a resumed prefill.
computed = self.prefetcher.compute_prefill(context_tails, chunk_tokens)
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
7a564ca to
9266880
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Critical configuration-handling defects and unresolved lifecycle, loading, caching, and overlap issues remain.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (8)
atom/model_engine/model_runner.py:910
- This fail-closed check is reached only after
load_model, but the registeredDeepseekV4ForCausalLMcurrently has nobuild_engram_hostand no skip prefixes forlayers.*.engram.*. A real V4.1 checkpoint will therefore either fail here after loading as much of the Engram tensors as the loader permits, or deserialize the shard containing the ~196 GB tables whenATOM_DISABLE_MMAP=true, before reporting the unsupported model. The target model needs to be wired (including loader skips) in this change, or Engram configs must be rejected before model loading; as written the advertised DeepSeek-V4.1 path cannot serve the model safely.
if self._config_declares_engram(self.config):
raise NotImplementedError(
f"{type(self.model).__name__} config declares engram layers "
"but the model does not implement build_engram_host; serving "
"would silently ignore the engram weights"
)
atom/model_engine/scheduler.py:1821
_drain_engram_dropped()is unconditional here, butEngineCoreskipsforwardand postprocesses an emptyScheduledBatchwhenreq_idsis empty. If the engine goes idle after a request finishes, this batch consumes the pending ids without callingEngramHost.drop_requests, leaving its cache/window alive and allowing stale state to survive until a later request. Drain only whenscheduled_seqsis non-empty, as the prefill path already does, so the ids remain pending for the next forwarded batch.
engram_dropped=self._drain_engram_dropped(),
atom/model_ops/engram.py:770
drop_requests()removes a sequence's rolling window before an in-flight_runcan finish, but this unconditionaladvance()recreates that window afterward. The drop marker only suppressescache.put, not_window, so a dropped request can remain in the LRU and evict a live request's context; a later cache miss can then hash with a stale/placeholder window. Carry a drop generation/state into the worker and suppress the window update (and publication) for requests dropped after submission.
def _run() -> None:
toks = token_source() if token_source is not None else token_ids
windows = self.advance(seq_ids, toks)
for key, value in self.compute(seq_ids, windows).items():
self.cache.put(key[0], key[1], value)
atom/model_ops/engram.py:1050
- This wait makes the side-stream H2D begin only after all work previously queued on the compute stream completes. Because each layer has a single device destination buffer, staging therefore serializes the copy after the previous forward and then makes the next forward wait for the copy; the H2D cannot overlap GPU work as described in
recipes/Engram.md. Achieving that overlap needs separate device buffers (and graph-safe selection), or the overlap claim should be narrowed.
self.copy_stream.wait_stream(compute_stream)
atom/model_ops/engram.py:841
- This comment says a timed-out worker causes rows to be recomputed inline, but
stage_embeddingsdeliberately waits a second time and then raisesRuntimeError; it never performs that fallback. Please make the comment describe the actual fail-fast behavior so the timeout's operational contract is not misleading.
# worker, at which point the rows are recomputed inline instead of hanging.
atom/model_ops/engram_layer.py:345
max_num_tokensis the per-step token budget, not the number of active sequences. A valid configuration can setmax_num_batched_tokens < max_num_seqs; then this cache/window capacity is too small and_evict_windows()removes contexts for active requests that were not selected in the current decode batch. When one is scheduled later, the miss path falls back to its current token and hashes without the preceding n-gram context, producing wrong embeddings. Size the per-request cache/window from the scheduler's active-sequence ceiling (or avoid evicting active windows), rather than onlymax_num_tokens.
# Size the prefetch cache / rolling-window store to the runner's hard
# concurrency ceiling (a decode step stages <= max_num_tokens rows,
# one per sequence), so an active request is never LRU-evicted; the
# capacity bound is then only a leak net for a missed drop_requests.
EngramPrefetcher(
self.hash_mapping, self._tables, cache_capacity=max_num_tokens
atom/model_ops/engram_layer.py:345
- The cache is keyed by
(seq_id, layer_id), so a decode row consumes one entry per Engram layer, butcache_capacityis onlymax_num_tokens(the row/token bound). In a legal configuration whose decode batch approaches that bound, inserting both layers evicts one layer for earlier sequences;_stage_embeddingsthen recomputes synchronously and the intended overlap is lost. Size this entry capacity for all layers (or otherwise account for the per-layer keys).
self.hash_mapping, self._tables, cache_capacity=max_num_tokens
recipes/Engram.md:120
- The documented integration point calls
self.engram[...]from the decoder layer loop, but the registered DeepSeek-V4 implementation places that loop insideDeepseekV4Model, which is marked@support_torch_compile(atom/models/deepseek_v4.py:4290). The repository's compile-boundary rule forbids modifying such model files; following this recipe would put the host-dependent lookup inside Dynamo/capture and can break compilation. Document or implement the hook at an allowed outer runner/call-site boundary instead.
for layer_id, layer in enumerate(self.layers):
if self.engram is not None and layer_id in self.engram:
if self.engram_host is None:
raise RuntimeError(
f"layer {layer_id} carries engram but no host was supplied; "
f"ModelRunner stages the embeddings before calling forward"
)
hidden = hidden + self.engram[layer_id](
hidden, self.engram_host.embeddings(layer_id)
)
hidden = layer(hidden, positions)
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
9266880 to
410c4a2
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved configuration, request-lifecycle, cache, performance, and integration-test issues remain.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (7)
atom/model_engine/model_runner.py:3343
- [verified] The new runner hooks are covered only by EngramHost/unit-level tests; no test drives
ModelRunner.forwardwith aScheduledBatchthrough decode, final/middle prefill, and dummy/CUDAGraph staging. Ordering or padding/stream handoff regressions here can pass the current suite and fail only in serving. Author must add a mocked integration test for the runner/scheduler contract.
if getattr(self, "engram", None) is not None:
self._stage_engram(batch)
logits, hidden_states = self.run_model(input_ids, batch)
atom/model_engine/scheduler.py:1827
- When no sequence is schedulable, this still drains
_engram_drop_pending, butEngineCoreskipsrunner.forward()for the resulting empty batch, so_stage_engramnever callsdrop_requests. The finished request's cache/window therefore survives until a later request-id reuse can consume stale n-gram state. Preserve the pending IDs on empty batches and drain them only into a batch that will be forwarded, as the prefill path already does.
engram_dropped=self._drain_engram_dropped(),
atom/model_ops/engram.py:640
drop_requests()removes a request's rolling window, but an in-flight worker can then reach thissetdefaultand recreate it. The cache tombstone prevents onlycache.put, not this window write; if the request ID is reused, the next prefetch appends to the stale tokens and hashes the wrong n-gram context. Synchronize request dropping with the worker or add a generation/tombstone check around window advancement.
w = self._window.setdefault(seq_id, deque(maxlen=self._ngram))
atom/model_ops/engram.py:707
- Every runtime prefetch uses
HostEmbeddingTable.gather's defaultfloat32output here, andEngramHost.stage_embeddingslater casts that full row into the model's bf16/fp16 staging slot. For a 6144-wide V4.1 row this adds an intermediate allocation and another full copy/conversion to every random host lookup, extending the worker work that is supposed to overlap GPU execution. Carry the target staging dtype into the prefetcher and gather directly in that dtype.
gathered = self._tables[layer_id].gather(rows[:, -1])
atom/model_ops/engram.py:739
- Prefill has the same avoidable dtype expansion as decode:
gatherdefaults tofloat32, thenstage_prefillcopies each row into the model-dtype staging buffer. This adds a full intermediate and conversion for every prompt position, which is especially costly for large chunks; pass the target staging dtype through the prefetcher and gather directly in it.
chunk_rows = self._tables[layer_id].gather(rows)[0, skip:]
atom/utils/envs.py:28
- [verified]
ATOM_ENGRAM_CACHE_DIRis a new operator-facing setting, but the repository's environment-variable reference does not list it. Deployments cannot discover how to relocate the compressed-vocabulary cache or what the default is. Author must document this variable indocs/environment_variables.mdwith its default and cache semantics.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
recipes/Engram.md:71
- [verified] The PR's central claim is that host lookup is hidden off the critical path, but the description provides no benchmark with units or reproducible command. Without target-model host latency, GPU-step overlap, and end-to-end throughput/latency data, a cache-miss fallback can turn this into a regression while all correctness tests still pass. Author must add a reproducible production-shape benchmark and report both steady-state hits and cold/missed-prefetch behavior.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
- Files reviewed: 7/7 changed files
- Comments generated: 3
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Critical staging-width and lint issues, plus unresolved integration, cache, cleanup, testing, and performance-validation concerns remain.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (8)
atom/model_engine/model_runner.py:880
- The registered DeepSeek-V4 model has no
build_engram_hostimplementation, so every real V4.1 config that declaresengram_layer_idsraises at this guard before model construction. The current tree therefore cannot serve the model named by this PR; either add the model-side Engram wiring (including the skipped table prefixes) or explicitly scope the PR as library groundwork.
if not hasattr(model_class, "build_engram_host"):
raise NotImplementedError(
f"{model_class.__name__} config declares engram layers but the "
"model class does not implement build_engram_host; serving "
"would silently ignore the engram weights"
atom/model_engine/model_runner.py:957
- The new shared runner hook is covered only by
tests/model_ops/test_engram.py; there is no integration coverage for_stage_engramthroughforward/postprocess, including dummy, prefill, decode, drop, and CUDAGraph-width behavior. A regression here can pair embeddings with the wrong rows or break capture while all host/unit tests pass. Add a runner-level test using a stub EngramHost that asserts staging order and row counts.
if getattr(batch, "engram_dropped", None):
self.engram.drop_requests(batch.engram_dropped)
seq_ids = list(batch.req_ids)
if batch.is_dummy_run or not seq_ids:
self.engram.stage_dummy(batch.total_tokens_num)
atom/model_engine/model_runner.py:3230
- [verified] The PR claims host work is off the critical path, but the only reported results are equality logs; this path explicitly waits for the worker in
stage_embeddingsand recomputes misses, so a gather longer than one GPU step becomes a synchronous stall. Author must provide a reproducible decode/prefill latency or throughput comparison showing the overlap on target hardware and production batch sizes.
# Prefetch on the just-sampled token (the NEXT step's input), not the
# deferred one, which would prefetch a step stale. `prefetch_next` gathers
# its embedding on the worker overlapping postprocess, and copies the
# token to the host async, so this thread pays no D2H sync.
atom/model_engine/scheduler.py:531
config_declares_engram()accepts a root-level declaration whentext_configexists without Engram, but this normalization unconditionally selects that nested block. Such a config reachestc["engram_max_ngram_size"]and raisesKeyError, even thoughEngramModules.from_checkpoint()handles the same shape. Check that the nested block actually containsengram_layer_idsbefore selecting it for both dict and object configs.
if isinstance(config.hf_config, dict):
tc = config.hf_config.get("text_config", config.hf_config)
else:
tc = getattr(config.hf_config, "text_config", None) or config.hf_config
atom/model_engine/scheduler.py:1827
_drain_engram_dropped()is called even whenscheduled_seqsis empty, butEngineCoreskipsrunner.forwardfor an empty batch, so_stage_engramnever consumes these IDs and the cache/window are left live. If a sequence ID is later reused, stale embeddings can be staged for the new request. Preserve the pending IDs by draining only for a non-empty batch, or process the drop explicitly on the empty-batch path.
engram_dropped=self._drain_engram_dropped(),
atom/model_ops/engram_layer.py:351
- The cache capacity counts
(seq_id, layer_id)entries, not sequences. With two Engram layers and a decode batch nearmax_num_tokens, inserting one row per layer evicts active rows immediately, forcingstage_embeddings()to recompute them synchronously and defeating the intended overlap. Size the cache for all layers (for example,max_num_tokens * len(self.config.layer_ids)).
# Size the prefetch cache / rolling-window store to the runner's hard
# concurrency ceiling (a decode step stages <= max_num_tokens rows,
# one per sequence), so an active request is never LRU-evicted; the
# capacity bound is then only a leak net for a missed drop_requests.
EngramPrefetcher(
self.hash_mapping, self._tables, cache_capacity=max_num_tokens
atom/utils/envs.py:28
ATOM_ENGRAM_CACHE_DIRis a new public setting, but it has no entry indocs/environment_variables.md, the repository's user-facing environment-variable reference. Operators cannot discover how to relocate the potentially large compressed-vocabulary cache. Add the variable with its default and cache semantics to the environment-variable documentation.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
recipes/Engram.md:73
stage_embeddingscan block waiting for the worker, yet this recipe presents the lookup as hidden behind GPU work without latency/tokens-per-second measurements or reproducible GPU, ROCm, and configuration details. The central overlap/performance claim is therefore unverified, especially for cold and cache-miss steps. Add production-shaped steady-state and miss/cold benchmark results with the reproduction command.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
double-buffered (a per-layer pinned ring) so the next step's fill never races the
previous step's in-flight H2D, while the device buffer stays fixed for CUDAGraph.
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Critical runner and Engram validation issues, along with additional integration and runtime gaps, remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (7)
atom/model_engine/model_runner.py:880
- This guard makes the advertised DeepSeek-V4.1 path unusable:
DeepseekV4ForCausalLMis the registered production class, but it does not implementbuild_engram_host(nor attachEngramModulesor skip the Engram checkpoint prefixes), so every V4.1 config declaring Engram raises here before serving. Either wire the model's three hooks and checkpoint loading, or scope the PR/title explicitly to the standalone library instead of rejecting the only advertised model.
if not hasattr(model_class, "build_engram_host"):
raise NotImplementedError(
f"{model_class.__name__} config declares engram layers but the "
"model class does not implement build_engram_host; serving "
"would silently ignore the engram weights"
atom/model_engine/model_runner.py:3230
- [verified] The PR's central claim is that host hashing/table gathers are hidden behind GPU work, but the supplied output only checks value equality; it contains no end-to-end latency/throughput or overlap measurement, and no comparison with synchronous/recomputed lookup at production table and batch sizes. Add a reproducible decode benchmark with units and the overlap-disabled baseline before relying on this optimization claim.
# Extract req_ids and token_ids from dict (key -1 is the is_deferred_out flag)
req_ids_out = [k for k in token_id_dict if k != -1]
token_ids_out = [token_id_dict[k] for k in req_ids_out]
atom/model_engine/model_runner.py:948
- The new tests exercise the host and device units, but no test constructs a model exposing
build_engram_hostand drives the runner's_stage_engram/postprocess path. That leaves scheduler-to-runner row ordering, final-vs-middle prefill behavior, CUDAGraph padding, and stream handoff unverified; add a mocked model/runner integration test covering one decode step and a chunked prefill.
def _stage_engram(self, batch: ScheduledBatch) -> None:
"""Stage this step's engram embeddings to the device before the forward.
Decode stages the rows the prefetch gathered; prefill gathers every
prompt position's n-gram (carrying the tokens preceding a chunk); a dummy
atom/model_engine/scheduler.py:541
- [verified] This branch does not handle the object-with-dict shape that
config_declares_engram()explicitly supports. For a config object whosetext_configis a dict,getattr(nested, "engram_layer_ids", None)returnsNone, sonested_hasis false andtcbecomes the outer object; the following access totc.engram_max_ngram_sizeraisesAttributeErroreven though Engram was detected. Check dicts with.get()here and add the corresponding scheduler regression test.
nested_has = (
nested is not None
and getattr(nested, "engram_layer_ids", None) is not None
)
atom/model_ops/engram.py:807
- [verified]
drop_requestsremoves the rolling window while an earlier_runmay still be insideadvance()(the runner calls this beforestage_embeddingswaits for the future). If the worker acquires_window_lockafter this pop,advance()recreates the dropped sequence's window;cache.drop()prevents a stale embedding from being stored, but does not prevent this window resurrection. Finished requests can therefore leak into the LRU and eventually evict a live request's context. Synchronize the worker before clearing the window or make the worker discard windows for dropped IDs.
def drop_requests(self, seq_ids: list[int]) -> None:
for seq_id in seq_ids:
self.cache.drop(seq_id)
with self._window_lock:
for seq_id in seq_ids:
atom/model_ops/engram_layer.py:257
- This factory only accepts a local checkpoint directory (
open(model_path/config.json)and latermodel.safetensors.index.json), but the model-facing recipe passesconfig.model, which ATOM documents as either a model name or path and which the normal loader supports by downloading a Hugging Face repo. A remote Engram model therefore loads its config/weights normally and then fails here withFileNotFoundError; resolve the model name to the same local snapshot (and handle the single-file checkpoint case) before opening these files.
if hf_config is None:
with open(os.path.join(model_path, "config.json")) as fh:
hf_config = json.load(fh)
# Match config_declares_engram's contract: the engram block may sit in a
# nested text_config OR at the root. Use the nested block only when it
# actually carries the declaration; otherwise a text_config without engram
# fields would make from_hf return None and build the model with no engram.
text_config = hf_config.get("text_config")
if text_config is None or "engram_layer_ids" not in text_config:
text_config = hf_config
config = EngramConfig.from_hf(text_config)
recipes/Engram.md:71
- The PR's central overlap claim is documented here, but the description and changes provide no end-to-end latency/throughput comparison or prefetch/H2D wait measurements. Since
stage_embeddingsstill blocks when host work misses the window, the correctness tests do not establish that overlap improves serving or quantify its fallback cost; please add a reproducible benchmark with units and configuration, or remove the performance claim.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved integration, lifecycle, cleanup, configuration, and coverage findings remain.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (7)
atom/model_engine/model_runner.py:880
DeepseekV4ForCausalLMis the registered class for the advertised V4.1 architecture, but it has nobuild_engram_hostand its decoder never constructs or invokesEngramModules. Any V4.1 config withengram_layer_idstherefore takes this guard and raises before serving, so the new host/device path is unreachable for the target model. Please wire the modules intoatom/models/deepseek_v4.py(including checkpoint skip prefixes and residual injection), or narrow the PR's supported-model claim.
if not hasattr(model_class, "build_engram_host"):
raise NotImplementedError(
f"{model_class.__name__} config declares engram layers but the "
"model class does not implement build_engram_host; serving "
"would silently ignore the engram weights"
atom/model_engine/scheduler.py:1838
decode_batchis still constructed whenscheduled_seqsis empty, andEngineCoreskipsforward/_stage_engramfor that batch (engine_core.py:398-401, 412-433). This drains_engram_drop_pendingwithout callingdrop_requests; if the last request is finished or preempted, its cache/window remains and can evict live entries or be hit by a reused ID. Retain the pending IDs until a non-empty batch (or explicitly process them in the empty-batch path).
engram_dropped=self._drain_engram_dropped(),
atom/model_engine/scheduler.py:541
config_declares_engramaccepts an object whosetext_configis a dict, but this branch only checks attributes onnested. For such a config, detection returns true,nested_hasis false,tcremains the outer object, and line 544 raisesAttributeErrorinstead of reading the nested Engram settings. Keep this selection logic consistent with the helper's supported shapes.
nested = getattr(hc, "text_config", None)
nested_has = (
nested is not None
and getattr(nested, "engram_layer_ids", None) is not None
)
atom/model_ops/engram.py:807
drop_requestsremoves the rolling window before waiting for the in-flight worker. If that worker has not reachedadvance()yet, it recreates the dropped sequence's deque after this method returns; the cache tombstone prevents its embedding put but does not protect the window. A stale context can consume an LRU slot or be appended to when an ID is reused, corrupting the next n-gram. Wait for the current future before clearing the cache/window, or add an equivalent generation check in the worker.
def drop_requests(self, seq_ids: list[int]) -> None:
for seq_id in seq_ids:
self.cache.drop(seq_id)
with self._window_lock:
for seq_id in seq_ids:
atom/model_ops/engram_layer.py:352
max_num_tokensis the token-budget capacity (ModelRunnerpassesmax_num_batched_tokens), not the request-concurrency limit. The cache/window are keyed per sequence, so using this value overstates their bound (16,384 vs the default 512 sequences) and lets missed drops retain far more host memory than necessary; pass the actual max-sequence limit separately while keeping the token budget for staging buffers.
# Size the prefetch store to the runner's hard concurrency ceiling (a
# decode step stages <= max_num_tokens rows, one per sequence), so an
# active request is never LRU-evicted; the prefetcher scales the cache
# by the engram layer count internally. The bound is then only a leak
# net for a missed drop_requests.
EngramPrefetcher(
self.hash_mapping, self._tables, max_concurrent_seqs=max_num_tokens
atom/utils/envs.py:28
ATOM_ENGRAM_CACHE_DIRis a new user-facing configuration knob, but the centraldocs/environment_variables.mdreference has no entry for it. Users therefore cannot discover the override, its default, or what the cache contains. Add it to the environment-variable documentation.
# Directory for the engram compressed-vocab cache (.npz). Empty = default
# ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
tests/model_ops/test_engram.py:618
- The new tests cover hashing, table decoding, the worker, and the device op in isolation, but do not exercise the changed scheduler→ModelRunner lifecycle. In particular, an empty decode batch can consume
engram_dropped, graph-bucket padding controls the staged row count, and postprocess must prefetch only final chunks; these interactions can regress while this suite remains green. Add a non-GPU integration/contract test covering aScheduledBatchthrough the runner stub, including the empty-batch cleanup case.
def make_runtime() -> EngramHost:
pf = make_prefetcher()
cfg = pf._hash_mapping.config
return EngramHost(
pf,
max_num_tokens=8,
num_hash_heads=cfg.num_hash_heads,
head_dim=8,
device=torch.device("cpu"),
)
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved integration and correctness issues remain in the reviewed implementation.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (9)
atom/model_engine/model_runner.py:881
- With the current tree, a DeepSeek-V4.1 Engram checkpoint is not actually serveable:
atom/models/deepseek_v4.pyhas nobuild_engram_hostor Engram module/residual wiring, so this guard rejects the config before loading. The recipe documents that no in-tree model uses the library, but that conflicts with this PR's V4.1-support title and description; either include the model wiring (including the loader skip prefixes) or scope the PR/title explicitly as a library/runner contract.
if not hasattr(model_class, "build_engram_host"):
raise NotImplementedError(
f"{model_class.__name__} config declares engram layers but the "
"model class does not implement build_engram_host; serving "
"would silently ignore the engram weights"
)
atom/model_ops/engram.py:874
- This comment says a timeout falls back to inline recomputation, but
stage_embeddingsactually waits for a second interval and then raises if the worker is still pending (lines 1003-1024); it never recomputes on timeout. Please make the description match the fail-fast behavior so operators do not expect a fallback for a stalled host gather.
miss (an evicted or just-admitted row) inline. The result is identical either
way -- the prefetch only decides whether the work was already done, never the
answer.
atom/model_ops/engram.py:823
drop_requestscan remove the window while the single worker is between submission andadvance(). The worker then acquires_window_lock, recreates this sequence's window, and its cache writes are only suppressed by the drop marker; when the sequence ID is reused, that resurrected window becomes stale context for the first n-gram. Wait for the in-flight future before removing the window (or add an equivalent window tombstone).
return False
def drop_requests(self, seq_ids: list[int]) -> None:
atom/model_ops/engram.py:125
config_declares_engramtreats a declaration whose value isNoneas absent, but this check only tests key presence. For a config carrying the optionalengram_layer_ids: Nonefield,from_hfproceeds totuple(None)(and the documented “None when the model has no engram” contract is broken). Check the value withtext_config.get("engram_layer_ids") is Noneinstead.
if "engram_layer_ids" not in text_config:
atom/model_ops/engram.py:218
- The documented empty-value behavior is not honored when the variable is explicitly set to an empty string:
os.environ.get(...)returns"", soPath("")points at the current working directory instead of the default cache directory. This can writecompressed_vocab_*.npzinto whatever directory launched the server (or fail there) and defeats cache reuse. Treat an empty environment value as unset before constructing the path.
# `or` on each step so an unset OR explicitly-empty env var (envs.py
# registers a "" default) falls through to the real default rather than
# writing the cache into the process's cwd.
cache_dir = (
atom/model_ops/engram_layer.py:257
- The nested-config selection repeats the same key-presence test. If a root config declares Engram but its nested
text_configcontainsengram_layer_ids: None, this selects the nested block and disables Engram (or raises before thefrom_hffix), even though the helper's contract says that value is not a declaration. Select the nested block only when the value is non-None.
text_config = hf_config.get("text_config")
if text_config is None or "engram_layer_ids" not in text_config:
text_config = hf_config
config = EngramConfig.from_hf(text_config)
atom/model_ops/engram_layer.py:353
max_num_tokensis the per-step token budget, but it is passed here asmax_concurrent_seqs. Ifmax_num_seqsis larger thanmax_num_batched_tokens, the rolling-window LRU evicts still-active requests; the next miss then falls back to only the current token and hashes without prompt context, producing wrong Engram rows. Pass a separatemax_num_seqsbound for the prefetcher/window while retaining the token budget for staging capacity.
EngramPrefetcher(
self.hash_mapping, self._tables, max_concurrent_seqs=max_num_tokens
),
atom/model_ops/engram_layer.py:184
- This added path eagerly dequantizes the checkpoint's 32x32 FP8 WKV matrix to float32 and then stores it in a regular
nn.Linear(BF16 after.to(dtype)). For each V4.1 layer that expands roughly 157M values to about 315 MB on HBM and loses the FP8 block-scaled GEMM path used elsewhere in ATOM, which undermines the stated FP8 device path and can materially hurt decode bandwidth. Use an appropriate quantized/AITER linear path or provide measured justification for this tradeoff.
wkv_vals = decode_fp8(wkv, torch.float32)
wkv = (
wkv_vals.reshape(rows // block, block, cols // block, block)
* ws.reshape(rows // block, 1, cols // block, 1)
).reshape(rows, cols)
atom/model_ops/engram_layer.py:139
- The tests cover shapes, errors, and weight dequantization, but none checks the numerical result of this gate against an independent reference. A projection split/transposition, RMS normalization, or signed-square-root sigmoid regression would still pass all current
EngramOptests while producing incorrect model outputs. Add a deterministic small-dimension value test for the fused WKV/gate calculation.
gate = (key * query).sum(dim=-1) / math.sqrt(self.hidden_size)
# Signed square root before the sigmoid: keeps the gate responsive for
# small scores without letting large ones saturate it.
gate = gate.abs().clamp_min(1e-6).sqrt() * gate.sign()
return gate.sigmoid().unsqueeze(-1) * value.unsqueeze(-2)
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate correctness, integration, lifecycle, overlap, and documentation issues remain.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (9)
atom/model_engine/model_runner.py:3346
- The new per-forward staging hook covers decode, chunked prefill, dropped requests, padding, and CUDAGraph warmup/capture, but the added tests only exercise the host objects and the device op in isolation. A batch-shape or stream-order regression in
forward()would therefore pass the suite without being detected. Add a mocked ModelRunner/scheduler integration test that drives at least decode, final/middle prefill, and a padded capture batch.
if getattr(self, "engram", None) is not None:
self._stage_engram(batch)
atom/model_engine/model_runner.py:3234
- This prefetch is launched only after
run_modeland sampling have finished enqueueing the current step, and_stage_engramwaits for its future before the next forward. Since speculative/drafter and pipeline paths are rejected above, there is no GPU work after this call that can hide the host hash/gather; the H2D is also ordered before compute. The advertised GPU overlap therefore does not occur, and decode still pays the lookup latency.
# Prefetch on the just-sampled token (the NEXT step's input), not the
# deferred one, which would prefetch a step stale. `prefetch_next` gathers
# its embedding on the worker overlapping postprocess, and copies the
# token to the host async, so this thread pays no D2H sync.
atom/model_engine/model_runner.py:880
- This fail-closed branch makes the advertised DeepSeek-V4.1 path unusable:
DeepseekV4ForCausalLMdoes not definebuild_engram_host, so an Engram-configured checkpoint is rejected before construction, and the model also has no Engram forward or loader integration. Either add the model-side hooks and skip/load the large host-resident tensors, or narrow the PR description/title to infrastructure-only support instead of claiming V4.1 support.
if not hasattr(model_class, "build_engram_host"):
raise NotImplementedError(
f"{model_class.__name__} config declares engram layers but the "
"model class does not implement build_engram_host; serving "
"would silently ignore the engram weights"
atom/model_engine/scheduler.py:1839
- This drains
engram_droppedeven whenscheduled_seqsis empty.EngineCoreskipsrunner.forwardfor that batch whenreq_idsis empty, so the IDs are discarded without callingdrop_requests(); a prefetch cache/window for a preempted request can then survive and contaminate a later reused ID. Preserve the pending IDs until a non-empty batch carries them, or explicitly process the drop on the empty-batch path.
engram_dropped=self._drain_engram_dropped(),
)
atom/model_engine/scheduler.py:541
- When
hf_configis an object whosetext_configis a dict,config_declares_engram()returns true (the new host tests explicitly support this shape), but this branch only probes object attributes.nested_hasbecomes false,tcremains the outer object, and the followingtc.engram_max_ngram_sizeaccess raises instead of reading the nested dict. Handle dict-valued nested configs here as well.
nested = getattr(hc, "text_config", None)
nested_has = (
nested is not None
and getattr(nested, "engram_layer_ids", None) is not None
)
atom/model_ops/engram.py:828
drop_requests()removes the rolling window without first joining the worker. If a prefetch is in flight,_run()can executeadvance()after this removal and recreate the window with the old request's context; when the request ID is reused, the next lookup hashes the wrong n-gram. The cache drop mark prevents stale embeddings from being inserted, but it does not protect_window. Wait for the in-flight future before removing the window, or add a generation check so stale work cannot mutate a reused ID.
def drop_requests(self, seq_ids: list[int]) -> None:
for seq_id in seq_ids:
self.cache.drop(seq_id)
with self._window_lock:
for seq_id in seq_ids:
self._window.pop(seq_id, None)
atom/model_ops/engram_layer.py:135
- The normalized key/query tensors are cast back to the model dtype before the dot product. On the bf16 path this makes the reduction and signed-square-root gate run in bf16, whereas the Engram reference keeps the normalization, elementwise product, reduction, and gate in fp32; the resulting gate error can change which memory values are admitted. Keep the gate math in fp32 and cast only the final residual delta to
hidden_states.dtype.
key = _rms_norm(keys, self.k_weight, self.norm_eps)
query = _rms_norm(hidden_states, self.q_weight, self.norm_eps)
gate = (key * query).sum(dim=-1) / math.sqrt(self.hidden_size)
atom/utils/envs.py:28
- The PR adds
ATOM_ENGRAM_CACHE_DIRto the centralized environment registry, but it is absent fromdocs/environment_variables.md, which the installation docs identify as the full list of runtime variables. Users cannot discover the cache override or its default from the documented configuration surface. Add it to that table with its type, default, and behavior.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
recipes/Engram.md:72
- The recipe presents the host lookup as off the critical path, but the PR provides only value-equality logs and no latency/throughput measurements or reproducible workload.
stage_embeddings()still waits for the worker when the prefetch misses its overlap window, so this claim is workload-dependent. Add steady-state decode and prefill measurements against the corresponding non-Engram path, including the host/cache conditions used.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
double-buffered (a per-layer pinned ring) so the next step's fill never races the
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved model integration, FP8 portability, lifecycle correctness, and test coverage issues block approval.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (8)
atom/model_engine/model_runner.py:881
- DeepSeek-V4.1 is not actually runnable from this tree:
DeepseekV4ForCausalLMhas nobuild_engram_host, and itsDeepseekV4Modelnever injects anEngramOpinto the layer loop, so a config containing the advertisedengram_*block always raises here before loading. The result is a fail-closed rejection rather than host Engram support. Add the model-side attachment/weight-skip/residual-injection path, or narrow the PR description to standalone plumbing and track model integration explicitly.
if not hasattr(model_class, "build_engram_host"):
raise NotImplementedError(
f"{model_class.__name__} config declares engram layers but the "
"model class does not implement build_engram_host; serving "
"would silently ignore the engram weights"
)
atom/model_engine/model_runner.py:950
- These new runner/scheduler hooks have no integration test with a model exposing
build_engram_host: the added tests exerciseEngramHostandEngramOpin isolation, so row ordering,engram_droppedcleanup, and CUDAGraph staging can regress while the suite stays green. Add a runner-level stub/integration test covering one decode prefetch, a chunked prefill, and request-id reuse.
def _stage_engram(self, batch: ScheduledBatch) -> None:
"""Stage this step's engram embeddings to the device before the forward.
Decode stages the rows the prefetch gathered; prefill gathers every
prompt position's n-gram (carrying the tokens preceding a chunk); a dummy
warmup/capture pass stages zeros.
atom/model_engine/model_runner.py:987
- When prefix caching serves the entire prompt, the scheduler skips prefill and sends this request directly to decode. In that path no
stage_prefill(..., final_mask=...)call seeds the rolling window, sostage_embeddingsfalls back to the one decode anchor and hashes[pad, ..., anchor]instead of the prompt's trailingngram-1tokens. The first decode Engram contribution is therefore wrong for full prefix-cache hits; carry the prompt tail into the first decode batch (or seed it before staging) whenever no local prefill ran.
# `tokens` recomputes a prefetch miss only for a missed row with no
# window (a cold/just-admitted row, where scheduled_tokens is its real
# anchor); a carried-over row uses its window, so its placeholder is never
# read.
tokens = batch.scheduled_tokens[: len(seq_ids)].astype(np.int64).reshape(-1, 1)
atom/model_engine/scheduler.py:1838
- The main scheduler can return an empty
ScheduledBatchwhen no decode sequence was selected, andEngineCorethen skips the forward (and therefore_stage_engram). Draining_engram_drop_pendinghere loses those IDs, leaving their cache/window state alive; a later reused request ID can consume stale embeddings. Keep the pending IDs whenscheduled_seqsis empty, as the prefill construction above already does, or explicitly process drops for empty batches.
engram_dropped=self._drain_engram_dropped(),
atom/model_engine/scheduler.py:541
config_declares_engramexplicitly accepts an object whose nestedtext_configis a dict (the new test covers_Obj(text_config={...})), but this branch only checks attributes onnested. For that supported shapenested_hasis false, sotcbecomes the root object and the next access totc.engram_max_ngram_sizefails instead of reading the nested block. Mirror the dict check used byconfig_declares_engram.
nested = getattr(hc, "text_config", None)
nested_has = (
nested is not None
and getattr(nested, "engram_layer_ids", None) is not None
)
atom/model_ops/engram.py:604
- The PR's purpose is to hide the host lookup behind GPU work, but the supplied equality logs only establish correctness; there are no steady-state latency/throughput numbers or a reproducible V4.1 GPU run showing that
stage_embeddingsdoes not routinely wait for the worker on the miss path. Add decode/prefill measurements with GPU, TP, checkpoint, and command details before treating the lookup as off the critical path.
class EngramPrefetcher:
"""Runs hash + host gather for the next step while the GPU works on this one.
One worker thread, not a thread per step: the work is numpy and torch gather
which release the GIL, and an unbounded thread-per-step (as in the reference)
both races with its own consumer and contends with the runner for the GIL.
`submit_compute` returns a Future so the consumer waits for the queued
prefetch before reading the cache; a genuine cache miss (an evicted or
just-admitted row) is then recomputed inline. The prefetch only decides
whether the work was already done, never the answer -- it is not a bounded
no-stall policy: `stage_embeddings` waits for the worker.
atom/model_ops/engram.py:828
drop_requestsremoves the rolling window without synchronizing with the worker. An in-flight_runcan executeadvance()after this pop, recreate the dropped sequence's window, and leave stale context behind; if the sequence id is reused,windows_for_recomputecan then hash that stale context instead of the new request's token. Wait for the current prefetch before marking/removing dropped sequences, or otherwise serializedrop_requestswithadvance().
def drop_requests(self, seq_ids: list[int]) -> None:
for seq_id in seq_ids:
self.cache.drop(seq_id)
with self._window_lock:
for seq_id in seq_ids:
self._window.pop(seq_id, None)
atom/utils/envs.py:28
ATOM_ENGRAM_CACHE_DIRis a new user-facingATOM_*setting, but it is absent fromdocs/environment_variables.md, whose header says that file documents the project's environment variables. Users therefore cannot discover how to relocate the compressed-vocabulary cache or its default path. Add the variable, default, and behavior to the public table.
# Directory for the engram compressed-vocab cache (.npz). Empty = default
# ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved correctness and integration issues remain in model wiring, prefetch lifecycle, and validation coverage.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (8)
atom/model_engine/model_runner.py:3241
- Every decode row is prefetched here, including rows whose sampled token immediately ends the request (EOS, a stop sequence, or the max-token limit). The scheduler only queues those ids for
engram_droppedon the following batch, so a terminal request still performs the very large random host-table gather for an embedding that will never be consumed; if the server goes idle, that work and its cached window also remain until another batch arrives. Filter terminal rows using the same stop decision as the scheduler before submitting the prefetch.
self.engram.prefetch_next(req_ids, sampled_tokens)
atom/model_engine/model_runner.py:1001
- The new runner/scheduler path is covered only by host-level unit tests; there is no test that drives
_stage_engramwith aScheduledBatchthrough a model forward, including the padded CUDAGraph row count and the asynchronous H2D/compute-stream handoff. A row-order or stale-buffer error can therefore passtests/model_ops/test_engram.pywhile failing or returning wrong values in serving. Author must add a runner/model integration test covering decode and chunked prefill (with mocked GPU primitives if needed).
padded_rows = len(seq_ids)
forward_mode = get_forward_context().context.forward_mode
if forward_mode is not None and forward_mode.use_cudagraph:
padded_rows = max(padded_rows, int(forward_mode.running_tokens))
self.engram.stage_embeddings(seq_ids, tokens, padded_rows=padded_rows)
atom/model_engine/model_runner.py:876
- [verified]
support_model_arch_dictresolvesDeepseekV4ForCausalLMtoatom.models.deepseek_v4, but no in-tree model definesbuild_engram_hostor attachesEngramModules. A V4.1 config withengram_layer_idstherefore reaches this guard and raises instead of serving the model, so the advertised DeepSeek-V4.1 path is not usable. Author must add the registered model's Engram wiring (including residual injection and loader exclusions) or scope this change explicitly as a library-only follow-up.
if not hasattr(model_class, "build_engram_host"):
atom/model_ops/engram.py:843
drop_requestscan race with the worker:cache.dropsuppresses the staleput, but the worker may still enteradvance()after this window is popped and recreate the sequence's rolling context. That stale window can consume the bounded window store, evict a live request's context, and make a later miss hash with the wrong history. Drain/wait for the in-flight prefetch before clearing the cache and windows.
self._inflight = self._pool.submit(_run)
return self._inflight
atom/model_ops/engram.py:152
engram_kernel_sizeis parsed intoEngramConfig, but no code consumeskernel_sizeandEngramOphas no short-convolution path. A checkpoint/config that declares a non-default kernel therefore appears supported while silently omitting part of the module. Remove this field for the V4.1-only implementation or reject configurations that require the unsupported convolution.
kernel_size=int(text_config.get("engram_kernel_size", 4)),
recipes/Engram.md:116
- This integration example puts the host lookup inside
DeepseekV4Model.forward, but that class is@support_torch_compile-decorated (atom/models/deepseek_v4.py:4290). The mutable host reference and dynamicembeddings()slice would be traced in the compiled boundary, contrary to the repository's compile-boundary rule and liable to stale/capture-time behavior. Author must revise the recipe to specify a supported non-compiled integration boundary and how staged embeddings are passed into the compiled model.
```python
for layer_id, layer in enumerate(self.layers):
if self.engram is not None and layer_id in self.engram:
if self.engram_host is None:
raise RuntimeError(
tests/model_ops/test_engram.py:195
- [verified] This supposedly independent scalar oracle takes
layer_multipliers,head_vocab_sizes, andhead_offsetsdirectly frommapping, so a wrong reference seed/RNG or prime assignment is reproduced by both the implementation and the test. The test can catch vectorization errors, but it cannot validate the claimed bit-exact reference mapping. Author must add fixed expected hashes/rows from the reference implementation, including at least one known multiplier/hash vector.
)
mult = mapping.layer_multipliers[layer_id]
sizes = mapping.head_vocab_sizes[layer_id]
tests/model_ops/test_engram.py:396
- [verified]
test_engram_op_and_checkpointchecks only shapes and exceptions; it never compares the gate, signed square-root, projection split, or value contribution against an independent numerical reference. A wrong Engram formula would therefore pass this suite while changing every model output. Author must add a fixed-weight numerical test against a hand-computed or reference implementation.
rt.stage_prefill([81], [np.arange(9)], [np.array([], dtype=np.int64)])
rt.shutdown()
def test_engram_op_and_checkpoint():
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
| environment_variables: dict[str, Callable[[], Any]] = { | ||
| # Directory for the engram compressed-vocab cache (.npz). Empty = default | ||
| # ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build. | ||
| "ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""), |
| The decode gather is prefetched a step early on a worker thread and its small | ||
| result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within | ||
| the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are | ||
| double-buffered (a per-layer pinned ring) so the next step's fill never races the | ||
| previous step's in-flight H2D, while the device buffer stays fixed for CUDAGraph. |
There was a problem hiding this comment.
🔵 Needs a closer look
Multiple unresolved correctness, integration, lifecycle, startup, testing, and performance risks require author changes and human review.
Review details
Suppressed comments (12)
atom/model_engine/model_runner.py:880
- [verified]
DeepseekV4ForCausalLMis the registered V4 implementation, but it does not exposebuild_engram_host(the recipe also states that no in-tree model wires the contract), so an actual V4.1 config reaches this guard and raises before serving any request. The advertised DeepSeek-V4.1 path is therefore not runnable. Author must wire the Engram modules, skipped table prefixes, and residual injection into the V4 model, or explicitly scope/rename this PR as library plumbing.
if not hasattr(model_class, "build_engram_host"):
raise NotImplementedError(
f"{model_class.__name__} config declares engram layers but the "
"model class does not implement build_engram_host; serving "
"would silently ignore the engram weights"
atom/model_engine/model_runner.py:3348
- [verified] The new runner staging/prefetch path is covered only by the host/device unit tests; there is no test that drives
ModelRunner.forward()through prefill, decode, CUDAGraph padding, and pending drops. A mismatch between scheduler batch rows and the staged Engram rows can therefore reach production as a crash or silent residual misalignment. Author must add a mocked integration test covering both prefill and decode hook ordering.
if getattr(self, "engram", None) is not None:
self._stage_engram(batch)
logits, hidden_states = self.run_model(input_ids, batch)
atom/model_engine/scheduler.py:1822
- This drains the pending drop IDs even when
scheduled_seqsis empty.EngineCoreskips the forward and postprocess for that batch whenlen(req_ids) == 0, so_stage_engramnever consumesengram_dropped; the finished request's cache/window can then survive and a reused ID may read stale embeddings. Preserve the list until a non-empty batch is actually forwarded.
engram_dropped=self._drain_engram_dropped(),
atom/model_engine/scheduler.py:288
- The new
prefill_contextslicing is correctness-critical for chunked/prefix-cached prompts, but the PR adds no scheduler or runner integration test for a nonzeronum_cached_tokens; the existing Engram tests callstage_prefilldirectly with hand-built contexts. An offset error here would pad the first n-grams and silently change the lookup rows. Author must add a scheduler-level test covering a cached middle chunk and verify the resulting staged embeddings/window.
self.prefill_context: list[list[int]] | None = None
if engram_ngram is not None and engram_ngram > 1:
k = engram_ngram - 1
self.prefill_context = [
(
atom/model_ops/engram.py:859
drop_requestsremoves the rolling window without first joining the worker. A prefetch that is already in_runcan executeadvance()after this removal, recreate the dropped request's window, and evict a live request when the window reaches capacity; the next prefetch for that live request then hashes with missing context. Wait for the in-flight future before removing the cache entry/window so the drop is atomic with respect to the worker.
def drop_requests(self, seq_ids: list[int]) -> None:
for seq_id in seq_ids:
self.cache.drop(seq_id)
with self._window_lock:
for seq_id in seq_ids:
self._window.pop(seq_id, None)
atom/model_ops/engram.py:289
- For the V4.1 config this loop runs once per raw tokenizer id (up to 16,000,000) and performs a separate
decodecall for each one. On a cold cache every serving rank therefore pays millions of Python/tokenizer calls before the model can start, far beyond the tens-of-seconds behavior of the 129k-token demo. Author must provide a precomputed compressed-vocabulary artifact or a genuinely batched/native build path, and measure cold-start cost for the 16M vocabulary.
key_to_new: dict[str, int] = {}
lookup = np.empty(vocab_size, dtype=np.int64)
next_id = 0
for tid in range(vocab_size):
text = self._tokenizer.decode([tid], skip_special_tokens=False)
atom/model_ops/engram.py:210
- Even on a cache hit,
_cache_key()serializes the entire fast tokenizer withbackend_tokenizer.to_str()before it can locate the.npz. With the V4.1 raw vocabulary this creates a large transient string on every rank and makes the cache lookup itself scale with the full tokenizer. Author must use a stable file/config fingerprint (or otherwise bound the serialization cost) and verify the memory/startup impact on a real V4.1 tokenizer.
serialized = (
backend.to_str()
if backend is not None and hasattr(backend, "to_str")
else None
)
atom/model_ops/engram_layer.py:304
EngramModules.from_checkpoint()is the production path that resolves the six per-layer tensor names, maps the ~98 GB table, and applies its scale layout, but the added tests only callEngramOp.load_checkpoint_weights()with in-memory tensors. A wrongweight_mapkey, shard handle lifetime, or table/scale shape would therefore survive CI until model startup. Author must add a minimal safetensors fixture that exercisesfrom_checkpoint()andbuild_engram_host()end to end.
ops: dict[int, EngramOp] = {}
tables: dict[int, HostEmbeddingTable] = {}
for layer_id, num_rows in zip(config.layer_ids, config.num_embeddings):
tables[layer_id] = HostEmbeddingTable(
_get(_EMBED.format(layer_id)),
num_rows=num_rows,
head_dim=config.head_dim,
scale=_get(_EMBED_SCALE.format(layer_id)),
)
atom/utils/envs.py:28
ATOM_ENGRAM_CACHE_DIRis a new user-facing environment variable, but it is not listed indocs/environment_variables.md(the recipe only describes the cache behavior, not this override). Users cannot discover how to place the large generated cache on suitable storage. Author must add the variable, default, and purpose to the environment-variable documentation.
# Directory for the engram compressed-vocab cache (.npz). Empty = default
# ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
recipes/Engram.md:72
- The central overlap claim is not backed by a latency/throughput measurement or a reproducible V4.1 workload. The implementation explicitly waits for the worker when the gather misses the step window, so this can become a synchronous host stall rather than an off-critical-path lookup. Author must add steady-state measurements (GPU, ROCm, model, batch/concurrency, and cold/warm cache) showing the overlap benefit and fallback cost.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
double-buffered (a per-layer pinned ring) so the next step's fill never races the
tests/model_ops/test_engram.py:400
- This only asserts output shapes for
EngramOp.forward; the key/value split, RMS-normalization, signed-root gate, and value broadcast can all be wrong while these assertions still pass. Because this is the device contribution that changes model logits, an independent deterministic reference is needed to catch numerical regressions. Author must add a fixed-weight CPU reference comparison for the forward formula.
def test_engram_op_and_checkpoint():
op = make_op()
# ATOM residual streams are [num_tokens, hc, dim], with optional leading dims.
assert op(torch.randn(5, 2, 16), torch.randn(5, 24)).shape == (5, 2, 16)
assert op(torch.randn(2, 3, 2, 16), torch.randn(2, 3, 24)).shape == (2, 3, 2, 16)
tests/model_ops/test_engram.py:186
- [verified] This "independent scalar reference" reads
layer_multipliers,head_vocab_sizes, andpad_iddirectly frommapping, so a wrong seed/stride, bucket layout, or pad mapping makes both implementations agree. The suite therefore does not verify the hash constants against the DeepSeek/reference vectors. Author must add fixed reference hash/row vectors from the upstream algorithm.
def test_hash_layer_matches_scalar_reference():
"""`hash_layer` + `to_row_indices` must match an independent scalar reimpl
(same constants, different code), catching vectorization/head/left-pad bugs."""
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Critical staging and computation issues, plus lifecycle cleanup and coverage gaps, remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (9)
atom/model_engine/model_runner.py:3347
- The new per-forward staging hook is in Tier-1
ModelRunner, but the added tests exercise only the host/op library; no scheduler/runner test covers prefill/decode ordering, finished-request drops, or padded CUDAGraph rows. A regression here can leave stale embeddings or fail only under capture. Add a CPU/source-level lifecycle test or an integration fixture for these paths.
if getattr(self, "engram", None) is not None:
self._stage_engram(batch)
atom/model_engine/scheduler.py:1822
_drain_engram_dropped()is unconditional here, butEngineCoreskipsrunner.forwardwhenscheduled_batch.req_idsis empty (atom/model_engine/engine_core.py:412-433). An idle tick therefore clears these IDs from_engram_drop_pendingwithout callingEngramHost.drop_requests(), leaving finished/preempted requests in the host cache/window until eviction and defeating stale-state cleanup. Drain only whenscheduled_seqsis non-empty, as the prefill branch already does, or route cleanup through an empty-batch path.
engram_dropped=self._drain_engram_dropped(),
atom/model_ops/engram.py:982
- This token D2H host buffer is created through
torch.empty()without a device while the runner's default device is CUDA during Engram host construction (model_runner.py:690). It therefore attempts to pin a CUDA tensor instead of allocating CPU-pinned storage, soEngramHostinitialization can fail before any request runs. Create the tensor withdevice="cpu"before pinning it.
self._token_host = [
torch.empty(max_num_tokens, dtype=torch.int64).pin_memory()
for _ in range(self._n_token_slots)
atom/model_ops/engram.py:60
engram_layer_idsis also defined as an empty tuple/list by ordinary DeepSeek-V4.1 config classes when Engram is disabled, but this predicate treats any non-Nonevalue—including[]—as an active declaration. That makes the runner reject non-Engram configs (and can cause a root wrapper's empty defaults to shadow a nested text config that actually contains Engram). Treat an empty layer list as absent here and keepEngramConfig.from_hfconsistent with that behavior.
return got is not None
atom/model_ops/engram.py:860
drop_requestscan remove the window while the single worker is still between submitting and enteringadvance(). In that interleaving,advance()recreates the dropped request's window after this method returns; if the sequence ID is reused, the new request inherits the stale context and hashes the wrong n-grams. Synchronize with the in-flight prefetch before removing the window, or add a generation/tombstone check under_window_lock.
for seq_id in seq_ids:
self.cache.drop(seq_id)
with self._window_lock:
for seq_id in seq_ids:
self._window.pop(seq_id, None)
atom/model_ops/engram_layer.py:140
- The new forward path is only tested for shapes and parameter dimensions; no assertion checks the signed-square-root sigmoid gate, per-branch RMS normalization, or shared-value broadcast against a hand-computed result. A transposed gain or incorrect gate formula would therefore pass the tests while producing incorrect logits. Add a small numerical reference case for this core computation.
gate = (key * query).sum(dim=-1) / math.sqrt(self.hidden_size)
# Signed square root before the sigmoid: keeps the gate responsive for
# small scores without letting large ones saturate it.
gate = gate.abs().clamp_min(1e-6).sqrt() * gate.sign()
return gate.sigmoid().unsqueeze(-1) * value.unsqueeze(-2)
atom/model_ops/engram_layer.py:185
- The checkpoint test exercises
load_checkpoint_weightswith float32 all-ones weights, so it never enters the actual raw-uint8 FP8 decode path used by the V4.1 checkpoint, nor does it validate the 32x32 block layout with nontrivial values. A byte reinterpretation or row/column scale error in this loader would still pass. Add a raw FP8/E8M0 fixture with distinct values and scales.
wkv_vals = decode_fp8(wkv, torch.float32)
wkv = (
wkv_vals.reshape(rows // block, block, cols // block, block)
* ws.reshape(rows // block, 1, cols // block, 1)
).reshape(rows, cols)
atom/utils/envs.py:28
- This adds a user-facing
ATOM_ENGRAM_CACHE_DIR, but the repository's environment-variable reference (docs/environment_variables.md) lists the supportedATOM_*settings and has no entry for it. Without documenting the override and its default, operators cannot discover how to place the compressed-vocab cache on a shared/local filesystem.
environment_variables: dict[str, Callable[[], Any]] = {
# Directory for the engram compressed-vocab cache (.npz). Empty = default
# ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
recipes/Engram.md:73
- The recipe claims the small H2D transfer overlaps GPU work, but
_embeddings_h2dfirst makescopy_streamwait for the compute stream andwait_for_embeddings()then makes compute wait for the copy event. This serializes the transfer between the previous and next forwards; only the host lookup can overlap prior work, so H2D remains on the critical path despite the documented claim. Either redesign the handoff with another safe device buffer or revise the claim and add measurements showing the actual overlap.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
double-buffered (a per-layer pinned ring) so the next step's fill never races the
previous step's in-flight H2D, while the device buffer stays fixed for CUDAGraph.
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
…sh oracle test Also address several review passes: - decode E8M0 code 0 as an exact zero on the raw-uint8 fallback (v4_quant's zero-scale sentinel) so an all-zero block stays zero; - make the decode timeout synchronize with the worker instead of recomputing inline from a rolling window the worker may not have advanced yet; - decode raw-uint8 E4M3 table rows and wkv weights as fp8 (reinterpret) rather than reading each byte as an integer 0..255; - config_declares_engram: probe both the root and a nested text_config, for a config object or a raw config.json dict; normalize the same dict shape in the scheduler before reading engram_max_ngram_size; - size the prefetch cache / rolling-window store to the runner's concurrency ceiling so an active request's window is never LRU-evicted; - fail closed before constructing/loading the model when the model class lacks build_engram_host, instead of after mapping ~200 GB of tables; - skip the compressed-vocab disk cache for a slow tokenizer (no fast- serialization fingerprint) rather than key it on the vocab map alone; - drain the outstanding prefetch before admitting a reused seq id and before reseeding a resumed request's window, so a stale worker cannot insert rows or mix the n-gram window.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved scheduler/cache lifecycle, synchronization, validation, performance, and documentation issues remain.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (5)
atom/model_engine/model_runner.py:3347
- [verified] The PR adds a Tier-1 scheduler/runner contract spanning
prefill_context, dropped-request delivery, CUDAGraph bucket padding, and two CUDA side streams, but the new tests cover only CPU hashing/gathering and the device op; they do not exercise a scheduler-to-runner prefill/decode transition, an empty batch with pending drops, or stream/event ordering. A row-order or synchronization regression can therefore silently corrupt Engram output despite the unit suite passing. Author must add an integration-level regression covering these runner hooks, including the CUDAGraph/padded decode case.
self._stage_engram(batch)
atom/model_engine/scheduler.py:1822
- [verified] The decode scheduler drains
_engram_drop_pendingeven whenscheduled_seqsis empty, butEngineCoreskipsrunner.forward()for that batch, so_stage_engram()never receives these IDs. A request preempted or finished on an otherwise idle step therefore retains its cache/window; later LRU pressure can evict an active rolling window and make the next multi-token lookup use incomplete context. Author must only drain this list when a batch will actually be forwarded and add an empty-batch regression test.
engram_dropped=self._drain_engram_dropped(),
atom/model_ops/engram.py:860
drop_requestsremoves the cache entry/sets its drop mark before removing the rolling window, but the worker can already be queued and acquire_window_lockafter this method releases it. In that interleavingadvance()recreates the dropped sequence's window; its subsequentcache.put()is suppressed, but the stale window remains and can be used if the sequence ID is reused (for example on a decode-only/prefix-cache admission). Drain or otherwise invalidate the in-flight prefetch before removing the window so a dropped request cannot resurrect host context.
def drop_requests(self, seq_ids: list[int]) -> None:
for seq_id in seq_ids:
self.cache.drop(seq_id)
with self._window_lock:
for seq_id in seq_ids:
self._window.pop(seq_id, None)
atom/model_ops/engram_layer.py:149
- [verified]
test_engram_op_and_checkpointchecks only output shapes, dtype, and checkpoint tensor dequantization; it never compares the device result with an independent numerical reference. A wrong key/value split, RMS-normalization epsilon, signed-square-root gate, or sigmoid can therefore pass the new suite while changing every Engram contribution. Author must add a deterministic reference calculation with nontrivial weights and inputs covering the gate and residual-dtype cast.
gate = gate.abs().clamp_min(1e-6).sqrt() * gate.sign()
contribution = gate.sigmoid().unsqueeze(-1) * value.float().unsqueeze(-2)
return contribution.to(hidden_states.dtype)
atom/utils/envs.py:28
- [verified] This adds the
ATOM_ENGRAM_CACHE_DIRruntime override, but the centraldocs/environment_variables.mdsays it is the full discoverable list and does not mention this variable. Operators cannot find the supported way to move the compressed-vocabulary cache off the default home directory. Author must add the variable, default, and cache-path behavior to the environment reference.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Lite
| # the same, so blocking is never worse here and is correct when the worker | ||
| # had simply not started; only a worker stuck well past the budget is a | ||
| # real fault worth failing on rather than serving a stale window. | ||
| if not self.prefetcher.wait(timeout=self._PREFETCH_WAIT_S): |
|
close for #2314 finished merged |
Refs #168, which prototyped the overlap on a standalone demo model.
engram: host-side n-gram lookup for DeepSeek-V4.1, prefetched and overlapped
DeepSeek-V4.1-Flash puts Engram modules on layers 1 and 14, backed by tables that cannot live in HBM: 384,006,168 and 384,016,682 rows of 256 fp8 values, about 98 GB each, 196B parameters in total. The lookup has to run on the host and it
has to be off the critical path, so this adds the host path, the device modules, and the runner hooks that hide the work behind the previous step's GPU time.
CPU precompute vs Engram recompute:
engram_demo_v1.py)Linear+ 1Linearwkv[25600, 6144]nn.Embedding)wkvEngram host-overlap (DeepSeek-V4.1)