Skip to content

engram: host-side n-gram lookup for DeepSeek-V4.1, prefetched and overlapped - #2185

Closed
ZhangLirong-amd wants to merge 22 commits into
mainfrom
lirzhang/engram-cpu-overlap
Closed

ZhangLirong-amd wants to merge 22 commits into
mainfrom
lirzhang/engram-cpu-overlap

Conversation

@ZhangLirong-amd

@ZhangLirong-amd ZhangLirong-amd commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #168, which prototyped the overlap on a standalone demo model.

engram: host-side n-gram lookup for DeepSeek-V4.1, prefetched and overlapped

DeepSeek-V4.1-Flash puts Engram modules on layers 1 and 14, backed by tables that cannot live in HBM: 384,006,168 and 384,016,682 rows of 256 fp8 values, about 98 GB each, 196B parameters in total. The lookup has to run on the host and it
has to be off the critical path, so this adds the host path, the device modules, and the runner hooks that hide the work behind the previous step's GPU time.

CPU precompute vs Engram recompute:

[atom.model_ops.engram_host 03:18:00] engram VERIFY VALUES step 1 seq=0 layer=1 token=32913 equal=True prefetched[:6]=[0.0703, -0.4375, 0.1875, -0.0049, -0.2344, -0.1719] recomputed[:6]=[0.0703, -0.4375, 0.1875, -0.0049, -0.2344, -0.1719]
[atom.model_ops.engram_host 03:18:00] engram VERIFY step 1: 0/2 rows mismatch, 0 wrong-token
[atom.model_ops.engram_host 03:18:00] engram VERIFY VALUES step 1 seq=0 layer=1 token=32913 equal=True prefetched[:6]=[0.0703, -0.4375, 0.1875, -0.0049, -0.2344, -0.1719] recomputed[:6]=[0.0703, -0.4375, 0.1875, -0.0049, -0.2344, -0.1719]
[atom.model_ops.engram_host 03:18:00] engram VERIFY step 1: 0/2 rows mismatch, 0 wrong-token
[atom.model_ops.engram_host 03:18:00] engram VERIFY VALUES step 1 seq=0 layer=1 token=32913 equal=True prefetched[:6]=[0.0703, -0.4375, 0.1875, -0.0049, -0.2344, -0.1719] recomputed[:6]=[0.0703, -0.4375, 0.1875, -0.0049, -0.2344, -0.1719]
[atom.model_ops.engram_host 03:18:00] engram VERIFY step 1: 0/2 rows mismatch, 0 wrong-token
[atom.model_ops.engram_host 03:18:00] engram VERIFY VALUES step 1 seq=0 layer=1 token=32913 equal=True prefetched[:6]=[0.0703, -0.4375, 0.1875, -0.0049, -0.2344, -0.1719] recomputed[:6]=[0.0703, -0.4375, 0.1875, -0.0049, -0.2344, -0.1719]
[atom.model_ops.engram_host 03:18:00] engram VERIFY step 1: 0/2 rows mismatch, 0 wrong-token

Paper demo (engram_demo_v1.py) DeepSeek-V4.1-Flash
Rows per hash head 129,280 × 5 ≈ 646k 16,000,000
N-gram orders max 3 → {2, 3} → 16 heads max 4 → {2, 3, 4} → 24 heads
Embedding dim per head 512 / 8 = 64 256
Rows per layer ≈ 10.3M 384,006,168 (layer 1) / 384,016,682 (layer 14)
Table size per layer ≈ 2.6 GB (fp32) 98.3 GB (fp8 e4m3 + e8m0 block scales)
Total engram parameters ≈ 1.3B 196B across both modules
Layer placement [1, 15] [1, 14]
Short causal conv present omitted (tech report §2.4.2)
Key/value projections 4 separate Linear + 1 Linear one fused wkv [25600, 6144]
Quantization none (fp32 nn.Embedding) fp8 throughout, 32-value blocks on the table, 32×32 on wkv

Engram host-overlap (DeepSeek-V4.1)

 step N      ┌─ model_runner ─────────────────────────────────────────┐
 (overlap    │  sampled token ─► CPU ─► hash ─► gather embedding       │
  postproc)  │                            └─ staging buffer ─► H2D ───┐ │
             └─────────────────────────────────────────────────────── │─┘
                                                                       ▼
 step N+1    ┌─ Engram op  (layers 1 & 14) ─────────────────────────────┐
 (forward)   │  load staged embedding ─► wkv + gate·value ─► += hidden   │
             └───────────────────────────────────────────────────────── ┘
image

Copilot AI lite review requested due to automatic review settings September 10, 2026 06:42
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2185 --add-label <label>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

It contains correctness/operational issues in the compressed-vocab cache key and prefetch lifecycle (drop/shutdown behavior) that can lead to wrong caches or racy/stale cache entries.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds the host-side components of DeepSeek Engram support to ATOM: configuration parsing, reference-compatible n‑gram hashing, and a host embedding-table gather path with an async prefetcher to overlap CPU table work with GPU execution. This lays the groundwork for later wiring into the DeepSeek-V4.1 model runner.

Changes:

  • Introduces EngramConfig, compressed-tokenizer table build/cache, and NgramHashMapping to reproduce the reference hashing and validate table layout.
  • Adds host-resident embedding table gather plus an LRU prefetch cache and single-worker prefetcher API.
  • Adds unit tests covering hashing/layout invariants, gather bounds, cache eviction/drop behavior, and prefetch-vs-inline equivalence.
File summaries
File Description
atom/model_ops/engram.py Engram config + compressed vocab caching + reference-compatible n‑gram hashing and layout validation.
atom/model_ops/engram_host.py Host embedding gather, LRU cache, and async prefetch worker for overlap.
tests/model_ops/test_engram.py Unit tests validating hashing/layout, gather correctness, cache behavior, and prefetch behavior.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 7
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram_host.py Resolved
Comment thread atom/model_ops/engram_host.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram_host.py Resolved
Comment thread tests/model_ops/test_engram.py Resolved
@ZhangLirong-amd ZhangLirong-amd changed the title engram: host-side n-gram hash and embedding prefetch engram: host-side n-gram lookup, prefetched and overlapped with the step Sep 10, 2026
@ZhangLirong-amd
ZhangLirong-amd force-pushed the lirzhang/engram-cpu-overlap branch 3 times, most recently from 810123f to a164e26 Compare September 10, 2026 07:24
@ZhangLirong-amd ZhangLirong-amd changed the title engram: host-side n-gram lookup, prefetched and overlapped with the step engram: host-side n-gram lookup for DeepSeek-V4.1, prefetched and overlapped Sep 10, 2026
@ZhangLirong-amd
ZhangLirong-amd force-pushed the lirzhang/engram-cpu-overlap branch 2 times, most recently from 9bcf5d1 to ce8a871 Compare September 10, 2026 09:15
…rlapped

DeepSeek-V4.1-Flash puts Engram modules on layers 1 and 14, backed by tables that
cannot live in HBM: 384,006,168 and 384,016,682 rows of 256 fp8 values, about
98 GB each, 196B parameters in total. The lookup has to run on the host and it
has to be off the critical path, so this adds the host path, the device modules,
and the runner hooks that hide the work behind the previous step's GPU time.

Everything here is written against the shipped checkpoint and the V4.1 tech
report, not against the January prototype's toy configuration.

Hashing -- atom/model_ops/engram.py
  EngramConfig reads the engram_* block of a HF text config and returns None for
  models without one. NgramHashMapping reproduces the reference hashing
  (github.com/deepseek-ai/Engram): per-layer odd multipliers from a seeded
  generator, an XOR mix over the n-gram window, one distinct prime modulus per
  (layer, n-gram order, head) from a search whose seen set is shared across
  layers. Any other hash indexes the wrong rows of a trained table.

  The layout is checkable rather than assumed. Summing each layer's 24 primes has
  to reproduce engram_num_embeddings, and does -- 384,006,168 and 384,016,682
  exactly -- and the compressed vocabulary built from the shipped tokenizer comes
  out at 99092, the declared value. A mismatched tokenizer or a wrong n-gram
  order fails at startup instead of quietly reading the wrong table. The
  compressed vocabulary is cached on disk rather than rebuilt by decoding 129,280
  tokens on every start.

Host tables and prefetch -- atom/model_ops/engram_host.py
  HostEmbeddingTable memory-maps a table, gathers only the rows a step asks for,
  and applies the block scales the checkpoint stores beside it: these tables are
  quantized per 32 values, and ignoring the scale does not fail, it returns
  values about 200x too large. EngramPrefetchCache is an LRU with an explicit
  drop for finished requests. EngramPrefetcher runs hash and gather on one worker
  and returns a Future. EngramRuntime owns the pinned staging buffers, issues the
  H2D on a side stream and orders the compute stream behind it; a sequence whose
  prefetch has not landed is computed inline, so the race decides latency and
  never the answer.

Device modules -- atom/model_ops/engram_layer.py
  EngramOp matches the checkpoint's six tensors per layer: one fused wkv holding
  hc_mult key projections followed by a single value projection, and stacked
  RMSNorm gains k_weight / q_weight. There is no short causal convolution --
  the tech report states V4.1 omits it, and the checkpoint carries no weights for
  one. The module takes embeddings as an argument instead of looking them up; a
  layer that indexed a 98 GB table inline would stall the step it belongs to.
  Token dimensions are flat, [num_tokens, hc_mult, hidden], matching what
  DeepseekV4ForCausalLM already returns.

Attachment -- atom/model_ops/engram_module.py
  EngramModules.from_checkpoint builds config, tokenizer, hash layout, mapped
  tables and loaded device modules in one call, so nothing about n-gram hashing
  or 98 GB tables appears in a model definition. A model attaches it and exposes
  build_engram_runtime; the staging buffers take their dtype from the loaded
  modules, since staging in float32 against bf16 weights is a dtype error at the
  first matmul.

Runner -- atom/model_engine/model_runner.py
  Three hooks, each a no-op for models without engram: build the runtime once the
  weights land, stage before run_model, and queue the next step's prefetch where
  the previous step's sampled ids are already on the host and already
  synchronized, so it adds no synchronization of its own.

Model -- atom/models/deepseek_v41.py, plus registration
  DeepSeek-V4.1 with only Engram implemented. Attention and the MoE are stubs
  that preserve the residual and compute nothing, so the output is not
  meaningful and this must not be used for accuracy -- but the model loads and
  runs through ATOM's ordinary path, which is the only way the engram host path
  gets exercised at all. Registration is three pieces: the architecture in the
  model registry, "deepseek_v41" in the multimodal text_config hoist, and its
  text half mapped onto the V3 schema the way deepseek_v4 already is.

  V4.1 replaced MLA with CSA2, so its config has head_dim / o_lora_rank /
  o_groups / index_* and no kv_lora_rank, qk_nope_head_dim or v_head_dim. ATOM
  sizes the KV cache and builds attention metadata from those names on the
  config, not from the model, so a V4.1 checkpoint cannot be loaded until they
  exist. _fill_v41_attention_shims synthesizes self-consistent values for that
  sizing only; they are explicitly not V4.1's attention geometry, and a model
  that actually runs CSA2 has to stop relying on them.

  The runtime is held on the model rather than passed in: run_model calls
  forward with a fixed (input_ids, positions) signature, so there is no argument
  to thread it through.

Loading is verified end to end on the shipped checkpoint: atom.config.Config
resolves deepseek_v41 through the text_config hoist, the registry resolves the
architecture, the engram modules load from the checkpoint at layers 1 and 14,
and forward returns [num_tokens, hc_mult, hidden] with logits over the real
vocabulary.

Verified against deepseek-ai/DeepSeek-V4.1-Flash rather than only in unit tests:
tables map instantly, a batched gather runs about 0.9 us/row against 8 us/row
row-at-a-time, a 32-sequence decode step costs about 1.9 ms per layer, and both
layers forward at the right shapes from the real wkv / k_weight / q_weight. The
two tables are 197 GB against 2.7 TB of host memory, so the page cache holds them
without the process ever allocating them.

Loading and startup are exercised on the shipped checkpoint through the real
server path: ModelRunner builds the runtime and logs "engram host path active on
layers [1, 14]", and the engram layers run inside forward. Getting there needed
four pieces of registration beyond the model itself -- the multimodal text_config
hoist, the V3 schema mapping, the synthesized attention fields, and
deepseek_v41_text in the per-request-cache set -- each marked as scaffolding a
real CSA2 model will replace.

Known gaps: prefill and dummy passes stage zeros, which propagate to exactly
zero through EngramOp, so engram contributes nothing during prefill. That is
wrong for a real model -- the prompt's n-grams are precisely what should be
looked up, and the addressing is deterministic so they could be -- and it has to
be replaced by a per-prompt-token gather before this serves anything for
accuracy. Decode uses the real path.The tables are also mapped whole on every rank, where the tech report describes
sharding them; and the overlap here is cross-step, where the report
describes prefetching within a forward, overlapping the first Transformer block.

Refs #168, which prototyped the overlap on a standalone toy model.

Tests: 40 cases in tests/model_ops/test_engram.py.
@ZhangLirong-amd
ZhangLirong-amd force-pushed the lirzhang/engram-cpu-overlap branch from ce8a871 to 00afbe3 Compare September 10, 2026 10:29
Route DeepseekV41* to the MLA attention family instead of V4 (a bare "DeepseekV4"
substring caught "DeepseekV41"), so the KV pool sizes off the real layer count
rather than dividing by zero; fall back to num_hidden_layers when the stub walks
no attention modules; drop deepseek_v41_text from the per-request state-cache
types (it is pure MLA, no per-req state pool).

Key the prefetch gather on the just-sampled token (next step's model input),
matching the deferred-output pipeline, so a cache hit equals an on-the-spot
recompute. Stage then reads a miss token off the scheduler's host
scheduled_tokens anchor rather than a device->host copy of input_ids: a hit is
keyed by seq id and needs no token, and the only misses are just-admitted rows
whose committed anchor already sits on the host -- so the hot path has no D2H.

Add an env-gated (ATOM_ENGRAM_DEBUG_VERIFY) check that recomputes a reference
from the ground-truth input and compares it to the async-prefetched rows; it
takes the ground-truth token separately, which is what lets it also catch a bad
host anchor feeding a miss.
The engram op was split across engram.py (hash primitives), engram_host.py
(tables + prefetch + runtime), engram_layer.py (the device op) and
engram_module.py (the model attachment) -- four files for one feature, with only
engram_module importing the other three.

Fold it along the host/device line: engram.py now holds the whole host pipeline
(config, tokenizer, hash, HostEmbeddingTable, prefetcher, cache, EngramRuntime),
and engram_layer.py holds the device side (EngramOp) plus the EngramModules that
builds the ops and their runtime. Behavior is unchanged -- classes moved
verbatim, only imports rewired; deepseek_v41 and the tests now import from the
two surviving modules.
@ZhangLirong-amd ZhangLirong-amd changed the title engram: host-side n-gram lookup for DeepSeek-V4.1, prefetched and overlapped engram: host-side Engram lookup for DeepSeek-V4.1, prefetched and overlapped Sep 11, 2026
@ZhangLirong-amd ZhangLirong-amd changed the title engram: host-side Engram lookup for DeepSeek-V4.1, prefetched and overlapped engram: host-side n-gram lookup for DeepSeek-V4.1, prefetched and overlapped Sep 11, 2026
The host-side object read "runtime" (generic) and its methods "stage" and
"build_engram_runtime" did not say what they act on. Rename for what they are:

  EngramRuntime            -> EngramHost          (it is engram's host half)
  EngramHost.stage         -> stage_embeddings    (stages this step's rows)
  *.build_engram_runtime   -> build_engram_host    (factory + getattr contract)
  DeepseekV41*.engram_runtime -> engram_host

`stage_dummy` keeps its name. Pure rename across the host path, the two model
build sites, the ModelRunner getattr string, and the tests; no behavior change.
Prefetching the just-sampled token used a blocking `.to("cpu").numpy()` in
postprocess, which synchronizes the compute thread on the sampling kernel every
decode step -- exactly the sampled-token sync deferred output exists to avoid.

Hand the raw sampled ids to EngramHost.prefetch_next instead: it slices the last
id per row and copies it to a pinned host buffer ASYNC on a side stream, records
an event, and the prefetch worker waits on that event before reading (a new
`token_source` hook on the prefetcher). The compute thread only launches the copy
and returns. `stage_embeddings`' existing wait on the worker keeps the single
reused buffer/event safe. The CPU test path submits the ids directly.

Verified with ATOM_ENGRAM_DEBUG_VERIFY: 0/N mismatch, 0 wrong-token, no hang.
Clearer names on the host path and its callers:
  EngramPrefetcher.submit  -> submit_compute   (queues an async compute)
  EngramHost.wait_for_copy -> wait_for_embeddings
  ModelRunner local `make_host` -> `build_engram_host` (matches the method)

Trim the verbose engram comments in ModelRunner down to the load-bearing why.
No behavior change.
The DeepSeek-V4.1 stub only ever existed to exercise and verify the Engram host
path end-to-end (done: ATOM_ENGRAM_DEBUG_VERIFY shows byte-exact prefetch, and it
serves under load). With correctness proven, drop the demo scaffolding and leave
Engram as a self-contained, tested library plus ModelRunner's generic drive hooks.

Removed: the stub model atom/models/deepseek_v41.py and its registration; and the
V4.1-arch-specific bring-up that only routed it -- the DeepseekV4(?!\\d) regexes in
selector/register/config (back to the plain "DeepseekV4" substring, correct again
with no V4.1 arch to mis-match), config's deepseek_v41_text registry entry +
_fill_v41_attention_shims + multimodal entry, aiter_mla's zero-module KV-pool
fallback, and llm_engine's per-request-cache note.

Kept: atom/model_ops/engram*.py and the (getattr-gated, inert-without-a-model)
ModelRunner hooks. recipes/Engram.md documents the model-side contract, using the
removed stub's wiring as the worked example. History keeps the bring-up commits
(00afbe3 onward) as the record of how it was hooked up and verified.
The compressed-vocab normalizer used a bare U+E000 (Private Use Area) character
as its lone-space sentinel, which renders as mojibake and reads as corruption.
Write it as chr(0xE000) with a comment; behavior is unchanged.
The verify check was bring-up scaffolding to prove the async prefetch equals an
inline recompute (done: byte-exact, 0 wrong-token). With correctness established,
remove it: the `verify_token_ids` parameter and the whole compare/log block in
stage_embeddings, the per-step counters, the prefetcher's _debug_prefetch_token,
the _stage_engram input_ids D2H it required, and the recipe's verify section.
@ZhangLirong-amd
ZhangLirong-amd marked this pull request as ready for review September 11, 2026 09:42
Copilot AI review requested due to automatic review settings September 11, 2026 09:42
EngramHost sorted out of place after the EngramRuntime->EngramHost rename; CI's
ruff runs the isort rule that the default local select does not.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Multiple unresolved critical correctness, integration, checkpoint-decoding, and staging issues remain.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (12)

atom/model_engine/model_runner.py:879

  • run_model can execute running_tokens rows rather than batch.total_tokens_num (piecewise graph buckets and DP padding do this; capture already stages the settled padded width). This stages only the shorter local prefix, so EngramOp can see a row-count mismatch or graph replay can consume stale tail embeddings. Stage the settled running width and initialize the padded tail consistently.
        self.engram.stage_embeddings(seq_ids, tokens)

atom/model_engine/model_runner.py:879

  • This fallback source is not always the current model input. In deferred decode, prepare_input_ids documents that scheduled_tokens is a placeholder for carried-over requests, while the actual input comes from prev_sampled_token_ids. If such a request misses the cache (for example by LRU eviction), this recomputation hashes the placeholder and silently uses the wrong Engram embedding. Pass the actual per-request input ids to the fallback.
        # `tokens` is read only to recompute a prefetch miss (a just-admitted
        # row), whose anchor is already in the host `scheduled_tokens` -- so the
        # hot path needs no D2H.
        tokens = batch.scheduled_tokens[: len(seq_ids)].astype(np.int64).reshape(-1, 1)
        self.engram.stage_embeddings(seq_ids, tokens)

atom/model_engine/model_runner.py:849

  • The runner says it owns EngramHost's lifecycle, but exit() never calls self.engram.shutdown() before deleting the model. The ThreadPoolExecutor worker is non-daemon and may still be scanning a 98 GB table, so server shutdown can block at interpreter exit; a running worker can also outlive the model's table handles. Close/join the prefetcher during runner exit before destroying model state.
        The model owns its tables (it is what loaded them) and hands back the
        EngramHost; the runner owns the lifecycle and the two hooks below. Models
        without engram leave this None and pay nothing but the attribute.
        """
        self.engram = None

atom/model_engine/model_runner.py:840

  • The new runner integration is not exercised by the added tests: they call the helper classes directly but never run _init_engram_host, _stage_engram, postprocess, or the capture paths through a ModelRunner. A regression in prefill/speculative row alignment or side-stream/cudagraph staging can therefore pass the entire new test file while producing wrong output or capture failures. Author must add a mocked runner-level integration test covering normal decode and the non-decode paths.
        self._init_engram_host()

atom/model_ops/engram.py:145

  • The cache key only hashes the first 1024 token strings and omits their token IDs. Two tokenizer revisions can therefore reuse the same .npz while mapping later IDs differently (or while reassigning IDs), producing silently wrong compressed IDs and table rows. Hash the complete token-to-ID mapping, along with the normalization/cache version, or use a tokenizer/checkpoint-specific cache key.
        for tok in sorted(vocab)[:1024]:
            h.update(tok.encode("utf-8", "replace"))

atom/model_ops/engram.py:663

  • torch.cuda.current_stream(self.device) is evaluated inside the copy_stream context, so it returns self.copy_stream rather than the compute stream. The wait_stream is therefore a self-wait and does not protect the previous forward from this buffer's next H2D overwrite; concurrent decode/PP forwards can read torn Engram embeddings. Capture the compute stream before entering the context and wait on that object.
            with torch.cuda.stream(self.copy_stream):
                self.copy_stream.wait_stream(torch.cuda.current_stream(self.device))

atom/model_ops/engram.py:686

  • The same stream-ordering bug is duplicated for dummy/capture staging: current_stream() is queried after entering copy_stream, making this wait a no-op. A subsequent dummy or real stage can overwrite the single GPU buffer while the preceding capture/forward still reads it. Capture the compute stream before the context here as well.
            with torch.cuda.stream(self.copy_stream):
                self.copy_stream.wait_stream(torch.cuda.current_stream(self.device))

atom/model_ops/engram.py:681

  • Clamping an oversized dummy request makes _staged_rows smaller than the num_tokens passed to the model, so EngramOp later sees a shape mismatch (or a future caller reads truncated rows) instead of a capacity violation. stage_embeddings already raises for this condition; stage_dummy should reject it too rather than silently changing the requested shape.
        num_rows = min(int(num_rows), self.max_num_tokens)

atom/model_ops/engram.py:738

  • This cleanup API is never called by ModelRunner (the only references are its definition and wrapper), so completed requests remain in the cache until LRU eviction. With two layers per request, high request churn can evict active prefetched rows and force stage_embeddings onto synchronous host gathers, defeating the overlap and making latency scale with the 98-GB tables. Author must invoke request cleanup from the request-finished/preempted lifecycle.
    def drop_requests(self, seq_ids: list[int]) -> None:
        self.prefetcher.drop_requests(seq_ids)

recipes/Engram.md:99

  • This recipe documents _stage_engram with an input_ids parameter, but the added runner API is def _stage_engram(self, batch). The example is the model-integration contract, so this mismatch can lead implementers to call a nonexistent argument; document the actual signature.
- `_stage_engram(batch, input_ids)` — before each forward: `stage_dummy` for

recipes/Engram.md:48

  • The prescribed model-side contract requires adding the Engram module/forward hook to the DeepSeek-V4 model, but DeepseekV4Model is @support_torch_compile-decorated; editing that compiled model path breaks Dynamo tracing. Integrate through an outer wrapper or another call site outside the compile boundary, and update this recipe so it does not direct users to modify the decorated model.
## Model-side contract (3 hooks)

A model opts in by doing three things. ModelRunner does the rest automatically,
gated on `getattr(model, "build_engram_host", None)` — a model without it pays
nothing.

tests/model_ops/test_engram.py:147

  • The hash tests only check shape and repeatability; they never compare a known input against reference hash outputs. A wrong multiplier RNG, padding convention, or XOR order would still satisfy every assertion here while indexing the wrong trained rows. Add golden vectors from the reference for each layer/order, including the left-padded prefix and compressed IDs.
def test_hash_shape_and_determinism():
    mapping = build_mapping()
    ids = np.array([[5, 9, 13, 21], [1, 2, 3, 4]], dtype=np.int64)
    first = mapping.hash_layer(ids, layer_id=0)
    assert first.shape == (2, 4, mapping.config.num_hash_heads)
    np.testing.assert_array_equal(first, mapping.hash_layer(ids, layer_id=0))
  • Files reviewed: 5/5 changed files
  • Comments generated: 10
  • Review effort level: Lite

Comment thread atom/model_engine/model_runner.py Resolved
Comment thread atom/model_engine/model_runner.py Resolved
Comment thread atom/model_engine/model_runner.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram_layer.py Resolved
Comment thread atom/model_ops/engram_layer.py Resolved
Comment thread recipes/Engram.md Resolved
Copilot AI review requested due to automatic review settings September 11, 2026 09:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical and moderate issues remain in FP8 decoding, configuration handling, batching, and request cleanup.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (7)

atom/model_engine/model_runner.py:984

  • On DP decode, ForwardMode.decide sets context.running_bs/running_tokens from the DP-wide maximum, not len(seq_ids); this rank can have one request while replaying the graph selected for a peer's larger batch. Choosing gbs from this local count can stage only two rows while run_model executes EngramOp at eight rows, causing a shape mismatch or stale embeddings. Derive the staging height from the settled forward shape (context.running_tokens) instead.
            gbs = next(
                (g for g in reversed(self.capture_sizes) if g >= padded_rows), None
            )
            if gbs is not None:
                padded_rows = max(padded_rows, int(gbs))

atom/model_engine/scheduler.py:1821

  • This drains the pending IDs even when scheduled_seqs is empty. EngineCore skips runner.forward() for such a batch (atom/model_engine/engine_core.py:412-433), so _stage_engram() never calls drop_requests() and the cache/window cleanup is lost. Only drain these IDs when a real batch will reach the runner, as the prefill construction already does.
            engram_dropped=self._drain_engram_dropped(),

atom/model_engine/scheduler.py:524

  • This extraction repeats the dictionary-handling bug: getattr leaves tc at the outer dict. Once nested-dict detection is corrected (or when the declaration is otherwise recognized), line 527 indexes the outer dict and raises KeyError instead of reading text_config["engram_max_ngram_size"]. Select the nested key explicitly for dictionary configs here as well.
        tc = getattr(config.hf_config, "text_config", None) or config.hf_config

atom/model_ops/engram.py:734

  • A queued worker can pass drop_requests() after the cache/window removal but still reach advance() here and recreate _window[seq_id]. Unlike cache.put, window updates have no tombstone or generation check, so a canceled request leaves stale rolling context behind and can associate it with a later reused sequence id. Coordinate the drop with the worker or reject updates from an older request generation.
            windows = self.advance(seq_ids, toks)

atom/model_ops/engram.py:572

  • The rolling-window cap is fixed at 4096, independently of the scheduler's configurable max_num_seqs, and the Engram host factory never passes the live-request limit. With more than 4096 active requests, _evict_windows drops still-live windows; a later cache miss then falls back to only the current token, so multi-token Engram hashes are wrong. Size this storage for the scheduler capacity or retain windows for live requests.
        self._window_capacity = cache_capacity

atom/model_ops/engram_layer.py:134

  • No test asserts the numerical result of this new forward path; the current EngramOp tests only check shapes, errors, and checkpoint loading. A wrong key/value split, RMSNorm/gate formula, or branch broadcasting would pass while changing logits. Add a deterministic forward case with independently calculated expected output, including multiple branches.
        gate = (key * query).sum(dim=-1) / math.sqrt(self.hidden_size)

tests/model_ops/test_engram.py:150

  • This test invokes hash_layer twice and compares the implementation with itself, so it cannot catch a wrong multiplier, padding/shift, XOR order, or layer hash that is still deterministic and in range. Add fixed vectors generated independently from the reference for both layers and representative padding/context windows; otherwise compatibility with the trained table row layout is not tested.
    np.testing.assert_array_equal(first, mapping.hash_layer(ids, layer_id=0))
  • Files reviewed: 6/6 changed files
  • Comments generated: 3
  • Review effort level: Lite

Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram_layer.py Resolved
… h2d_done

Extract the slot rotation + reuse barrier into _SlotRing, used by both the
staging H2D ring and the token D2H ring -- they differ only in the per-slot
barrier (a CUDA event synchronize vs a threading.Event wait). Rename copy_done to
h2d_done to name what the event marks (the host-to-device staging copy).
Copilot AI review requested due to automatic review settings September 15, 2026 01:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical and moderate findings affect correctness, integration, lifecycle cleanup, and the claimed overlap.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (5)

atom/model_engine/model_runner.py:903

  • For the architecture registered as DeepseekV4ForCausalLM, no in-tree model implements build_engram_host or injects EngramOp into the decoder, so a V4.1 config reaches this fail-closed branch and startup aborts. As written, the host/device implementation is unreachable for the model named in the PR; either wire the model (including its checkpoint prefixes and forward path) in this change or scope/retitle this as infrastructure-only.
        build_engram_host = getattr(self.model, "build_engram_host", None)
        if build_engram_host is None:
            # Fail closed rather than silently serve the base model: a config that
            # declares engram layers but resolves to a model that does not wire
            # build_engram_host would otherwise ignore the engram weights and

atom/model_engine/model_runner.py:3220

  • This hook is invoked only after run_model and sampling, while prefetch_next's CUDA path explicitly makes its D2H stream wait on the current compute stream. Therefore the token cannot reach the worker until the preceding GPU step has completed; stage_embeddings then waits for the worker before launching the next forward. The host gather is overlapped with CPU postprocessing/scheduling, not the previous GPU work described by this PR, so the claimed critical-path hiding is not achieved for a host lookup that exceeds that CPU gap.
        # Prefetch on the just-sampled token (the NEXT step's input), not the
        # deferred one, which would prefetch a step stale. `prefetch_next` gathers
        # its embedding on the worker overlapping postprocess, and copies the
        # token to the host async, so this thread pays no D2H sync.
        if getattr(self, "engram", None) is not None and not batch.is_dummy_run:

atom/model_engine/scheduler.py:1821

  • When decode scheduling produces an empty batch, EngineCore skips the forward and postprocess entirely, so this drains the pending IDs without ever calling _stage_engram. The finished request's cache/window is therefore left alive (and a reused ID can observe stale state). Keep the pending IDs until a real batch is built, as the prefill branch already does.
            engram_dropped=self._drain_engram_dropped(),

atom/model_ops/engram.py:48

  • When hf_config is a raw dict with the usual nested text_config, getattr always returns None, so tc becomes the outer dict and this function misses text_config.engram_layer_ids. The dict branch suggests raw configs are supported, but such a config can bypass the runner's Engram setup; read the text_config mapping before falling back to the root.
    tc = getattr(hf_config, "text_config", None) or hf_config
    return (
        tc.get("engram_layer_ids") is not None
        if isinstance(tc, dict)
        else getattr(tc, "engram_layer_ids", None) is not None

atom/model_ops/engram.py:734

  • drop_requests can remove a request's window while this queued worker is still running, but advance() recreates it unconditionally before cache.put checks the drop mark. A normal finish during an in-flight prefetch therefore leaks a stale rolling window and can contaminate a reused request id; the worker needs to observe cancellation/generation before advancing, not only before cache insertion.
        def _run() -> None:
            toks = token_source() if token_source is not None else token_ids
            windows = self.advance(seq_ids, toks)
            for key, value in self.compute(seq_ids, windows).items():
                self.cache.put(key[0], key[1], value)
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Copilot AI review requested due to automatic review settings September 15, 2026 02:57
@ZhangLirong-amd
ZhangLirong-amd force-pushed the lirzhang/engram-cpu-overlap branch from cef87c7 to 7a564ca Compare September 15, 2026 02:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical Engram quantization/cache issues and moderate integration, DP-shape, and lifecycle issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (5)

atom/model_engine/model_runner.py:905

  • For the in-tree DeepseekV4ForCausalLM, no build_engram_host hook or residual Engram injection is present, so a V4.1 config takes this branch and raises after model loading instead of serving Engram. That leaves the advertised DeepSeek-V4.1 path unusable; either add the model-side integration (including skipped host-table weights and layer injection) in this change or scope the PR as library-only.
            if self._config_declares_engram(self.config):

atom/model_engine/model_runner.py:947

  • A DP dummy also goes through ForwardMode.decide: dummy_execution() creates a one-token batch, but graph replay uses the DP-wide context.running_tokens settled from the peer. Staging only batch.total_tokens_num therefore leaves the Engram buffer at one row while the dummy hidden states have the running height, causing a shape mismatch or stale padded rows. Stage the context's settled running height here as well.
        if batch.is_dummy_run or not seq_ids:
            self.engram.stage_dummy(batch.total_tokens_num)
            self.engram.wait_for_embeddings()

atom/model_engine/model_runner.py:984

  • The decode shape can be widened to the DP group's running_bs/running_tokens, while len(seq_ids) is only this rank's local batch. On a rank with fewer requests, choosing the bucket from the local count stages too few rows; the graph then executes the larger padded rectangle and EngramOp receives hidden states and embeddings with different leading dimensions. Use the already-set forward mode's actual running row count for graph-backed decode (or explicitly reject this DP mode).
            gbs = next(
                (g for g in reversed(self.capture_sizes) if g >= padded_rows), None
            )
            if gbs is not None:
                padded_rows = max(padded_rows, int(gbs))

atom/model_engine/scheduler.py:1821

  • This decode path drains _engram_drop_pending even when scheduled_seqs is empty. EngineCore skips runner.forward for such a batch, so the IDs are cleared from the scheduler without ever reaching EngramHost.drop_requests; their cache/window (and any in-flight stale state) then remains indefinitely and can consume LRU capacity or contaminate a later request. Keep the IDs pending until a non-empty batch is forwarded, as the prefill path already does.
            engram_dropped=self._drain_engram_dropped(),

atom/model_ops/engram.py:1057

  • stage_prefill can run while the previous decode's prefetch future is still in flight. On preemption/resume, drop_requests removes the current window but does not prevent that worker's advance() from recreating and mutating it; the worker can therefore append its stale sampled token after this method seeds the new prompt tail, making the next decode lookup use the wrong n-gram (and cache rows may also be stale). Synchronize or generation-guard the in-flight worker before reseeding and staging a resumed prefill.
        computed = self.prefetcher.compute_prefill(context_tails, chunk_tokens)
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Copilot AI review requested due to automatic review settings September 15, 2026 03:32
@ZhangLirong-amd
ZhangLirong-amd force-pushed the lirzhang/engram-cpu-overlap branch from 7a564ca to 9266880 Compare September 15, 2026 03:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical configuration-handling defects and unresolved lifecycle, loading, caching, and overlap issues remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (8)

atom/model_engine/model_runner.py:910

  • This fail-closed check is reached only after load_model, but the registered DeepseekV4ForCausalLM currently has no build_engram_host and no skip prefixes for layers.*.engram.*. A real V4.1 checkpoint will therefore either fail here after loading as much of the Engram tensors as the loader permits, or deserialize the shard containing the ~196 GB tables when ATOM_DISABLE_MMAP=true, before reporting the unsupported model. The target model needs to be wired (including loader skips) in this change, or Engram configs must be rejected before model loading; as written the advertised DeepSeek-V4.1 path cannot serve the model safely.
            if self._config_declares_engram(self.config):
                raise NotImplementedError(
                    f"{type(self.model).__name__} config declares engram layers "
                    "but the model does not implement build_engram_host; serving "
                    "would silently ignore the engram weights"
                )

atom/model_engine/scheduler.py:1821

  • _drain_engram_dropped() is unconditional here, but EngineCore skips forward and postprocesses an empty ScheduledBatch when req_ids is empty. If the engine goes idle after a request finishes, this batch consumes the pending ids without calling EngramHost.drop_requests, leaving its cache/window alive and allowing stale state to survive until a later request. Drain only when scheduled_seqs is non-empty, as the prefill path already does, so the ids remain pending for the next forwarded batch.
            engram_dropped=self._drain_engram_dropped(),

atom/model_ops/engram.py:770

  • drop_requests() removes a sequence's rolling window before an in-flight _run can finish, but this unconditional advance() recreates that window afterward. The drop marker only suppresses cache.put, not _window, so a dropped request can remain in the LRU and evict a live request's context; a later cache miss can then hash with a stale/placeholder window. Carry a drop generation/state into the worker and suppress the window update (and publication) for requests dropped after submission.
        def _run() -> None:
            toks = token_source() if token_source is not None else token_ids
            windows = self.advance(seq_ids, toks)
            for key, value in self.compute(seq_ids, windows).items():
                self.cache.put(key[0], key[1], value)

atom/model_ops/engram.py:1050

  • This wait makes the side-stream H2D begin only after all work previously queued on the compute stream completes. Because each layer has a single device destination buffer, staging therefore serializes the copy after the previous forward and then makes the next forward wait for the copy; the H2D cannot overlap GPU work as described in recipes/Engram.md. Achieving that overlap needs separate device buffers (and graph-safe selection), or the overlap claim should be narrowed.
                self.copy_stream.wait_stream(compute_stream)

atom/model_ops/engram.py:841

  • This comment says a timed-out worker causes rows to be recomputed inline, but stage_embeddings deliberately waits a second time and then raises RuntimeError; it never performs that fallback. Please make the comment describe the actual fail-fast behavior so the timeout's operational contract is not misleading.
    # worker, at which point the rows are recomputed inline instead of hanging.

atom/model_ops/engram_layer.py:345

  • max_num_tokens is the per-step token budget, not the number of active sequences. A valid configuration can set max_num_batched_tokens < max_num_seqs; then this cache/window capacity is too small and _evict_windows() removes contexts for active requests that were not selected in the current decode batch. When one is scheduled later, the miss path falls back to its current token and hashes without the preceding n-gram context, producing wrong embeddings. Size the per-request cache/window from the scheduler's active-sequence ceiling (or avoid evicting active windows), rather than only max_num_tokens.
            # Size the prefetch cache / rolling-window store to the runner's hard
            # concurrency ceiling (a decode step stages <= max_num_tokens rows,
            # one per sequence), so an active request is never LRU-evicted; the
            # capacity bound is then only a leak net for a missed drop_requests.
            EngramPrefetcher(
                self.hash_mapping, self._tables, cache_capacity=max_num_tokens

atom/model_ops/engram_layer.py:345

  • The cache is keyed by (seq_id, layer_id), so a decode row consumes one entry per Engram layer, but cache_capacity is only max_num_tokens (the row/token bound). In a legal configuration whose decode batch approaches that bound, inserting both layers evicts one layer for earlier sequences; _stage_embeddings then recomputes synchronously and the intended overlap is lost. Size this entry capacity for all layers (or otherwise account for the per-layer keys).
                self.hash_mapping, self._tables, cache_capacity=max_num_tokens

recipes/Engram.md:120

  • The documented integration point calls self.engram[...] from the decoder layer loop, but the registered DeepSeek-V4 implementation places that loop inside DeepseekV4Model, which is marked @support_torch_compile (atom/models/deepseek_v4.py:4290). The repository's compile-boundary rule forbids modifying such model files; following this recipe would put the host-dependent lookup inside Dynamo/capture and can break compilation. Document or implement the hook at an allowed outer runner/call-site boundary instead.
for layer_id, layer in enumerate(self.layers):
    if self.engram is not None and layer_id in self.engram:
        if self.engram_host is None:
            raise RuntimeError(
                f"layer {layer_id} carries engram but no host was supplied; "
                f"ModelRunner stages the embeddings before calling forward"
            )
        hidden = hidden + self.engram[layer_id](
            hidden, self.engram_host.embeddings(layer_id)
        )
    hidden = layer(hidden, positions)
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/model_engine/scheduler.py Resolved
Comment thread atom/model_ops/engram.py Resolved
Copilot AI review requested due to automatic review settings September 15, 2026 05:57
@ZhangLirong-amd
ZhangLirong-amd force-pushed the lirzhang/engram-cpu-overlap branch from 9266880 to 410c4a2 Compare September 15, 2026 05:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved configuration, request-lifecycle, cache, performance, and integration-test issues remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (7)

atom/model_engine/model_runner.py:3343

  • [verified] The new runner hooks are covered only by EngramHost/unit-level tests; no test drives ModelRunner.forward with a ScheduledBatch through decode, final/middle prefill, and dummy/CUDAGraph staging. Ordering or padding/stream handoff regressions here can pass the current suite and fail only in serving. Author must add a mocked integration test for the runner/scheduler contract.
        if getattr(self, "engram", None) is not None:
            self._stage_engram(batch)
        logits, hidden_states = self.run_model(input_ids, batch)

atom/model_engine/scheduler.py:1827

  • When no sequence is schedulable, this still drains _engram_drop_pending, but EngineCore skips runner.forward() for the resulting empty batch, so _stage_engram never calls drop_requests. The finished request's cache/window therefore survives until a later request-id reuse can consume stale n-gram state. Preserve the pending IDs on empty batches and drain them only into a batch that will be forwarded, as the prefill path already does.
            engram_dropped=self._drain_engram_dropped(),

atom/model_ops/engram.py:640

  • drop_requests() removes a request's rolling window, but an in-flight worker can then reach this setdefault and recreate it. The cache tombstone prevents only cache.put, not this window write; if the request ID is reused, the next prefetch appends to the stale tokens and hashes the wrong n-gram context. Synchronize request dropping with the worker or add a generation/tombstone check around window advancement.
                w = self._window.setdefault(seq_id, deque(maxlen=self._ngram))

atom/model_ops/engram.py:707

  • Every runtime prefetch uses HostEmbeddingTable.gather's default float32 output here, and EngramHost.stage_embeddings later casts that full row into the model's bf16/fp16 staging slot. For a 6144-wide V4.1 row this adds an intermediate allocation and another full copy/conversion to every random host lookup, extending the worker work that is supposed to overlap GPU execution. Carry the target staging dtype into the prefetcher and gather directly in that dtype.
            gathered = self._tables[layer_id].gather(rows[:, -1])

atom/model_ops/engram.py:739

  • Prefill has the same avoidable dtype expansion as decode: gather defaults to float32, then stage_prefill copies each row into the model-dtype staging buffer. This adds a full intermediate and conversion for every prompt position, which is especially costly for large chunks; pass the target staging dtype through the prefetcher and gather directly in it.
                chunk_rows = self._tables[layer_id].gather(rows)[0, skip:]

atom/utils/envs.py:28

  • [verified] ATOM_ENGRAM_CACHE_DIR is a new operator-facing setting, but the repository's environment-variable reference does not list it. Deployments cannot discover how to relocate the compressed-vocabulary cache or what the default is. Author must document this variable in docs/environment_variables.md with its default and cache semantics.
    "ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),

recipes/Engram.md:71

  • [verified] The PR's central claim is that host lookup is hidden off the critical path, but the description provides no benchmark with units or reproducible command. Without target-model host latency, GPU-step overlap, and end-to-end throughput/latency data, a cache-miss fallback can turn this into a regression while all correctness tests still pass. Author must add a reproducible production-shape benchmark and report both steady-state hits and cold/missed-prefetch behavior.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
  • Files reviewed: 7/7 changed files
  • Comments generated: 3
  • Review effort level: Lite

Comment thread atom/model_ops/engram_layer.py Resolved
Comment thread atom/model_engine/scheduler.py Resolved
Comment thread atom/model_ops/engram_layer.py Resolved
Copilot AI review requested due to automatic review settings September 15, 2026 06:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical staging-width and lint issues, plus unresolved integration, cache, cleanup, testing, and performance-validation concerns remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (8)

atom/model_engine/model_runner.py:880

  • The registered DeepSeek-V4 model has no build_engram_host implementation, so every real V4.1 config that declares engram_layer_ids raises at this guard before model construction. The current tree therefore cannot serve the model named by this PR; either add the model-side Engram wiring (including the skipped table prefixes) or explicitly scope the PR as library groundwork.
            if not hasattr(model_class, "build_engram_host"):
                raise NotImplementedError(
                    f"{model_class.__name__} config declares engram layers but the "
                    "model class does not implement build_engram_host; serving "
                    "would silently ignore the engram weights"

atom/model_engine/model_runner.py:957

  • The new shared runner hook is covered only by tests/model_ops/test_engram.py; there is no integration coverage for _stage_engram through forward/postprocess, including dummy, prefill, decode, drop, and CUDAGraph-width behavior. A regression here can pair embeddings with the wrong rows or break capture while all host/unit tests pass. Add a runner-level test using a stub EngramHost that asserts staging order and row counts.
        if getattr(batch, "engram_dropped", None):
            self.engram.drop_requests(batch.engram_dropped)
        seq_ids = list(batch.req_ids)
        if batch.is_dummy_run or not seq_ids:
            self.engram.stage_dummy(batch.total_tokens_num)

atom/model_engine/model_runner.py:3230

  • [verified] The PR claims host work is off the critical path, but the only reported results are equality logs; this path explicitly waits for the worker in stage_embeddings and recomputes misses, so a gather longer than one GPU step becomes a synchronous stall. Author must provide a reproducible decode/prefill latency or throughput comparison showing the overlap on target hardware and production batch sizes.
        # Prefetch on the just-sampled token (the NEXT step's input), not the
        # deferred one, which would prefetch a step stale. `prefetch_next` gathers
        # its embedding on the worker overlapping postprocess, and copies the
        # token to the host async, so this thread pays no D2H sync.

atom/model_engine/scheduler.py:531

  • config_declares_engram() accepts a root-level declaration when text_config exists without Engram, but this normalization unconditionally selects that nested block. Such a config reaches tc["engram_max_ngram_size"] and raises KeyError, even though EngramModules.from_checkpoint() handles the same shape. Check that the nested block actually contains engram_layer_ids before selecting it for both dict and object configs.
            if isinstance(config.hf_config, dict):
                tc = config.hf_config.get("text_config", config.hf_config)
            else:
                tc = getattr(config.hf_config, "text_config", None) or config.hf_config

atom/model_engine/scheduler.py:1827

  • _drain_engram_dropped() is called even when scheduled_seqs is empty, but EngineCore skips runner.forward for an empty batch, so _stage_engram never consumes these IDs and the cache/window are left live. If a sequence ID is later reused, stale embeddings can be staged for the new request. Preserve the pending IDs by draining only for a non-empty batch, or process the drop explicitly on the empty-batch path.
            engram_dropped=self._drain_engram_dropped(),

atom/model_ops/engram_layer.py:351

  • The cache capacity counts (seq_id, layer_id) entries, not sequences. With two Engram layers and a decode batch near max_num_tokens, inserting one row per layer evicts active rows immediately, forcing stage_embeddings() to recompute them synchronously and defeating the intended overlap. Size the cache for all layers (for example, max_num_tokens * len(self.config.layer_ids)).
            # Size the prefetch cache / rolling-window store to the runner's hard
            # concurrency ceiling (a decode step stages <= max_num_tokens rows,
            # one per sequence), so an active request is never LRU-evicted; the
            # capacity bound is then only a leak net for a missed drop_requests.
            EngramPrefetcher(
                self.hash_mapping, self._tables, cache_capacity=max_num_tokens

atom/utils/envs.py:28

  • ATOM_ENGRAM_CACHE_DIR is a new public setting, but it has no entry in docs/environment_variables.md, the repository's user-facing environment-variable reference. Operators cannot discover how to relocate the potentially large compressed-vocabulary cache. Add the variable with its default and cache semantics to the environment-variable documentation.
    "ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),

recipes/Engram.md:73

  • stage_embeddings can block waiting for the worker, yet this recipe presents the lookup as hidden behind GPU work without latency/tokens-per-second measurements or reproducible GPU, ROCm, and configuration details. The central overlap/performance claim is therefore unverified, especially for cold and cache-miss steps. Add production-shaped steady-state and miss/cold benchmark results with the reproduction command.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
double-buffered (a per-layer pinned ring) so the next step's fill never races the
previous step's in-flight H2D, while the device buffer stays fixed for CUDAGraph.
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/model_engine/model_runner.py Resolved
Comment thread atom/model_ops/engram_layer.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical runner and Engram validation issues, along with additional integration and runtime gaps, remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (7)

atom/model_engine/model_runner.py:880

  • This guard makes the advertised DeepSeek-V4.1 path unusable: DeepseekV4ForCausalLM is the registered production class, but it does not implement build_engram_host (nor attach EngramModules or skip the Engram checkpoint prefixes), so every V4.1 config declaring Engram raises here before serving. Either wire the model's three hooks and checkpoint loading, or scope the PR/title explicitly to the standalone library instead of rejecting the only advertised model.
            if not hasattr(model_class, "build_engram_host"):
                raise NotImplementedError(
                    f"{model_class.__name__} config declares engram layers but the "
                    "model class does not implement build_engram_host; serving "
                    "would silently ignore the engram weights"

atom/model_engine/model_runner.py:3230

  • [verified] The PR's central claim is that host hashing/table gathers are hidden behind GPU work, but the supplied output only checks value equality; it contains no end-to-end latency/throughput or overlap measurement, and no comparison with synchronous/recomputed lookup at production table and batch sizes. Add a reproducible decode benchmark with units and the overlap-disabled baseline before relying on this optimization claim.
        # Extract req_ids and token_ids from dict (key -1 is the is_deferred_out flag)
        req_ids_out = [k for k in token_id_dict if k != -1]
        token_ids_out = [token_id_dict[k] for k in req_ids_out]

atom/model_engine/model_runner.py:948

  • The new tests exercise the host and device units, but no test constructs a model exposing build_engram_host and drives the runner's _stage_engram/postprocess path. That leaves scheduler-to-runner row ordering, final-vs-middle prefill behavior, CUDAGraph padding, and stream handoff unverified; add a mocked model/runner integration test covering one decode step and a chunked prefill.
    def _stage_engram(self, batch: ScheduledBatch) -> None:
        """Stage this step's engram embeddings to the device before the forward.

        Decode stages the rows the prefetch gathered; prefill gathers every
        prompt position's n-gram (carrying the tokens preceding a chunk); a dummy

atom/model_engine/scheduler.py:541

  • [verified] This branch does not handle the object-with-dict shape that config_declares_engram() explicitly supports. For a config object whose text_config is a dict, getattr(nested, "engram_layer_ids", None) returns None, so nested_has is false and tc becomes the outer object; the following access to tc.engram_max_ngram_size raises AttributeError even though Engram was detected. Check dicts with .get() here and add the corresponding scheduler regression test.
                nested_has = (
                    nested is not None
                    and getattr(nested, "engram_layer_ids", None) is not None
                )

atom/model_ops/engram.py:807

  • [verified] drop_requests removes the rolling window while an earlier _run may still be inside advance() (the runner calls this before stage_embeddings waits for the future). If the worker acquires _window_lock after this pop, advance() recreates the dropped sequence's window; cache.drop() prevents a stale embedding from being stored, but does not prevent this window resurrection. Finished requests can therefore leak into the LRU and eventually evict a live request's context. Synchronize the worker before clearing the window or make the worker discard windows for dropped IDs.
    def drop_requests(self, seq_ids: list[int]) -> None:
        for seq_id in seq_ids:
            self.cache.drop(seq_id)
        with self._window_lock:
            for seq_id in seq_ids:

atom/model_ops/engram_layer.py:257

  • This factory only accepts a local checkpoint directory (open(model_path/config.json) and later model.safetensors.index.json), but the model-facing recipe passes config.model, which ATOM documents as either a model name or path and which the normal loader supports by downloading a Hugging Face repo. A remote Engram model therefore loads its config/weights normally and then fails here with FileNotFoundError; resolve the model name to the same local snapshot (and handle the single-file checkpoint case) before opening these files.
        if hf_config is None:
            with open(os.path.join(model_path, "config.json")) as fh:
                hf_config = json.load(fh)
        # Match config_declares_engram's contract: the engram block may sit in a
        # nested text_config OR at the root. Use the nested block only when it
        # actually carries the declaration; otherwise a text_config without engram
        # fields would make from_hf return None and build the model with no engram.
        text_config = hf_config.get("text_config")
        if text_config is None or "engram_layer_ids" not in text_config:
            text_config = hf_config
        config = EngramConfig.from_hf(text_config)

recipes/Engram.md:71

  • The PR's central overlap claim is documented here, but the description and changes provide no end-to-end latency/throughput comparison or prefetch/H2D wait measurements. Since stage_embeddings still blocks when host work misses the window, the correctness tests do not establish that overlap improves serving or quantify its fallback cost; please add a reproducible benchmark with units and configuration, or remove the performance claim.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/model_engine/model_runner.py Resolved
Comment thread atom/model_ops/engram.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved integration, lifecycle, cleanup, configuration, and coverage findings remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (7)

atom/model_engine/model_runner.py:880

  • DeepseekV4ForCausalLM is the registered class for the advertised V4.1 architecture, but it has no build_engram_host and its decoder never constructs or invokes EngramModules. Any V4.1 config with engram_layer_ids therefore takes this guard and raises before serving, so the new host/device path is unreachable for the target model. Please wire the modules into atom/models/deepseek_v4.py (including checkpoint skip prefixes and residual injection), or narrow the PR's supported-model claim.
            if not hasattr(model_class, "build_engram_host"):
                raise NotImplementedError(
                    f"{model_class.__name__} config declares engram layers but the "
                    "model class does not implement build_engram_host; serving "
                    "would silently ignore the engram weights"

atom/model_engine/scheduler.py:1838

  • decode_batch is still constructed when scheduled_seqs is empty, and EngineCore skips forward/_stage_engram for that batch (engine_core.py:398-401, 412-433). This drains _engram_drop_pending without calling drop_requests; if the last request is finished or preempted, its cache/window remains and can evict live entries or be hit by a reused ID. Retain the pending IDs until a non-empty batch (or explicitly process them in the empty-batch path).
            engram_dropped=self._drain_engram_dropped(),

atom/model_engine/scheduler.py:541

  • config_declares_engram accepts an object whose text_config is a dict, but this branch only checks attributes on nested. For such a config, detection returns true, nested_has is false, tc remains the outer object, and line 544 raises AttributeError instead of reading the nested Engram settings. Keep this selection logic consistent with the helper's supported shapes.
                nested = getattr(hc, "text_config", None)
                nested_has = (
                    nested is not None
                    and getattr(nested, "engram_layer_ids", None) is not None
                )

atom/model_ops/engram.py:807

  • drop_requests removes the rolling window before waiting for the in-flight worker. If that worker has not reached advance() yet, it recreates the dropped sequence's deque after this method returns; the cache tombstone prevents its embedding put but does not protect the window. A stale context can consume an LRU slot or be appended to when an ID is reused, corrupting the next n-gram. Wait for the current future before clearing the cache/window, or add an equivalent generation check in the worker.
    def drop_requests(self, seq_ids: list[int]) -> None:
        for seq_id in seq_ids:
            self.cache.drop(seq_id)
        with self._window_lock:
            for seq_id in seq_ids:

atom/model_ops/engram_layer.py:352

  • max_num_tokens is the token-budget capacity (ModelRunner passes max_num_batched_tokens), not the request-concurrency limit. The cache/window are keyed per sequence, so using this value overstates their bound (16,384 vs the default 512 sequences) and lets missed drops retain far more host memory than necessary; pass the actual max-sequence limit separately while keeping the token budget for staging buffers.
            # Size the prefetch store to the runner's hard concurrency ceiling (a
            # decode step stages <= max_num_tokens rows, one per sequence), so an
            # active request is never LRU-evicted; the prefetcher scales the cache
            # by the engram layer count internally. The bound is then only a leak
            # net for a missed drop_requests.
            EngramPrefetcher(
                self.hash_mapping, self._tables, max_concurrent_seqs=max_num_tokens

atom/utils/envs.py:28

  • ATOM_ENGRAM_CACHE_DIR is a new user-facing configuration knob, but the central docs/environment_variables.md reference has no entry for it. Users therefore cannot discover the override, its default, or what the cache contains. Add it to the environment-variable documentation.
    # Directory for the engram compressed-vocab cache (.npz). Empty = default
    # ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build.
    "ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),

tests/model_ops/test_engram.py:618

  • The new tests cover hashing, table decoding, the worker, and the device op in isolation, but do not exercise the changed scheduler→ModelRunner lifecycle. In particular, an empty decode batch can consume engram_dropped, graph-bucket padding controls the staged row count, and postprocess must prefetch only final chunks; these interactions can regress while this suite remains green. Add a non-GPU integration/contract test covering a ScheduledBatch through the runner stub, including the empty-batch cleanup case.
def make_runtime() -> EngramHost:
    pf = make_prefetcher()
    cfg = pf._hash_mapping.config
    return EngramHost(
        pf,
        max_num_tokens=8,
        num_hash_heads=cfg.num_hash_heads,
        head_dim=8,
        device=torch.device("cpu"),
    )
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread atom/model_ops/engram.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved integration and correctness issues remain in the reviewed implementation.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (9)

atom/model_engine/model_runner.py:881

  • With the current tree, a DeepSeek-V4.1 Engram checkpoint is not actually serveable: atom/models/deepseek_v4.py has no build_engram_host or Engram module/residual wiring, so this guard rejects the config before loading. The recipe documents that no in-tree model uses the library, but that conflicts with this PR's V4.1-support title and description; either include the model wiring (including the loader skip prefixes) or scope the PR/title explicitly as a library/runner contract.
            if not hasattr(model_class, "build_engram_host"):
                raise NotImplementedError(
                    f"{model_class.__name__} config declares engram layers but the "
                    "model class does not implement build_engram_host; serving "
                    "would silently ignore the engram weights"
                )

atom/model_ops/engram.py:874

  • This comment says a timeout falls back to inline recomputation, but stage_embeddings actually waits for a second interval and then raises if the worker is still pending (lines 1003-1024); it never recomputes on timeout. Please make the description match the fail-fast behavior so operators do not expect a fallback for a stalled host gather.
    miss (an evicted or just-admitted row) inline. The result is identical either
    way -- the prefetch only decides whether the work was already done, never the
    answer.

atom/model_ops/engram.py:823

  • drop_requests can remove the window while the single worker is between submission and advance(). The worker then acquires _window_lock, recreates this sequence's window, and its cache writes are only suppressed by the drop marker; when the sequence ID is reused, that resurrected window becomes stale context for the first n-gram. Wait for the in-flight future before removing the window (or add an equivalent window tombstone).
            return False

    def drop_requests(self, seq_ids: list[int]) -> None:

atom/model_ops/engram.py:125

  • config_declares_engram treats a declaration whose value is None as absent, but this check only tests key presence. For a config carrying the optional engram_layer_ids: None field, from_hf proceeds to tuple(None) (and the documented “None when the model has no engram” contract is broken). Check the value with text_config.get("engram_layer_ids") is None instead.
        if "engram_layer_ids" not in text_config:

atom/model_ops/engram.py:218

  • The documented empty-value behavior is not honored when the variable is explicitly set to an empty string: os.environ.get(...) returns "", so Path("") points at the current working directory instead of the default cache directory. This can write compressed_vocab_*.npz into whatever directory launched the server (or fail there) and defeats cache reuse. Treat an empty environment value as unset before constructing the path.
        # `or` on each step so an unset OR explicitly-empty env var (envs.py
        # registers a "" default) falls through to the real default rather than
        # writing the cache into the process's cwd.
        cache_dir = (

atom/model_ops/engram_layer.py:257

  • The nested-config selection repeats the same key-presence test. If a root config declares Engram but its nested text_config contains engram_layer_ids: None, this selects the nested block and disables Engram (or raises before the from_hf fix), even though the helper's contract says that value is not a declaration. Select the nested block only when the value is non-None.
        text_config = hf_config.get("text_config")
        if text_config is None or "engram_layer_ids" not in text_config:
            text_config = hf_config
        config = EngramConfig.from_hf(text_config)

atom/model_ops/engram_layer.py:353

  • max_num_tokens is the per-step token budget, but it is passed here as max_concurrent_seqs. If max_num_seqs is larger than max_num_batched_tokens, the rolling-window LRU evicts still-active requests; the next miss then falls back to only the current token and hashes without prompt context, producing wrong Engram rows. Pass a separate max_num_seqs bound for the prefetcher/window while retaining the token budget for staging capacity.
            EngramPrefetcher(
                self.hash_mapping, self._tables, max_concurrent_seqs=max_num_tokens
            ),

atom/model_ops/engram_layer.py:184

  • This added path eagerly dequantizes the checkpoint's 32x32 FP8 WKV matrix to float32 and then stores it in a regular nn.Linear (BF16 after .to(dtype)). For each V4.1 layer that expands roughly 157M values to about 315 MB on HBM and loses the FP8 block-scaled GEMM path used elsewhere in ATOM, which undermines the stated FP8 device path and can materially hurt decode bandwidth. Use an appropriate quantized/AITER linear path or provide measured justification for this tradeoff.
            wkv_vals = decode_fp8(wkv, torch.float32)
            wkv = (
                wkv_vals.reshape(rows // block, block, cols // block, block)
                * ws.reshape(rows // block, 1, cols // block, 1)
            ).reshape(rows, cols)

atom/model_ops/engram_layer.py:139

  • The tests cover shapes, errors, and weight dequantization, but none checks the numerical result of this gate against an independent reference. A projection split/transposition, RMS normalization, or signed-square-root sigmoid regression would still pass all current EngramOp tests while producing incorrect model outputs. Add a deterministic small-dimension value test for the fused WKV/gate calculation.
        gate = (key * query).sum(dim=-1) / math.sqrt(self.hidden_size)
        # Signed square root before the sigmoid: keeps the gate responsive for
        # small scores without letting large ones saturate it.
        gate = gate.abs().clamp_min(1e-6).sqrt() * gate.sign()
        return gate.sigmoid().unsqueeze(-1) * value.unsqueeze(-2)
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread atom/model_engine/scheduler.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical and moderate correctness, integration, lifecycle, overlap, and documentation issues remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (9)

atom/model_engine/model_runner.py:3346

  • The new per-forward staging hook covers decode, chunked prefill, dropped requests, padding, and CUDAGraph warmup/capture, but the added tests only exercise the host objects and the device op in isolation. A batch-shape or stream-order regression in forward() would therefore pass the suite without being detected. Add a mocked ModelRunner/scheduler integration test that drives at least decode, final/middle prefill, and a padded capture batch.
        if getattr(self, "engram", None) is not None:
            self._stage_engram(batch)

atom/model_engine/model_runner.py:3234

  • This prefetch is launched only after run_model and sampling have finished enqueueing the current step, and _stage_engram waits for its future before the next forward. Since speculative/drafter and pipeline paths are rejected above, there is no GPU work after this call that can hide the host hash/gather; the H2D is also ordered before compute. The advertised GPU overlap therefore does not occur, and decode still pays the lookup latency.
        # Prefetch on the just-sampled token (the NEXT step's input), not the
        # deferred one, which would prefetch a step stale. `prefetch_next` gathers
        # its embedding on the worker overlapping postprocess, and copies the
        # token to the host async, so this thread pays no D2H sync.

atom/model_engine/model_runner.py:880

  • This fail-closed branch makes the advertised DeepSeek-V4.1 path unusable: DeepseekV4ForCausalLM does not define build_engram_host, so an Engram-configured checkpoint is rejected before construction, and the model also has no Engram forward or loader integration. Either add the model-side hooks and skip/load the large host-resident tensors, or narrow the PR description/title to infrastructure-only support instead of claiming V4.1 support.
            if not hasattr(model_class, "build_engram_host"):
                raise NotImplementedError(
                    f"{model_class.__name__} config declares engram layers but the "
                    "model class does not implement build_engram_host; serving "
                    "would silently ignore the engram weights"

atom/model_engine/scheduler.py:1839

  • This drains engram_dropped even when scheduled_seqs is empty. EngineCore skips runner.forward for that batch when req_ids is empty, so the IDs are discarded without calling drop_requests(); a prefetch cache/window for a preempted request can then survive and contaminate a later reused ID. Preserve the pending IDs until a non-empty batch carries them, or explicitly process the drop on the empty-batch path.
            engram_dropped=self._drain_engram_dropped(),
        )

atom/model_engine/scheduler.py:541

  • When hf_config is an object whose text_config is a dict, config_declares_engram() returns true (the new host tests explicitly support this shape), but this branch only probes object attributes. nested_has becomes false, tc remains the outer object, and the following tc.engram_max_ngram_size access raises instead of reading the nested dict. Handle dict-valued nested configs here as well.
                nested = getattr(hc, "text_config", None)
                nested_has = (
                    nested is not None
                    and getattr(nested, "engram_layer_ids", None) is not None
                )

atom/model_ops/engram.py:828

  • drop_requests() removes the rolling window without first joining the worker. If a prefetch is in flight, _run() can execute advance() after this removal and recreate the window with the old request's context; when the request ID is reused, the next lookup hashes the wrong n-gram. The cache drop mark prevents stale embeddings from being inserted, but it does not protect _window. Wait for the in-flight future before removing the window, or add a generation check so stale work cannot mutate a reused ID.
    def drop_requests(self, seq_ids: list[int]) -> None:
        for seq_id in seq_ids:
            self.cache.drop(seq_id)
        with self._window_lock:
            for seq_id in seq_ids:
                self._window.pop(seq_id, None)

atom/model_ops/engram_layer.py:135

  • The normalized key/query tensors are cast back to the model dtype before the dot product. On the bf16 path this makes the reduction and signed-square-root gate run in bf16, whereas the Engram reference keeps the normalization, elementwise product, reduction, and gate in fp32; the resulting gate error can change which memory values are admitted. Keep the gate math in fp32 and cast only the final residual delta to hidden_states.dtype.
        key = _rms_norm(keys, self.k_weight, self.norm_eps)
        query = _rms_norm(hidden_states, self.q_weight, self.norm_eps)
        gate = (key * query).sum(dim=-1) / math.sqrt(self.hidden_size)

atom/utils/envs.py:28

  • The PR adds ATOM_ENGRAM_CACHE_DIR to the centralized environment registry, but it is absent from docs/environment_variables.md, which the installation docs identify as the full list of runtime variables. Users cannot discover the cache override or its default from the documented configuration surface. Add it to that table with its type, default, and behavior.
    "ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),

recipes/Engram.md:72

  • The recipe presents the host lookup as off the critical path, but the PR provides only value-equality logs and no latency/throughput measurements or reproducible workload. stage_embeddings() still waits for the worker when the prefetch misses its overlap window, so this claim is workload-dependent. Add steady-state decode and prefill measurements against the corresponding non-Engram path, including the host/cache conditions used.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
double-buffered (a per-layer pinned ring) so the next step's fill never races the
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/model_ops/engram_layer.py Resolved
Comment thread recipes/Engram.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved model integration, FP8 portability, lifecycle correctness, and test coverage issues block approval.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (8)

atom/model_engine/model_runner.py:881

  • DeepSeek-V4.1 is not actually runnable from this tree: DeepseekV4ForCausalLM has no build_engram_host, and its DeepseekV4Model never injects an EngramOp into the layer loop, so a config containing the advertised engram_* block always raises here before loading. The result is a fail-closed rejection rather than host Engram support. Add the model-side attachment/weight-skip/residual-injection path, or narrow the PR description to standalone plumbing and track model integration explicitly.
            if not hasattr(model_class, "build_engram_host"):
                raise NotImplementedError(
                    f"{model_class.__name__} config declares engram layers but the "
                    "model class does not implement build_engram_host; serving "
                    "would silently ignore the engram weights"
                )

atom/model_engine/model_runner.py:950

  • These new runner/scheduler hooks have no integration test with a model exposing build_engram_host: the added tests exercise EngramHost and EngramOp in isolation, so row ordering, engram_dropped cleanup, and CUDAGraph staging can regress while the suite stays green. Add a runner-level stub/integration test covering one decode prefetch, a chunked prefill, and request-id reuse.
    def _stage_engram(self, batch: ScheduledBatch) -> None:
        """Stage this step's engram embeddings to the device before the forward.

        Decode stages the rows the prefetch gathered; prefill gathers every
        prompt position's n-gram (carrying the tokens preceding a chunk); a dummy
        warmup/capture pass stages zeros.

atom/model_engine/model_runner.py:987

  • When prefix caching serves the entire prompt, the scheduler skips prefill and sends this request directly to decode. In that path no stage_prefill(..., final_mask=...) call seeds the rolling window, so stage_embeddings falls back to the one decode anchor and hashes [pad, ..., anchor] instead of the prompt's trailing ngram-1 tokens. The first decode Engram contribution is therefore wrong for full prefix-cache hits; carry the prompt tail into the first decode batch (or seed it before staging) whenever no local prefill ran.
        # `tokens` recomputes a prefetch miss only for a missed row with no
        # window (a cold/just-admitted row, where scheduled_tokens is its real
        # anchor); a carried-over row uses its window, so its placeholder is never
        # read.
        tokens = batch.scheduled_tokens[: len(seq_ids)].astype(np.int64).reshape(-1, 1)

atom/model_engine/scheduler.py:1838

  • The main scheduler can return an empty ScheduledBatch when no decode sequence was selected, and EngineCore then skips the forward (and therefore _stage_engram). Draining _engram_drop_pending here loses those IDs, leaving their cache/window state alive; a later reused request ID can consume stale embeddings. Keep the pending IDs when scheduled_seqs is empty, as the prefill construction above already does, or explicitly process drops for empty batches.
            engram_dropped=self._drain_engram_dropped(),

atom/model_engine/scheduler.py:541

  • config_declares_engram explicitly accepts an object whose nested text_config is a dict (the new test covers _Obj(text_config={...})), but this branch only checks attributes on nested. For that supported shape nested_has is false, so tc becomes the root object and the next access to tc.engram_max_ngram_size fails instead of reading the nested block. Mirror the dict check used by config_declares_engram.
                nested = getattr(hc, "text_config", None)
                nested_has = (
                    nested is not None
                    and getattr(nested, "engram_layer_ids", None) is not None
                )

atom/model_ops/engram.py:604

  • The PR's purpose is to hide the host lookup behind GPU work, but the supplied equality logs only establish correctness; there are no steady-state latency/throughput numbers or a reproducible V4.1 GPU run showing that stage_embeddings does not routinely wait for the worker on the miss path. Add decode/prefill measurements with GPU, TP, checkpoint, and command details before treating the lookup as off the critical path.
class EngramPrefetcher:
    """Runs hash + host gather for the next step while the GPU works on this one.

    One worker thread, not a thread per step: the work is numpy and torch gather
    which release the GIL, and an unbounded thread-per-step (as in the reference)
    both races with its own consumer and contends with the runner for the GIL.

    `submit_compute` returns a Future so the consumer waits for the queued
    prefetch before reading the cache; a genuine cache miss (an evicted or
    just-admitted row) is then recomputed inline. The prefetch only decides
    whether the work was already done, never the answer -- it is not a bounded
    no-stall policy: `stage_embeddings` waits for the worker.

atom/model_ops/engram.py:828

  • drop_requests removes the rolling window without synchronizing with the worker. An in-flight _run can execute advance() after this pop, recreate the dropped sequence's window, and leave stale context behind; if the sequence id is reused, windows_for_recompute can then hash that stale context instead of the new request's token. Wait for the current prefetch before marking/removing dropped sequences, or otherwise serialize drop_requests with advance().
    def drop_requests(self, seq_ids: list[int]) -> None:
        for seq_id in seq_ids:
            self.cache.drop(seq_id)
        with self._window_lock:
            for seq_id in seq_ids:
                self._window.pop(seq_id, None)

atom/utils/envs.py:28

  • ATOM_ENGRAM_CACHE_DIR is a new user-facing ATOM_* setting, but it is absent from docs/environment_variables.md, whose header says that file documents the project's environment variables. Users therefore cannot discover how to relocate the compressed-vocabulary cache or its default path. Add the variable, default, and behavior to the public table.
    # Directory for the engram compressed-vocab cache (.npz). Empty = default
    # ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build.
    "ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread atom/model_ops/engram.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved correctness and integration issues remain in model wiring, prefetch lifecycle, and validation coverage.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (8)

atom/model_engine/model_runner.py:3241

  • Every decode row is prefetched here, including rows whose sampled token immediately ends the request (EOS, a stop sequence, or the max-token limit). The scheduler only queues those ids for engram_dropped on the following batch, so a terminal request still performs the very large random host-table gather for an embedding that will never be consumed; if the server goes idle, that work and its cached window also remain until another batch arrives. Filter terminal rows using the same stop decision as the scheduler before submitting the prefetch.
                self.engram.prefetch_next(req_ids, sampled_tokens)

atom/model_engine/model_runner.py:1001

  • The new runner/scheduler path is covered only by host-level unit tests; there is no test that drives _stage_engram with a ScheduledBatch through a model forward, including the padded CUDAGraph row count and the asynchronous H2D/compute-stream handoff. A row-order or stale-buffer error can therefore pass tests/model_ops/test_engram.py while failing or returning wrong values in serving. Author must add a runner/model integration test covering decode and chunked prefill (with mocked GPU primitives if needed).
        padded_rows = len(seq_ids)
        forward_mode = get_forward_context().context.forward_mode
        if forward_mode is not None and forward_mode.use_cudagraph:
            padded_rows = max(padded_rows, int(forward_mode.running_tokens))
        self.engram.stage_embeddings(seq_ids, tokens, padded_rows=padded_rows)

atom/model_engine/model_runner.py:876

  • [verified] support_model_arch_dict resolves DeepseekV4ForCausalLM to atom.models.deepseek_v4, but no in-tree model defines build_engram_host or attaches EngramModules. A V4.1 config with engram_layer_ids therefore reaches this guard and raises instead of serving the model, so the advertised DeepSeek-V4.1 path is not usable. Author must add the registered model's Engram wiring (including residual injection and loader exclusions) or scope this change explicitly as a library-only follow-up.
            if not hasattr(model_class, "build_engram_host"):

atom/model_ops/engram.py:843

  • drop_requests can race with the worker: cache.drop suppresses the stale put, but the worker may still enter advance() after this window is popped and recreate the sequence's rolling context. That stale window can consume the bounded window store, evict a live request's context, and make a later miss hash with the wrong history. Drain/wait for the in-flight prefetch before clearing the cache and windows.

        self._inflight = self._pool.submit(_run)
        return self._inflight

atom/model_ops/engram.py:152

  • engram_kernel_size is parsed into EngramConfig, but no code consumes kernel_size and EngramOp has no short-convolution path. A checkpoint/config that declares a non-default kernel therefore appears supported while silently omitting part of the module. Remove this field for the V4.1-only implementation or reject configurations that require the unsupported convolution.
            kernel_size=int(text_config.get("engram_kernel_size", 4)),

recipes/Engram.md:116

  • This integration example puts the host lookup inside DeepseekV4Model.forward, but that class is @support_torch_compile-decorated (atom/models/deepseek_v4.py:4290). The mutable host reference and dynamic embeddings() slice would be traced in the compiled boundary, contrary to the repository's compile-boundary rule and liable to stale/capture-time behavior. Author must revise the recipe to specify a supported non-compiled integration boundary and how staged embeddings are passed into the compiled model.
```python
for layer_id, layer in enumerate(self.layers):
    if self.engram is not None and layer_id in self.engram:
        if self.engram_host is None:
            raise RuntimeError(

tests/model_ops/test_engram.py:195

  • [verified] This supposedly independent scalar oracle takes layer_multipliers, head_vocab_sizes, and head_offsets directly from mapping, so a wrong reference seed/RNG or prime assignment is reproduced by both the implementation and the test. The test can catch vectorization errors, but it cannot validate the claimed bit-exact reference mapping. Author must add fixed expected hashes/rows from the reference implementation, including at least one known multiplier/hash vector.
    )
    mult = mapping.layer_multipliers[layer_id]
    sizes = mapping.head_vocab_sizes[layer_id]

tests/model_ops/test_engram.py:396

  • [verified] test_engram_op_and_checkpoint checks only shapes and exceptions; it never compares the gate, signed square-root, projection split, or value contribution against an independent numerical reference. A wrong Engram formula would therefore pass this suite while changing every model output. Author must add a fixed-weight numerical test against a hand-computed or reference implementation.
        rt.stage_prefill([81], [np.arange(9)], [np.array([], dtype=np.int64)])
    rt.shutdown()


def test_engram_op_and_checkpoint():
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/utils/envs.py
environment_variables: dict[str, Callable[[], Any]] = {
# Directory for the engram compressed-vocab cache (.npz). Empty = default
# ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
Comment thread recipes/Engram.md
Comment on lines +69 to +73
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
double-buffered (a per-layer pinned ring) so the next step's fill never races the
previous step's in-flight H2D, while the device buffer stays fixed for CUDAGraph.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Multiple unresolved correctness, integration, lifecycle, startup, testing, and performance risks require author changes and human review.

Review details

Suppressed comments (12)

atom/model_engine/model_runner.py:880

  • [verified] DeepseekV4ForCausalLM is the registered V4 implementation, but it does not expose build_engram_host (the recipe also states that no in-tree model wires the contract), so an actual V4.1 config reaches this guard and raises before serving any request. The advertised DeepSeek-V4.1 path is therefore not runnable. Author must wire the Engram modules, skipped table prefixes, and residual injection into the V4 model, or explicitly scope/rename this PR as library plumbing.
            if not hasattr(model_class, "build_engram_host"):
                raise NotImplementedError(
                    f"{model_class.__name__} config declares engram layers but the "
                    "model class does not implement build_engram_host; serving "
                    "would silently ignore the engram weights"

atom/model_engine/model_runner.py:3348

  • [verified] The new runner staging/prefetch path is covered only by the host/device unit tests; there is no test that drives ModelRunner.forward() through prefill, decode, CUDAGraph padding, and pending drops. A mismatch between scheduler batch rows and the staged Engram rows can therefore reach production as a crash or silent residual misalignment. Author must add a mocked integration test covering both prefill and decode hook ordering.
        if getattr(self, "engram", None) is not None:
            self._stage_engram(batch)
        logits, hidden_states = self.run_model(input_ids, batch)

atom/model_engine/scheduler.py:1822

  • This drains the pending drop IDs even when scheduled_seqs is empty. EngineCore skips the forward and postprocess for that batch when len(req_ids) == 0, so _stage_engram never consumes engram_dropped; the finished request's cache/window can then survive and a reused ID may read stale embeddings. Preserve the list until a non-empty batch is actually forwarded.
            engram_dropped=self._drain_engram_dropped(),

atom/model_engine/scheduler.py:288

  • The new prefill_context slicing is correctness-critical for chunked/prefix-cached prompts, but the PR adds no scheduler or runner integration test for a nonzero num_cached_tokens; the existing Engram tests call stage_prefill directly with hand-built contexts. An offset error here would pad the first n-grams and silently change the lookup rows. Author must add a scheduler-level test covering a cached middle chunk and verify the resulting staged embeddings/window.
        self.prefill_context: list[list[int]] | None = None
        if engram_ngram is not None and engram_ngram > 1:
            k = engram_ngram - 1
            self.prefill_context = [
                (

atom/model_ops/engram.py:859

  • drop_requests removes the rolling window without first joining the worker. A prefetch that is already in _run can execute advance() after this removal, recreate the dropped request's window, and evict a live request when the window reaches capacity; the next prefetch for that live request then hashes with missing context. Wait for the in-flight future before removing the cache entry/window so the drop is atomic with respect to the worker.
    def drop_requests(self, seq_ids: list[int]) -> None:
        for seq_id in seq_ids:
            self.cache.drop(seq_id)
        with self._window_lock:
            for seq_id in seq_ids:
                self._window.pop(seq_id, None)

atom/model_ops/engram.py:289

  • For the V4.1 config this loop runs once per raw tokenizer id (up to 16,000,000) and performs a separate decode call for each one. On a cold cache every serving rank therefore pays millions of Python/tokenizer calls before the model can start, far beyond the tens-of-seconds behavior of the 129k-token demo. Author must provide a precomputed compressed-vocabulary artifact or a genuinely batched/native build path, and measure cold-start cost for the 16M vocabulary.
        key_to_new: dict[str, int] = {}
        lookup = np.empty(vocab_size, dtype=np.int64)
        next_id = 0
        for tid in range(vocab_size):
            text = self._tokenizer.decode([tid], skip_special_tokens=False)

atom/model_ops/engram.py:210

  • Even on a cache hit, _cache_key() serializes the entire fast tokenizer with backend_tokenizer.to_str() before it can locate the .npz. With the V4.1 raw vocabulary this creates a large transient string on every rank and makes the cache lookup itself scale with the full tokenizer. Author must use a stable file/config fingerprint (or otherwise bound the serialization cost) and verify the memory/startup impact on a real V4.1 tokenizer.
        serialized = (
            backend.to_str()
            if backend is not None and hasattr(backend, "to_str")
            else None
        )

atom/model_ops/engram_layer.py:304

  • EngramModules.from_checkpoint() is the production path that resolves the six per-layer tensor names, maps the ~98 GB table, and applies its scale layout, but the added tests only call EngramOp.load_checkpoint_weights() with in-memory tensors. A wrong weight_map key, shard handle lifetime, or table/scale shape would therefore survive CI until model startup. Author must add a minimal safetensors fixture that exercises from_checkpoint() and build_engram_host() end to end.
        ops: dict[int, EngramOp] = {}
        tables: dict[int, HostEmbeddingTable] = {}
        for layer_id, num_rows in zip(config.layer_ids, config.num_embeddings):
            tables[layer_id] = HostEmbeddingTable(
                _get(_EMBED.format(layer_id)),
                num_rows=num_rows,
                head_dim=config.head_dim,
                scale=_get(_EMBED_SCALE.format(layer_id)),
            )

atom/utils/envs.py:28

  • ATOM_ENGRAM_CACHE_DIR is a new user-facing environment variable, but it is not listed in docs/environment_variables.md (the recipe only describes the cache behavior, not this override). Users cannot discover how to place the large generated cache on suitable storage. Author must add the variable, default, and purpose to the environment-variable documentation.
    # Directory for the engram compressed-vocab cache (.npz). Empty = default
    # ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build.
    "ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),

recipes/Engram.md:72

  • The central overlap claim is not backed by a latency/throughput measurement or a reproducible V4.1 workload. The implementation explicitly waits for the worker when the gather misses the step window, so this can become a synchronous host stall rather than an off-critical-path lookup. Author must add steady-state measurements (GPU, ROCm, model, batch/concurrency, and cold/warm cache) showing the overlap benefit and fallback cost.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
double-buffered (a per-layer pinned ring) so the next step's fill never races the

tests/model_ops/test_engram.py:400

  • This only asserts output shapes for EngramOp.forward; the key/value split, RMS-normalization, signed-root gate, and value broadcast can all be wrong while these assertions still pass. Because this is the device contribution that changes model logits, an independent deterministic reference is needed to catch numerical regressions. Author must add a fixed-weight CPU reference comparison for the forward formula.
def test_engram_op_and_checkpoint():
    op = make_op()
    # ATOM residual streams are [num_tokens, hc, dim], with optional leading dims.
    assert op(torch.randn(5, 2, 16), torch.randn(5, 24)).shape == (5, 2, 16)
    assert op(torch.randn(2, 3, 2, 16), torch.randn(2, 3, 24)).shape == (2, 3, 2, 16)

tests/model_ops/test_engram.py:186

  • [verified] This "independent scalar reference" reads layer_multipliers, head_vocab_sizes, and pad_id directly from mapping, so a wrong seed/stride, bucket layout, or pad mapping makes both implementations agree. The suite therefore does not verify the hash constants against the DeepSeek/reference vectors. Author must add fixed reference hash/row vectors from the upstream algorithm.
def test_hash_layer_matches_scalar_reference():
    """`hash_layer` + `to_row_indices` must match an independent scalar reimpl
    (same constants, different code), catching vectorization/head/left-pad bugs."""
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical staging and computation issues, plus lifecycle cleanup and coverage gaps, remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (9)

atom/model_engine/model_runner.py:3347

  • The new per-forward staging hook is in Tier-1 ModelRunner, but the added tests exercise only the host/op library; no scheduler/runner test covers prefill/decode ordering, finished-request drops, or padded CUDAGraph rows. A regression here can leave stale embeddings or fail only under capture. Add a CPU/source-level lifecycle test or an integration fixture for these paths.
        if getattr(self, "engram", None) is not None:
            self._stage_engram(batch)

atom/model_engine/scheduler.py:1822

  • _drain_engram_dropped() is unconditional here, but EngineCore skips runner.forward when scheduled_batch.req_ids is empty (atom/model_engine/engine_core.py:412-433). An idle tick therefore clears these IDs from _engram_drop_pending without calling EngramHost.drop_requests(), leaving finished/preempted requests in the host cache/window until eviction and defeating stale-state cleanup. Drain only when scheduled_seqs is non-empty, as the prefill branch already does, or route cleanup through an empty-batch path.
            engram_dropped=self._drain_engram_dropped(),

atom/model_ops/engram.py:982

  • This token D2H host buffer is created through torch.empty() without a device while the runner's default device is CUDA during Engram host construction (model_runner.py:690). It therefore attempts to pin a CUDA tensor instead of allocating CPU-pinned storage, so EngramHost initialization can fail before any request runs. Create the tensor with device="cpu" before pinning it.
            self._token_host = [
                torch.empty(max_num_tokens, dtype=torch.int64).pin_memory()
                for _ in range(self._n_token_slots)

atom/model_ops/engram.py:60

  • engram_layer_ids is also defined as an empty tuple/list by ordinary DeepSeek-V4.1 config classes when Engram is disabled, but this predicate treats any non-None value—including []—as an active declaration. That makes the runner reject non-Engram configs (and can cause a root wrapper's empty defaults to shadow a nested text config that actually contains Engram). Treat an empty layer list as absent here and keep EngramConfig.from_hf consistent with that behavior.
        return got is not None

atom/model_ops/engram.py:860

  • drop_requests can remove the window while the single worker is still between submitting and entering advance(). In that interleaving, advance() recreates the dropped request's window after this method returns; if the sequence ID is reused, the new request inherits the stale context and hashes the wrong n-grams. Synchronize with the in-flight prefetch before removing the window, or add a generation/tombstone check under _window_lock.
        for seq_id in seq_ids:
            self.cache.drop(seq_id)
        with self._window_lock:
            for seq_id in seq_ids:
                self._window.pop(seq_id, None)

atom/model_ops/engram_layer.py:140

  • The new forward path is only tested for shapes and parameter dimensions; no assertion checks the signed-square-root sigmoid gate, per-branch RMS normalization, or shared-value broadcast against a hand-computed result. A transposed gain or incorrect gate formula would therefore pass the tests while producing incorrect logits. Add a small numerical reference case for this core computation.
        gate = (key * query).sum(dim=-1) / math.sqrt(self.hidden_size)
        # Signed square root before the sigmoid: keeps the gate responsive for
        # small scores without letting large ones saturate it.
        gate = gate.abs().clamp_min(1e-6).sqrt() * gate.sign()
        return gate.sigmoid().unsqueeze(-1) * value.unsqueeze(-2)

atom/model_ops/engram_layer.py:185

  • The checkpoint test exercises load_checkpoint_weights with float32 all-ones weights, so it never enters the actual raw-uint8 FP8 decode path used by the V4.1 checkpoint, nor does it validate the 32x32 block layout with nontrivial values. A byte reinterpretation or row/column scale error in this loader would still pass. Add a raw FP8/E8M0 fixture with distinct values and scales.
            wkv_vals = decode_fp8(wkv, torch.float32)
            wkv = (
                wkv_vals.reshape(rows // block, block, cols // block, block)
                * ws.reshape(rows // block, 1, cols // block, 1)
            ).reshape(rows, cols)

atom/utils/envs.py:28

  • This adds a user-facing ATOM_ENGRAM_CACHE_DIR, but the repository's environment-variable reference (docs/environment_variables.md) lists the supported ATOM_* settings and has no entry for it. Without documenting the override and its default, operators cannot discover how to place the compressed-vocab cache on a shared/local filesystem.
environment_variables: dict[str, Callable[[], Any]] = {
    # Directory for the engram compressed-vocab cache (.npz). Empty = default
    # ~/.cache/atom/engram. Read by CompressedTokenizer._load_or_build.
    "ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),

recipes/Engram.md:73

  • The recipe claims the small H2D transfer overlaps GPU work, but _embeddings_h2d first makes copy_stream wait for the compute stream and wait_for_embeddings() then makes compute wait for the copy event. This serializes the transfer between the previous and next forwards; only the host lookup can overlap prior work, so H2D remains on the critical path despite the documented claim. Either redesign the handoff with another safe device buffer or revise the claim and add measurements showing the actual overlap.
The decode gather is prefetched a step early on a worker thread and its small
result is H2D'd on a side stream, so it overlaps GPU work **when it finishes within
the step**; `stage_embeddings` waits for it otherwise. The host staging buffers are
double-buffered (a per-layer pinned ring) so the next step's fill never races the
previous step's in-flight H2D, while the device buffer stays fixed for CUDAGraph.
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/model_ops/engram.py Resolved
Comment thread atom/model_ops/engram_layer.py Resolved
…sh oracle test

Also address several review passes:
- decode E8M0 code 0 as an exact zero on the raw-uint8 fallback (v4_quant's
  zero-scale sentinel) so an all-zero block stays zero;
- make the decode timeout synchronize with the worker instead of recomputing
  inline from a rolling window the worker may not have advanced yet;
- decode raw-uint8 E4M3 table rows and wkv weights as fp8 (reinterpret) rather
  than reading each byte as an integer 0..255;
- config_declares_engram: probe both the root and a nested text_config, for a
  config object or a raw config.json dict; normalize the same dict shape in the
  scheduler before reading engram_max_ngram_size;
- size the prefetch cache / rolling-window store to the runner's concurrency
  ceiling so an active request's window is never LRU-evicted;
- fail closed before constructing/loading the model when the model class lacks
  build_engram_host, instead of after mapping ~200 GB of tables;
- skip the compressed-vocab disk cache for a slow tokenizer (no fast-
  serialization fingerprint) rather than key it on the vocab map alone;
- drain the outstanding prefetch before admitting a reused seq id and before
  reseeding a resumed request's window, so a stale worker cannot insert rows or
  mix the n-gram window.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved scheduler/cache lifecycle, synchronization, validation, performance, and documentation issues remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (5)

atom/model_engine/model_runner.py:3347

  • [verified] The PR adds a Tier-1 scheduler/runner contract spanning prefill_context, dropped-request delivery, CUDAGraph bucket padding, and two CUDA side streams, but the new tests cover only CPU hashing/gathering and the device op; they do not exercise a scheduler-to-runner prefill/decode transition, an empty batch with pending drops, or stream/event ordering. A row-order or synchronization regression can therefore silently corrupt Engram output despite the unit suite passing. Author must add an integration-level regression covering these runner hooks, including the CUDAGraph/padded decode case.
            self._stage_engram(batch)

atom/model_engine/scheduler.py:1822

  • [verified] The decode scheduler drains _engram_drop_pending even when scheduled_seqs is empty, but EngineCore skips runner.forward() for that batch, so _stage_engram() never receives these IDs. A request preempted or finished on an otherwise idle step therefore retains its cache/window; later LRU pressure can evict an active rolling window and make the next multi-token lookup use incomplete context. Author must only drain this list when a batch will actually be forwarded and add an empty-batch regression test.
            engram_dropped=self._drain_engram_dropped(),

atom/model_ops/engram.py:860

  • drop_requests removes the cache entry/sets its drop mark before removing the rolling window, but the worker can already be queued and acquire _window_lock after this method releases it. In that interleaving advance() recreates the dropped sequence's window; its subsequent cache.put() is suppressed, but the stale window remains and can be used if the sequence ID is reused (for example on a decode-only/prefix-cache admission). Drain or otherwise invalidate the in-flight prefetch before removing the window so a dropped request cannot resurrect host context.
    def drop_requests(self, seq_ids: list[int]) -> None:
        for seq_id in seq_ids:
            self.cache.drop(seq_id)
        with self._window_lock:
            for seq_id in seq_ids:
                self._window.pop(seq_id, None)

atom/model_ops/engram_layer.py:149

  • [verified] test_engram_op_and_checkpoint checks only output shapes, dtype, and checkpoint tensor dequantization; it never compares the device result with an independent numerical reference. A wrong key/value split, RMS-normalization epsilon, signed-square-root gate, or sigmoid can therefore pass the new suite while changing every Engram contribution. Author must add a deterministic reference calculation with nontrivial weights and inputs covering the gate and residual-dtype cast.
        gate = gate.abs().clamp_min(1e-6).sqrt() * gate.sign()
        contribution = gate.sigmoid().unsqueeze(-1) * value.float().unsqueeze(-2)
        return contribution.to(hidden_states.dtype)

atom/utils/envs.py:28

  • [verified] This adds the ATOM_ENGRAM_CACHE_DIR runtime override, but the central docs/environment_variables.md says it is the full discoverable list and does not mention this variable. Operators cannot find the supported way to move the compressed-vocabulary cache off the default home directory. Author must add the variable, default, and cache-path behavior to the environment reference.
    "ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv("ATOM_ENGRAM_CACHE_DIR", ""),
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread atom/model_ops/engram.py
# the same, so blocking is never worse here and is correct when the worker
# had simply not started; only a worker stuck well past the budget is a
# real fault worth failing on rather than serving a stale window.
if not self.prefetcher.wait(timeout=self._PREFETCH_WAIT_S):
@ZhangLirong-amd

Copy link
Copy Markdown
Collaborator Author

close for #2314 finished merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants