Repository navigation
Description
Purpose
This is a discussion issue, not a bug report. I mapped each of PDD's major design commitments against how frontier models are actually served today, using published research plus evidence from this codebase at commit 817abe2. The goal is to separate the parts of PDD that are well-aligned with serving reality from the parts that are betting against it, so the roadmap can be explicit about which bets it is making.
Short version: the verification machinery is well-aligned and is what makes PDD work. The regeneration-as-reproducibility framing is the part that collides with infrastructure — and the codebase already contains most of the mitigations, they are just described in the docs as cost or overflow measures rather than as fidelity measures.
Static inspection only; no PDD commands were run and no LLM calls were made.
1. Regenerative core vs. non-batch-invariant inference
pdd sync regenerates on prompt change; split includes a regen gate proving prompts can regenerate. This treats the model as a compiler.
Production inference is not batch-invariant: batch size varies with server load, and non-associative floating-point reduction in normalization, matmul, and attention kernels yields different logits at different batch sizes, so near-ties flip under greedy decoding (Thinking Machines, Defeating Nondeterminism in LLM Inference; batch-invariant kernels).
PDD already concedes this correctly — docs/prompting_guide.md:1453: replay "does not guarantee bit-for-bit identical generated code, because the LLM call itself may still be nondeterministic." The gap is that the README's headline framing ("regenerate software from durable intent") reads to an evaluator as a build-reproducibility guarantee, while the implementation provides prompt-level reproducibility plus cache-scoped response stability.
Worth tracking: if providers ship deterministic kernels + pinned weights + seeds as a product, this collision largely dissolves and PDD's model gets substantially stronger.
2. Context assembly vs. effective context
Covered in detail in #2341 — summary: context is modelled as an overflow cliff rather than a quality slope, and the documented remedy (escalate to a 1M-token model) moves prompts further into the degraded regime described by NoLiMa and Chroma's context-rot study.
3. Multi-session batch orientation vs. prefill economics
Prefill is O(n²) in prompt length, and sync runs eight sequential phases while agentic workflows run 11-18 steps over largely overlapping context. PDD never uses provider prompt caching (grep -rn "cache_control" pdd/ --include='*.py' → 0). Also detailed in #2341.
4. Dynamic tags vs. cacheability and reproducibility
<shell> and <web> mutate prompt bytes on every run, which is at odds with both reproducibility and any future prefix caching.
PDD handles the reproducibility half well — --snapshot / pdd replay, evidence manifests with uses_nondeterministic_tags, context.shell_snapshots / context.web_snapshots, secret redaction before hashing, and pdd checkup snapshot as a CI policy gate. If prefix caching is adopted, the ordering rule follows naturally: stable prefix first, dynamic content last.
5. Model cascade vs. fleet heterogeneity
--strength interpolates over the credential-filtered catalog (260 rows in pdd/data/llm_model.csv); overflow/refusal/rate-limit advance to the next candidate; the agentic path chains Claude → Google → Codex → OpenCode.
Serving fleets are already heterogeneous underneath (GPU generations, quantized serving variants, MoE routing under load). Quantization and KV-cache eviction concentrate errors in rare tokens — identifiers, numerals, API names — which prose absorbs and code does not. PDD layers voluntary heterogeneity on top of that involuntary heterogeneity.
Partially instrumented: generation.agentic_fallback and the PDD_PROVIDER_LIMIT marker cover the agentic layer; the llm_invoke candidate loop has no allowlist, no fail-closed mode, and no substitution record.
6. Batch lanes and preemptible compute
Batch APIs and Cloud Batch spot VMs are cheap because they run on preemptible capacity with no mid-course correction. sync --durable exists precisely because workers die from "timeout, crash, ephemeral cloud checkout deletion," and the PDD-Sync-Checkpoint-V1 trailer design is a sound response. The residual cost is latency of error discovery: a bad generation surfaces only after the batch returns, and the fix loop then re-spends across providers.
7. Verification machinery — the part that fits
crash/verify/test/fix loops, mock_contract_validation.py, the deterministic checkup_gates.py suite, atomic FingerprintTransaction writes, and test accumulation are all well-matched to infrastructure reality. They assume the generator is unreliable and verify empirically.
This repo carries 464,952 lines of tests against 230,155 lines of implementation (2:1), with recorded run reports showing 5,856 tests passing at 77.9% average coverage across 40 modules. That ratio is not overhead — it is the mechanism that makes regeneration safe.
Synthesis, offered for discussion
Reading the code rather than the README, PDD's stability comes from the accumulated test suite and the deterministic gates, not from prompts being an authoritative source. Prompts expand to code at 10.1x across 181 matched pairs in this repo, so roughly 90% of every artifact is re-derived each regeneration and is pinned only by tests. #1012 ("pdd sync over-regenerates mature modules and breaks public compatibility") is the same observation from the inside.
That places PDD, operationally, in what Böckeler's analysis on martinfowler.com calls spec-anchored — specs and code kept in sync, code regenerated when specs change — rather than spec-as-source. That analysis also notes nondeterminism persists despite precise specs, echoing model-driven architecture's history.
My suggestion — offered as an outsider, so weight it accordingly — is that aligning the framing with the implementation would make PDD easier to adopt, not harder. "Prompts are the durable intent; accumulated tests are the contract; code is regenerable output" is both more accurate and more defensible to an enterprise evaluator than "prompts are source." The verification machinery is the real differentiator, and the current framing undersells it.
Concrete asks
- Publish measured regeneration fidelity as a function of assembled context size. PDD is uniquely positioned: 181 prompt/code pairs, an accumulated suite, and cost instrumentation already in place.
- State the determinism boundary in the README, matching what
docs/prompting_guide.md:1453andpdd replayalready say correctly. - Adopt prefix caching and order prompts stable-prefix-first (Context limits are treated as a hard overflow boundary, not a quality gradient; and provider prompt caching is never used #2341).
- Track the deterministic-inference work — batch-invariant kernels plus pinned weights and seeds would materially strengthen the regenerative model.
Happy to be corrected on any of this; several readings come from outside the project and maintainers will know where they are wrong.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity