Harden the two-axis analysis: uncertainty, spatial-robust SEs, tests, CI

Statistics. Report LAD-clustered 95% CIs on every headline ratio, with a
delta-method and cluster-bootstrap layer in the new stats/inference.py; switch
the access Poisson from freq_weights to var_weights so effective N is the OA
count. Add common-support companions to the pure-type contrasts, the rate range
(harmonised and circularity-robust variants beside the headline), and heat-DV
and travel-elasticity sensitivities. Recast the size decomposition as a
descriptive covariate adjustment ("account for"), not causal mediation. Point
estimates are unchanged; this adds uncertainty and disclosure around them.

Paper and documents. Fold the CIs into PAPER.md; correct Table 2 so the total
column is a genuine per-OA median of household energy (a median is not additive)
and caption it as such. De-stale ROADMAP/README/CLAUDE/REPRODUCTION (drop the
retracted 6.3x rate and the superseded lock-in figures). Repair references.bib
and engage the counter-literature. Regenerate paper/results_snapshot.txt
end-to-end.

Engineering. Add a pytest suite (26 tests, data-free) and GitHub Actions CI;
validate every data-layer join with match-rate floors that raise; make EPC
dedup, boundary-postcode resolution, and NTS latest-year selection
deterministic; factor shared fetch/text helpers; add data/manifest.json,
.env.example, and stronger multi-output done-detection in the pipeline
orchestrator.

Author: songololo

.env.example

@@ -0,0 +1,10 @@
+# Copy this file to `.env` at the repo root and set the path for your machine.
+# The .env is gitignored; only this example is committed.
+#
+# URBAN_ENERGY_DATA_DIR is the base directory for every downloaded and derived
+# dataset (see CLAUDE.md "Storage layout"). Point it at a disk with roughly
+# 80+ GB free: the raw EPC certificates (~55 GB) and OS Open UPRN dominate the
+# footprint. Caches live under `$URBAN_ENERGY_DATA_DIR/cache`.
+#
+# Use an absolute path.
+URBAN_ENERGY_DATA_DIR=/absolute/path/to/urban-energy-data

.github/workflows/ci.yml

@@ -0,0 +1,43 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  lint-type-test:
+    runs-on: ubuntu-latest
+    env:
+      # The data layer is not downloaded in CI; tests use synthetic frames only.
+      # A placeholder keeps urban_energy.paths importable (it requires the var).
+      URBAN_ENERGY_DATA_DIR: ${{ runner.temp }}/urban-energy-data
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          # cityseer has no wheel for Python 3.14 yet, so the project pins <3.14.
+          python-version: "3.13"
+          enable-cache: true
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Ruff lint
+        run: uv run ruff check .
+
+      - name: Ruff format check
+        run: uv run ruff format --check .
+
+      - name: Type check (ty)
+        # Non-blocking. ty reports a single known pandas-stubs false positive (the
+        # Series.sort_values overload in data/download_fsa.py) and nothing else; the
+        # check runs for visibility but does not gate the build because that one
+        # diagnostic cannot be suppressed without a stub fix. Gate it once the stub
+        # is fixed upstream.
+        run: uv run ty check
+        continue-on-error: true
+
+      - name: Tests
+        run: uv run pytest

.gitignore

@@ -71,6 +71,8 @@ dmypy.json
 *.json
 !package.json
 !*.config.json
+# data-source manifest (tracked; verified by src/urban_energy/fetch.py)
+!data/manifest.json
 *.kml
 *.kmz
 *.gml

CLAUDE.md

@@ -35,7 +35,7 @@ urban-energy/
 │   ├── lock_in.py             # Residual energy gap after best fabric + full EV
 │   ├── form_size_decomposition.py # Heat vs dwelling/household-size decomposition
 │   ├── argument_figures.py    # The two paper/summary.md figures
-│   ├── figures/{oa,nepi}/     # Legacy three-surface PNGs (referenced only by the deferred PAPER)
+│   ├── figures/{oa,nepi}/     # Legacy three-surface PNGs (not referenced by the current two-axis PAPER)
 │   └── archive/               # Archived LSOA analysis scripts
 ├── tests/                     # pytest framework configured, tests pending
 ├── temp/                      # Default $URBAN_ENERGY_DATA_DIR (gitignored)
@@ -174,7 +174,9 @@ are run on demand rather than wired as pipeline stages.
 > **⏸ Pending.** The earlier three-surface / A–G scorecard, the empirical access-penalty model,
 > and the Atlas (XGBoost planning models + static site) were removed from the tree in the
 > two-axis migration (git history holds them). The Atlas scoring + models are **pending
-> reevaluation** for the two-axis frame; the paper is deferred. See
+> reevaluation** for the two-axis frame; the paper ([PAPER.md](PAPER.md)) is now drafted on that
+> frame, with the old three-surface draft archived at
+> [`paper/archive/PAPER_three_surface_deferred.md`](paper/archive/PAPER_three_surface_deferred.md). See
 > [`paper/summary.md`](paper/summary.md) and [ROADMAP.md](ROADMAP.md).
 
 ---

PAPER.md

@@ -9,8 +9,8 @@ computed from open data.
 removed pending a fresh two-axis rebuild — see below).*
 
 > **⏸ Current focus.** The live work is the **[argument](paper/summary.md)** (the canonical
-> two-axis statement) and the **data + analysis pipeline**. The **paper ([PAPER.md](PAPER.md)) is
-> deferred**, and the **Atlas is pending** — its scoring and the XGBoost planning models are to
+> two-axis statement) and the **data + analysis pipeline**. The **paper ([PAPER.md](PAPER.md))** is now
+> drafted on the two-axis frame (finalisation pending), and the **Atlas is pending** — its scoring and the XGBoost planning models are to
 > be reevaluated for the two-axis frame (that code lives in git history). The theory + headline
 > below are the current two-axis frame.
 
@@ -68,7 +68,7 @@ less access per Joule.** This is the carbon/infrastructure lock-in (Seto et al.
 | --- | ---: | ---: | ---: |
 | Heat (metered) | 10,194 | 15,020 | 1.6× |
 | Car travel (NTS-anchored) | 3,240 | 9,272 | 3.1× |
-| **Total energy** | **13,674** | **23,832** | **2.1×** |
+| **Total energy** (per-OA median) | **13,674** | **23,832** | **2.1×** |
 
 The Flat/Detached columns are observed medians; the gap is the compositional flat-to-detached
 estimate per dwelling, so it is not the column quotient. Energy is modelled per dwelling with
@@ -103,7 +103,7 @@ homes and longer trips, while the access deficit is **100% unchanged**. Built fo
 
 ### ⏸ Pending (next phase)
 
-1. **The paper** — deferred ([PAPER.md](PAPER.md)).
+1. **The paper** ([PAPER.md](PAPER.md)): drafted on the two-axis frame; finalisation and submission pending.
 2. **The NEPI Atlas + planning tool** — pending: reevaluate the place-scoring and the XGBoost
    planning models for the two-axis frame (their code lives in git history).
 
@@ -114,7 +114,7 @@ homes and longer trips, while the access deficit is **100% unchanged**. Built fo
 | Path | Purpose |
 | ---- | ------- |
 | [paper/summary.md](paper/summary.md) | **The argument** — canonical two-axis statement (single source of truth) |
-| [PAPER.md](PAPER.md) | The formal IMRaD paper — **⏸ deferred** (old three-surface framing) |
+| [PAPER.md](PAPER.md) | The formal IMRaD paper (current two-axis draft; finalisation pending). Old three-surface draft archived at [paper/archive/PAPER_three_surface_deferred.md](paper/archive/PAPER_three_surface_deferred.md) |
 | [CLAUDE.md](CLAUDE.md) | **Technical brief** — codebase layout, data, architecture, conventions |
 | [REPRODUCTION.md](REPRODUCTION.md) | **How to rebuild** — orchestrator-driven recipe, manual downloads |
 | [ROADMAP.md](ROADMAP.md) | **Status, scope & open work** — incl. the methodology decisions |
@@ -163,7 +163,7 @@ from the tree in the migration (in git history, pending reevaluation).
 
 **Current focus:** keeping the argument + processing pipeline watertight.
 
-**⏸ Pending (next phase):** the paper ([PAPER.md](PAPER.md)); reevaluating the Atlas scoring +
+**⏸ Pending (next phase):** finalising the paper ([PAPER.md](PAPER.md)) (drafted on the two-axis frame); reevaluating the Atlas scoring +
 planning models for the two-axis frame.
 
 ---

README.md

@@ -56,8 +56,8 @@ Stages skip when their declared outputs already exist (`--force` to rebuild).
 
 ```bash
 uv run python stats/oa_network_access.py        # build network-access cache (cityseer, ~12 min)
-uv run python stats/lock_in.py                  # energy 2.0× → optimised 1.5× per household (1.5× → 1.15× per person)
-uv run python stats/access_profile.py           # network 6.3× access per kWh + on-foot gap ~24×
+uv run python stats/lock_in.py                  # energy per dwelling 2.12× → optimised 1.51× (equal family size 1.71× → 1.18×)
+uv run python stats/access_profile.py           # network 3.6× access per kWh + on-foot gap ~24×
 uv run python stats/form_size_decomposition.py  # heat vs dwelling/household size
 ```
 

REPRODUCTION.md

@@ -4,7 +4,7 @@ Single source of truth for status, open work, and methodology decisions.
 Updated 2026-06-15 (two-axis reframe; paper + Atlas deferred).
 
 > **⏸ Current focus.** The live work is the **[argument](paper/summary.md)** and the
-> **data + analysis pipeline**. The **paper ([PAPER.md](PAPER.md)) is deferred**; the **Atlas is
+> **data + analysis pipeline**. The **paper ([PAPER.md](PAPER.md))** is drafted on the two-axis frame (finalisation pending); the **Atlas is
 > pending** — its place-scoring and XGBoost planning models are to be reevaluated for the
 > two-axis frame (that code lives in git history).
 
@@ -19,7 +19,9 @@ Updated 2026-06-15 (two-axis reframe; paper + Atlas deferred).
 
 ### ⏸ Pending (next phase)
 
-1. **The paper** — deferred ([PAPER.md](PAPER.md)).
+1. **The paper** ([PAPER.md](PAPER.md)): drafted on the two-axis frame and matching the canonical
+   `summary.md` numbers; finalisation and submission pending. The old three-surface draft is archived
+   at [paper/archive/PAPER_three_surface_deferred.md](paper/archive/PAPER_three_surface_deferred.md).
 2. **The NEPI Atlas + planning tool** — pending: reevaluate the place-scoring and the XGBoost
    planning models for the two-axis frame (code in git history).
 
@@ -46,11 +48,12 @@ The rebuild targets only what the two-axis analysis consumes:
 
 - **National OA dataset** — assembled in the stats layer.
 - **Network access** (`stats/oa_network_access.py`): cityseer over OS Open Roads, national network
-  built **once** and queried per catchment band, each OA at its own NTS car-trip catchment;
-  rate **6.3× access per kWh**, validated to ~2% of a literal per-OA computation (~12 min).
+  built **once** and queried per catchment band, each OA at its own NTS car-trip catchment,
+  the built-once counts matching a literal per-OA computation to ~2% (~12 min); the rate is
+  **3.6× access per kWh** (access advantage × energy saving, `access_profile.py`).
 - **Two-axis analysis** ([paper/summary.md](paper/summary.md)): NTS-anchored
-  car-travel energy, lock-in (per household 2.0× → 1.5×; per person 1.5× → 1.15×), network access
-  rate (6.3× access per kWh) + on-foot gap (~24×), heat-vs-size decomposition — all on the shared
+  car-travel energy, lock-in (per dwelling 2.12× → 1.51×; at equal family size 1.71× → 1.18×), network
+  access rate (3.6× access per kWh) + on-foot gap (~24×), heat-vs-size decomposition, all on the shared
   `stats/oa_data.py` core.
 - **Two-axis migration cleanup:** stripped the retired three-surface / A–G code and
   the old Atlas; unified the EPC→OA aggregation (`data/aggregate_epc_oa.py`); lean
@@ -78,8 +81,9 @@ These are the contestable scientific choices; none gate acquisition.
 - **Under-recording robustness.** Flat metered energy omits communal/bulk gas (district
   heating); detached omits off-gas. Re-add an under-recording check before publishing the
   heat gradient (the old `form_bias` flags were removed with the pipeline).
-- **Spatial autocorrelation.** Consider spatial error / lag models (or spatially-clustered SEs)
-  on the form/size regression.
+- **Spatial autocorrelation.** LAD-clustered SEs are now the delivered primary inference
+  (every headline ratio carries a LAD-clustered 95% CI; composites carry cluster-bootstrap
+  CIs). What remains open is an optional spatial error / lag model as a cross-check.
 
 ### Forward work (out of current scope)
 - **Atlas (pending):** reevaluate the place-scoring and the XGBoost planning models for the
@@ -90,5 +94,5 @@ These are the contestable scientific choices; none gate acquisition.
 
 ### Paper / repo
 - Finalise `paper/references.bib`.
-- Expand the pytest suite (form-bias tests landed; pipeline/stats coverage next).
+- (Re)establish the pytest suite: none currently (the earlier form-bias tests were removed); pipeline and stats coverage pending.
 - Pre-submission cover-letter framing for the target journal.

ROADMAP.md

@@ -18,7 +18,9 @@
 
 import pandas as pd
 
+from urban_energy.checks import assert_match_rate
 from urban_energy.paths import DATA_DIR
+from urban_energy.text import normalise_postcode as _normalise_postcode
 
 OUTPUT_DIR = DATA_DIR / "statistics"
 
@@ -33,8 +35,10 @@ def normalise_postcode(s: pd.Series) -> pd.Series:
     """
     Normalise postcode format for matching.
 
-    Strips whitespace, converts to uppercase. Does NOT remove the internal
-    space — both DESNZ and Code-Point use the standard "AB10 1AU" format.
+    Strips whitespace, converts to uppercase. Keeps the internal space — both
+    DESNZ and the Code-Point-derived lookup use the standard "AB10 1AU" format,
+    so both sides of the join must retain the space. Thin wrapper over
+    :func:`urban_energy.text.normalise_postcode` with ``keep_space=True``.
 
     Parameters
     ----------
@@ -46,7 +50,7 @@ def normalise_postcode(s: pd.Series) -> pd.Series:
     pd.Series
         Normalised postcode strings.
     """
-    return s.str.strip().str.upper()
+    return _normalise_postcode(s, keep_space=True)
 
 
 def aggregate_postcode_to_oa(
@@ -75,13 +79,13 @@ def aggregate_postcode_to_oa(
     energy["Postcode"] = normalise_postcode(energy["Postcode"])
     lookup["Postcode"] = normalise_postcode(lookup["Postcode"])
 
-    # Join energy to OA lookup
+    # Join energy to OA lookup. m:1 — the lookup holds one OA per postcode, so a
+    # duplicate-postcode lookup can never fan out the energy rows.
     lookup_cols = ["Postcode", "OA21CD", "LSOA21CD"]
-    merged = energy.merge(lookup[lookup_cols], on="Postcode", how="inner")
-    n_matched = len(merged)
-    n_total = len(energy)
-    pct = n_matched / n_total
-    print(f"  Matched {n_matched:,}/{n_total:,} postcodes ({pct:.1%})")
+    merged = energy.merge(
+        lookup[lookup_cols], on="Postcode", how="inner", validate="m:1"
+    )
+    assert_match_rate(len(energy), len(merged), name="energy postcode ↔ OA lookup")
 
     # Compute meter counts per postcode (use max of gas/elec meters)
     elec_meters = pd.to_numeric(
@@ -117,8 +121,14 @@ def aggregate_postcode_to_oa(
     # Count metrics
     results["oa_num_meters"] = merged.groupby("OA21CD")["_meters"].sum()
     results["oa_num_postcodes"] = merged.groupby("OA21CD")["Postcode"].count()
-    # Per-fuel meter counts (feed the Form under-recording flags, TODO #6):
-    # low gas-meter coverage flags off-gas / communal-gas OAs.
+    # Per-fuel meter counts. NOTE (2026-07-08): these once fed the Form-surface
+    # "under-recording" flags (off-gas / communal-gas OAs, methodology item #6),
+    # removed with the form_bias module in the two-axis migration (git 624c50d).
+    # The counts are still emitted because low gas-meter coverage relative to
+    # electricity is a useful off-gas signal. A re-add would derive, per OA:
+    # gas-meter coverage = oa_gas_num_meters / oa_elec_num_meters, plus the
+    # Census TS044 flat share, and flag OAs below a coverage floor as likely
+    # under-recording metered gas (see the archived tests/test_form_bias.py).
     results["oa_elec_num_meters"] = elec_meters.groupby(merged["OA21CD"]).sum()
     results["oa_gas_num_meters"] = gas_meters.groupby(merged["OA21CD"]).sum()
 

data/aggregate_energy_oa.py

@@ -31,7 +31,9 @@
 
 import pandas as pd
 
+from urban_energy.checks import assert_match_rate
 from urban_energy.paths import DATA_DIR
+from urban_energy.text import normalise_postcode
 
 EPC_PATH = DATA_DIR / "epc" / "epc_domestic_spatial.parquet"
 LOOKUP_PATH = DATA_DIR / "statistics" / "postcode_oa_lookup.parquet"
@@ -45,7 +47,25 @@
 
 
 def _band_to_year(band: object) -> float:
-    """Midpoint construction year from an EPC age band (NaN if unparseable)."""
+    """
+    Midpoint construction year from an EPC age band.
+
+    Closed bands ("1900-1929") return the arithmetic midpoint. Open bands return
+    their single boundary year: "2007 onwards" gives 2007, and "before 1900" gives
+    1900 (its upper bound, a conservative lower-bound proxy for the age of a
+    pre-1900 dwelling). This preserves the old-housing signal in the build-year
+    confound rather than discarding it. A band with no parseable year returns NaN.
+
+    Parameters
+    ----------
+    band : object
+        Raw EPC ``CONSTRUCTION_AGE_BAND`` value.
+
+    Returns
+    -------
+    float
+        Representative year, or NaN when no year is parseable.
+    """
     years = [int(y) for y in re.findall(r"\b(1[89]\d{2}|20\d{2})\b", str(band))]
     return sum(years) / len(years) if years else float("nan")
 
@@ -64,7 +84,7 @@ def main() -> None:
             "CONSTRUCTION_AGE_BAND",
         ],
     )
-    epc["POSTCODE"] = epc["POSTCODE"].astype(str).str.strip().str.upper()
+    epc["POSTCODE"] = normalise_postcode(epc["POSTCODE"], keep_space=True)
     floor = pd.to_numeric(epc["TOTAL_FLOOR_AREA"], errors="coerce")
     cur = pd.to_numeric(epc["ENERGY_CONSUMPTION_CURRENT"], errors="coerce")
     pot = pd.to_numeric(epc["ENERGY_CONSUMPTION_POTENTIAL"], errors="coerce")
@@ -76,9 +96,13 @@ def main() -> None:
     )
 
     lookup = pd.read_parquet(LOOKUP_PATH, columns=["Postcode", "OA21CD"])
-    lookup["Postcode"] = lookup["Postcode"].astype(str).str.strip().str.upper()
-    merged = epc.merge(lookup, left_on="POSTCODE", right_on="Postcode", how="inner")
-    print(f"  {len(merged):,} certificates matched to an OA")
+    lookup["Postcode"] = normalise_postcode(lookup["Postcode"], keep_space=True)
+    # m:1 — many certificates share a postcode, but the lookup holds one OA per
+    # postcode, so the lookup cannot fan out the certificate rows.
+    merged = epc.merge(
+        lookup, left_on="POSTCODE", right_on="Postcode", how="inner", validate="m:1"
+    )
+    assert_match_rate(len(epc), len(merged), name="EPC postcode ↔ OA lookup")
 
     # Floor area: median over valid-floor certs, OA kept only if ≥ MIN_EPC_PER_OA.
     floor_oa = (
@@ -112,15 +136,29 @@ def main() -> None:
         .rename(columns={"year": "oa_median_build_year"})
     )
 
+    # Outer merge — each column has its own OA coverage. Floor area is gated by
+    # MIN_EPC_PER_OA (≥5 certs), while intensity and build year keep any OA with
+    # at least one valid cert. So an OA can carry an intensity/build-year value
+    # but a NaN floor area (fewer than 5 floor certs). Downstream consumers must
+    # treat the columns as independently populated, not row-complete.
     oa = (
-        floor_oa.merge(pot_oa, on="OA21CD", how="outer")
-        .merge(cur_oa, on="OA21CD", how="outer")
-        .merge(year_oa, on="OA21CD", how="outer")
+        floor_oa.merge(pot_oa, on="OA21CD", how="outer", validate="1:1")
+        .merge(cur_oa, on="OA21CD", how="outer", validate="1:1")
+        .merge(year_oa, on="OA21CD", how="outer", validate="1:1")
     )
     OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True)
     oa.to_parquet(OUTPUT_PATH, index=False)
 
     print(f"  wrote {len(oa):,} OAs → {OUTPUT_PATH}")
+    print("    per-column coverage (populated / total OAs):")
+    for col in (
+        "oa_median_floor_area_m2",
+        "epc_current_kwh_m2",
+        "epc_potential_kwh_m2",
+        "oa_median_build_year",
+    ):
+        n_pop = int(oa[col].notna().sum())
+        print(f"      {col:<24s} {n_pop:,} / {len(oa):,} ({n_pop / len(oa):.1%})")
     print(f"    floor median     {oa['oa_median_floor_area_m2'].median():.0f} m²")
     print(f"    current median   {oa['epc_current_kwh_m2'].median():.0f} kWh/m²/yr")
     print(f"    potential median {oa['epc_potential_kwh_m2'].median():.0f} kWh/m²/yr")