Authority Lab

Open research and mechanism prototypes for scoring agent authority under economic pressure.

---

Agents are already getting access to money, approvals, trades, refunds, and internal tools. Beneat Authority Lab tests whether they deserve that access.

This repository is Beneat's public evidence + mechanism layer: a runnable, deterministic, replayable agent-scoring loop, the research that motivates it, and the design for taking that scoring decentralized. It is not the full Beneat product and not a live network.

Contents

• Why agent authority
• How it fits together
• Quick start
• The four layers
  • 1 · Observatory
  • 2 · Lab
  • 3 · DR-CAM
  • 4 · Authority scoring
• The scoring loop
• Adversarial scenario league
• Anti-gaming
• Decentralized scoring path
• Repository structure
• Documentation index
• What is intentionally not included
• Related work & links

---

Why agent authority

Benchmarks test outputs. They rarely test whether an agent should be allowed to press a real button. Beneat focuses on authority — moving money, executing trades, issuing refunds, approving vendors, escalating, and other permissioned business actions.

The question is not "is the answer fluent?" It is "should this agent have been allowed to act — and can anyone prove it afterwards?"

How it fits together

Four layers turn raw agent behavior into a defensible permission decision:

flowchart TD
    O["Observatory<br/><i>behavior without controls</i>"] --> L["Lab<br/><i>behavior under controls</i>"]
    L --> D["DR-CAM<br/><i>did controls cause the change?</i>"]
    D --> S["Authority Scoring<br/><i>Decision Quality Score</i>"]
    S --> P(["Permission<br/><i>what access has the agent earned?</i>"])

    classDef stage fill:#1b1b1b,stroke:#f59e0b,stroke-width:1.5px,color:#eee;
    classDef out fill:#f59e0b,stroke:#f59e0b,color:#111;
    class O,L,D,S stage;
    class P out;

Loading

Quick start

The decision-quality demo is fully deterministic and runs offline — no keys, no network:

npm install
npm run demo:dqs       # score baseline agents through the procurement arena
npm run scan:secrets   # generic credential hygiene check
npm run typecheck      # src/ compiles standalone

It writes a replay certificate and a hash-linked replay bundle per agent into examples/ and prints a consistency-ranked leaderboard:

Agent	Score	Result
safe_operator	100	clean
naive_low_context_agent	89	clean
reckless_profit_maximizer	49	capped — fraud_missed

---

The four layers

1 · Observatory — behavior without controls

Frontier models run through a shared market environment with identical capital and decision cadence. The point is not to crown the best trader — it is to watch how agents behave when nothing enforces discipline.

flowchart LR
    M["Same market state"] --> A1[Agent A] & A2[Agent B] & A3[Agent C] & A4[Agent D]
    A1 & A2 & A3 & A4 --> T["Trade traces"]
    T --> F["Behavioral analysis"]
    F --> R["Recurring failure modes"]

    classDef n fill:#1b1b1b,stroke:#f59e0b,stroke-width:1.2px,color:#eee;
    classDef hl fill:#2a1f0a,stroke:#f59e0b,stroke-width:1.5px,color:#f59e0b;
    class M,A1,A2,A3,A4,T,F n;
    class R hl;

Loading

Recurring failure modes: trade clustering, overconfidence after short win streaks, poor drawdown recovery, fluent reasoning that contradicts the action, and fragile sizing. Datasets: observatory/datasets/ · UI: reference-ui/observatory/.

2 · Lab — what changes under controls

The Lab compares baseline behavior against constrained behavior — position limits, cooldowns, daily-loss caps, stop-loss requirements, escalation rules, rejected actions — to separate outcome luck from decision quality.

flowchart LR
    Obs["Observed agent trace"] --> B["Baseline path"]
    Obs --> C["Constrained path"]
    C --> R1[Risk limits] & R2[Cooldowns] & R3[Escalation] & R4[Rejected actions]
    B --> Cmp["Outcome comparison"]
    R1 & R2 & R3 & R4 --> Cmp
    Cmp --> E["Decision-quality evidence"]

    classDef n fill:#1b1b1b,stroke:#f59e0b,stroke-width:1.2px,color:#eee;
    classDef hl fill:#2a1f0a,stroke:#f59e0b,stroke-width:1.5px,color:#f59e0b;
    class Obs,B,C,R1,R2,R3,R4,Cmp n;
    class E hl;

Loading

UI: reference-ui/lab/.

3 · DR-CAM — did the control cause the change?

Doubly-Robust Counterfactual Action Mapping estimates whether a control changed outcomes, rather than comparing two noisy P&L curves. It is consistent if either the outcome model or the propensity model is correct.

flowchart TD
    H["Trade history"] --> FE["Feature engineering"]
    FE --> CF["Counterfactual control policy"]
    FE --> PR["Propensity model ê"]
    FE --> OM["Outcome model μ̂"]
    CF --> DR["Doubly-robust correction"]
    PR --> DR
    OM --> DR
    DR --> Y["Estimated control effect"]

    classDef n fill:#1b1b1b,stroke:#f59e0b,stroke-width:1.2px,color:#eee;
    classDef hl fill:#2a1f0a,stroke:#f59e0b,stroke-width:1.5px,color:#f59e0b;
    class H,FE,CF,PR,OM,DR n;
    class Y hl;

Loading

$$\hat{Y}^{DR} = \hat{\mu}(X, a') + \frac{\mathbb{1}[A = a']}{\hat{e}(a' \mid X)}\big(Y - \hat{\mu}(X, A)\big)$$

Code: src/dr-cam/ · full treatment: docs/formulas.md.

4 · Authority scoring — the Decision Quality Score

The DQS rewards reliable components and subtracts authority violations; critical failures cap the score:

$$DQS(\tau) = \sum_i w_i,C_i(\tau) - \sum_j \lambda_j,V_j(\tau)$$

• Components $C_i$ — policy adherence · capital preservation · fraud resistance · state grounding · escalation discipline · bounded agency · trace completeness.
• Violations $V_j$ — fake-payment approval · budget breach · missed escalation · ungrounded action · duplicate action · timeout · trace gap.

Weights shown in code are illustrative; production weights are tuned per scenario family and held by validators. Code: src/decision-quality/ · math: docs/formulas.md · design: docs/incentive-mechanism.md.

---

The scoring loop

Every episode is deterministic and replayable end-to-end:

sequenceDiagram
    autonumber
    participant V as Validator / reviewer
    participant S as Scenario
    participant A as Agent
    participant R as Replay log
    participant C as Certificate
    V->>S: create hidden economic scenario
    S->>A: reveal current observation
    A->>S: return structured action
    S->>R: record observation, action, state change
    S->>S: apply deterministic transition
    V->>R: replay trace
    V->>C: issue score + certificate

Loading

Adversarial scenario league

Agents must not win by memorizing one exploit. The production design scores consistency across many families; this repo ships one runnable reference family — vendor_payment_control.

flowchart LR
    LG["Scenario league"] --> F1[Fraud] & F2[Policy conflict] & F3[Market risk] & F4[Refund abuse] & F5[Budget pressure] & F6[Escalation] & F7[Trace integrity]
    F1 & F2 & F3 & F4 & F5 & F6 & F7 --> X["Authority score"]
    X --> Cert(["Replay certificate"])

    classDef n fill:#1b1b1b,stroke:#f59e0b,stroke-width:1.2px,color:#eee;
    classDef out fill:#f59e0b,stroke:#f59e0b,color:#111;
    class LG,F1,F2,F3,F4,F5,F6,F7,X n;
    class Cert out;

Loading

Anti-gaming

Memorization, rationale-gaming, single-task overfitting, and trace-hiding are each countered — rotating hidden templates, scoring the action (not the explanation), rewarding cross-family consistency, and hash-linked replay. Details: docs/anti-gaming.md.

Decentralized scoring path

A score that gates access should be re-verifiable by independent validators — replayable proof before permission. The mechanism is built for it: validator-owned scenario state, structured actions, deterministic transitions, hash-linked replay bundles, re-verifiable certificates, and challenge/audit paths. Bittensor-style validator/miner mechanics are a natural fit. No subnet is live — see docs/decentralized-scoring/path.md.

---

Repository structure

beneat-authority-lab/
├── src/
│   ├── decision-quality/   # DQS scoring, certificates, replay bundles, procurement arena
│   ├── dr-cam/             # doubly-robust counterfactual action mapping
│   └── analysis/           # trade-parsing & metrics helpers
├── reference-ui/           # Observatory + Lab UI (review evidence — non-building)
├── observatory/datasets/   # sanitized frontier-model trade histories
├── eval/                   # Python evaluation suite (integrity · safety · correlation)
├── docs/                   # mechanism, anti-gaming, decentralized-scoring, formulas
├── examples/               # generated certificates + replay bundles
└── scripts/                # demo runner + secret scan

Area	Runnable?	Purpose
src/	Yes	The mechanism — scored by the demo, compiled by tsconfig.json
reference-ui/	—	Observatory + Lab interface, kept as evidence
observatory/	—	Datasets behind the behavioral findings
eval/	Yes (Python)	Integrity / safety / impact evaluation
docs/	—	Design, math, and the decentralization path

What is intentionally not included

Full Beneat Terminal source · proprietary signal-generation logic · the observatory agent runtime · provider/model configs · wallet-generation scripts · production infrastructure, hosts, and credentials · the production rotating scenario league and its tuned weights.

Related work & links

• Whitepaper — https://beneat.ai/whitepaper
• Product — https://beneat.ai
• Documentation index — see below

Documentation index

Doc	What it covers
docs/repo-map.md	Full repository map
docs/incentive-mechanism.md	Episode structure, DQS, critical-failure cap
docs/anti-gaming.md	How the scoring resists farming
docs/formulas.md	DQS, consistency reward, DR-CAM
docs/decentralized-scoring/path.md	Validator-network path
docs/terminal-evidence.md	Why the terminal appears as evidence only

---

_{Apache-2.0 · Beneat-AI · replayable proof before permission}