Skip to content

Show certification to the ReAct agent and prefer certified definitions (#268, draft) - #573

Draft
cafzal wants to merge 2 commits into
mainfrom
feat/answer-time-certification
Draft

cafzal wants to merge 2 commits into
mainfrom
feat/answer-time-certification

Conversation

@cafzal

@cafzal cafzal commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Draft for team discussion. Not ready to merge. This is step 1 of #268's answer-time plan, put up so the team can react to working code. Evaluation results will be added here before review.

What it does

Certification is a review flag today: retrieve never renders it, so the ReAct agent uses an uncertified description or SQL attribute exactly like a certified one. This PR:

  • Shows it: retrieve marks certified table descriptions, column descriptions and SQL attributes with [certified]. The mark goes after the dash, so a definition's name reads back unchanged.
  • Ranks it: competing definitions are listed certified-first. The sort is stable, so search order holds within each group.
  • Explains it: one prompt rule: when a certified and an uncertified definition could both answer, use the certified one.
  • Gets the source right: a shown description often comes from a fallback (a table shows its Term's description; a column shows its attribute's). The new sql_fragments expressions follow the same fallback, so the flag belongs to whichever source supplied the text. fetch_tables_by_ids returns description_certified for tables and columns, as an additive field.

Not in this PR (for discussion)

  • Step 2: report which definitions an answer used, and whether each is certified.
  • Step 3: a strict mode that refuses or flags answers relying on uncertified definitions.
  • An existing query (datasources.py, the per-table column page) pairs the fallback description with the column's own flag, which is the mismatch this PR avoids. It's left unchanged.

Collision note

The ReAct changes are small: about 40 lines in retrieve.py and 3 in prompts.py. Those files are also changed by #516, #549 and #567, so this will be rebased onto whichever lands first.

Testing so far

  • New tests: 6 (rendering, ordering, name read-back, and the flag following the description's source).
  • Snapshots: the DAL surface snapshot and goldens were regenerated, and their diffs contain only the two new expressions and the new description_certified fields.
  • Full suite: passes, except 18 Kumo wiring tests that fail the same way on a clean main (environmental).

Planned evaluation

  • Regression: Auto-Ontology Eval on WWI, main vs this branch with nothing certified. Answers should be unchanged.
  • Effect: competing definitions planted with the correct one certified, main vs this branch, measuring answer accuracy and which definition the agent used.

Certification was a review flag only: retrieve never rendered it, so an
uncertified description or SQL attribute was used exactly like a
certified one (#268, answer-time use, step 1).

- retrieve marks certified table and column descriptions and SQL
  attributes with [certified], after the dash so a definition's name
  reads back unchanged, and lists certified definitions before
  uncertified ones that compete.
- The system prompt says what the mark means: when a certified and an
  uncertified definition could both answer, use the certified one.
- fetch_tables_by_ids returns description_certified for tables and
  columns. The new sql_fragments expressions follow the description
  fallback, so the flag is that of whichever source supplied the text:
  a table showing its Term's description reports the Term's flag.

Reporting which definitions an answer used (step 2) and a strict mode
(step 3) are left for discussion.

Signed-off-by: Cameron Afzal <cafzal@users.noreply.github.com>
@cafzal cafzal added enhancement New feature or request correctness Answers, queries, and predictions that are right and explainable labels Nov 19, 2026
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

retrieve orders its output new tables, joins, definitions, then tables
seen before, and the size limit cuts what comes last. On Wide World
Importers a revenue search filled the limit with eleven tables, so both
revenue definitions were only named under ALSO FOUND: certifying one
changed nothing the model saw.

Certified definitions now lead. They are short and are the deployment's
reviewed answer to how a measure is calculated. Uncertified definitions
keep their place, so a catalog with nothing certified reads as before.

Signed-off-by: Cameron Afzal <cafzal@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

correctness Answers, queries, and predictions that are right and explainable enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant