Repository navigation
Conversation
Certification was a review flag only: retrieve never rendered it, so an uncertified description or SQL attribute was used exactly like a certified one (#268, answer-time use, step 1). - retrieve marks certified table and column descriptions and SQL attributes with [certified], after the dash so a definition's name reads back unchanged, and lists certified definitions before uncertified ones that compete. - The system prompt says what the mark means: when a certified and an uncertified definition could both answer, use the certified one. - fetch_tables_by_ids returns description_certified for tables and columns. The new sql_fragments expressions follow the description fallback, so the flag is that of whichever source supplied the text: a table showing its Term's description reports the Term's flag. Reporting which definitions an answer used (step 2) and a strict mode (step 3) are left for discussion. Signed-off-by: Cameron Afzal <cafzal@users.noreply.github.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
|
retrieve orders its output new tables, joins, definitions, then tables seen before, and the size limit cuts what comes last. On Wide World Importers a revenue search filled the limit with eleven tables, so both revenue definitions were only named under ALSO FOUND: certifying one changed nothing the model saw. Certified definitions now lead. They are short and are the deployment's reviewed answer to how a measure is calculated. Uncertified definitions keep their place, so a catalog with nothing certified reads as before. Signed-off-by: Cameron Afzal <cafzal@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft for team discussion. Not ready to merge. This is step 1 of #268's answer-time plan, put up so the team can react to working code. Evaluation results will be added here before review.
What it does
Certification is a review flag today:
retrievenever renders it, so the ReAct agent uses an uncertified description or SQL attribute exactly like a certified one. This PR:retrievemarks certified table descriptions, column descriptions and SQL attributes with[certified]. The mark goes after the dash, so a definition's name reads back unchanged.sql_fragmentsexpressions follow the same fallback, so the flag belongs to whichever source supplied the text.fetch_tables_by_idsreturnsdescription_certifiedfor tables and columns, as an additive field.Not in this PR (for discussion)
datasources.py, the per-table column page) pairs the fallback description with the column's own flag, which is the mismatch this PR avoids. It's left unchanged.Collision note
The ReAct changes are small: about 40 lines in
retrieve.pyand 3 inprompts.py. Those files are also changed by #516, #549 and #567, so this will be rebased onto whichever lands first.Testing so far
description_certifiedfields.main(environmental).Planned evaluation
mainvs this branch with nothing certified. Answers should be unchanged.mainvs this branch, measuring answer accuracy and which definition the agent used.