Repository navigation
Feat/several sqls generation - #325
Draft
ftatiana-nv wants to merge 3 commits into
Draft
ftatiana-nv wants to merge 3 commits into
ftatiana-nv wants to merge 3 commits into
Conversation
One question produced one query, so a wrong derivation had no competition: the only recovery was the retry loop re-running the same generator on the same prompt. This builds a pool instead. Slot 0 stays on the base client and the base prompt, so BIRD_NCAND=1 is bit-for-bit the old single-candidate path; slots 1..N-1 sample at a raised temperature and each take a different derivation. The strategies are two-stage on purpose. Asking the one SQL call for a richer `thought` field does not create an independent reasoning path -- the model writes the same query and narrates it differently. Emitting the plan or the decomposition as its own structured artifact, in its own call, is what makes the second call arrive somewhere else, so models.py gains the stage-1 artifacts (query plan, decomposition, decomposition tree, synthetic same-schema examples) that never reach the caller. Strategy variation alone leaves the candidates agreeing on which tables to read, which is where a pool is least useful. Schema readings are therefore a second, orthogonal axis -- join-preferring, alternative-binding, alternative base table -- and slot 0 stays free of them, so enabling the axis can only add readings and never drops the one already obtained. Every flag in candidate_flags.py defaults to the previous behaviour and is read per call rather than captured at import, so a sweep can change one between questions without reloading the module. `_extract_answer` now exposes the candidate SQL strings (absent, not empty, when nothing generated SQL, leaving the entry router and the information agent unchanged) so a caller can score the best-of-N ceiling without reading the generator's debug log.
Both of these called the embedding library directly while the catalog path
went through `_embed_with_retry`. The library catches per frame rather than
per request, so one refused request zeroes every row it was given, and
neither caller could tell that from a frame that genuinely had nothing to
embed -- the rows were dropped and the run failed later, somewhere else.
The frames here make that worse rather than better. A CustomAnalysis frame is
usually a handful of rows, so a single transient 5xx empties the whole thing
and aborts the ingest at whichever database it landed on; the catalog refresh
runs after a semantic compile has already paid for its descriptions, and
throwing it away for a 429 wastes all of it.
The `0/{len(embedded)}` in both error messages read the embedded frame, which
is empty or None exactly when the message fires, so every failure reported
0/0. They now count the input frame.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
liavnave
reviewed
Oct 1, 2026
Comment on lines
+50
to
+117
| def num_candidates() -> int: | ||
| """How many SQL candidates to generate per question (``BIRD_NCAND``). | ||
|
|
||
| 1 keeps the single-call path, retries included. | ||
| """ | ||
| return max(1, _int("BIRD_NCAND", 1)) | ||
|
|
||
|
|
||
| def candidate_temperature() -> float: | ||
| """Sampling temperature for slots 1..N-1 (``BIRD_NCAND_TEMP``). | ||
|
|
||
| Slot 0 stays on the base client so ``BIRD_NCAND=1`` is bit-for-bit the old | ||
| behaviour. | ||
| """ | ||
| return _float("BIRD_NCAND_TEMP", 0.8) | ||
|
|
||
|
|
||
| def max_parallel() -> int: | ||
| """Cap on candidate threads (``BIRD_NCAND_PARALLEL``). | ||
|
|
||
| The shared in-flight semaphore in ``llm_invoke`` is what actually protects | ||
| the endpoint; this only bounds thread count. | ||
| """ | ||
| return max(1, _int("BIRD_NCAND_PARALLEL", 4)) | ||
|
|
||
|
|
||
| def schema_slots_enabled() -> bool: | ||
| """Whether slots also vary how they read the schema (``BIRD_SCHEMA_SLOTS``).""" | ||
| return _bool("BIRD_SCHEMA_SLOTS", False) | ||
|
|
||
|
|
||
| def pin_strategy() -> str: | ||
| """Force every slot onto one strategy (``BIRD_PIN_STRATEGY``). | ||
|
|
||
| Trades the pool's breadth for repeated sampling of a single generator. | ||
| """ | ||
| return (os.getenv("BIRD_PIN_STRATEGY") or "").strip() | ||
|
|
||
|
|
||
| def slot_plan() -> tuple[str, ...]: | ||
| """Explicit strategy per slot, comma separated (``BIRD_SLOT_PLAN``). | ||
|
|
||
| Round-robin caps a strategy at its share of the slots; this is the knob | ||
| that moves that share. | ||
| """ | ||
| raw = (os.getenv("BIRD_SLOT_PLAN") or "").strip() | ||
| if not raw: | ||
| return () | ||
| return tuple(tag.strip() for tag in raw.split(",") if tag.strip()) | ||
|
|
||
|
|
||
| def synthetic_n() -> int: | ||
| """How many same-schema demonstrations to invent (``BIRD_SYNTHETIC_N``).""" | ||
| return max(2, _int("BIRD_SYNTHETIC_N", 3)) | ||
|
|
||
|
|
||
| def synthetic_reasoning() -> bool: | ||
| """Whether invented demonstrations carry their derivation (``BIRD_SYNTHETIC_REASONING``).""" | ||
| return _bool("BIRD_SYNTHETIC_REASONING", False) | ||
|
|
||
|
|
||
| def decomposition_tree() -> bool: | ||
| """Whether decomposition recurses into a tree (``BIRD_DECOMP_TREE``). | ||
|
|
||
| The flat variant yields a linear list of table-access steps, which is | ||
| close enough to the query plan that the two candidates often agree. | ||
| """ | ||
| return _bool("BIRD_DECOMP_TREE", False) |
Collaborator
There was a problem hiding this comment.
all functions are directly related to the bird
can we chnage it?
liavnave
reviewed
Oct 1, 2026
|
|
||
|
|
||
| # Round-robin order for slots 1..N-1. Slot 0 is always baseline so that | ||
| # BIRD_NCAND=1 reproduces single-candidate behaviour exactly. |
liavnave
reviewed
Oct 1, 2026
Comment on lines
+175
to
+203
| def strategy_for_slot(index: int, n_candidates: int) -> str: | ||
| """Which strategy slot *index* runs. | ||
|
|
||
| ``BIRD_PIN_STRATEGY`` wins over ``BIRD_SLOT_PLAN``, which wins over the | ||
| round-robin. A pin also makes slot 0 sample; otherwise one draw of N would | ||
| be deterministic and understate the spread the pin is meant to measure. | ||
| """ | ||
| if n_candidates < 2: | ||
| return "baseline" | ||
|
|
||
| pinned = flags.pin_strategy() | ||
| if pinned: | ||
| if pinned in STRATEGY_TAGS: | ||
| return pinned | ||
| logger.warning( | ||
| "BIRD_PIN_STRATEGY=%r is not one of %s; ignoring", | ||
| pinned, | ||
| ", ".join(STRATEGY_TAGS), | ||
| ) | ||
|
|
||
| plan = flags.slot_plan() | ||
| if plan: | ||
| unknown = sorted({tag for tag in plan if tag not in STRATEGY_TAGS}) | ||
| if unknown: | ||
| logger.warning( | ||
| "BIRD_SLOT_PLAN names unknown strategies %s; ignoring the plan", | ||
| ", ".join(unknown), | ||
| ) | ||
| else: |
Collaborator
There was a problem hiding this comment.
this is coupled to bird only?
liavnave
reviewed
Oct 1, 2026
Comment on lines
+440
to
+447
| def generate_candidate(index: int) -> tuple: | ||
| """Produce one SQL candidate, returning ``(response, messages, tag)``. | ||
|
|
||
| Slot 0 uses the base client and the untouched schema order, so | ||
| ``BIRD_NCAND=1`` reproduces the previous behaviour exactly. Other | ||
| slots run a two-stage method: build an explicit intermediate | ||
| artifact, then translate that artifact into SQL in a second call. | ||
| """ |
liavnave
reviewed
Oct 1, 2026
Comment on lines
+143
to
+158
| def test_single_candidate_is_the_untouched_path(recorder, monkeypatch): | ||
| """An unset BIRD_NCAND must not add a strategy message anywhere.""" | ||
| monkeypatch.delenv("BIRD_NCAND", raising=False) | ||
|
|
||
| out = SQLFromCandidatesAgent().execute(_state()) | ||
|
|
||
| assert out["decision"] == "constructable" | ||
| assert len(recorder.calls) == 1 | ||
| assert recorder.strategy_texts() == [] | ||
| assert len(out["path_state"]["sql_candidates"]) == 1 | ||
|
|
||
|
|
||
| def test_pool_runs_one_distinct_strategy_per_slot(recorder, monkeypatch): | ||
| monkeypatch.setenv("BIRD_NCAND", "6") | ||
|
|
||
| out = SQLFromCandidatesAgent().execute(_state()) |
Collaborator
There was a problem hiding this comment.
this is coupled to bird only?
liavnave
reviewed
Oct 1, 2026
Comment on lines
+172
to
+176
| def test_a_slot_that_fails_drops_out_of_the_pool(monkeypatch): | ||
| """One dead slot must cost one candidate, not the whole question.""" | ||
| monkeypatch.setenv("BIRD_NCAND", "3") | ||
| calls = {"n": 0} | ||
|
|
liavnave
reviewed
Oct 1, 2026
Comment on lines
+193
to
+212
| def test_unusable_candidates_are_filtered_before_selection(monkeypatch): | ||
| monkeypatch.setenv("BIRD_NCAND", "2") | ||
|
|
||
| def half_empty(_client, messages, schema): | ||
| if schema is SQLQueryPlanModel: | ||
| return SQLQueryPlanModel(plan="p") | ||
| return _ok_sql() if schema is SQLGenerationModel else None | ||
|
|
||
| monkeypatch.setattr( | ||
| sql_from_semantic, "safe_invoke_with_structured_output", half_empty | ||
| ) | ||
| out = SQLFromCandidatesAgent().execute(_state()) | ||
| assert all(c.sql_code.strip() for c in out["path_state"]["sql_candidates"]) | ||
|
|
||
|
|
||
| def test_every_slot_generating_nothing_is_unconstructable(monkeypatch): | ||
| monkeypatch.setenv("BIRD_NCAND", "2") | ||
| monkeypatch.setattr( | ||
| sql_from_semantic, | ||
| "safe_invoke_with_structured_output", |
Generating a pool only helps if something chooses from it; until now slot 0 shipped and the other candidates were discarded after being paid for. This node runs each candidate and picks by agreement between their *results*, on the grounds that candidates which disagree about the SQL but agree about the answer are evidence for that answer. No LLM calls: the cost is one execution per candidate. Three rules, in descending order of what they are worth. Candidates that fail to execute or come back empty are dropped -- empty being the expensive case, since an over-constrained predicate returns no rows, several candidates make that same mistake, and their agreement would outvote the one candidate that found the data. Survivors are grouped by result signature and the largest group wins, different SQL returning the same rows being the same answer. Near-ties break on structural fit against the reference examples, which is the only signal here that is not candidates agreeing with each other, and so the only one that can move a decision voting gets wrong. Comparing a candidate against examples written for another schema needs the schema-specific part gone, which is what sql_skeleton.py does: identifiers become C/T, literals L, numbers N, keywords and punctuation survive, and the comparison runs over token sequences so renaming a column cannot outweigh a structural difference. BIRD dev, 1517 questions at 6 candidates: 73.96% shipping slot 0 against 75.54% with this node, 35 questions fixed against 11 broken, p=0.0005. The bound is structural -- when one candidate is right and five are wrong, no vote elects it, which leaves this well below the pool's ceiling. Off by default (BIRD_SQL_SELECT) and the node is not built when it is off, so the graph keeps the shape it had before candidates existed.
Collaborator
|
@ftatiana-nv, suggestion: close, or rebuild on ReAct. #488 removed |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.