Skip to content

Feat/several sqls generation - #325

Draft
ftatiana-nv wants to merge 3 commits into
mainfrom
feat/several-sqls-generation
Draft

ftatiana-nv wants to merge 3 commits into
mainfrom
feat/several-sqls-generation

Conversation

@ftatiana-nv

Copy link
Copy Markdown
Collaborator

No description provided.

One question produced one query, so a wrong derivation had no competition:
the only recovery was the retry loop re-running the same generator on the
same prompt. This builds a pool instead. Slot 0 stays on the base client and
the base prompt, so BIRD_NCAND=1 is bit-for-bit the old single-candidate
path; slots 1..N-1 sample at a raised temperature and each take a different
derivation.

The strategies are two-stage on purpose. Asking the one SQL call for a richer
`thought` field does not create an independent reasoning path -- the model
writes the same query and narrates it differently. Emitting the plan or the
decomposition as its own structured artifact, in its own call, is what makes
the second call arrive somewhere else, so models.py gains the stage-1
artifacts (query plan, decomposition, decomposition tree, synthetic
same-schema examples) that never reach the caller.

Strategy variation alone leaves the candidates agreeing on which tables to
read, which is where a pool is least useful. Schema readings are therefore a
second, orthogonal axis -- join-preferring, alternative-binding, alternative
base table -- and slot 0 stays free of them, so enabling the axis can only
add readings and never drops the one already obtained.

Every flag in candidate_flags.py defaults to the previous behaviour and is
read per call rather than captured at import, so a sweep can change one
between questions without reloading the module.

`_extract_answer` now exposes the candidate SQL strings (absent, not empty,
when nothing generated SQL, leaving the entry router and the information
agent unchanged) so a caller can score the best-of-N ceiling without reading
the generator's debug log.
Both of these called the embedding library directly while the catalog path
went through `_embed_with_retry`. The library catches per frame rather than
per request, so one refused request zeroes every row it was given, and
neither caller could tell that from a frame that genuinely had nothing to
embed -- the rows were dropped and the run failed later, somewhere else.

The frames here make that worse rather than better. A CustomAnalysis frame is
usually a handful of rows, so a single transient 5xx empties the whole thing
and aborts the ingest at whichever database it landed on; the catalog refresh
runs after a semantic compile has already paid for its descriptions, and
throwing it away for a 429 wastes all of it.

The `0/{len(embedded)}` in both error messages read the embedded frame, which
is empty or None exactly when the message fires, so every failure reported
0/0. They now count the input frame.
@coderabbitai

coderabbitai Bot commented Oct 1, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Comment @coderabbitai help to get the list of available commands.

Comment on lines +50 to +117
def num_candidates() -> int:
"""How many SQL candidates to generate per question (``BIRD_NCAND``).

1 keeps the single-call path, retries included.
"""
return max(1, _int("BIRD_NCAND", 1))


def candidate_temperature() -> float:
"""Sampling temperature for slots 1..N-1 (``BIRD_NCAND_TEMP``).

Slot 0 stays on the base client so ``BIRD_NCAND=1`` is bit-for-bit the old
behaviour.
"""
return _float("BIRD_NCAND_TEMP", 0.8)


def max_parallel() -> int:
"""Cap on candidate threads (``BIRD_NCAND_PARALLEL``).

The shared in-flight semaphore in ``llm_invoke`` is what actually protects
the endpoint; this only bounds thread count.
"""
return max(1, _int("BIRD_NCAND_PARALLEL", 4))


def schema_slots_enabled() -> bool:
"""Whether slots also vary how they read the schema (``BIRD_SCHEMA_SLOTS``)."""
return _bool("BIRD_SCHEMA_SLOTS", False)


def pin_strategy() -> str:
"""Force every slot onto one strategy (``BIRD_PIN_STRATEGY``).

Trades the pool's breadth for repeated sampling of a single generator.
"""
return (os.getenv("BIRD_PIN_STRATEGY") or "").strip()


def slot_plan() -> tuple[str, ...]:
"""Explicit strategy per slot, comma separated (``BIRD_SLOT_PLAN``).

Round-robin caps a strategy at its share of the slots; this is the knob
that moves that share.
"""
raw = (os.getenv("BIRD_SLOT_PLAN") or "").strip()
if not raw:
return ()
return tuple(tag.strip() for tag in raw.split(",") if tag.strip())


def synthetic_n() -> int:
"""How many same-schema demonstrations to invent (``BIRD_SYNTHETIC_N``)."""
return max(2, _int("BIRD_SYNTHETIC_N", 3))


def synthetic_reasoning() -> bool:
"""Whether invented demonstrations carry their derivation (``BIRD_SYNTHETIC_REASONING``)."""
return _bool("BIRD_SYNTHETIC_REASONING", False)


def decomposition_tree() -> bool:
"""Whether decomposition recurses into a tree (``BIRD_DECOMP_TREE``).

The flat variant yields a linear list of table-access steps, which is
close enough to the query plan that the two candidates often agree.
"""
return _bool("BIRD_DECOMP_TREE", False)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all functions are directly related to the bird

can we chnage it?



# Round-robin order for slots 1..N-1. Slot 0 is always baseline so that
# BIRD_NCAND=1 reproduces single-candidate behaviour exactly.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove bird

Comment on lines +175 to +203
def strategy_for_slot(index: int, n_candidates: int) -> str:
"""Which strategy slot *index* runs.

``BIRD_PIN_STRATEGY`` wins over ``BIRD_SLOT_PLAN``, which wins over the
round-robin. A pin also makes slot 0 sample; otherwise one draw of N would
be deterministic and understate the spread the pin is meant to measure.
"""
if n_candidates < 2:
return "baseline"

pinned = flags.pin_strategy()
if pinned:
if pinned in STRATEGY_TAGS:
return pinned
logger.warning(
"BIRD_PIN_STRATEGY=%r is not one of %s; ignoring",
pinned,
", ".join(STRATEGY_TAGS),
)

plan = flags.slot_plan()
if plan:
unknown = sorted({tag for tag in plan if tag not in STRATEGY_TAGS})
if unknown:
logger.warning(
"BIRD_SLOT_PLAN names unknown strategies %s; ignoring the plan",
", ".join(unknown),
)
else:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is coupled to bird only?

Comment on lines +440 to +447
def generate_candidate(index: int) -> tuple:
"""Produce one SQL candidate, returning ``(response, messages, tag)``.

Slot 0 uses the base client and the untouched schema order, so
``BIRD_NCAND=1`` reproduces the previous behaviour exactly. Other
slots run a two-stage method: build an explicit intermediate
artifact, then translate that artifact into SQL in a second call.
"""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove bird

Comment on lines +143 to +158
def test_single_candidate_is_the_untouched_path(recorder, monkeypatch):
"""An unset BIRD_NCAND must not add a strategy message anywhere."""
monkeypatch.delenv("BIRD_NCAND", raising=False)

out = SQLFromCandidatesAgent().execute(_state())

assert out["decision"] == "constructable"
assert len(recorder.calls) == 1
assert recorder.strategy_texts() == []
assert len(out["path_state"]["sql_candidates"]) == 1


def test_pool_runs_one_distinct_strategy_per_slot(recorder, monkeypatch):
monkeypatch.setenv("BIRD_NCAND", "6")

out = SQLFromCandidatesAgent().execute(_state())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is coupled to bird only?

Comment on lines +172 to +176
def test_a_slot_that_fails_drops_out_of_the_pool(monkeypatch):
"""One dead slot must cost one candidate, not the whole question."""
monkeypatch.setenv("BIRD_NCAND", "3")
calls = {"n": 0}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Comment on lines +193 to +212
def test_unusable_candidates_are_filtered_before_selection(monkeypatch):
monkeypatch.setenv("BIRD_NCAND", "2")

def half_empty(_client, messages, schema):
if schema is SQLQueryPlanModel:
return SQLQueryPlanModel(plan="p")
return _ok_sql() if schema is SQLGenerationModel else None

monkeypatch.setattr(
sql_from_semantic, "safe_invoke_with_structured_output", half_empty
)
out = SQLFromCandidatesAgent().execute(_state())
assert all(c.sql_code.strip() for c in out["path_state"]["sql_candidates"])


def test_every_slot_generating_nothing_is_unconstructable(monkeypatch):
monkeypatch.setenv("BIRD_NCAND", "2")
monkeypatch.setattr(
sql_from_semantic,
"safe_invoke_with_structured_output",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Generating a pool only helps if something chooses from it; until now slot 0
shipped and the other candidates were discarded after being paid for. This
node runs each candidate and picks by agreement between their *results*, on
the grounds that candidates which disagree about the SQL but agree about the
answer are evidence for that answer. No LLM calls: the cost is one execution
per candidate.

Three rules, in descending order of what they are worth. Candidates that
fail to execute or come back empty are dropped -- empty being the expensive
case, since an over-constrained predicate returns no rows, several candidates
make that same mistake, and their agreement would outvote the one candidate
that found the data. Survivors are grouped by result signature and the
largest group wins, different SQL returning the same rows being the same
answer. Near-ties break on structural fit against the reference examples,
which is the only signal here that is not candidates agreeing with each
other, and so the only one that can move a decision voting gets wrong.

Comparing a candidate against examples written for another schema needs the
schema-specific part gone, which is what sql_skeleton.py does: identifiers
become C/T, literals L, numbers N, keywords and punctuation survive, and the
comparison runs over token sequences so renaming a column cannot outweigh a
structural difference.

BIRD dev, 1517 questions at 6 candidates: 73.96% shipping slot 0 against
75.54% with this node, 35 questions fixed against 11 broken, p=0.0005. The
bound is structural -- when one candidate is right and five are wrong, no
vote elects it, which leaves this well below the pool's ceiling.

Off by default (BIRD_SQL_SELECT) and the node is not built when it is off, so
the graph keeps the shape it had before candidates existed.
@liofnv liofnv added the enhancement New feature or request label Nov 15, 2026
@cafzal

cafzal commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

@ftatiana-nv, suggestion: close, or rebuild on ReAct. #488 removed text_to_sql_graph and sql_from_semantic, which this draft changes. Generating several SQL candidates could map to multiple run_sql drafts inside the ReAct loop.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants