Safety valve for Structured Output ?

[Mr-Ruben]

Context: Prompt Eng. / Design.

In prompts with structured output, what recommendations/techniques are out there to handle cases where a correct output cannot be provided because the input is wrong and/or a valid answer cannot be provided?

Example:
Input has correct values, but they don't make sense. (e.g. what the numeric result of 3 + K ? )
Therefore an output that correctly considers the Input is not possible.
The output is processed by Pydantic.

---

I can think on one:

class Output(BaseModel):
   <other stuff>
   answer: Union[str,None] 
    <other stuff>

In the prompt, specify that if any kind of error is found, the answer can be None. Being None a value that would not occur otherwise.

---

What approaches have you seen or used?

Note: Union[str,None] = None is the same as Optional[str] = None

[ivanbelenky]

disclaimer

I think it depends greatly on the domain you are working. I am not sure about my answer either, I did not test this thoroughly, so everything below is both intuition and experience guided.

invalid Input

Your example is symbolically specifiable but no further computation can be made. If the goal of this structured output was to compute values, then the output type would be numbers.Number | Uncomputable. Where

class Uncomputable(BaseModel):
    reason = str

Another example. If your goal was to symbolically reduce to the "most compact form" an equation or algebraic expression, then you would try to bias the LLM to recognize incorrect inputs. For instance 2x + 2y --> 2(x+y). But if for some reason your input has a typo or is meaningless like (2x + (2>y)) --> SymbolicError.

Where SymbolicError as in Uncomputable is expressive of the problem at hand and its deterministic input validation.

Needless to say, validating Input is a well known topic. In this context (python + instructor) I would suggest the following taxonomy for how input is validated

• language validation: built in into the language (like function receiving fewer arguments than expected, python will cover this)

• type validation: pydantic will take care of this with the default validators for each built-in type and/or user runtime defined type

• semantic validation: pydantic and instructor can help you in this regard. What I call semantic validation is just, for the lack of a better word, a way to describe every validation on variables that look like

  •   def is_valid(variable: Any, what_means_to_be_valid: str) -> bool: ...

for this instructor counts with the llm_validator that can work in tandem with pydantic off the shelf BeforeValidator for instance. I would suggest reading this.

invalid Output

Output can be incorrect in the following senses

• type: pydantic will throw an error (instructor will usually retry, unless you turned this off)
• semantic: the human will throw an error

If impossible to solve input is common in your InputSet I would suggest once again biasing the LLM to recognize that fact. This can be done either beforehand or specifically in the output model with some field like this_prompt_is_impossible_to_solve: bool (probably a more prompt engineered variable name, after all naming matters.

Needless to say, if you want to use llm_validator for output, this is totally possible. The only thing here is that this will trigger a pydantic error. Usually for I prefer to see the gibberish in the output first (if any) and then validate it, and improve it if possible.

[Mr-Ruben]

https://python.useinstructor.com/concepts/maybe/#defining-the-model