AIFunction Extensibility - JSON Parsing

[luisquintanilla]

If a consumer wants to create their own AIFunction, they need to know that to have the correct types / names / schemas, they need to override initialization properties because that is how you provide / set schema, description, and name.

Utilities to create JSON, especially the creation of schema and manipulation of JSON Elements and JSON Nodes is done through classes that are private / internal. This means that if users want to change or orverrider functionality, they have to copy/paste from M.E.AI source into their project. This makes it difficult to land rich tool calling and union-like parsing in C# with existing implementation.

Previous implementations which included Metadata made it easier to abstract tool calling regardless of provider.

Sample project:

https://github.com/SteveSandersonMS/dotnet-ai-workshop/tree/main/exercises/CorrectiveRetrievalAugmentedGeneration

https://github.com/SteveSandersonMS/dotnet-ai-workshop/blob/main/exercises/CorrectiveRetrievalAugmentedGeneration/End/StructuredPrediction/AIParserFunction.cs

[stephentoub]

Utilities to create JSON, especially the creation of schema and manipulation of JSON Elements and JSON Nodes is done through classes that are private / internal.

Can you elaborate? What's missing from the public AIJsonUtilities class that you need? AIFunctionFactory just uses those public utilities.

[stephentoub]

cc: @eiriktsarpalis

[colombod]

Will talk with @eiriktsarpalis asap then.

[colombod]

The core issue here is that we are implementing those custom functions to simulate python ability to parse union types, in the repo you can see the planner needs to emit a plan or a result, so that uses the custom made StructuredChatClient (similar behaviour to python llama index structured llm) because in current state of the M.E.AI we can only use parsed results via the generic attribute which allows only 1 type.

[stephentoub]

because in current state of the M.E.AI we can only use parsed results via the generic attribute which allows only 1 type.

I'm obviously missing some context, as I don't follow what that has to do with AIFunction metadata. Can you elaborate on that part, where AIFunctions are coming into play, and on how the previous metadata approach helped here?

I don't understand how this relates to the comments "Utilities to create JSON, especially the creation of schema and manipulation of JSON Elements and JSON Nodes is done through classes that are private / internal" nor which utilities it's referring to are private / internal. AIFunctionFactory is just using public API on AIJsonUtilities to create the schema:

extensions/src/Libraries/Microsoft.Extensions.AI/Functions/AIFunctionFactory.cs

Lines 262 to 267 in 6def38f

	JsonSchema = AIJsonUtilities.CreateFunctionJsonSchema(
	key.Method,
	Name,
	Description,
	serializerOptions,
	key.SchemaOptions);

with AIJsonUtilities providing support for creating schemas for either .NET types or whole MethodInfos, all of which is layered on top of System.Text.Json's public JsonSchemaExporter. What specifically is missing there?

I'd like us to be able to help, but I'm still unclear what the problem is.

we are implementing those custom functions to simulate python ability to parse union types, in the repo you can see the planner needs to emit a plan or a result, so that uses the custom made StructuredChatClient (similar behaviour to python llama index structured llm)

Why are AIFunctions necessary for this? If you're using a service that supports structured output, can you just use that directly? e.g. a quick and dirty example:

using Microsoft.Extensions.AI;
using OpenAI;
using System.Text.Json;

using IChatClient client = new OpenAIClient(Environment.GetEnvironmentVariable("AI:OpenAI:ApiKey")!).AsChatClient("gpt-4o-mini");

Console.WriteLine(await GetResponseAsync<string, int>(client, "Parse this data into json: \"this is a test\""));
Console.WriteLine(await GetResponseAsync<string, int>(client, "Parse this data into json: 42"));

static async Task<(T1, T2)> GetResponseAsync<T1, T2>(IChatClient client, string prompt)
{
    JsonElement je1 = AIJsonUtilities.CreateJsonSchema(typeof(T1));
    JsonElement je2 = AIJsonUtilities.CreateJsonSchema(typeof(T2));

    ChatOptions options = new()
    {
        ResponseFormat = ChatResponseFormat.ForJsonSchema(JsonSerializer.Deserialize<JsonElement>($$"""
            {
              "type": "object",
              "properties": {
                "output": {
                  "oneOf": [
                    {{je1}},
                    {{je2}}
                  ]
                }
              },
              "required": ["output"]
            }
            """)),
    };

    var response = await client.GetResponseAsync(prompt, options);

    using JsonDocument json = JsonDocument.Parse(response.Message.Text);
    JsonElement output = json.RootElement.GetProperty("output");

    try
    {
        return (output.Deserialize<T1>()!, default);
    }
    catch { }

    return (default, output.Deserialize<T2>()!);
}

[stephentoub]

cc: @SteveSandersonMS

[colombod]

would your sample work on non open ai models?
have you looked how llama index and langchain handle structured parsing in general?

[stephentoub]

would your sample work on non open ai models?

It should work with any service that supports structured output. And for models/services that don't support an actual schema but do support JSON, it could be done via prompting, just as GetResponseAsync<T> does when specifying not to use native structured output support.

But that aside, I'm still not clear on how this relates to the original request in this issue. I'd love to better understand. What's missing? What was the code doing with the old metadata that's hard to do now? Etc.

[stephentoub]

Still trying to reverse engineer the concern. Is it that you want a helper function that does the equivalent of this:
https://github.com/SteveSandersonMS/dotnet-ai-workshop/blob/d1b92ebf53d21806df3e5a5fe71ed0a23c8cefbf/exercises/CorrectiveRetrievalAugmentedGeneration/End/StructuredPrediction/AIParserFunction.cs#L24-L35

        JsonElement schemaElement = AIJsonUtilities.CreateJsonSchema(
            type: type,
            serializerOptions: AIJsonUtilities.DefaultOptions,
            inferenceOptions: s_inferenceOptions);

        JsonElement propertiesElement = schemaElement.GetProperty("properties");
        List<AIFunctionParameterMetadata> parameters = [];
        foreach (JsonProperty p in propertiesElement.EnumerateObject())
        {
            AIFunctionParameterMetadata parameterSchema = new(p.Name) { Schema = p.Value, };
            parameters.Add(parameterSchema);
        }

that produces a JsonElement schema for a function whose parameters are the properties of the type?

[colombod]

is doing the same pydantic programs in the python ecosystem, given a type it creates a function that is the factory for that model. in that case they can perform structured parsing even when the model doesn't have parsed response and with prompt tool calling (the react loop) even when tool instructions are not available.

The approach you propose works as long as you have access to model with structured parsing.

[colombod]

look at this docs https://docs.llamaindex.ai/en/stable/module_guides/querying/structured_outputs/

[stephentoub]

I've read all that.

What, very specifically, are you hoping M.E.AI adds here? Are you asking for a method:

public static class AIJsonUtilities
{
    public static JsonElement CreateFunctionJsonSchemaFromType(Type type, ...);
}

? Something else?

[colombod]

That sounds about right, but then still need the factory part for it, so it is more about creating an AIFunction from a type (something a pydantic program), the json schema is a requirement but not the end goal. The goal is to have a way to fallback onto tool calling when structured output is not available natively.

Have you dug into the implementation is the structured parser too?