Feature Request: gbnfgen port

[JohnGalt1717]

https://github.com/IntrinsicLabsAI/gbnfgen

This would be HUGELY useful. You could take C# classes/records and have it generate JSON grammars for them that guaranteed your responses (if you didn't run out of context) would be valid always.

I can envision a wrapper around the whole thing that basically does instruct, but takes a generic which will be the response type and it will return that deserialized from json using this tech.

(the current json grammar while ok, still has lots of cases of inconsistency in results where it doesn't follow the format requested.)

[martindevans]

This would be an amazingly useful thing to have! I'm not sure we'd want to have it in the core LLamaSharp library, but it could definitely be something included in an extra package (e.g. LLamaSharp.GrammarGenerator).

Is it something you're interested in working on?

[JohnGalt1717]

I'll take a look and see how difficult it is to convert the typescript to C# on Monday.

[martindevans]

I had a bit of a look at how it's done in the gbnfgen project, it looks like it's all based on strings?

Just thinking out loud here, I'd love an API for C# which looked more like this:

Grammar g = GrammarBuilder.Generate<T>();

i.e. it generates a grammar for a type, instead of a magic string.

[JohnGalt1717]

I’m thinking exactly that.

And also instruct<TRequest, TResponse>(string systemMessage, TRequest request, CancellationToken cancellationToken)

(and a version without TRequest so that the user message could just be text)

I’d love to be able to use this same approach to tell the AI that the message will be json in the specific format and what it means so it understands is perfectly. The. I could give it system instructions on how to handle the request, the object that is the input and get an object out as the response.

This makes pipelining amazingly powerful if done right. It also makes AI planning work REALLY well b/c chaining calls becomes trivial.

[JohnGalt1717]

Update: I have a handle on the syntax. I'm using their code as a reference and building the same for C# but it will output an actual grammar that will load into LLamaSharp (unless you choose to save it to disk)

What I'd love to do is figure out a way that the same can be used for inputs so that it understands the structure that's coming in.

[martindevans]

What I'd love to do is figure out a way that the same can be used for inputs so that it understands the structure that's coming in.

I'm not sure what you mean by this bit? sorry.

[JohnGalt1717]

In a perfect world your user prompt would also be json and defined by a separate grammar so it understood the language of the user. Ie something like putting the gbnf definition in the system prompt as what it should expect from the user as input.

[martindevans]

oh I see, as far as I'm aware there's nothing like that in llama.cpp. I guess it would have to be part of the model (e.g. trained on JSON inputs).

[github-actions]

This issue has been automatically marked as stale due to inactivity. If no further activity occurs, it will be closed in 7 days.