Add Multi model support to java-cfenv #299
Conversation
| */ | ||
| public class GenAIModelInfo { | ||
|
|
||
| public enum Capability { |
There was a problem hiding this comment.
I personally wouldn't make this an enum. We're expecting the capabilities returned by the ai-server to increase over time. If this is an enum we need to ensure this is updated in lock-step with ai-server.
There was a problem hiding this comment.
OK, I have made Capability a non-enum class.
|
|
||
| return models.stream() | ||
| .filter(model -> model.hasCapability(requiredCapability)) | ||
| .findFirst(); |
There was a problem hiding this comment.
With this approach, its only possible to get the first model that matches
There was a problem hiding this comment.
This is true. I originally experimented with allowing for other matching options (getting the last model that matches, preferring a certain model, etc), but in practice this doesn't work well. It's not intuitive how an end user configures that sort of preference, and for first/last model matching, the user can't reliably predict the ordering of models anyway.
I don't think it's best practice for a multi-model plan to publish multiple models with the same capability. If this does happen, and an end user needs to exert precise control over which model is selected, I think that needs to be done through code, not through an auto-config library like java-cfenv.
There was a problem hiding this comment.
We're actually assuming that multiple models within an endpoint with the same capabilities will be used quite frequently, especially when using things like mcp sampling where we want to select a model based on labels rather than an exact model name.
We're also expecting these models to change over time and we want to allow apps to be able to handle this without restarting - hence the introduction of the GenaiLocator which can be used to dynamically be query the models available rather than statically wiring in a model that requires a restart to pick up any changes.
There was a problem hiding this comment.
This is all good, but again, I don't think MCP Sampling or the selection of individual models is handled elegantly within an autoconfig startup library like java-cfenv.
I think java-cfenv is for developers who want an easy button default option on startup. Fine-grained model selection (and certainly runtime model selection) needs to be handled in other code.
Similarly, java-cfenv doesn't provide a mechanism for selecting a specific RabbitMQ server if multiple instances are bound on startup. That's outside the scope of java-cfenv.
There was a problem hiding this comment.
The only other option would be to refuse to bind any models if multiple models with matching capabilities are found. I can implement that if you like.
These changes fully support both the single-model and multi-model plans in the Tanzu Platform GenAI tile.
It uses the
/v1/modelsendpoint for model discovery as described here:https://techdocs.broadcom.com/us/en/vmware-tanzu/platform-services/genai-on-tanzu-platform-for-cloud-foundry/10-2/ai-cf/how-to-guides-discover-models-and-send-openai-requests-to-them.html