Ollama API doesn't include models from Admin mode directory

[samfundev]

It would be nice if the Ollama API returned the models listed in the Admin mode directory, so that they could be swapped between. Instead of just returning the active model.

I'd be willing to implement this change.

[LostRuins]

It's not easy to do a mid-request swap in koboldcpp, because the server is shutdown and restarted whenever a model is reloaded, so all existing connections will be dropped.

[samfundev]

Right, I forgot that the model wouldn't be swapped until you sent the completion request. So you would have to send the request twice. Would you want kobold to be able to swap without dropping connections? I could look into implementing that first.

[LostRuins]

You can explore that, but it will be harder than it looks. Let me know if you have any plans.

[samfundev]

I think the way I would implement it would be using a proxy process. The main process would become a proxy, forwarding requests to the correct server processes. There would be one server process for each modality (tts, text, stt, image). This way if you want you can easily swap one modality without having to reload all the others.

[LostRuins]

possible, but that would make streaming a lot more of a headache.

[henk717]

Its definately something we have wanted for a while now, not just for ollama but also for openai. Like lostruins said the reason we dont have it at the moment is because its hard to build. If you can come up with a well functioning solution that would be very welcome.