Ollama Support

[k2an]

I love your project, I want to use it with local ollama+llava and i tried many way including asking chat gpt.
I am on Windows 11, i tried docker and no go. changed api address from settings in frontend also

api key as "ollama" and 
Api Url as http://localhost:11434/v1/ 

and i tested my local ollama+llava answering and running with postman.

changed frontend\src\lib\models.ts

// Keep in sync with backend (llm.py)
// Order here matches dropdown order
export enum CodeGenerationModel {
  GPT_4O_2024_05_13 = "llava", --> HERE
  GPT_4_TURBO_2024_04_09 = "gpt-4-turbo-2024-04-09",
  GPT_4_VISION = "gpt_4_vision",
  CLAUDE_3_SONNET = "claude_3_sonnet",
}

// Will generate a static error if a model in the enum above is not in the descriptions
export const CODE_GENERATION_MODEL_DESCRIPTIONS: {
  [key in CodeGenerationModel]: { name: string; inBeta: boolean };
} = {
  "llava": { name: "LLava", inBeta: false }, --> and here
  "gpt-4-turbo-2024-04-09": { name: "GPT-4 Turbo (Apr 2024)", inBeta: false },
  gpt_4_vision: { name: "GPT-4 Vision (Nov 2023)", inBeta: false },
  claude_3_sonnet: { name: "Claude 3 Sonnet", inBeta: false },
};

also backend\llm.py

Actual model versions that are passed to the LLMs and stored in our logs

class Llm(Enum):
    GPT_4_VISION = "gpt-4-vision-preview"
    GPT_4_TURBO_2024_04_09 = "gpt-4-turbo-2024-04-09"
    GPT_4O_2024_05_13 = "llava"
    CLAUDE_3_SONNET = "claude-3-sonnet-20240229"
    CLAUDE_3_OPUS = "claude-3-opus-20240229"
    CLAUDE_3_HAIKU = "claude-3-haiku-20240307"


 Will throw errors if you send a garbage string
def convert_frontend_str_to_llm(frontend_str: str) -> Llm:
    if frontend_str == "gpt_4_vision":
        return Llm.GPT_4_VISION
    elif frontend_str == "claude_3_sonnet":
        return Llm.CLAUDE_3_SONNET
    elif frontend_str == "llava":
        return Llm.GPT_4O_2024_05_13
    else:
        return Llm(frontend_str)

console and backend errors below

INFO:     ('127.0.0.1', 57364) - "WebSocket /generate-code" [accepted]
Incoming websocket connection...        
INFO:     connection open
Received params
Generating html_tailwind code for uploaded image using Llm.GPT_4O_2024_05_13 model...
Using OpenAI API key from client-side settings dialog
Using OpenAI Base URL from client-side settings dialog
generating code...
ERROR:    Exception in ASGI application
Traceback (most recent call last):      
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\uvicorn\protocols\websockets\websockets_impl.py", line 250, in run_asgi
    result = await self.app(self.scope, self.asgi_receive, self.asgi_send)      
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^      
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\errors.py", line 149, in __call__    
    await self.app(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\cors.py", line 75, in __call__       
    await self.app(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__ 
    raise exc
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__ 
    await self.app(scope, receive, sender)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\routing.py", line 341, in handle
    await self.app(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\routing.py", line 82, in app
    await func(session)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\routing.py", line 289, in app
    await dependant.call(**values)      
  File "C:\Users\k\screenshot-to-code\backend\routes\generate_code.py", line 262, in stream_code
    completion = await stream_openai_response(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\k\screenshot-to-code\backend\llm.py", line 60, in stream_openai_response
    stream = await client.chat.completions.create(**params)  # type: ignore     
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\openai\resources\chat\completions.py", line 1334, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
y\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\openai\_base_client.py", line 1532, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
INFO:     connection closed

If can be use on local server it'll be awesome!
Thanks for consideration

[abi]

We're definitely interested in adding Ollama support to this project. Thanks for opening this issue.

[isaganijaen]

I'm also looking forward to this feature! ✨

[cognitivetech]

👀

[HuangKaibo2017]

Yeah, it will would great to support ollama, LM studio, llama.cpp and more well-known opensource LLMs, like MiniCPM for vision.

[Yitianw]

[immmor]

how's it going now?

[DStarEpoch]

👀

[pdufour]

Ollama already has a OpenAI-compatible API, so all you have to do is change the env variable and create model aliases.

Add to backend/.env

export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_API_KEY=fake-key

Then create aliases for the models that screenshot-to-code-uses

for model in \
   claude-3-5-sonnet-20240620 \
   gpt-4o-2024-05-13 \
   gpt-4-turbo-2024-04-09 \
   gpt_4_vision \
   claude_3_sonnet; do
   ollama cp x/llama3.2-vision $model
done

Then it will call your local models when it makes LLM requests. Change x/llama3.2-vision to whatever model you want.

I've not gotten very good success because all I can run is a 13b model but perhaps the more powerful models will work well.

Note: you need 0.4.0 release of ollama to run the llama-vision model https://medium.com/@tapanbabbar/how-to-run-llama-3-2-vision-on-ollama-a-game-changer-for-edge-ai-80cb0e8d8928.

[abi]

@pdufour thanks for the tip! I'll include this on the README.

[immmor]

export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_API_KEY=fake-key

Got this error:
openai.InternalServerError: Error code: 500 - {'error': {'message': 'llama runner process has terminated: error:Missing required key: clip.has_text_encoder', 'type': 'api_error', 'param': None, 'code': None}}

[pdufour]

@immmor see ollama/ollama#7300

[immmor]

ollama/ollama#7300

Thought my ollama is already 0.4.0... Still 0.3.14...

[immmor]

@immmor see ollama/ollama#7300

I tried this model: minicpm-v:latest
It works!