CodeGate doesn't work with either LiteLLM or OpenRouter

[tan-yong-sheng]

I am trying to use codegate with cline, and connect LLM api via LiteLLM proxy (hosted on cloud)

Environment:

• Windows 11

Here are the steps I did:

1. On laptop, I run a docker container for codegate:

docker run --name codegate -d -p 8989:8989 -p 9090:9090 -e CODEGATE_OPENAI_URL=https://<MY_LITELLM_URL>/v1 --restart unless-stopped ghcr.io/stacklok/codegate

2. Then, I tried to set up the credentials for cline in VS code, as follows:

I thought as I set the environment variable to CODEGATE_OPENAI_URL=https://litellm.tanyongsheng.site/v1, it will change the default environment variable for openai url

However, I still get this error:

And when I tried to switch to gemini 2.0 that I set up in litellm proxy. There is still error coming up:

Hope for help, thanks

[tan-yong-sheng]

I think how it works aldy

As my litellm proxy uses os.environ/GEMINI_API_KEY for the config.yaml file, I need to pass in GEMINI's api key instead of LiteLLM master key: https://docs.litellm.ai/docs/providers/gemini

Extract of sample config.yaml file for my self-hosted litellm proxy

model_list:
  - model_name: gemini-pro
    litellm_params:
      model: gemini/gemini-1.5-pro
      api_key: os.environ/GEMINI_API_KEY 

Here is the amendment I made for CodeGate to work:

Should be working now, and I will close this issue.

[tan-yong-sheng]

But, it seems like leading to another error, which it doesn't trigger when there are secrets inside the file:

Here is the same credentials in my config.ini file

And when I check codegate via docker logs, and I found a lot of errors there:

And not sure if it's related to the errors above, I found there are some 304 errors for my litellm proxy's UI? But, I think I didn't access my litellm proxy's UI that time.... (Not that sure)

I am thinking it's better for me to try to configure it via openai API first, and then transfer to litellm API ...

[jhrozek]

@tan-yong-sheng thanks for filing the issue!

The rate limits were a codegate bug that was fixed in main but not released yet. We'll get a release out shortly.

I've not used the litellm proxy myself, but I assume it works somewhat like openrouter? Sounds like you expected the litellm master key to work? If yes, it might be worth to reopen this issue and investigate.

[tan-yong-sheng]

I see, noted on the info and thanks.

Yes, litellm proxy is kinda similar to openrouter which is to unify the llm endpoints from different providers.

Ya, I previously expect LITELLM_MASTER_KEY to work, but in the end, I found I need to pass the GEMINI_API_KEY here I want to Gemini llm. Also, I could pass in MISTRAL_API_KEY if I want to Mistral llm. It means LITELLM_MASTER_KEY not being used to fill in the image below, but it still works (note: perhaps this is how litellm proxy works and I would add on more info about this when I am available)

As shown in my previous images:

[jhrozek]

ok, reopening for further triage. We need to make sure codegate works well without the workarounds.

Thanks again for finding the bug and filing the issue.

[tan-yong-sheng]

Hi @jhrozek, could I request any docker container (e.g., ghcr.io/stacklok/codegate:test) for testing? I am not that sure if I am doing right to build the docker image for codegate's main repository:

I tried to git clone https://github.com/stacklok/codegate.git and then follow your guide to build docker container on my linux cloud, as shown below:

Yet, when I use cline (even with openai endpoint: https://api.openai.com/v1), it still couldn't trigger and detect credentials. The prompt I used to reproduce the error is read config.ini file

However, for docker logs, no error logs for codegate's docker container as before:

[jhrozek]

Hi @jhrozek, could I request any docker container (e.g., ghcr.io/stacklok/codegate:test) for testing? I am not that sure if I am doing right to build the docker image for codegate's main repository:

I don't think you need to build a local container (unless you want to!) we release several times a month and the current release is just a couple of days old.

I tried to git clone https://github.com/stacklok/codegate.git and then follow your guide to build docker container on my linux cloud, as shown below:

Yet, when I use cline (even with openai endpoint: https://api.openai.com/v1), it still couldn't trigger and detect credentials. The prompt I used to reproduce the error is read config.ini file

I think this is a configuration issue. In order for cline to talk to codegate, you should point it to codegate as the base URL, not to the LLM

However, for docker logs, no error logs for codegate's docker container as before:

I think this would be solved by pointing the extension to codegate.

btw does it still stand that the main issue is not being able to use litellm through the main key?

[jhrozek]

Just a quick note, I think I was able to reproduce your issue (and I think it was the same issue that @danbarr pinged me about some days ago).

Will keep digging to fix this!

[tan-yong-sheng]

I see, thanks a lot. I will test again when the bug (= codegate is not triggered when I ask LLM to read config.ini file to safeguard my credentials) is fixed.

Thanks a lot for your efforts in delivering this product. Tbh, I am impressed with the idea of this project to safeguard credentials and filter risky dependencies when doing AI coding.

[tan-yong-sheng]

btw does it still stand that the main issue is not being able to use litellm through the main key?

Ya, initially I thought we can't use LiteLLM proxy with Cline and Codegate, and it keeps saying authentication error. That's the reason I create this issue.

But, for now, I think it's not that much matter, as I found the workaround to solve this issue. We could use the GEMINI_API_KEY or CODESTRAL_API_KEY for respective LLM models, instead of LITELLM_MASTER_KEY. And it works!

---

Demonstration

Just for your info, here is what I did to use Cline (together with LiteLLM proxy) with and without codegate:

(1) WITHOUT codegate and with LiteLLM proxy

(2) WITH codegate and with LiteLLM proxy

Reply to issue

I hope this explanation is clear. I believe we can maintain the existing configuration, since we have a workaround to enable Cline and Codegate to work together with LiteLLM. Therefore, I suggest closing this issue.

Additional info

(i) the config.yaml for LiteLLM proxy

You could ignore this, but it's for clarify for my setup to self-host LiteLLM proxy on my Linux cloud server. Here is what I setup for config.yaml file for my litellm proxy to access Gemini and Codestral models:

model_list:
  - model_name: gemini/*
    litellm_params:
      # to include all gemini models available
      model: gemini/*
      api_key: <GEMINI_API_KEY_IN_TEXT_FORM>
  
  - model_name: codestral/*
    litellm_params:
      # to include all codestral models available
      model: codestral/*
      api_key: <CODESTRAL_API_KEY_IN_TEXT_FORM>

Reference: https://docs.litellm.ai/docs/proxy/configs

(ii) How to run codegate via docker container

Note: I change CODEGATE_OPENAI_URL from https://api.openai.com/v1 to https://<MY_LITELLM_URL>/v1

docker run --name codegate -d -p 8989:8989 -p 9090:9090 -e CODEGATE_OPENAI_URL=https://<MY_LITELLM_URL>/v1 --mount type=volume,src=codegate_volume,dst=/app/codegate_volume --restart unless-stopped ghcr.io/stacklok/codegate