feat: Allow overriding default LLM provider endpoints

[zhengkezhou1]

Description

• Motivation: Add new configuration fields to allow customizing the full API endpoint for LLM providers. This enables more flexible integration with various LLM services by allowing custom endpoint.
• What changed: Add PathOverride and AuthHeaderOverride fields to LLMProvider.
• Related issues: Fixes hostOverride setting causes incorrect requests in AI Gateway #11225

Change Type

/kind new_feature

Changelog

Add `PathOverride` and `AuthHeaderOverride` fields for custom LLM provider endpoints

Additional Notes

[zhengkezhou1]

I will add E2E tests soon...

[npolshakova]

Yep, a unit test at the plugin level would be good to add here. You can also test the output envoy config by adding a setup test similar to how this deepseek test is structured: https://github.com/kgateway-dev/kgateway/blob/main/internal/kgateway/setup/testdata/standard/ai-deepseek-prompt-guard.yaml

If you run the setup tests with the input config defined, it will write the output file for you. This can be a quick way to validate the path replacement is correct!

[npolshakova]

Ah! Nevermind I see you've already created internal/kgateway/setup/testdata/standard/ai-custom-url-out.yaml 🎉

[zhengkezhou1]

It works after adding the following:

...
fullPathOverride:
    path: "/api/v1/chat/completions"
...

[zhengkezhou1]

In current testing, it seems that there is no way to validate these new inputs.

[npolshakova]

We can confirm this at the setup_test level by checking the path is correctly replaced!

[andy-fong]

Suggested change

	// Only supported for OpenAI and Anthropic compatible APIs for now.
	FullPath *string `json:"path"`
	FullPath *string `json:"fullPath"`

[zhengkezhou1]

I believe I made an error; it should be base_url = TEST_OVERRIDE_BASE_URL if overrideProvider else TEST_OPENAI_BASE_URL. 😳

@npolshakova @andy-fong Additionally, should we test Routing, Streaming, PromptGuard, etc., under the condition of TEST_OVERRIDE_PROVIDER? 🤔 Please let me know. Thank you!

[npolshakova]

None of the PromptGuard settings should change the path, so it's fine to skip adding a test there.

Gemini and VertexAI change the path depending on whether streaming is enabled or not, but OpenAI and other providers just use stream: true in the body. Since we're now always overriding the path if the user sets the override here, let's add a small streaming test with gemini to show that the override path is set correctly in that case.

[andy-fong]

@zhengkezhou1 again, thank you for taking the time for the contribution. That's a good question and I understand why you use an env to set an override base url; however, the various TEST_*_BASE_URL are meant to test the different providers on the various features. Adding a TEST_OVERRIDE_BASE_URL and following the test "pattern" lead you to the question of should you also add tests for Routing, Streaming, PromptGuard ...etc. I think you are on the right thinking but it's also a red flag because it basically will double all the tests we need to write for all existing and future features.

So, step back and think about it. This override applies to all providers; however, it's not going to change the response from the LLM providers for various features. So, instead of adding the path override tests to all existing test suites, it might be better to test this as a separate feature for each provider once. That way, you would just follow the existing test pattern and just use the TEST_OPENAI_BASE_URL for example and have a new yaml file under testdata to turn on this path override feature for each provider. However, for upstream, you don't need a mock LLM response but rather can use something like http-echo to return you the request headers and path as a json object, then the test just verify the request path is the one we expect.

[npolshakova]

Actually, I agree with Andy. Let's keep the current approach with TEST_OVERRIDE_BASE_URL for now for just the routing e2e test, then as a follow up let's clean up the tests and create a standalone test suite for the path override.

The follow up test PR should have a separate test for each provider once (keep original TEST_OPENAI_BASE_URL and create a new testdata file with values to use).

[npolshakova]

Looks good! Let's just add the streaming test and this looks good to go!

Let's handle this in a follow up PR: #11282 (comment)