feat(ai-proxy): add models & image generation support for gemini #2380

daixijun · 2025-06-06T03:33:49Z

Ⅰ. Describe what this PR did

新增功能:

支持模型列表接口 /v1/models
支持生图接口 /v1/images/generations Generate images using Imagen 3
支持对话接口文生图能力 /v1/chat/completions Generate images using Gemini

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

docker-compose.yml

services:
  envoy:
    image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:v2.1.3
    entrypoint: /usr/local/bin/envoy
    command: -c /etc/envoy/envoy.yaml --component-log-level wasm:debug
    networks:
      - wasmtest
    ports:
      - '10000:10000'
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml
      - ./plugin.wasm:/etc/envoy/plugin.wasm

networks:
  wasmtest: {}

envoy.yml

# File generated by hgctl. Modify as required.

admin:
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 9901
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 10000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                scheme_header_transformation:
                  scheme_to_overwrite: https
                stat_prefix: ingress_http
                # Output envoy logs to stdout
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                # Modify as required
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: google
                            timeout: 300s
                http_filters:
                  - name: wasmtest
                    typed_config:
                      "@type": type.googleapis.com/udpa.type.v1.TypedStruct
                      type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
                      value:
                        config:
                          name: wasmtest
                          vm_config:
                            runtime: envoy.wasm.runtime.v8
                            code:
                              local:
                                filename: /etc/envoy/plugin.wasm
                          configuration:
                            "@type": "type.googleapis.com/google.protobuf.StringValue"
                            value: |
                              {
                                "provider": {
                                    "type": "gemini",
                                    "apiTokens": [
                                        "YOUR API KEY"
                                    ]
                                }
                              }
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
    - name: google
      connect_timeout: 30s
      type: LOGICAL_DNS
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: google
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: generativelanguage.googleapis.com
                      port_value: 443
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          "sni": "generativelanguage.googleapis.com"

非流式对话

$ curl -s http://127.0.0.1:10000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
      "messages": [
        {
          "role": "user",
          "content": "A white kitten"
        }
      ],
      "model": "gemini-2.0-flash"
    }'
{
  "id": "chatcmpl-58283637-d938-4d51-8ebb-2a022897d9dd",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Okay! Here are some ways I can respond to your prompt \"Awhitekitten\":\n\n**1. Descriptive:**\n\n*   A tiny, fluffy cloud with bright blue eyes.\n*   A pristine white kitten, its fur like freshly fallen snow.\n*   A little snowball of a kitten, purring softly.\n*   A white kitten, almost translucent in the sunlight, playing with a dust bunny.\n*   A delicate, porcelain-like kitten with pink ears and nose.\n\n**2. Fictional/Story Starter:**\n\n*   The white kitten was different. While its siblings tumbled and played, it stared out the window, dreaming of adventures.\n*   Old Maggie found the white kitten shivering on her doorstep, a silent plea in its wide eyes.\n*   The prophecy spoke of a white kitten born under a blood moon, destined to either save or destroy the kingdom.\n*   She named him \"Ghost,
        \" because that's what he resembled, a fleeting apparition of white fur and mischief.\n\n**3. Questioning:**\n\n*   What breed of white kitten are you thinking of?\n*   What is the white kitten doing?\n*   Is the white kitten a long-haired or short-haired?\n*   Does the white kitten have any distinctive markings?\n\n**4. Suggesting Images:**\n\n*   I could generate an image of a white kitten for you. Would you like that? What kind of scene would you like? (e.g., \"awhitekittenplayinginafieldofdaisies,
        \" or \"awhitekittensleepinginasunbeam\").\n\n**5. Informative:**\n\n*   White kittens can be prone to deafness, especially if they have blue eyes. This is due to a genetic link.\n\n**To give you the best response, tell me what you'd like me to do with the prompt \"Awhitekitten.\"  For example, are you looking for:**\n\n*   **A description?**  If so, give me more details (e.g., \"aplayfulwhitekitten,
        \" or \"asadwhitekitten\").\n*   **A story?**\n*   **Information?**\n*   **An image?**\n"
      },
      "finish_reason": "STOP"
    }
  ],
  "created": 1749179754,
  "model": "gemini-2.0-flash",
  "object": "chat.completion",
  "usage": {
    "prompt_tokens": 3,
    "completion_tokens": 457,
    "total_tokens": 460
  }
}

流式对话

$ curl -s http://127.0.0.1:10000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
      "messages": [
        {
          "role": "user",
          "content": "A white kitten"
        }
      ],
      "model": "gemini-2.0-flash", 
      "stream": true
    }'
data: {"id":"chatcmpl-c07bdee6-c40e-488b-848c-733fc30bfe87","choices":[{"index":0,"delta":{"content":"That"}}],"created":1749179886,"model":"gemini-2.0-flash","object":"chat.completion.chunk","usage":{"prompt_tokens":4,"total_tokens":4}}

data: {"id":"chatcmpl-6740ca56-c44a-4e30-8cf7-e1ff78cbb138","choices":[{"index":0,"delta":{"content":"'s a lovely image!  Here are a few ways we could describe a white"}}],"created":1749179887,"model":"gemini-2.0-flash","object":"chat.completion.chunk","usage":{"prompt_tokens":4,"total_tokens":4}}

data: {"id":"chatcmpl-b924155d-39b9-4c3b-b19b-3271aa2bcaf0","choices":[{"index":0,"delta":{"content":" kitten, depending on what you'd like to focus on:\n\n**General"}}],"created":1749179887,"model":"gemini-2.0-flash","object":"chat.completion.chunk","usage":{"prompt_tokens":4,"total_tokens":4}}

data: {"id":"chatcmpl-9b271bcf-eaa1-42bd-97e9-811d9c85d4bb","choices":[{"index":0,"delta":{"content":" Descriptions:**\n\n*   A fluffy white kitten.\n*   A tiny, white kitten.\n*   A pure white kitten.\n*   A snow"}}],"created":1749179887,"model":"gemini-2.0-flash","object":"chat.completion.chunk","usage":{"prompt_tokens":4,"total_tokens":4}}

data: {"id":"chatcmpl-0d84061c-0b6f-4946-9781-3bda94d2d850","choices":[{"index":0,"delta":{"content":"-white kitten.\n\n**Focusing on Appearance:**\n\n*   A white kitten with bright blue eyes.\n*   A white kitten with a pink nose.\n*"}}],"created":1749179887,"model":"gemini-2.0-flash","object":"chat.completion.chunk","usage":{"prompt_tokens":4,"total_tokens":4}}

data: {"id":"chatcmpl-112936ac-4767-472d-a158-480790042a86","choices":[{"index":0,"delta":{"content":"   A white kitten with soft, downy fur.\n*   A small white kitten with a long fluffy tail.\n*   A pristine white kitten with delicate pink ears.\n\n**Focusing on Behavior/Personality:**\n\n*   A playful"}}],"created":1749179887,"model":"gemini-2.0-flash","object":"chat.completion.chunk","usage":{"prompt_tokens":4,"total_tokens":4}}

data: {"id":"chatcmpl-d091ce57-5041-445a-9aad-ab85a7b44d6b","choices":[{"index":0,"delta":{"content":" white kitten.\n*   A shy white kitten.\n*   A curious white kitten.\n*   A sleepy white kitten, curled up in a ball.\n*   A mischievous white kitten, batting at a toy.\n\n**More"}}],"created":1749179888,"model":"gemini-2.0-flash","object":"chat.completion.chunk","usage":{"prompt_tokens":4,"total_tokens":4}}

data: {"id":"chatcmpl-adef4500-87fb-49ed-b678-3c2626cbdb6c","choices":[{"index":0,"delta":{"content":" Evocative Descriptions:**\n\n*   A cloud of white fur, barely bigger than a teacup, with wide, innocent eyes.\n*   A little snowdrift of a kitten, purring contentedly.\n*   Like a miniature polar bear, a white kitten explores its world.\n*   A fragile"}}],"created":1749179888,"model":"gemini-2.0-flash","object":"chat.completion.chunk","usage":{"prompt_tokens":4,"total_tokens":4}}

data: {"id":"chatcmpl-11bb1d8d-968b-417f-9632-c127a8d376ff","choices":[{"index":0,"delta":{"content":" white kitten, a tiny bundle of life.\n\nIf you want a more specific description, tell me what details you want to emphasize.  For example, are you writing a story, painting a picture, or just admiring the idea of a white kitten?\n"}}],"created":1749179888,"model":"gemini-2.0-flash","object":"chat.completion.chunk","usage":{"prompt_tokens":3,"completion_tokens":314,"total_tokens":317}}

使用 Gemini 模型生成图片

$ curl -s http://127.0.0.1:10000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
    "messages": [
    {
        "role": "user",
        "content": "A white kitten"
    }
    ],
    "model": "gemini-2.0-flash-preview-image-generation",
    "modalities": ["TEXT", "IMAGE"]
}'
{
  "id": "chatcmpl-41ef1115-9545-4ee2-9f1d-7d90029b5d5a",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "A fluffy white kitten with bright blue eyes sits playfully batting at a dangling toy mouse. Its tiny pink nose twitches with curiosity against a soft, out-of-focus green background.\n"
      },
      "finish_reason": "STOP"
    },
    {
      "index": 1,
      "message": {
        "role": "assistant",
        "content": "BASE64 ENCRYPTED IMAGE DATA"
      },
      "finish_reason": "STOP"
    }
  ],
  "created": 1749179153,
  "model": "gemini-2.0-flash-preview-image-generation",
  "object": "chat.completion",
  "usage": {
    "prompt_tokens": 4,
    "completion_tokens": 1327,
    "total_tokens": 1331
  }
}

使用 Imagen 3 模型生成图片

$ curl -s "http://127.0.0.1:10000/v1/images/generations" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "imagen-3.0-generate-002",
        "prompt": "a portrait of a sheepadoodle wearing a cape",
        "n": 1
      }' | jq -r '.data[0].b64_json' | base64 --decode > imagen-image.jpg

列出模型清单

$ curl -s "http://127.0.0.1:10000/v1/models"
{
  "models": [
    {
      "name": "models/gemini-1.0-pro-vision-latest",
      "version": "001",
      "displayName": "Gemini 1.0 Pro Vision",
      "description": "The original Gemini 1.0 Pro Vision model version which was optimized for image understanding. Gemini 1.0 Pro Vision was deprecated on July 12, 2024. Move to a newer Gemini version.",
      "inputTokenLimit": 12288,
      "outputTokenLimit": 4096,
      "supportedGenerationMethods": [
        "generateContent",
        "countTokens"
      ],
      "temperature": 0.4,
      "topP": 1,
      "topK": 32
    },
    {
      "name": "models/gemini-pro-vision",
      "version": "001",
      "displayName": "Gemini 1.0 Pro Vision",
      "description": "The original Gemini 1.0 Pro Vision model version which was optimized for image understanding. Gemini 1.0 Pro Vision was deprecated on July 12, 2024. Move to a newer Gemini version.",
      "inputTokenLimit": 12288,
      "outputTokenLimit": 4096,
      "supportedGenerationMethods": [
        "generateContent",
        "countTokens"
      ],
      "temperature": 0.4,
      "topP": 1,
      "topK": 32
    }
  ]
}

Ⅴ. Special notes for reviews

Signed-off-by: Xijun Dai <[email protected]>

codecov-commenter · 2025-06-06T03:36:37Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.06%. Comparing base (ef31e09) to head (3fda9e4).
Report is 542 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2380       +/-   ##
===========================================
+ Coverage   35.91%   46.06%   +10.15%     
===========================================
  Files          69       81       +12     
  Lines       11576    13010     +1434     
===========================================
+ Hits         4157     5993     +1836     
+ Misses       7104     6671      -433     
- Partials      315      346       +31

see 78 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

johnlanni · 2025-06-06T05:20:18Z

cc @rinfx @mirror58229

CH3CHO · 2025-06-06T09:10:39Z

plugins/wasm-go/extensions/ai-proxy/provider/gemini.go


 const (
 	geminiApiKeyHeader             = "x-goog-api-key"
+	geminiApiVersion               = "v1beta" // 可选: v1 ,v1beta


这个需要支持用户配置吗？

我是想支持下的，没想好是不是要单开一个配置项，还是用一个通用的
gemini的部分能力不支持v1版本接口

Signed-off-by: Xijun Dai <[email protected]>

CH3CHO · 2025-06-08T05:43:59Z

plugins/wasm-go/extensions/ai-proxy/provider/gemini.go

 		}
 	}
-	shouldAddDummyModelMessage := false
+	// shouldAddDummyModelMessage := false


这个变量要是不用了的话就删掉吧

CH3CHO · 2025-06-08T05:46:25Z

plugins/wasm-go/extensions/ai-proxy/provider/provider.go

+	c.apiVersion = json.Get("claudeVersion").String()
+	if c.apiVersion == "" {
+		// 增加获取 version 字段，用于适配其他模型的配置，并保持与结构体中定义的 tag 一致
+		c.apiVersion = json.Get("version").String()


感觉这个配置项在 JSON 里还是叫 apiVersion 比较好？另外，对应的 README 需要更新一下。

结构体里面的 json tag 原先就是叫 version, 我只是做了兼容
如果要修改为 apiVersion 不确定会不会影响到已有用户的配置

我想了下，虽然 tag 中为 version，文档中并没有说明可以通过 version 来指定版本。
所以这里应该是可以修改为 apiVersion

Signed-off-by: Xijun Dai <[email protected]>

CH3CHO

LGTM

…baba#2380) Signed-off-by: Xijun Dai <[email protected]> Co-authored-by: Kent Dong <[email protected]>

feat(ai-proxy): add models & image generation support for gemini

0ac0b9d

Signed-off-by: Xijun Dai <[email protected]>

daixijun requested review from CH3CHO, cr7258 and rinfx as code owners June 6, 2025 03:33

CH3CHO reviewed Jun 6, 2025

View reviewed changes

daixijun added 2 commits June 8, 2025 13:20

feat(ai-proxy): Added specified apiVersion support for gemini

e443fad

Signed-off-by: Xijun Dai <[email protected]>

feat(ai-proxy): 修复接口版本结构定义与实际获取字段不一致问题

960543e

Signed-off-by: Xijun Dai <[email protected]>

CH3CHO reviewed Jun 8, 2025

View reviewed changes

daixijun and others added 2 commits June 8, 2025 14:19

feat(ai-proxy): 调整字段 version 为 apiVersion 并更新 README

e183787

Signed-off-by: Xijun Dai <[email protected]>

Update README.md

9607cd0

CH3CHO approved these changes Jun 8, 2025

View reviewed changes

Merge branch 'main' into feat/gemini

3fda9e4

CH3CHO merged commit e674c78 into alibaba:main Jun 8, 2025
12 checks passed

daixijun deleted the feat/gemini branch June 8, 2025 07:26

lingma-agents bot mentioned this pull request Jun 17, 2025

add release-notes of 2.1.4 #2433

Merged

ink-hz pushed a commit to ink-hz/higress-ai-capability-auth that referenced this pull request Nov 5, 2025

feat(ai-proxy): add models & image generation support for gemini (ali…

632c033

…baba#2380) Signed-off-by: Xijun Dai <[email protected]> Co-authored-by: Kent Dong <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ai-proxy): add models & image generation support for gemini #2380

feat(ai-proxy): add models & image generation support for gemini #2380

Uh oh!

daixijun commented Jun 6, 2025

Uh oh!

codecov-commenter commented Jun 6, 2025 •

edited

Loading

Uh oh!

johnlanni commented Jun 6, 2025

Uh oh!

CH3CHO Jun 6, 2025

Uh oh!

daixijun Jun 6, 2025

Uh oh!

CH3CHO Jun 8, 2025

Uh oh!

CH3CHO Jun 8, 2025

Uh oh!

daixijun Jun 8, 2025

Uh oh!

daixijun Jun 8, 2025

Uh oh!

CH3CHO left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat(ai-proxy): add models & image generation support for gemini #2380

feat(ai-proxy): add models & image generation support for gemini #2380

Uh oh!

Conversation

daixijun commented Jun 6, 2025

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

Uh oh!

codecov-commenter commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

johnlanni commented Jun 6, 2025

Uh oh!

CH3CHO Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

daixijun Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

CH3CHO Jun 8, 2025

Choose a reason for hiding this comment

Uh oh!

CH3CHO Jun 8, 2025

Choose a reason for hiding this comment

Uh oh!

daixijun Jun 8, 2025

Choose a reason for hiding this comment

Uh oh!

daixijun Jun 8, 2025

Choose a reason for hiding this comment

Uh oh!

CH3CHO left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Jun 6, 2025 •

edited

Loading