Skip to content

Conversation

@daixijun
Copy link
Contributor

Ⅰ. Describe what this PR did

增加 Openai 兼容 Provider 对图片生成、图片编辑、音频处理等接口对模型 Mapping 的能力

Ⅱ. Does this pull request fix one issue?

fixes #2338

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

envoy.yaml 内容

# File generated by hgctl. Modify as required.

admin:
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 9901
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 10000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                scheme_header_transformation:
                  scheme_to_overwrite: https
                stat_prefix: ingress_http
                # Output envoy logs to stdout
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                # Modify as required
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: google
                            timeout: 300s
                http_filters:
                  - name: wasmtest
                    typed_config:
                      "@type": type.googleapis.com/udpa.type.v1.TypedStruct
                      type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
                      value:
                        config:
                          name: wasmtest
                          vm_config:
                            runtime: envoy.wasm.runtime.v8
                            code:
                              local:
                                filename: /etc/envoy/plugin.wasm
                          configuration:
                            "@type": "type.googleapis.com/google.protobuf.StringValue"
                            value: |
                              {
                                "provider": {
                                    "type": "openai",
                                    "openaiCustomUrl": "generativelanguage.googleapis.com/v1beta/openai/",
                                    "modelMapping": {
                                      "gpt-image-1": "imagen-3.0-generate-002"
                                    },
                                    "apiTokens": [
                                        "{YOUR-API-KEY}"
                                    ]
                                }
                              }
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
    - name: google
      connect_timeout: 30s
      type: LOGICAL_DNS
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: google
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: generativelanguage.googleapis.com
                      port_value: 443
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          "sni": "generativelanguage.googleapis.com"

docker-compose.yaml 文件内容如下

services:
  envoy:
    image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:v2.1.3
    entrypoint: /usr/local/bin/envoy
    command: -c /etc/envoy/envoy.yaml --component-log-level wasm:debug
    networks:
      - wasmtest
    ports:
      - '10000:10000'
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml
      - ./plugin.wasm:/etc/envoy/plugin.wasm

networks:
  wasmtest: {}

使用 openai 的 gpt-image-1 模型请求 gemini ,验证请求是否会被正确 mapping 到 imagen-3.0-generate-002 模型

curl -v "http://localhost:10000/v1/images/generations" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-image-1",
        "prompt": "a portrait of a sheepadoodle wearing a cape",
        "response_format": "b64_json",
        "n": 1, 
        "size": "1024x1024"      
      }' | jq -r '.data[0].b64_json' | base64 --decode > gemini-image.jpeg

wasm 日志如下

image

生成的效果图

image

Ⅴ. Special notes for reviews

@codecov-commenter
Copy link

codecov-commenter commented May 29, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.01%. Comparing base (ef31e09) to head (0292650).
Report is 529 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##             main    #2341       +/-   ##
===========================================
+ Coverage   35.91%   46.01%   +10.10%     
===========================================
  Files          69       81       +12     
  Lines       11576    13010     +1434     
===========================================
+ Hits         4157     5986     +1829     
+ Misses       7104     6678      -426     
- Partials      315      346       +31     

see 78 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@daixijun daixijun force-pushed the feat/add-image-audio-modelmapping branch from 84b015b to c8c6212 Compare May 29, 2025 17:25
Copy link
Collaborator

@CH3CHO CH3CHO left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CH3CHO CH3CHO merged commit a73c33f into alibaba:main May 30, 2025
12 checks passed
@daixijun daixijun deleted the feat/add-image-audio-modelmapping branch May 30, 2025 05:10
ink-hz pushed a commit to ink-hz/higress-ai-capability-auth that referenced this pull request Nov 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ai-proxy] 文生图接口 Fallback 后无法正确 mapping 到目标模型

3 participants