higress-plugin-server开发方案 || higress-plugin-server development plan

[NorthernBob]

关联 issue： #2232 #2280

1 需求分析与架构设计

1.1 核心需求

• 功能：提供 HTTP 文件服务器，支持网关通过 http://file-server/plugins/plugin-name/version/plugin.wasm 下载 Wasm 插件。

• 高可用性：文件服务器需作为核心组件部署，保障 SLA。

• 版本管理：镜像需与插件版本绑定，响应 wasm 插件相关 PR ，支持自动构建和人工部署。

• 兼容性：与现有 OCI 插件加载方式共存，用户可配置插件拉取策略（如 IfNotPresent ），需要测试验证 Always 策略时，文件服务器的方式是否能感知到变化，并更新插件内容。（例如对比文件的 md5 ）。

1.2 架构设计

• 独立组件：新建 higress-plugin-server 仓库，包含：

  • Dockerfile：构建 Nginx 镜像，注入插件文件。
  • 插件目录结构：按 plugins/plugin-name/version/ 组织文件。
  • nginx 配置文件：nginx.conf
  • K8s 部署配置文件
  • 全部的 wasm plugin 文件
  • 插件初始化相关脚本

• 集成方式：

  • 个人或测试版用户直接使用 docker build 构建。
  • 用户通过 Helm Chart 安装 higress-plugin-server 组件。
  • higress-gateway 通过环境变量或配置文件指定文件服务器地址。

2 具体开发方案

2.1 新建仓库与代码结构

• 创建仓库
  在 higress-group 下新建仓库 higress-plugin-server 。（我先在本地和个人 github 账号下新建一个仓库）

• 目录结构（ plugins 全部下载放到仓库里，同时提供从 OCI 仓库拉取插件的脚本，可配置参数选择直接打包或者拉取远程仓库中的插件）

higress-plugin-server/
├── Dockerfile          # 构建 Nginx 镜像
├── nginx.conf          # Nginx 配置文件
├── pull_plugins.py     # 插件下载脚本，生成插件元数据
├── generate_metadata.py# 生成插件元数据的脚本
├── plugins/            # 插件静态文件目录
│   └── ai-proxy/
│       └── 1.0.0/
│           └── plugin.wasm
│           └── metadata.txt
├── deploy/             # Kubernetes 部署文件
│   └── higress-plugin-server.yaml
└── .github/workflows/  # GitHub Action 流水线定义（可选）
    └── build.yaml

• Dockerfile

FROM nginx:alpine
COPY nginx.conf /etc/nginx/nginx.conf
COPY plugins /usr/share/nginx/html/plugins
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

• nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    # 访问日志格式
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    keepalive_timeout  65;

    # 启用目录浏览
    autoindex on;

    server {
        listen       80;
        server_name  localhost;

        # 静态文件根目录
        root /usr/share/nginx/html;

        # 隐藏 Nginx 版本号
        server_tokens off;

        # 错误页面
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
}

2.2 实现插件和镜像版本管理

• 插件文件目录结构，按插件名称和版本组织文件：

plugins/
└── ai-proxy/
    └── 1.0.0/
        └── plugin.wasm
        └── metadata.txt
    └── 1.1.0/
        └── plugin.wasm
        └── metadata.txt
└── ai-search/
    └── 0.9.0/
        └── plugin.wasm
        └── metadata.txt

• 插件元数据（ metadata.txt ）结构

Plugin Name: ai-proxy
Size: 123456 bytes
Last Modified: 2025-04-05T12:34:56.789012
Created: 2025-04-04T10:11:12.345678
MD5: XXXXXX

• 版本更新策略

  • 流水线1：alibaba/higress main 分支插件相关 PR 触发，触发时从 OCI 拉取最新插件并 push 到仓库，同步 main 分支的插件PR。
  • 流水线2：higress-plugin-server 相关 PR 触发，触发时重新构建镜像并推送到镜像仓库。
  • 用户侧需不需要自动升级？如何感知到镜像版本变化？
  • 人工更新：可手动替换插件文件，执行元数据生成脚本后，重新构建镜像。

• 镜像版本管理

  • 直接用 latest ( higress-plugin-server:latest )
  • 与插件版本关联 ( higress-plugin-server:1.0.0 )

2.3 配置插件拉取策略

• 默认策略 IfNotPresent

  • 在 higress-gateway 的 Helm Chart 中，默认设置 imagePullPolicy: IfNotPresent，避免频繁拉取影响性能。
  • 示例配置：

wasm:
  plugin:
    imagePullPolicy: IfNotPresent

2.4 K8s 部署，要编排进 alibaba/higress 仓库的 helm 中

• Kubernetes 部署文件 deploy/higress-plugin-server.yaml 示例：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: higress-plugin-server
  namespace: higress-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: higress-plugin-server
  template:
    metadata:
      labels:
        app: higress-plugin-server
    spec:
      containers:
        - name: nginx
          image: registry.higress.io/higress-plugin-server:latest
          ports:
            - containerPort: 80
          resources:
            limits:
              cpu: 500m
              memory: 256Mi
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - higress-plugin-server
            topologyKey: "kubernetes.io/hostname"
---
apiVersion: v1
kind: Service
metadata:
  name: higress-plugin-server
  namespace: higress-system
spec:
  selector:
    app: higress-plugin-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

• 健康检查,在 Deployment 中添加健康检查探针：

livenessProbe:
  httpGet:
    path: /plugins/
    port: 80
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /plugins/
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 5

2.5 测试与验证

• 本地测试镜像，构建运行镜像，并测试：

docker build -t higress-plugin-server .
docker run -d -p 8080:80 higress-plugin-server
curl http://localhost:8080/plugins/ai-proxy/1.0.0/plugin.wasm --output plugin.wasm

• Kubernetes 部署测试

kubectl apply -f deploy/higress-plugin-server.yaml
kubectl get pods -n higress-system
kubectl exec -it <gateway-pod> -- curl http://higress-plugin-server.higress-system.svc/plugins/ai-proxy/1.0.0/plugin.wasm

• higress 集成环境测试
  部署到 higress 测试环境中，配置 higress-gateway 使其从 higress-plugin-server 节点中下载插件，测试是否 OK 。

2.6 全流程

higress-console 等管控组件支持环境变量，支持默认下发插件使用文件服务器方案。

2.7 用户文档

• 安装指南：如何通过 Helm 安装 higress-plugin-server。
• 配置说明：如何配置 higress-gateway 的插件下载地址。
• 自定义构建：如何替换插件文件并构建自定义镜像。
• 故障排查：常见问题（如 404 插件文件、性能问题）的解决方案。

---

Related issue: #2232 #2280

#1 Requirements Analysis and Architecture Design

1.1 Core Requirements

• Function: Provides an HTTP file server, and supports the gateway to download the Wasm plug-in through http://file-server/plugins/plugin-name/version/plugin.wasm.

• High Availability: The file server needs to be deployed as a core component to ensure SLA.

• Version management: The mirror needs to be bound to the plug-in version, respond to the wasm plug-in-related PR, and supports automatic construction and manual deployment.

• Compatibility: **Coexist with existing OCI plug-in loading methods. Users can configure plug-in pulling policies (such as IfNotPresent). It is necessary to test and verify whether the file server's method can perceive changes when verifying the Always policy and update the plug-in content. (e.g. md5 of comparison file). **

1.2 Architectural Design

• Independent components: Create a new higress-plugin-server repository, including:

  • Dockerfile: Build an Nginx image and inject plugin files.
  • Plugin directory structure: organize files by plugins/plugin-name/version/.
  • nginx configuration file: nginx.conf
  • K8s deployment configuration file
  • All wasm plugin files
  • Plugin initialization related scripts

• Integration method:

  • Build directly with docker build by individual or beta users.
  • Users install the higress-plugin-server component through Helm Chart.
  • hgress-gateway Specifies the file server address through environment variables or configuration files.

#2 Specific development plan

2.1 Create a new repository and code structure

• Create a warehouse
  Create a new repository under higress-group higress-plugin-server. (I will first create a new warehouse under my local and personal github account)

• Directory structure (all plugins are downloaded and placed in the repository, and scripts are provided to pull plugins from the OCI repository. You can configure parameters to directly package or pull plugins in the remote repository)

hgress-plugin-server/
├── Dockerfile # Build Nginx images
├── nginx.conf # Nginx configuration file
├── pull_plugins.py # Plugin download script to generate plug-in metadata
├── generate_metadata.py# Script to generate plugin metadata
├── plugins/ # plugin static file directory
│ └── ai-proxy/
│ └── 1.0.0/
│ └── plugin.wasm
│ └── metadata.txt
├── deploy/ # Kubernetes deployment files
│ └── higress-plugin-server.yaml
└── .github/workflows/ # GitHub Action Pipeline Definition (optional)
    └── build.yaml

• Dockerfile

FROM nginx:alpine
COPY nginx.conf /etc/nginx/nginx.conf
COPY plugins /usr/share/nginx/html/plugins
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

• nginx.conf

user nginx;
worker_processes auto;

error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    # Access log format
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /var/log/nginx/access.log main;

    sendfile on;
    keepalive_timeout 65;

    # Enable directory browsing
    autoindex on;

    server {
        listen 80;
        server_name localhost;

        # Static file root directory
        root /usr/share/nginx/html;

        # Hide Nginx version number
        server_tokens off;

        # Error page
        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
            root /usr/share/nginx/html;
        }
    }
}

2.2 Implement plug-in and mirror version management

• Plugin file directory structure, organize files by plugin name and version:

plugins/
└── ai-proxy/
    └── 1.0.0/
        └── plugin.wasm
        └── metadata.txt
    └── 1.1.0/
        └── plugin.wasm
        └── metadata.txt
└── ai-search/
    └── 0.9.0/
        └── plugin.wasm
        └── metadata.txt

• Plugin metadata ( metadata.txt ) structure

Plugin Name: ai-proxy
Size: 123456 bytes
Last Modified: 2025-04-05T12:34:56.789012
Created: 2025-04-04T10:11:12.345678
MD5: XXXXXX

• Version update strategy

  • Pipeline 1: alibaba/higress main branch plug-in related PR triggers. When triggering, pull the latest plug-in from OCI and push it to the repository, and synchronize the plug-in PR of the main branch.
  • Pipeline 2: Higress-plugin-server related PR triggers, rebuild the image when triggered and push it to the mirror repository.
  • **Does the user side need to automatically upgrade? How to perceive the change in the mirror version? **
  • Manual update: You can manually replace the plug-in file, execute the metadata generation script, and rebuild the image.

• Mirror version management

  • Use latest (higress-plugin-server:latest)
  • Associated with plugin version (higress-plugin-server:1.0.0)

2.3 Configuring plug-in pull policy

• Default policy IfNotPresent

  • In Helm Chart in the higress-gateway, the default setting imagePullPolicy: IfNotPresent is avoided from frequent pulling affecting performance.
  • Sample configuration:

wasm:
  plugin:
    imagePullPolicy: IfNotPresent

2.4 K8s deployment should be orchestrated into the helm of the alibaba/higress warehouse

• Kubernetes deployment file deploy/higress-plugin-server.yaml example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: higress-plugin-server
  namespace: higress-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: higress-plugin-server
  template:
    metadata:
      labels:
        app: higress-plugin-server
    spec:
      containers:
        - name: nginx
          image: registry.higress.io/higress-plugin-server:latest
          Ports:
            - containerPort: 80
          resources:
            limits:
              cpu: 500m
              memory: 256Mi
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - hgress-plugin-server
            topologyKey: "kubernetes.io/hostname"
---
apiVersion: v1
kind: Service
metadata:
  name: higress-plugin-server
  namespace: higress-system
spec:
  selector:
    app: higress-plugin-server
  Ports:
    - protocol: TCP
      port: 80
      targetPort: 80

• Health Check, add a health check probe in Deployment:

livenessProbe:
  httpGet:
    path: /plugins/
    port: 80
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /plugins/
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 5

2.5 Testing and Verification

• Local test image, build the running image, and test:

docker build -t higress-plugin-server.
docker run -d -p 8080:80 higress-plugin-server
curl http://localhost:8080/plugins/ai-proxy/1.0.0/plugin.wasm --output plugin.wasm

• Kubernetes Deployment Testing

kubectl apply -f deploy/higress-plugin-server.yaml
kubectl get pods -n hgress-system
kubectl exec -it <gateway-pod> -- curl http://higress-plugin-server.higress-system.svc/plugins/ai-proxy/1.0.0/plugin.wasm

• higress integrated environment testing
  Deploy into the higress test environment, configure higress-gateway to download the plug-in from the higress-plugin-server node, and test whether it is OK.

2.6 The entire process

Control components such as higress-console support environment variables and support the default issuance of plug-ins to use file server solutions.

2.7 User Documentation

• Installation Guide: How to install higress-plugin-server through Helm.
• Configuration instructions: How to configure the plug-in download address of higress-gateway.
• Custom Build: How to replace plugin files and build custom images.
• Troubleshooting: Solutions for common problems (such as 404 plugin files, performance issues).

[johnlanni]

感谢，很详细的设计。我来补充一些背景：

higress 的 wasm 插件机制，是让插件的生命周期（版本）跟网关的生命周期（版本）解耦，从而无需升级网关，即可灵活更新插件。

但从社区一些用户的反馈看，仍然存在希望让插件的版本跟随网关版本一起的需求，例如出于稳定一致性考虑，以及网络隔离环境下的私有化部署考虑。

我会重点关注后续 PR 中，对于这些用户实际需求的解决情况，比如跟 Higress Console 的联动可能是一个需要注意的地方。

---

Thanks, very detailed design. Let me add some background:

The wasm plug-in mechanism of higress is to decouple the life cycle (version) of the plug-in from the life cycle (version) of the gateway, so that the plug-in can be flexibly updated without upgrading the gateway.

However, judging from feedback from some community users, there is still a need to let the plug-in version follow the gateway version, such as for the sake of stability and consistency, as well as privatization deployment in a network isolation environment.

I will focus on the solution to the actual needs of these users in subsequent PRs, such as linkage with Higress Console, which may be a point to pay attention to.