[Feature] Use sgl.Engine along with fastapi

[OpenJarvisAI]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Is there any example can be referenced use sgl.Engine with fastapi?

Am using it with fastapi, always got uvloop.event already error when call engine.generate, how to do that?

Related resources

Is there any example can be referenced use sgl.Engine with fastapi?

Am using it with fastapi, always got uvloop.event already error when call engine.generate, how to do that?

[ravi03071991]

Hello @OpenJarvisAI,

Following code worked for me.

1. Create main.py file:

from fastapi import FastAPI, Request
import sglang as sgl
import uvicorn
import asyncio
from contextlib import asynccontextmanager

engine = None

# Use FastAPI's lifespan manager to initialize/shutdown the engine
@asynccontextmanager
async def lifespan(app: FastAPI):
    global engine
    # Initialize the SGLang engine when the server starts
    # Adjust model_path and other engine arguments as needed
    print("Loading SGLang engine...")
    engine = sgl.Engine(model_path="Qwen/Qwen2-7B-Instruct", tp_size=1) 
    print("SGLang engine loaded.")
    yield
    # Clean up engine resources when the server stops (optional, depends on engine needs)
    print("Shutting down SGLang engine...")
    # engine.shutdown() # Or other cleanup if available/necessary
    print("SGLang engine shutdown.")


app = FastAPI(lifespan=lifespan)

@app.post("/generate")
async def generate_text(request: Request):
    global engine
    if not engine:
        return {"error": "Engine not initialized"}, 503

    try:
        data = await request.json()
        prompt = data.get("prompt")
        max_new_tokens = data.get("max_new_tokens", 128)
        temperature = data.get("temperature", 0.7)

        if not prompt:
            return {"error": "Prompt is required"}, 400

        # Use async_generate for non-blocking generation
        state = await engine.async_generate(
            prompt,
            sampling_params={
                "max_new_tokens": max_new_tokens,
                "temperature": temperature,
            }
            # Add other parameters like stop, top_p etc. as needed
        )

        return {"generated_text": state["text"]}
    except Exception as e:
        return {"error": str(e)}, 500

if __name__ == "__main__":
     # Required for async_generate if running in a nested loop environment like Jupyter
     # import nest_asyncio
     # nest_asyncio.apply()

    uvicorn.run(app, host="0.0.0.0", port=8000)

2. Start the server using uvicorn:

    uvicorn main:app --host 0.0.0.0 --port 8000 --reload

3. Example curl request to test the endpoint:

    curl -X POST http://localhost:8000/generate \
         -H "Content-Type: application/json" \
         -d '{
               "prompt": "What is the capital of France?",
               "max_new_tokens": 50,
               "temperature": 0.5
             }'

[ravi03071991]

@zhyncs / @zhaochenyang20 should we add it to docs?

[zhaochenyang20]

@ravi03071991 Make a PR and add it to https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine

[OpenJarvisAI]

Hi, thank u so much for the quick answer, what about Qwen2.5 VL, I want infer VL model and the request is a OpenAI like requests