feature request: add support for gpt4all, agieval, and bigbench

[aigeek0x0]

Here is the development roadmap for 2025 H1. Contributions and feedback are welcome (Join Bi-weekly Development Meeting). The previous 2024 Q4 roadmap can be found in #1487

Focus

• Throughput-oriented large-scale deployment similar to the deepseek inference system
• Long context optimizations
• Low latency speculative decoding
• Reinforcement learning training framework integration
• Kernel optimizations

Parallelism

• Support PD disaggregation @ByronHsu
• Support expert parallelism and load balancer https://github.com/deepseek-ai/eplb
• Support pipeline parallelism @Ying1123
• Support data parallelism attention compatible with all other parallelism @ispobock @merrymercy
• Support sequence parallelism [Feature] Add initial support for sequence parallelism #1436. Related paper. @Ying1123
• Support overlap communication in TP/EP @tom
• Improvements of sgl-router for better data parallelism

Caching

• Optimize Hierarchical cache (GPU/CPU/Disk) Hierarchical Caching for SGLang #2693 Hierarchical Caching supports MLA #4009 @xiezhq-hermann
• Integrate DeepSeek 3FS integration

Kernel

• Integrate DeepGemm add deepgemm into sgl-kernel #3957
• Integrate FlashMLA https://github.com/deepseek-ai/FlashMLA
• Integrate cuDNN attention. reference
• Integrate TransformerEngine layers
• Start to maintain performant attention in sgl-kernel
• Start to maintain more sparse attention ops in sgl-kernel

Quantization

• W8A8 (FP8 and INT8) implementation in sgl-kernel, removing vllm dependency
• Integration of awq and gptq in sgl-kernel, removing vllm dependency
• TorchAO support extension to additional models
• Blackwell FP4 support
• Optional quantization support using vllm's implementation (e.g. bnb, gguf)
• Communication quant

RL Framework integration

• veRL integration SGLang + Verl #3852
• OpenRLHF integration Integrate SGLang into OpenRLHF. Non-Hybrid Engine Only OpenRLHF/OpenRLHF#661

Core refactor

• Support page size > 1
• Simplify scheduler.py and model_runner.py to make them more modular
• Remove the dependency of vLLM @zhyncs @ByronHsu @yizhang2077 [Track] progress in removing vLLM dependencies #2245

Speculative decoding

• Optimizations for large batch
• Adaptive speculative decoding according to batch sizes
• Reference-based speculative decoding Reference speculative decoding #270 Speculative decoding with lookahead #2790

Multi-LoRA serving

@Fridge003

Hardware

• Blackwell support @merrymercy
• AMD GPU @HaiShaw
  • CK kernels
  • aiter integration
• More backends (Intel XPU, TPU)

Model coverage

• Multi-modal models
  • DeepSeek VL2 [Feature] Support DeepSeek VL 2 #2653
  • mistralai/Pixtral [Feature] Support mistralai/Pixtral #2351
  • GLM 4V Add GLM-4v Multimodal Model support for SGLang #1641
  • VILA https://arxiv.org/abs/2412.04468
• Language models
  • Mamba models @rahulbatra85 @HaiShaw

Others

• A padded batch mode to make results more deterministic

  sglang/docs/references/faq.md

  Line 3 in 8912b76

  ## The results are not deterministic, even with a temperature of 0