Skip to content
Robin930Public

About

A profiling-based simulator for vLLM inference

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

vLLM-Sim

Install

uv venv .venv --python 3.10
source .venv/bin/activate
uv pip install -r requirements.txt
uv pip install .

输出结果依赖 chromium

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ./google-chrome-stable_current_amd64.deb

Quick Start

python -m vllm_sim.main  \
    --replica_config_device h800 \
    --replica_config_network_device h800_dgx \
    --replica_config_model_name NousResearch/Llama-2-7b-hf \
    --cluster_config_num_replicas 1 \
    --replica_config_tensor_parallel_size 2 \
    --replica_config_num_pipeline_stages 1 \
    --replica_scheduler_config_type vllm \
    --length_generator_config_type trace \
    --trace_request_length_generator_config_trace_file ./data/processed_traces/cnn_dailymail2000.csv \
    --interval_generator_config_type trace \
    --trace_request_interval_generator_config_trace_file ./data/processed_traces/cnn_dailymail2000.csv

切换模型、GPU、节点架构后需要重新训练预测模型,需要的时间比较长,更换 workload 不需要重新训练

Arguments

  • --replica_config_device:GPU,支持 h800
  • replica_config_network_device:节点架构,支持 h800-dgx
  • replica_config_model_name:模型,支持 NousResearch/Llama-2-7b-hf,NousResearch/Llama-2-70b-hf,Qwen/Qwen2.5-32B,Qwen/Qwen2.5-72B-Instruct,添加新模型需要跑 profiling
  • cluster_config_num_replicas:dp size
  • replica_config_tensor_parallel_size:tp size
  • replica_config_num_pipeline_stages:pp size
  • replica_scheduler_config_type:调度器,默认是 vLLM
  • trace_request_length_generator_config_trace_file 和 trace_request_interval_generator_config_trace_file:workload,请指定为同一个文件

Workload

workload 文件格式如下

arrival_time,num_prefill_tokens,num_decode_tokens
<arrival_time>,<num_prefill_tokens>,<num_decode_tokens>
...
  • arrival_time:请求的到达时间,格式为YYYY-MM-DD HH:mm:ss
  • num_prefill_tokens:输入token数量
  • num_decode_tokens:输出token数量

Output

所有输出的文件位于./simulator_output/<timestamp>/下

config.json为本次仿真的所有配置项

request_metrics.csv包含所有的预测指标,每一行对应一个请求

  • Request Id
  • request_e2e_time:请求的端到端执行时间
  • request_e2e_time_normalized:归一化的端到端执行时间,为request_e2e_time/(num_prefill_tokens+num_decode_tokens)
  • request_model_execution_time:模型执行时间,不包含输入预处理、调度器等执行时间
  • request_model_execution_time_normalized:为request_model_execution_time/(num_prefill_tokens+num_decode_tokens)
  • request_preemption_time:请求因被抢占的等待时间
  • request_scheduling_delay:请求从到达到第一次被执行的排队时间
  • request_execution_plus_preemption_time:请求从开始处理到完成的时间,为request_model_execution_time+request_preemption_time
  • request_execution_plus_preemption_time_normalized:为request_execution_plus_preemption_time/(num_prefill_tokens+num_decode_tokens)
  • prefill_e2e_time:prefill阶段的端到端执行时间
  • prefill_time_execution_plus_preemption:prefill阶段执行和被抢占的总时间
  • prefill_time_execution_plus_preemption_normalized:为prefill_time_execution_plus_preemption/num_prefill_tokens
  • decode_time_execution_plus_preemption_normalized:为decode_time_execution_plus_preemption/num_decode_tokens
  • request_inter_arrival_delay:请求到达的间隔
  • request_num_tokens:请求输入和输出的总token数
  • request_num_prefill_tokens:请求输入的token数
  • request_num_decode_tokens:请求输出的token数
  • request_pd_ratio:为request_num_prefill_tokens/request_num_decode_tokens
  • request_num_restarts:请求由于调度被重新执行的次数

plots目录下有单个预测指标的记录和可视化

About

A profiling-based simulator for vLLM inference

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages