vllm-bench

vllm-bench is a Rust benchmark client for online LLM serving. It does not run the model. It drives an already running OpenAI-compatible or vLLM endpoint and reports TTFT, TPOT, ITL, E2EL, throughput, and multi-turn per-turn breakdowns.

Use this workflow for multi-turn work:

1. Install or locate vllm-bench.
2. Run vllm-bench --help and use it as the authority for current flags.
3. Confirm the server base URL and model id from /v1/models.
4. Dry-run the dataset locally.
5. Run a tiny smoke benchmark.
6. Scale conversations, turns, and concurrency only after the smoke passes.
7. Save JSON for any result that may be compared later.

Install

vllm-bench has moved into the main vLLM repository as a crate in the Rust workspace (vllm-project/vllm → rust/src/bench); the standalone vllm-project/vllm-bench repo is archived and read-only. Build the maintained version from the new home:

git clone --depth 1 https://github.com/vllm-project/vllm.git
cd vllm/rust
cargo build --release -p vllm-bench   # -> target/release/vllm-bench

--depth 1 keeps the clone small; vLLM full history is large and not needed for a build.

To install onto PATH from the clone (the build above left the shell in vllm/rust):

cargo install --path src/bench        # from vllm/rust -> ~/.cargo/bin/vllm-bench

Note: upstream rust/src/bench/README.md may still show the archived standalone-repo install commands; the migration notice on the archived repo is the authoritative pointer, and --help on a fresh build is the flag authority.

After install, run:

vllm-bench --version
vllm-bench --help

Choose Backend

For multi-turn chat, always use:

--backend openai-chat

Common backends:

Prefer http://127.0.0.1:<port> over localhost; some servers bind IPv4 while localhost resolves to IPv6.

Confirm the model id before benchmarking:

curl -s http://127.0.0.1:8000/v1/models

Use the returned id as --model, or omit --model and let vllm-bench auto-fetch it.

Multi-Turn Flags

Core flags:

--multi-turn
--multi-turn-num-turns <N>
--num-prompts <CONVERSATIONS>
--multi-turn-concurrency <CONCURRENT_CONVERSATIONS>

In multi-turn mode, --num-prompts means conversation count, not request count. A run with --num-prompts 50 --multi-turn-num-turns 5 can send up to 250 chat requests.

Synthetic random conversations:

--dataset-name random
--random-input-len <TOKENS_FOR_TURN_1>
--per-turn-input-len <TOKENS_FOR_TURNS_2_PLUS>
--random-output-len <OUTPUT_TOKENS_PER_TURN>

If --per-turn-input-len is omitted or 0, later turns reuse --random-input-len. Use --multi-turn-min-turns and --multi-turn-max-turns when conversation lengths should vary.

Think time and prefix sharing:

--multi-turn-delay-ms 500
--multi-turn-prefix-global-ratio 0.2
--multi-turn-prefix-conversation-ratio 0.5

Prefix sharing works only with --dataset-name random; the two ratios must sum to less than 1.0.

When either prefix-sharing flag is non-zero, vllm-bench switches to fixed- length per-turn prompts with no accumulated chat history. Use this only for prefix-cache experiments; leave both ratios at 0 for realistic multi-turn conversation growth.

Dry Run

Dry-run before sending traffic. It loads the tokenizer and builds conversations without hitting the server:

vllm-bench \
  --backend openai-chat \
  --model Qwen/Qwen3-4B \
  --tokenizer <model-path> \
  --dataset-name random \
  --multi-turn --multi-turn-num-turns 2 \
  --random-input-len 16 --random-output-len 4 \
  --num-prompts 2 \
  --dry-run

<model-path> is a local tokenizer/model directory; for a public example, replace it with the served model id or any local model directory.

Example: Generic vLLM Multi-Turn

Use this after a vLLM OpenAI server is already running on port 8000:

vllm-bench \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --model <model-id-from-/v1/models> \
  --dataset-name random \
  --multi-turn \
  --multi-turn-num-turns 5 \
  --random-input-len 512 \
  --per-turn-input-len 256 \
  --random-output-len 128 \
  --num-prompts 50 \
  --multi-turn-concurrency 10 \
  --ready-check-timeout-sec 60 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,99 \
  --save-result

Read the output as:

• Conversations completed/total: end-to-end conversation success.
• Successful requests: successful turns.
• Total input tokens: includes growing chat history in normal multi-turn mode.
• per_turn_metrics in JSON: TTFT/TPOT/ITL/E2EL by turn index.

Expect later turns to have larger input token counts because each request sends the prior user/assistant history plus the next user message.

Example: Local PegaInfer Qwen3-4B Smoke

This was validated locally with PegaInfer Qwen3-4B and vllm-bench built from this repo. Start PegaInfer:

cd <pegainfer-repo>
cargo run --release -p pegainfer-server -- \
  --model-path <model-path> \
  --served-model-name Qwen/Qwen3-4B \
  --port 18080 \
  --cuda-graph=false

Confirm the model:

curl -s http://127.0.0.1:18080/v1/models

Run a tiny multi-turn benchmark:

cd <vllm-repo>/rust
target/release/vllm-bench \
  --backend openai-chat \
  --base-url http://127.0.0.1:18080 \
  --model Qwen/Qwen3-4B \
  --tokenizer <model-path> \
  --dataset-name random \
  --multi-turn \
  --multi-turn-min-turns 2 \
  --multi-turn-max-turns 3 \
  --random-input-len 32 \
  --per-turn-input-len 12 \
  --random-output-len 8 \
  --num-prompts 3 \
  --multi-turn-concurrency 2 \
  --ready-check-timeout-sec 30 \
  --extra-body '{"min_tokens":null}' \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90 \
  --save-result \
  --result-dir /tmp \
  --result-filename vllm-bench-multi-turn-smoke.json

The --extra-body '{"min_tokens":null}' is for PegaInfer compatibility. The random multi-turn path auto-adds min_tokens unless overridden; this pins output length on vLLM but PegaInfer currently rejects that field. --ignore-eos also skips min_tokens and worked in local smoke testing, but it asks the server to ignore EOS and can grow contexts more aggressively.

The local smoke run completed 3/3 conversations, 8/8 turns, and wrote /tmp/vllm-bench-multi-turn-smoke.json with per_turn_metrics.

Scale Safely

After the smoke passes:

1. Increase --num-prompts first to improve measurement stability.
2. Increase --multi-turn-concurrency to find the service knee.
3. Increase --random-input-len, --per-turn-input-len, and --random-output-len to match the target workload.
4. Save every serious run with --save-result --metadata KEY=VALUE.

For deterministic latency A/B runs, pass --temperature 0 explicitly. Omit it only when the benchmark is intentionally measuring server default generation behavior.

Use a sweep when searching for the concurrency knee:

vllm-bench \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --model <model-id> \
  --dataset-name random \
  --multi-turn --multi-turn-num-turns 5 \
  --random-input-len 512 --per-turn-input-len 256 --random-output-len 128 \
  --num-prompts 200 \
  --sweep-max-concurrency 1,2,4,8,16,32 \
  --sweep-summary-percentiles 90,99 \
  --save-result

For A/B comparisons, keep the workload flags identical and compare result JSON:

vllm-bench --compare baseline.json candidate.json

Troubleshooting

• min_tokens is not supported by this engine: pass --extra-body '{"min_tokens":null}' or, if appropriate, --ignore-eos.
• All-zero metrics with failed requests: inspect the first HTTP error printed above the result block; it usually names the unsupported request field.
• Connection fails on localhost: switch to 127.0.0.1.
• Model id mismatch: use curl /v1/models and set --model to that exact id.
• Tokenizer load fails: set --tokenizer to a local model directory with tokenizer.json, or rely on server-side tokenize/detokenize fallback if the server implements it.
• Dataset construction is suspicious: run the same command with --dry-run.
• Long serious runs need reproducibility: set --seed, --result-filename, and --metadata entries for server commit, model, GPU, and key flags.

Metrics

Use database-style thinking: requests are transactions, prompt tokens are read set size, output tokens are write volume, and concurrency is the number of in-flight conversations.

• TTFT: queueing plus prefill latency until the first output token.
• TPOT: decode time per output token after the first token.
• ITL: per-token latency distribution.
• E2EL: total turn latency.
• Request throughput: turns per second.
• Output throughput: generated tokens per second.
• Multi-turn JSON: inspect per_turn_metrics to see how latency changes as history grows.