title: the vLLM pool and one stats line at a cap of 64
source: manuals/sglang-101/capture/gpu/out-l4/server-h2h-vllm.log
lang: text
note: Count the running and waiting requests, and read the KV cache usage and the prompt throughput.
---
INFO 10-17 20:13:17 [kv_cache_utils.py:2464] GPU KV cache size: 24,096 tokens, Maximum concurrency for 8,192 tokens per request: 2.94x
INFO 10-17 22:35:43 [loggers.py:320] Engine 000: Avg prompt throughput: 460.3 tokens/s, Avg generation throughput: 323.6 tokens/s, Running: 40 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 8.7%
