Throughput

The number of tokens an inference system generates per unit of time, usually measured in tokens per second. There is overall throughput (all concurrent requests) and single-user latency: two numbers that tell different stories and should be measured separately.