Calculate token-per-second generation throughput for autoregressive models given KV cache size, memory bandwidth, and hardware specifications.
Enter the values for the patient or scenario you are assessing. Calculate token-per-second generation throughput for autoregressive models given KV cache size, memory bandwidth, and hardware specifications. Use the gpt throughput result to inform your clinical assessment.