Batch Size, Throughput and Latency
~14 mincode completion
Implement batching_tradeoff(t0, c, arrival_rate, batch_sizes) returning an array of shape (len(batch_sizes), 2) whose rows are [throughput, latency].
Where this shows up
~14 min
••••••••••••••••
8 employers weight this skill
4 big tech firms, 1 AI product company, 1 enterprise vendor, 1 defense company, 1 quant fund. Top match scores 93.
Python
import numpy as np
def batching_tradeoff(t0, c, arrival_rate, batch_sizes):
"""
Throughput and average per-request latency at several batch sizes.
Args:
t0: fixed cost of one model call, in ms
c: extra cost per row in the batch, in ms
arrival_rate: requests arriving per second
batch_sizes: list of batch sizes B
Returns:
(len(batch_sizes), 2) array of [throughput in req/s, latency in ms]
"""
# YOUR CODE HERE
pass