Tech_Interview_Prep

Model Serving & Inference Infrastructure

The infrastructure choices behind getting predictions out of a trained model at production latency and scale.

OverviewQ&AMCQ

What it is

Model serving is the infrastructure layer that takes a trained model and exposes it to make predictions in production, with the latency and throughput characteristics the use case actually needs.

Key points

  • Online vs. batch inference: online serving answers individual requests in real time (a recommendation on page load); batch inference precomputes predictions for many inputs at once on a schedule — the right choice depends entirely on whether the use case needs a fresh, per-request prediction.
  • Latency vs. throughput tradeoffs: techniques like request batching improve throughput but add latency per request — serving infrastructure design is largely about tuning this tradeoff for the specific use case's requirements.
  • Hardware considerations: GPU inference is often necessary for large models but is expensive and needs careful utilization management; CPU inference is cheaper and sufficient for many smaller/classical models.
  • Model compression (quantization, distillation) trades some accuracy for significantly lower latency and infrastructure cost — often the deciding factor for serving large models within a latency budget.