NVIDIA Triton Inference Server for production ML serving — dynamic batching, concurrent model execution, multiple backends, and the GPU-utilization and model-management problems it solves between a trained model and serving at scale.
Ml-Infrastructure
-
Triton Inference Server: Production ML Serving That Actually Scales