#offloading

Transfer cost vs recompute cost. If moving data off GPU costs less than recomputing it, offload. If not, keep it. The math is straightforward.

GPU memory is precious. CPU memory is cheap. Moving the right data at the right time can 2x your concurrent requests.