Run Llama3 inference on API server

[nwatab]

Does anyone know how to run HTTP server that runs Llama inference on it? I searched ending up find no helpful resource about integration with application/WSGI server (ex. Flask, gunicorn). The Llama3 tutorial uses torchrun, but what it does under the hood seems a bit complicated.

Edited:
LLM server scale is limited to the number of GPUs, so we might not need WSGI server in most cases