Post #3536498

Found vLLM to be the most efficient local runtime service. And “ray” as a good (but complicated) way to distribute the load: docs.ray.io