vLLM🤝🤗! You can now deploy any @huggingface language model with vLLM's speed. This integration makes it possible for one consistent implementation of the model in HF for both training and inference. 🧵
Concretely, vLLM now integrates @huggingface Transformers as a backend, combining its vast model ecosystem with vLLM's inference optimizations. You can now deploy any HF model with vLLM's speed—lower latency, better memory usage & OpenAI-compatible API. Best of both worlds!
3
It's not a trivial task to port a model implementation in vLLM. In the ideal world, we would be able to use the new model in vLLM as soon as it is added to transformers. We are taking a step into that.
4
vLLM's new `--model-impl transformers` flag switch the model implementation to the transformers model implementation, without losing performance.
5
By pairing Transformers’ model ecosystem with vLLM’s inference optimizations, you unlock a workflow that’s both flexible and scalable.
P.S. Multimodal support coming soon.
Save this thread — create a free accountSave this thread