The Future of vLLM (or Similar Projects) for Arm64 CPU Inference?

General Discussion
,
1

Recently noticed that vLLM a fast and easy-to-use library for LLM inference and serving.
AI explained to me the difference of llama.cpp vs. vLLM :slight_smile: :

image
image1866Ă—780 134 KB

While vLLM is primarily GPU-driven, given the rise of highly optimized small LLMs and quantized models, do you think there’s a future for vLLM (or a similar project) to offer highly efficient CPU inference backends, especially on powerful Arm64 server CPUs like Ampere One M? What would be the challenges?
The number of AI projects like libraries/Frameworks/MLOps is growing rapidly. Can you share more information about them?

2 Likes
2

I think we should start from the question “Why do people use ARM CPU to run inference workloads instead of using specialize hardware like GPUs” ?

2 Likes
3

I am all for it! One of vLLM’s slogans (“Any model, any hardware, any cloud”) certainly fits. Right now I believe the Arm64 CPU implementation is not compelling - FP32 for working data, and very little SIMD acceleration - and some of its dependencies do not have Ampere performance improvements - but I hope that changes very soon!

2 Likes
4

I’m seeing already a problem with the llama.cpp improvement from ampere not landing upstream, making hard to keep up their improvement and their extended model support.

vLLM would have even larger problems since it is so far more focused on cuda and just cuda.