一直使用mindie框架进行大模型推理，现在vllm可以提供监控指标给Prometheus/Grafana，而且支持Ascend了。
一、镜像：下个镜像真不容易，最后从渡渡鸟上下了：swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/ascend/vllm-ascend:v0.9.2rc1-linuxarm64
二、启动：修改官网命令：其中权重文件映射到/model/QwQ-32B。并行2张npu
docker run -itd --name qwq-32b-vllm \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-v /app/model/:/model/:ro \
-p 18034:8000 \
swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/ascend/vllm-ascend:v0.9.2rc1-linuxarm64 \
vllm serve /model/QwQ-32B --max_model_len 32768 -tp 2 --served-model-name qwq32b
三、测试：
curl -H "Accept: application/json" \ -H "Content-type: application/json" \ -X POST \ -d '{"model": "qwq32b","messages": [{"role": "user", "content": "你是哪个模型？"},{"role": "assistant", "content": "你好"}],"stream": false}' \ http://localhost:18034/v1/chat/completions
四、参考：
https://agent.blog.csdn.net/article/details/155446713
https://agent.blog.csdn.net/article/details/155539176
https://quay.io/repository/ascend/vllm-ascend?tab=tags
https://github.com/vllm-project/vllm-ascend
https://docs.vllm.ai/projects/ascend/en/latest/tutorials
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。