再次刷新端侧多模态天花板，面壁「小钢炮」 MiniCPM-V 2.6 模型重磅上新！发布即支持 llama.cpp、ollama、vllm 推理！仅 8B 参数，取得 20B 以下单图、多图、视频理解 3 SOTA 成绩，一举将端侧AI多模态能力拉升至全面对标 GPT-4V 水平。更有多项功能首次上「端」：小钢炮一口气将实时视频理解、多图联合理解、多图 ICL 等能力首次搬上端侧多模态模型，更接近充斥着复杂、模糊、连续实时视觉信息的多模态真实世界，更能充分发挥端侧 AI 传感器富集、贴近用户的优势。 ➤ 模型介绍
? 多图、视频首上端！3 SOTA 面壁小钢炮，创 GPT-4V 端侧全面对标新时代！? https://github.com/OpenBMB/MiniCPM-V
? https://huggingface.co/openbmb/MiniCPM-V-2_6
➤ B站配套视频 ，搭配食用更佳
? https://www.bilibili.com/video/BV1sM4m1172r/?spm_id_from=333.999.0.0&vd_source=1534be4f756204643265d5f6aaa38c7b 
vLLM推理
使用Python进行推理
首先前往huggingface下载pytorch权重：
git clone https://huggingface.co/openbmb/MiniCPM-V-2_6
使用git下载安装vllm
git clone https://github.com/vllm-project/vllm.gitcd vllmpip install e .
创建一个python代码调用vllm
from PIL import Imagefrom transformers import AutoTokenizerfrom vllm import LLM, SamplingParams
# 图像文件路径列表IMAGES = ["/root/ld/ld_project/MiniCPM-V/assets/airplane.jpeg",# 本地图片路径]
# 模型名称或路径MODEL_NAME = "/root/ld/ld_model_pretrained/Minicpmv2_6"# 本地模型路径或Hugging Face模型名称
# 打开并转换图像image = Image.open(IMAGES[0]).convert("RGB")
# 初始化分词器tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
# 初始化语言模型llm = LLM(model=MODEL_NAME, gpu_memory_utilization=1,# 使用全部GPU内存 trust_remote_code=True, max_model_len=2048)# 根据内存状况可调整此值
# 构建对话消息messages = [{'role': 'user', 'content': '(<image>./</image>)\n' + '请描述这张图片'}]
# 应用对话模板到消息prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# 设置停止符ID# 2.0# stop_token_ids = [tokenizer.eos_id]# 2.5#stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]# 2.6 stop_tokens = ['<|im_end|>', '<|endoftext|>']stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]
# 设置生成参数sampling_params = SamplingParams(stop_token_ids=stop_token_ids,# temperature=0.7,# top_p=0.8,# top_k=100,# seed=3472,max_tokens=1024,# min_tokens=150,temperature=0,use_beam_search=True,# length_penalty=1.2,best_of=3)
# 获取模型输出outputs = llm.generate({"prompt": prompt,"multi_modal_data": {"image": image}}, sampling_params=sampling_params)print(outputs[0].outputs[0].text)
4. 使用视频推理
from transformers import AutoTokenizerfrom decord import VideoReader, cpufrom PIL import Imagefrom vllm import LLM, SamplingParams
# 进行图片推理MAX_NUM_FRAMES = 64def encode_video(filepath):def uniform_sample(l, n):gap = len(l) / nidxs = [int(i * gap + gap / 2) for i in range(n)]return [l[i] for i in idxs]vr = VideoReader(filepath, ctx=cpu(0))sample_fps = round(vr.get_avg_fps() / 1)# FPSframe_idx = [i for i in range(0, len(vr), sample_fps)]if len(frame_idx)>MAX_NUM_FRAMES:frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)video = vr.get_batch(frame_idx).asnumpy()video = [Image.fromarray(v.astype('uint8')) for v in video]return video
MODEL_NAME = "openbmb/MiniCPM-V-2_6" # or local model path
video = encode_video("xxx.mp4")messages = [{"role":"user","content":"".join(["(<image>./</image>)"] * len(video)) + \"\nPlease describe this video."}]prompt = tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True)tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)llm = LLM(model=MODEL_NAME,gpu_memory_utilization=1,max_model_len=4096)
stop_tokens = ['<|im_end|>', '<|endoftext|>']stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]
sampling_params = SamplingParams(stop_token_ids=stop_token_ids, use_beam_search=Truetemperature=0, max_tokens=64)
outputs = llm.generate({"prompt": prompt,"multi_modal_data": {"image": {"images": video,"use_image_id": False,"max_slice_nums": 1 if len(video) > 16 else 2}}}, sampling_params=sampling_params_beam)
vLLM API Server
使用git下载安装vllm
git clone https://github.com/vllm-project/vllm.gitcd vllmpip install e .
命令行启动vllm服务
gpu_memory_utilization gpu使用率更多参数请访问：https://docs.vllm.ai/en/latest/models/engine_args.htmlvllm serve /root/ld/ld_model_pretrained/Minicpmv2_6 --dtype auto --max-model-len 2048 --api-key token-abc123 --gpu_memory_utilization 1 --trust-remote-code
使用python代码调用vllm的http服务
3.1 传入网络图片
from openai import OpenAIopenai_api_key = "token-abc123" # your api key set in launch serveropenai_api_base = "http://localhost:8000/v1" # http idclient = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)
chat_response = client.chat.completions.create(model="/root/ld/ld_model_pretrained/Minicpmv2_6", # model_local_path or huggingface idmessages=[{"role": "user","content": [# NOTE: 使用图像令牌 <image> 的提示格式是不必要的，因为提示将由API服务器自动处理。# 由于提示将由API服务器自动处理，因此不需要使用包含 <image> 图像令牌的提示格式。{"type": "text", "text": "请描述这张图片"},{"type": "image_url","image_url": {"url": "https://air-example-data-2.s3.us-west-2.amazonaws.com/vllm_opensource_llava/stop_sign.jpg",},},],}],extra_body={"stop_token_ids": [151645, 151643]})print("Chat response:", chat_response)print("Chat response content:", chat_response.choices[0].message.content)
3.2 传入本地图片
from openai import OpenAIopenai_api_key = "token-abc123" openai_api_base = "http://localhost:8000/v1" client = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)
with open('your/local/pic/path','rb') as file:image = "data:image/jpeg;base64,"+ base64.b64encode(file.read()).decode('utf-8')
chat_response = client.chat.completions.create(model="/root/ld/ld_model_pretrained/Minicpmv2_6", messages=[{"role": "user","content": [{"type": "text", "text": "请描述这张图片"},{"type": "image_url","image_url": {"url": image,},},],}],extra_body={"stop_token_ids": [151645, 151643]})print("Chat response:", chat_response)print("Chat response content:", chat_response.choices[0].message.content)
llama.cpp推理
设备要求：运行非量化版内存超过19g，运行量化版超过8g内存
1. 下载依赖包
brew install ffmpegbrew install pkg-config
2. 获取openbmb的llama.cpp分支
git clone -b minicpm-v2.5 https://github.com/OpenBMB/llama.cpp.git
3. 编译llama.cpp
4. 获取MiniCPM-V 2.6的gguf权重
方法一：
a. 首先前往huggingface或者modelscope下载pytorch权重：
git clone https://huggingface.co/openbmb/MiniCPM-V-2_6
b. 使用上述llama.cpp将模型权重转化为gguf文件：# 第一行为获得模型中间输出，为转换为gguf作准备python ./examples/llava/minicpmv-convert/minicpmv2_6-surgery.py -m ../MiniCPM-V-2_6# 将siglip模型转换为ggufpython ./examples/llava/minicpmv-convert/minicpmv2_6-convert-image-encoder-to-gguf.py -m ../MiniCPM-V-2_6 --minicpmv-projector ../MiniCPM-V-2_6/minicpmv.projector --output-dir ../MiniCPM-V-2_6/ --image-mean 0.5 0.5 0.5 --image-std 0.5 0.5 0.5# 将语言模型转换为ggufpython ./convert-hf-to-gguf.py ../MiniCPM-V-2_6/model
# quantize int4 version./llama-quantize ../MiniCPM-V-2_6/model/ggml-model-f16.gguf ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf Q4_K_M
方法二：
直接前往MiniCPM-V 2.6-gguf官方仓库下载模型，ggml-model-Q4_K_M.gguf(量化版)和ggml-model-f16.gguf二者选其一。下载地址：https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf/tree/main5. 开始推理：
./llama-minicpmv-cli -m ./Minicpmv2_6gguf/ggml-model-Q4_K_M.gguf --mmproj ./Minicpmv2_6gguf/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image ./Minicpmv2_6gguf/42.jpg -p "这张图片中有什么？"
5.2 视频推理指令
./llama-minicpmv-cli -m /Users/liudan/Downloads/Minicpmv2_6gguf/ggml-model-Q4_K_M.gguf --mmproj /Users/liudan/Downloads/Minicpmv2_6gguf/mmproj-model-f16.gguf -c 8192 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --video ./Minicpmv2_6gguf/test_vedieo.mp4 -p "我接下来会给你一个视频，请告诉我视频中描述了什么"
Ollama推理
ollma官方github即将合并我们的ollama分支，请关注我们的 GitHub设备要求：运行非量化版本需要19g以上内存，运行量化版本需要8g以上内存1. 按照上述llama.cpp教程获取gguf模型
语言模型最好是量化模型。
2. 获取openbmb官方ollma分支
git clone -b minicpm-v2.6 https://github.com/OpenBMB/ollama.gitcd ollama/llm
3. 环境需求
- cmake version 3.24 or above
- gcc version 11.4.0 or above
brew install go cmake gcc
4. 安装大型模型依赖项:
go generate ./...
5. 编译ollama
go build .
6. 编译成功，在ollama主路径启动ollama：
./ollama serve
7. 创建一个Modelfile:
vim minicpmv2_6.Modelfile
FROM ./MiniCPM-V-2_6/model/ggml-model-Q4_K_M.ggufFROM ./MiniCPM-V-2_6/mmproj-model-f16.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>{{ end }}
{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>{{ end }}
<|im_start|>assistant<|im_end|>
{{ .Response }}<|im_end|>"""
PARAMETER stop "<|endoftext|>"PARAMETER stop "<|im_end|>"PARAMETER num_ctx 2048
8. 创建ollama模型实例：
ollama create minicpm2.6 -f minicpmv2_6.Modelfile
9. 另起一个命令行窗口，运行ollama模型实例：
ollama run minicpm2.6
10. 输入问题和图片 URL，以空格分隔
What is described in this picture? /Users/liudan/Desktop/11.jpg
粤ICP备14082021号
免费POC，
零成本试错
