Qwen 发布 Qwen2-VL-2B-Instruct 视觉语言模型
Qwen/Qwen2-VL-2B-Instruct
Qwen 发布 Qwen2-VL 系列视觉语言模型,包含 2B、7B、72B 三个参数规模,该仓库为指令微调的 2B 版本。模型支持任意分辨率图像输入、20 分钟以上视频理解,以及图像内多语种文字识别,并采用 Naive Dynamic Resolution 与 M-ROPE 两项架构更新。
模型卡给出了 2B 版本在图像与视频基准上的具体分数,可与同量级模型直接对照。
简介
我们很高兴推出 Qwen2-VL,这是 Qwen-VL 模型的最新版,凝聚了我们近一年的创新成果。
Qwen2-VL 的新特性
主要增强功能:
对多种分辨率和比例图像的顶尖理解能力:Qwen2-VL 在视觉理解基准测试中取得了最先进的性能,包括 MathVista、DocVQA、RealWorldQA、MTVQA 等。
支持理解超过 20 分钟的视频:Qwen2-VL 能够理解时长超过 20 分钟的视频,用于高质量的基于视频的问题回答、对话、内容创作等。
可操作手机、机器人等设备的智能体:凭借复杂的推理和决策能力,Qwen2-VL 可与手机、机器人等设备集成,根据视觉环境和文本指令实现自动操作。
多语言支持:为了服务全球用户,除了英语和中文,Qwen2-VL 现在还支持理解图像中不同语言的文本,包括大多数欧洲语言、日语、韩语、阿拉伯语、越南语等。
模型架构更新:
- 原生动态分辨率:与以往不同,Qwen2-VL 可以处理任意分辨率的图像,将其映射为动态数量的视觉 token,提供更接近人类视觉的处理体验。
- 多模态旋转位置嵌入(M-ROPE):将位置嵌入分解为多个部分,以捕捉一维文本、二维视觉和三维视频的位置信息,从而增强其多模态处理能力。
我们提供了三个参数量分别为 20 亿、70 亿和 720 亿的模型。本仓库包含经过指令微调的 20 亿参数版 Qwen2-VL 模型。更多信息请访问我们的 博客 和 GitHub。
评估结果
图像基准测试
| 基准测试 | InternVL2-2B | MiniCPM-V 2.0 | Qwen2-VL-2B |
|---|---|---|---|
| MMMUval | 36.3 | 38.2 | 41.1 |
| DocVQAtest | 86.9 | - | 90.1 |
| InfoVQAtest | 58.9 | - | 65.5 |
| ChartQAtest | 76.2 | - | 73.5 |
| TextVQAval | 73.4 | - | 79.7 |
| OCRBench | 781 | 605 | 794 |
| MTVQA | - | - | 20.0 |
| VCRen easy | - | - | 81.45 |
| VCRzh easy | - | - | 46.16 |
| RealWorldQA | 57.3 | 55.8 | 62.9 |
| MMEsum | 1876.8 | 1808.6 | 1872.0 |
| MMBench-ENtest | 73.2 | 69.1 | 74.9 |
| MMBench-CNtest | 70.9 | 66.5 | 73.5 |
| MMBench-V1.1test | 69.6 | 65.8 | 72.2 |
| MMT-Benchtest | - | - | 54.5 |
| MMStar | 49.8 | 39.1 | 48.0 |
| MMVetGPT-4-Turbo | 39.7 | 41.0 | 49.5 |
| HallBenchavg | 38.0 | 36.1 | 41.7 |
| MathVistatestmini | 46.0 | 39.8 | 43.0 |
| MathVision | - | - | 12.4 |
视频基准测试
| 基准测试 | Qwen2-VL-2B |
|---|---|
| MVBench | 63.2 |
| PerceptionTesttest | 53.9 |
| EgoSchematest | 54.9 |
| Video-MMEwo/w subs | 55.6/60.4 |
环境要求
Qwen2-VL 的代码已合并至最新的 Hugging Face transformers 库中,我们建议使用命令 pip install git+https://github.com/huggingface/transformers 从源码构建,否则可能会遇到以下错误:
KeyError: 'qwen2_vl'
快速开始
我们提供了一个工具包,帮助您更方便地处理各种类型的视觉输入,包括 base64、URL 以及图文/视频交错的内容。您可以使用以下命令安装该工具包:
pip install qwen-vl-utils
此处展示了一段代码示例,说明如何使用带有 transformers 和 qwen_vl_utils 的聊天模型:
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info
# default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2-VL-2B-Instruct", torch_dtype="auto", device_map="auto"
)
# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
# model = Qwen2VLForConditionalGeneration.from_pretrained(
# "Qwen/Qwen2-VL-2B-Instruct",
# torch_dtype=torch.bfloat16,
# attn_implementation="flash_attention_2",
# device_map="auto",
# )
# default processer
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")
# The default range for the number of visual tokens per image in the model is 4-16384. You can set min_pixels and max_pixels according to your needs, such as a token count range of 256-1280, to balance speed and memory usage.
# min_pixels = 256*28*28
# max_pixels = 1280*28*28
# processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
},
{"type": "text", "text": "Describe this image."},
],
}
]
# Preparation for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
不使用 qwen_vl_utils
from PIL import Image
import requests
import torch
from torchvision import io
from typing import Dict
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
# Load the model in half-precision on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2-VL-2B-Instruct", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")
# Image
url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
image = Image.open(requests.get(url, stream=True).raw)
conversation = [
{
"role": "user",
"content": [
{
"type": "image",
},
{"type": "text", "text": "Describe this image."},
],
}
]
# Preprocess the inputs
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe this image.<|im_end|>\n<|im_start|>assistant\n'
inputs = processor(
text=[text_prompt], images=[image], padding=True, return_tensors="pt"
)
inputs = inputs.to("cuda")
# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [
output_ids[len(input_ids) :]
for input_ids, output_ids in zip(inputs.input_ids, output_ids)
]
output_text = processor.batch_decode(
generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
)
print(output_text)
多图推理
# Messages containing multiple images and a text query
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": "file:///path/to/image1.jpg"},
{"type": "image", "image": "file:///path/to/image2.jpg"},
{"type": "text", "text": "Identify the similarities between these images."},
],
}
]
# Preparation for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")
# Inference
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
视频推理
# Messages containing a images list as a video and a text query
messages = [
{
"role": "user",
"content": [
{
"type": "video",
"video": [
"file:///path/to/frame1.jpg",
"file:///path/to/frame2.jpg",
"file:///path/to/frame3.jpg",
"file:///path/to/frame4.jpg",
],
"fps": 1.0,
},
{"type": "text", "text": "Describe this video."},
],
}
]
# Messages containing a video and a text query
messages = [
{
"role": "user",
"content": [
{
"type": "video",
"video": "file:///path/to/video1.mp4",
"max_pixels": 360 * 420,
"fps": 1.0,
},
{"type": "text", "text": "Describe this video."},
],
}
]
# Preparation for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")
# Inference
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
批量推理
# Sample messages for batch inference
messages1 = [
{
"role": "user",
"content": [
{"type": "image", "image": "file:///path/to/image1.jpg"},
{"type": "image", "image": "file:///path/to/image2.jpg"},
{"type": "text", "text": "What are the common elements in these pictures?"},
],
}
]
messages2 = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who are you?"},
]
# Combine messages for batch processing
messages = [messages1, messages1]
# Preparation for batch inference
texts = [
processor.apply_chat_template(msg, tokenize=False, add_generation_prompt=True)
for msg in messages
]
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=texts,
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")
# Batch Inference
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_texts = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_texts)
更多使用技巧
对于输入图像,我们支持本地文件、base64 和 URL。对于视频,目前仅支持本地文件。
# You can directly insert a local file path, a URL, or a base64-encoded image into the position where you want in the text.
## Local file path
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": "file:///path/to/your/image.jpg"},
{"type": "text", "text": "Describe this image."},
],
}
]
## Image URL
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": "http://path/to/your/image.jpg"},
{"type": "text", "text": "Describe this image."},
],
}
]
## Base64 encoded image
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": "data:image;base64,/9j/..."},
{"type": "text", "text": "Describe this image."},
],
}
]
提升性能的图片分辨率设置
该模型支持多种分辨率的输入。默认情况下,它使用原生分辨率作为输入,但更高的分辨率可以在增加计算量的同时提升性能。用户可以设置最小和最大像素数以优化配置,例如将 token 数量范围设定为 256-1280,以平衡速度与内存占用。
min_pixels = 256 * 28 * 28
max_pixels = 1280 * 28 * 28
processor = AutoProcessor.from_pretrained(
"Qwen/Qwen2-VL-2B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels
)
此外,我们提供两种方法对输入模型的图像尺寸进行细粒度控制:
定义 min_pixels 和 max_pixels:图像将被调整大小以保持其宽高比在 min_pixels 和 max_pixels 范围内。
指定精确尺寸:直接设置
resized_height和resized_width。这些值将四舍五入到最接近的 28 的倍数。
# min_pixels and max_pixels
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "file:///path/to/your/image.jpg",
"resized_height": 280,
"resized_width": 420,
},
{"type": "text", "text": "Describe this image."},
],
}
]
# resized_height and resized_width
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "file:///path/to/your/image.jpg",
"min_pixels": 50176,
"max_pixels": 50176,
},
{"type": "text", "text": "Describe this image."},
],
}
]
局限性
虽然 Qwen2-VL 适用于广泛的视觉任务,但了解其局限性同样重要。以下是一些已知的限制:
- 缺乏音频支持:当前模型无法理解视频中的音频信息。
- 数据时效性:我们的图像数据集更新至 2023 年 6 月,此日期之后的信息可能未被涵盖。
- 个人与知识产权(IP)识别限制:模型识别特定个人或 IP 的能力有限,可能无法全面覆盖所有知名人物或品牌。
- 复杂指令处理能力有限:面对复杂的分步指令时,模型的理解和执行能力有待提升。
- 计数准确性不足:特别是在复杂场景中,物体计数的准确率不高,需要进一步改进。
- 空间推理能力较弱:特别是在 3D 空间中,模型对物体位置关系的推断不足,难以准确判断物体的相对位置。
这些局限性是模型持续优化和改进的方向,我们致力于不断提升模型的性能和应用范围。
引用
如果您觉得我们的工作有帮助,欢迎引用。
@article{Qwen2VL,
title={Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
author={Wang, Peng and Bai, Shuai and Tan, Sinan and Wang, Shijie and Fan, Zhihao and Bai, Jinze and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and Fan, Yang and Dang, Kai and Du, Mengfei and Ren, Xuancheng and Men, Rui and Liu, Dayiheng and Zhou, Chang and Zhou, Jingren and Lin, Junyang},
journal={arXiv preprint arXiv:2409.12191},
year={2024}
}
@article{Qwen-VL,
title={Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond},
author={Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren},
journal={arXiv preprint arXiv:2308.12966},
year={2023}
}
- 上月下载量
- 942,624
Safetensors
Qwen/Qwen2-VL-2B-Instruct 的模型树
微调版
(25)
此模型
使用 Qwen/Qwen2-VL-2B-Instruct 的 Spaces 100
包含 Qwen/Qwen2-VL-2B-Instruct 的集合
关于 Qwen/Qwen2-VL-2B-Instruct 的论文
来源:Qwen 模型发布 · hf-mirror.com