跳到正文
字节 Seed 模型发布·· 2026-07-29AI 评分48

字节发布 Sa2VA-Qwen3-VL-4B-SAM3:基于 Qwen3-VL-4B 与 SAM3 的分割模型

ByteDance/Sa2VA-Qwen3-VL-4B-SAM3

AI 导读

字节发布 Sa2VA-Qwen3-VL-4B-SAM3,在 Qwen3-VL-4B-Instruct 上接入 SAM3 PVS-tracker grounding 编码器,由 MLLM 预测 [SEG] token 驱动 SAM3 mask 解码器,支持图像与视频的指代分割及开放式对话。

正文 · 原文

Sa2VA-Qwen3-VL-4B-SAM3 is a Sa2VA model built on Qwen3-VL-4B-Instruct with a SAM3 PVS-tracker grounding encoder. The MLLM predicts a [SEG] token whose hidden state conditions the SAM3 mask decoder, producing dense image and video referring segmentation alongside open-ended chat.

This checkpoint is self-contained: the SAM3 grounding code is vendored into the repository, so it loads with trust_remote_code=True without any extra packages.

Results

Image referring segmentation (cIoU):

RefCOCO val / testA / testB RefCOCO+ val / testA / testB RefCOCOg val / test
83.7 / 84.7 / 81.9 79.4 / 83.2 / 76.0 83.0 / 83.0

Video referring segmentation (J&F):

MeViS (val_u) ReVOS Ref-DAVIS17
65.3 66.3 77.1

Grounded conversation generation (GCG, val):

AP50 mIoU Recall
37.1 70.1 49.4

Usage

import torch
from transformers import AutoModel, AutoProcessor
from PIL import Image

path = "HarborYuan/Sa2VA-Qwen3-VL-4B-SAM3"
model = AutoModel.from_pretrained(
    path, torch_dtype=torch.bfloat16, trust_remote_code=True, low_cpu_mem_usage=True,
).eval().cuda()
processor = AutoProcessor.from_pretrained(path, trust_remote_code=True)

image = Image.open("your_image.jpg").convert("RGB")
out = model.predict_forward(
    image=image,
    text="Please segment the dog in the image.",
    processor=processor,
)
print(out["prediction"])          # text response containing [SEG]
masks = out["prediction_masks"]   # list of boolean masks at the original image size

Load in bfloat16 (the lm_head is kept in higher precision; torch_dtype="auto" mixes dtypes and fails).

For video, pass video=[frame0, frame1, ...] (a list of PIL images) instead of image.

Notes

  • SAM3 grounding input resolution is 1008.
  • Built on Sa2VA.

Citation

If you find this project useful in your research, please consider citing:

@article{sa2va,
  title={Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos},
  author={Yuan, Haobo and Li, Xiangtai and Zhang, Tao and Sun, Yueyi and Huang, Zilong and Xu, Shilin and Ji, Shunping and Tong, Yunhai and Qi, Lu and Feng, Jiashi and Yang, Ming-Hsuan},
  journal={IEEE TPAMI},
  year={2026}
}

来源:字节 Seed 模型发布 · hf-mirror.com