ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding

Zoey Guo, Yiwen Tang, Ray Zhang, Dong Wang, Zhigang Wang, Bin Zhao, Xuelong Li

科研成果: 书/报告/会议事项章节会议稿件同行评审

20 引用 (Scopus)

摘要

Understanding 3D scenes from multi-view inputs has been proven to alleviate the view discrepancy issue in 3D visual grounding. However, existing methods normally neglect the view cues embedded in the text modality and fail to weigh the relative importance of different views. In this paper, we propose ViewRefer, a multi-view framework for 3D visual grounding exploring how to grasp the view knowledge from both text and 3D modalities. For the text branch, ViewRefer leverages the diverse linguistic knowledge of large-scale language models, e.g., GPT, to expand a single grounding text to multiple geometry-consistent descriptions. Meanwhile, in the 3D modality, a transformer fusion module with inter-view attention is introduced to boost the interaction of objects across views. On top of that, we further present a set of learnable multi-view prototypes, which memorize scene-agnostic knowledge for different views, and enhance the framework from two perspectives: a view-guided attention module for more robust text features, and a view-guided scoring strategy during the final prediction. With our designed paradigm, ViewRefer achieves superior performance on three benchmarks and surpasses the second-best by +2.8%, +1.5%, and +1.35% on Sr3D, Nr3D, and ScanRefer.

源语言英语
主期刊名Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
出版商Institute of Electrical and Electronics Engineers Inc.
15326-15337
页数12
ISBN(电子版)9798350307184
DOI
出版状态已出版 - 2023
活动2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Paris, 法国
期限: 2 10月 20236 10月 2023

出版系列

姓名Proceedings of the IEEE International Conference on Computer Vision
ISSN(印刷版)1550-5499

会议

会议2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
国家/地区法国
Paris
时期2/10/236/10/23

指纹

探究 'ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding' 的科研主题。它们共同构成独一无二的指纹。

引用此