Rethinking the refinement stage of 3D object detection: A multi-task learning perspective with Mixture-of-Experts

Two-stage LiDAR-based 3D object detectors have achieved state-of-the-art accuracy, yet their performance is often limited by the refinement stage. In this work, we revisit 3D object refinement from a multi-task learning perspective and identify two independent sources of negative transfer: an inter-attribute conflict, where heterogeneous regression objectives, such as center, size, and orientation, interfere during joint optimization, and an inter-sample conflict, where proposals with varying point densities lead to gradient imbalance. To address these issues, we introduce two specialized Mixture-of-Experts architectures. The Attribute-MoE decouples regression objectives into dedicated expert branches to alleviate feature conflicts, while the Sparsity-MoE employs density-aware experts to adaptively refine proposals according to point sparsity. Integrated into strong two-stage baselines, our modules consistently improve performance on the KITTI dataset and the Waymo Open Dataset. Beyond empirical gains, our analysis reveals that Attribute-MoE and Sparsity-MoE solve largely independent problems, offering a practical “toolbox” for mitigating negative transfer in 3D object refinement and advancing adaptive, task-aware detector design. Code will be released at https://github.com/12e21/RefineMoE.