RCF-TP: Radar-Camera Fusion with Temporal Priors for 3D Object Detection

Yakov Miron, Florian Drews, Florian Faion, Dotan Di Castro, Itzik Klein

Research output: Contribution to journalArticlepeer-review

Abstract

Sensor fusion is an important method for achieving robust perception systems in autonomous driving, Internet of things, and robotics. Most multi-modal 3D detection models assume the data is synchronized between the sensors and do not necessarily have real-time capabilities. We propose RCF-TP, an asynchronous, modular, real-time multi-modal architecture, to fuse cameras and radars for 3D object detection, with sensor fault mitigation and extreme weather conditions handling. Our dedicated feature extractors can be trained assuming either a regular or an irregular bird’s-eye-view grid or with different grid resolutions, such that the fusion module is agnostic to both. These extracted features are correlated to the other modality features or to another sensor of the same modality, and eventually a detection head that exploits rich multi-modal features could be applied at any time to produce bounding box predictions. Experimental results show the effectiveness of our fusion module. It improves detection performance for higher radar grid resolution, can operate under sensor faults without performance degradation, and improves pedestrian detection when our dataset combination strategy is implemented during training.

Original languageEnglish
Pages (from-to)1
Number of pages1
JournalIEEE Access
DOIs
StateAccepted/In press - 2024

Bibliographical note

Publisher Copyright:
Authors

Keywords

  • 3D Object Detection
  • Cameras
  • Feature extraction
  • Radar
  • Radar detection
  • Radar imaging
  • Self-Supervised Learning
  • Sensor Dropout
  • Sensor Fusion
  • Sensor phenomena and characterization
  • Three-dimensional displays

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering

Fingerprint

Dive into the research topics of 'RCF-TP: Radar-Camera Fusion with Temporal Priors for 3D Object Detection'. Together they form a unique fingerprint.

Cite this