In the rapidly evolving domain of autonomous driving, the accuracy and resilience of perception systems are paramount. Recent advancements, particularly in bird's eye view (BEV) representations and LiDAR sensing technologies, have significantly improved in-vehicle 3D scene perception.
Yet, the robustness of 3D scene perception methods under varied and challenging conditions — integral to ensuring safe operations — has been insufficiently assessed. To fill in the existing gap, we introduce The RoboDrive Challenge, seeking to push the frontiers of robust autonomous driving perception.
RoboDrive is one of the first benchmarks that targeted probing the Out-of-Distribution (OoD) robustness of state-of-the-art autonomous driving perception models, centered around two mainstream topics: common corruptions and sensor failures.
Topic One: Corruptions
There are eighteen real-world corruption types in total, ranging from three perspectives:
- Weather and lighting conditions, such as bright, low-light, foggy, and snowy conditions.
- Movement and acquisition failures, such as potential blurs caused by vehicle motions.
- Data processing issues, such as noises and quantizations happen due to hardware malfunctions.
Topic Two: Sensor Failures
Additionally, we aim to probe the 3D scene perception robustness under camera and LiDAR sensor failures:
- Loss of certain camera frames during the driving system sensing process.
- Loss of one or more camera views during the driving system sensing process.
- Loss of the roof-top LiDAR view during the driving system sensing process.
There are five tracks in this RoboDrive Challenge, with emphasis on the following 3D scene perception tasks:
- Track 1: Robust BEV Detection.
- Track 2: Robust Map Segmentation.
- Track 3: Robust Occupancy Prediction.
- Track 4: Robust Depth Estimation.
- Track 5: Robust Multi-Modal BEV Detection.
The RoboDrive Challenge is affiliated with the 41st IEEE Conference on Robotics and Automation (ICRA 2024).
ICRA is IEEE Robotics and Automation Society's flagship conference. ICRA 2024 will be held from May 13th to 17th, 2024, in Yokohama, Japan.
The ICRA competitions provide a unique venue for state-of-the-art technical demonstrations from research labs throughout academia and industry. For additional details, kindly refer to the ICRA 2024 website.
Register for your team by filling in this Google Form.
Download the data from the competition toolkit.
Shortlisted teams are invited to participate in the next phase.
Don't forget to include the code link in your submissions.
Associated with the ICRA 2024 conference formality.
In this competition, all participants are expected to adopt the official nuScenes dataset for model training and our robustness probing sets for model evaluation. Additional data sources are NOT allowed in this competition.
Kindly refer to DATA_PREPARE.md for the details to prepare the training and evaluation data.
For all five tracks in this RoboDrive competition, the participants are expected to submit their predictions to the EvalAI servers for model evaluation.
To facilitate the training and evaluation, we have provided detailed instructions and a baseline model for each track. Kindly refer to GET_STARTED.md for additional details.
Please refer to Frequently Asked Questions for more detailed rules and conditions of this competition.
This competition is supported by HUAWEI Noah's Ark Lab.
This competition is hosted by Shanghai AI Laboratory.
This competition is made freely available to academic and non-academic entities for non-commercial purposes such as academic research, teaching, scientific publications, or personal experimentation. Permission is granted to use the data given that you agree:
1. That the data in this competition comes “AS IS”, without express or implied warranty. Although every effort has been made to ensure accuracy, we do not accept any responsibility for errors or omissions.
2. That you may not use the data in this competition or any derivative work for commercial purposes as, for example, licensing or selling the data, or using the data with a purpose to procure a commercial gain.
3. That you include a reference to RoboDrive (including the benchmark data and the specially generated data for academic challenges) in any work that makes use of the benchmark. For research papers, please cite our preferred publications as listed on our webpage.
To ensure a fair comparison among all participants, we require:
1. All participants must follow the exact same data configuration when training and evaluating their algorithms. Please do not use any public or private datasets other than those specified for model training.
2. The theme of this competition is to probe the out-of-distribution robustness of autonomous driving perception models. Theorefore, any use of the corruption and sensor failure types designed in this benchmark is strictly prohibited, including any atomic operation that is comprising any one of the mentioned corruptions.
3. To ensure the above two rules are followed, each participant is requested to submit the code with reproducible results before the final result is announced; the code is for examination purposes only and we will manually verify the training and evaluation of each participant's model.