SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving

Shaoyuan Xie^1,*   Zihan Zhang^2,*   Jingxuan Wang¹   Jiashu Qu³   Xiaoqing Liang¹
Lingdong Kong⁴   Junchi Lu¹   Henrik I. Christensen²   Qi Alfred Chen¹

¹ UC Irvine   ·   ² UC San Diego   ·   ³ U. Cincinnati   ·   ⁴ NUS
^*Equal contribution

About

Vision-Language-Action (VLA) models widen the navigation interface of end-to-end driving from a fixed command set to free-form natural language. This open interface also exposes a new vulnerability surface: a navigation signal can be unsafe given the current scene, whether issued by an inattentive driver or on purpose by a malicious user.

We first build a benchmark suite that evaluates driving VLAs along three axes: navigation following (CARLA-F), navigation-scene conflict awareness (B2D-C and NavSim-C), and robustness to adversarial instructions (B2D-Adv). Existing policies fall short on both safe following and conflict awareness.

We then propose SafeDriveVLA, which decouples conflict reasoning from action generation:

• Driving mode. Every expert frame is relabeled with a discrete driving mode (<STRICT>, <CAUTIOUS> or <FALLBACK>) that states whether the trajectory executes, cautiously follows, or overrides the navigation signal. The model emits this mode token before its actions, so conflict awareness is supervised directly.
• Navigation-conditioned world model dreaming. A frozen latent world model (V-JEPA 2 encoder with an action-conditioned predictor) rolls the scene forward under the action anchor of the instructed meta-command. The VLA reads the dreamed world tokens and sees whether the maneuver is feasible before it commits.

Citation

If you find this work helpful, please consider citing:

@inproceedings{xie2026safedrivevla,
  title     = {SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving},
  author    = {Xie, Shaoyuan and Zhang, Zihan and Wang, Jingxuan and Qu, Jiashu and Liang, Xiaoqing and Kong, Lingdong and Lu, Junchi and Christensen, Henrik I. and Chen, Qi Alfred},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}

Updates

• [2026.10] The code of SafeDriveVLA (CoRL 2026) is released: world-model pre-training, VLA training, and closed-loop evaluation on Bench2Drive, CARLA-F, B2D-C and B2D-Adv.
• [2026.10] The CARLA-F and B2D-C datasets have been extended to Track 3 of the RoboWorld Challenge at NeurIPS 2026 RoboPAD workshop. See the track homepage and GitHub repo for more details

Outline

• Installation
• Getting Started
• Benchmarks
• Main Results
• TODO List
• License
• Acknowledgements

Installation

Please refer to install.md for the environment setup and data preparation.

Getting Started

Please refer to getting_started.md for training and closed-loop evaluation, including the released checkpoints.

Benchmarks

Benchmark	Scale	Navigation signal	Evaluates	Metrics
Bench2Drive	220 routes	command, target waypoint	closed-loop driving	DS, SR, Efficiency, Comfortness
CARLA-F	210 routes	command, target waypoint, language	navigation following	NCR per meta-command, Speed Error
B2D-C	150 routes	language	navigation-scene conflict	DS, SR, collisions, traffic violations, out-of-route
NavSim-C	300 frames	command	navigation-scene conflict (real-world sensors)	PDMS (coming soon)
B2D-Adv	15 routes	language	adversarial instructions	DS

• CARLA-F rebuilds each Bench2Drive route on the OpenDRIVE graph so that the ground-truth path performs the sampled meta-command (turn left / right, go straight, lane change left / right, lane follow), with target-speed instructions. Background traffic is disabled and all lights are green, so every navigation signal is safe and should be executed.
• B2D-C injects an unsafe natural-language instruction at the safety-critical event of each Bench2Drive scenario; the same instruction would be safe in a different scene.
• B2D-Adv extends B2D-C with adversarial instructions that are unsafe under any circumstance.

The route files are in benchmark/data, the generation toolkit in benchmark/generation, and the metrics in benchmark/metrics.

Main Results

SafeDriveVLA is trained on the full PDM-Lite data. The Bench2Drive results of SimLingo are taken from its paper.

Bench2Drive

Method	Nav.	Expert	DS ↑	SR (%) ↑	Efficiency ↑	Comfortness ↑
UniAD	CMD	Think2Drive	45.81	16.36	129.21	43.58
VAD	CMD	Think2Drive	42.35	15.00	157.94	46.01
ReCogDrive	CMD	Think2Drive	71.36	45.45	138.18	17.45
ORION	CMD	Think2Drive	77.74	54.62	151.48	17.38
MindDrive	CMD	Think2Drive	78.04	55.09	-	-
AutoVLA	Lan	PDM-Lite	78.84	57.73	146.93	39.33
DriveMoE	WP	Think2Drive	74.22	48.64	175.96	15.31
SimLingo	CMD	PDM-Lite	86.08	65.78	-	-
SimLingo	WP	PDM-Lite	85.07	67.27	259.23	33.67
SafeDriveVLA	CMD	PDM-Lite	87.29	68.18	263.51	31.34
SafeDriveVLA	WP	PDM-Lite	88.16	68.64	264.51	30.14

CARLA-F (Navigation Compliance Rate, %)

Model	Nav.	Speed Error ↓	Turn left	Turn right	Go straight	Left lane	Right lane	Lane follow	Avg. ↑
ORION	CMD	-	100.0	100.0	94.1	60.6	58.6	86.2	87.8
MindDrive	CMD	-	100.0	92.2	94.1	33.3	37.9	70.7	78.8
AutoMoT	CMD	-	98.2	100.0	97.6	21.2	27.6	89.7	82.0
SimLingo	Lan	1.44	83.6	80.4	45.9	0.0	0.0	63.8	52.4
SimLingo-IF	Lan	1.31	89.1	82.3	49.4	0.0	0.0	69.0	55.6
SimLingo-Safe	Lan	1.67	92.7	84.3	50.6	0.0	0.0	62.1	55.6
SafeDriveVLA	Lan	4.28	92.7	90.2	84.7	63.6	72.4	81.0	83.0

Avg. is weighted by the number of instructions per meta-command and excludes speed.

B2D-C

Model	DS ↑	SR (%) ↑	Collision ↓	Traffic Violation ↓	Out of Route ↓
SimLingo	72.8	36.7	66	60	18
SimLingo-IF	56.3	12.7	166	57	32
SimLingo-Safe	72.8	38.0	60	61	11
SafeDriveVLA	85.9	67.3	38	9	4

B2D-Adv (Driving Score on the 15 routes)

Model	without attack ↑	B2D-Adv ↑
SimLingo	95.7	76.0
SimLingo-IF	84.3	38.6
SimLingo-Safe	95.3	74.3
SafeDriveVLA	94.7	83.9

"without attack" is the Driving Score of the same 15 routes in the Bench2Drive evaluation.

TODO List

• World-model pre-training and VLA training code
• Closed-loop evaluation on Bench2Drive, CARLA-F, B2D-C and B2D-Adv
• Benchmark route files and generation toolkit
• Installation and data preparation guide
• Getting-started guide for training and evaluation
• Pretrained checkpoints
• NavSim-C benchmark

License

This project is released under the Apache License 2.0. Third-party code keeps its original license:

• world_model/vjepa2: V-JEPA 2, MIT (LICENSE).
• third_party/bench2drive: the Bench2Drive leaderboard and scenario runner are MIT; other Bench2Drive material is CC BY-NC-ND 4.0 (LICENSE).
• Parts of the data pipeline and the CARLA agent are adapted from SimLingo, Apache-2.0.

Acknowledgements

This work builds on SimLingo, Bench2Drive, CARLA, V-JEPA 2, InternVL, AutoVLA, and LMDrive. We thank the authors for releasing their code and data.