Robust scene understanding under varying environmental conditions is essential for safe autonomous driving. While cameras and LiDAR degrade in adverse weather and poor lighting, Frequency-Modulated Continuous-Wave (FMCW) spinning radar provides dependable 360° perception at extended ranges with minimal compute cost. However, real-time FMCW radar scene understanding is challenging due to complex signal returns, limited temporal resolution, and scarce annotated datasets. We introduce StreamRPS, a novel framework for streaming FMCW radar panoptic segmentation, comprising an automated label-generation pipeline via LiDAR scene completion and a real-time streaming network for semantic and panoptic segmentation. Our encoder-decoder streaming architecture comprises a radar-specific hierarchical Transformer encoder and task-specific decoders for semantic or panoptic prediction. Our encoder introduces a stem to capture range-dependent signatures, anisotropic patch embeddings to preserve angular resolution, and axial polar attention to separately model the periodic azimuth and bounded range axes. We further introduce azimuth roll rotation augmentation to simulate ego-vehicle rotation during training. Comprehensive experiments on the SAX and BOREAS datasets validate both our auto-labeling methodology and streaming segmentation network against established baselines. We outperform the strongest baseline by at least 1.4pp while processing sectors at over 400 FPS in a 4-sector streaming setting. We additionally provide manual annotations for 100 SAX radar scans, enabling independent evaluation of the entire framework.

StreamRPS is, to the best of our knowledge, the first streaming semantic and panoptic segmentation framework for spinning FMCW radar. It consists of two parts:

Cross-modal dense label generation. Annotating radar scans is hard, and projecting sparse LiDAR points yields very few labelled radar pixels. Instead, we run the pre-trained LiDAR scene completion network PaSCo to obtain dense voxel grids, collapse them to bird's-eye-view labels, project them to polar radar coordinates and synchronise them per azimuth. Panoptic instances are obtained from connected components.

Streaming radar segmentation. The polar radar image is split into sectors that are processed sequentially as the sensor rotates, so predictions are available before the full 360° scan is complete. Our hierarchical Transformer encoder is tailored to radar: a RadarStem captures range-dependent signatures, anisotropic patch embeddings preserve angular resolution, and axial polar attention models the periodic azimuth and bounded range axes separately. Azimuth roll rotation augmentation simulates ego-vehicle rotation during training. Task-specific decoders produce either semantic or panoptic predictions.

We evaluate on the SAX and Boreas datasets. StreamRPS outperforms the strongest baseline by at least 1.4pp while processing sectors at over 400 FPS in a 4-sector streaming setting (NVIDIA GeForce RTX 2080 Ti). Please see the paper for full quantitative results and ablations.

Santosh Parit, Julia Hindel, Rohit Mohan, Georgi Pramatarov, Paul Newman, Daniele De Martini, and Abhinav Valada
Real-Time Radar Semantic and Panoptic Scene Understanding
arXiv preprint arXiv:2610.26816

@article{parit26streamrps,
			title={Real-Time Radar Semantic and Panoptic Scene Understanding},
			author={Santosh Parit and Julia Hindel and Rohit Mohan and Georgi Pramatarov and Paul Newman and Daniele De Martini and Abhinav Valada},
			journal={arXiv preprint arXiv:2610.26816},
			volume={arXiv:2610.26816},
			year={2026}
		  }