ROBUSTNESS-AWARE MODEL COMPRESSION FOR MULTIMODAL PERCEPTION UNDER CHALLENGING CONDITIONS

Authors

  • FengYuan Zhang (Corresponding Author) New York University, New York 10012, NY, USA.
  • JieSi Yang The University of Utah, Salt Lake City 84112, UT, USA.

Keywords:

Model compression, Multimodal perception, RGB-thermal segmentation, Robustness, Mixed-precision quantization, Structured pruning, Calibration

Abstract

Compressed multimodal perception models are usually validated on clean benchmarks, yet they are deployed where visibility is low, sensors are occluded and measurements are noisy. This paper studies RGB-thermal semantic segmentation on the MSRS road-scene benchmark and asks three questions: which predictions break after compression under challenging conditions, how compression, calibration and multimodal reliability cues can repair them, and how accuracy, hard-scenario performance and real latency trade off. We first show that 4-bit quantization and structured pruning remove 2.8-4.9 mIoU points on clean data but up to 7.4 points under unseen corruptions, and that the resulting harmful prediction flips concentrate on low-margin pixels, object boundaries and rare classes. We then propose RAMC, a robustness-aware multimodal compression framework that combines condition-aware Hessian sensitivity and minimax integer programming for mixed-precision post-training quantization with condition-aware Taylor pruning, paired clean-to-degraded distillation and fusion-gate consistency, and we analyze post-compression confidence calibration. At a tight budget of 24 BitOPs per MAC, RAMC improves night mIoU by 1.2 points and mean corrupted mIoU by 0.4 points over HAWQ while reducing harmful flips by 17%. With 75% of the hidden channels removed, it improves mean corrupted mIoU by 1.2 points over distillation; at 50% pruning it exceeds the uncompressed model in clean, night and mean corrupted mIoU while running 1.24-1.47 times faster in ONNX Runtime. Distilled students become underconfident, and a single temperature fitted on held-out degraded data nearly halves their calibration error on unseen corruptions.

References

[1] Ha Q, Watanabe K, Karasawa T, et al. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.

[2] Tang L, Yuan J, Zhang H, et al. PIAFusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 2022, 83-84: 79-92.

[3] Sun Y, Zuo W, Liu M. RTFNet: RGB-thermal fusion network for semantic segmentation of urban scenes. IEEE Robotics and Automation Letters, 2019, 4(3): 2576-2583.

[4] Zhang J, Liu H, Yang K, et al. CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(12): 14679-14694.

[5] Shin U, Lee K, Kweon I S, et al. Complementary random masking for RGB-thermal semantic segmentation. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2024.

[6] Hendrycks D, Dietterich T. Benchmarking neural network robustness to common corruptions and perturbations. In: Proceedings of the International Conference on Learning Representations (ICLR), 2019.

[7] Michaelis C, Mitzkus B, Geirhos R, et al. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv:1907.07484, 2019.

[8] Dong Y, Kang C, Zhang J, et al. Benchmarking robustness of 3D object detection to common corruptions in autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023: 1022-1032. DOI: 10.1109/CVPR52729.2023.00105.

[9] Kong L, Liu Y, Li X, et al. Robo3D: Towards robust and reliable 3D perception against corruptions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023: 19994-20006.

[10] Sakaridis C, Dai D, Van Gool L. Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 2018, 126(9): 973-992.

[11] Hooker S, Courville A, Clark G, et al. What do compressed deep neural networks forget? arXiv:1911.05248, 2019.

[12] Diffenderfer J, Bartoldson B, Chaganti S, et al. A winning hand: Compressing deep networks can improve out-of-distribution robustness. In: Advances in Neural Information Processing Systems (NeurIPS), 2021, 34.

[13] Jacob B, Kligys S, Chen B, et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018: 2704-2713.

[14] Dong Z, Yao Z, Arfeen D, et al. HAWQ-V2: Hessian aware trace-weighted quantization of neural networks. In: Advances in Neural Information Processing Systems (NeurIPS), 2020, 33.

[15] Liu J, Niu L, Yuan Z, et al. PD-Quant: Post-training quantization based on prediction difference metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023: 24427-24437.

[16] Li H, Kadav A, Durdanovic I, et al. Pruning filters for efficient ConvNets. In: Proceedings of the International Conference on Learning Representations (ICLR), 2017.

[17] Molchanov P, Mallya A, Tyree S, et al. Importance estimation for neural network pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019: 11264-11272.

[18] Fang G, Ma X, Song M, et al. DepGraph: Towards any structural pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023: 16091-16101.

[19] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.

[20] Sun S, Ren W, Li J, et al. Logit standardization in knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024: 15731-15740.

[21] Zhang Q, Wu H, Zhang C, et al. Provable dynamic fusion for low-quality multimodal data. In: Proceedings of the International Conference on Machine Learning (ICML), PMLR, 2023, 202: 41753-41769.

[22] Ma H, Zhang Q, Zhang C, et al. Calibrating multimodal learning. In: Proceedings of the International Conference on Machine Learning (ICML), PMLR, 2023, 202: 23429-23450.

[23] Guo C, Pleiss G, Sun Y, et al. On calibration of modern neural networks. In: Proceedings of the International Conference on Machine Learning (ICML), PMLR, 2017, 70: 1321-1330.

[24] Joy T, Pinto F, Lim S N, et al. Sample-dependent adaptive temperature scaling for improved calibration. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2023, 37(12): 14919-14926.

[25] Qin D, Leichner C, Delakis M, et al. MobileNetV4: Universal models for the mobile ecosystem. In: Proceedings of the European Conference on Computer Vision (ECCV), 2024, 15098: 78-96.

Downloads

Published

2024-12-31

Issue

Section

Research Article

DOI:

How to Cite

FengYuan Zhang, JieSi Yang. Robustness-Aware Model Compression For Multimodal Perception Under Challenging Conditions. AI and Data Science Journal. 2024, 1(1): 103-111. DOI: https://doi.org/10.61784/adsj4042.