ACCURACY-AWARE MODEL COMPRESSION FOR REAL-TIME VISUAL PERCEPTION UNDER RESOURCE CONSTRAINTS
Keywords:
Model compression, Mixed-precision quantization, Structured pruning, Real-time object detection, Edge inference, Latency modelingAbstract
Deploying real-time visual perception models on CPU-only edge devices requires compression that respects measured latency while preserving task accuracy. Uniform quantization and pruning ignore the uneven sensitivity of network components, and existing allocation methods optimize a single compression dimension with additive per-layer cost models that miss the precision-switch overheads of CPU inference runtimes. This paper presents ACORN, a post-training, accuracy-aware compression framework that jointly allocates channel-pruning ratios and numerical precision to the components of a pretrained detector under explicit latency and memory budgets. ACORN estimates the accuracy cost of every option as a perturbation of the uniformly quantized deployment model with a confidence-weighted discrepancy restricted to task-relevant categories, predicts latency from per-node profiles of the runtime-optimized graph augmented with pairwise quantize/dequantize costs, prunes channels without training through ridge least-squares reconstruction, and solves the allocation exactly as an integer linear program. On YOLOv8n deployed for PASCAL VOC detection on a single CPU core with ONNX Runtime, ACORN reaches 63.47% mAP@[.5:.95] at the latency of uniform INT8 quantization (62.77%), within 0.07 points of the FP32 model while running 2.28× faster with a 3.6× smaller model. At a 2.51× speedup it retains 63.17%, whereas latency-aware knapsack pruning and uniform DepGraph pruning at similar latency fall to 52.54% and 30.33%. The complete allocation uses 32 unlabeled images and about ten minutes of single-core computation.References
[1] Jocher G, Chaurasia A, Qiu J. Ultralytics YOLOv8. Version 8.0.0, 2023. https://github.com/ultralytics/ultralytics.
[2] Wang C Y, Bochkovskiy A, Liao H Y M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023: 7464-7475.
[3] Li C, Wang Z, Yang C, et al. YOLOv6: A single-stage object detection framework for industrial applications. arXiv:2209.02976, 2022.
[4] Lv W, Xu S, Zhao Y, et al. DETRs beat YOLOs on real-time object detection. arXiv:2304.08069, 2023.
[5] Gholami A, Kim S, Dong Z, et al. A survey of quantization methods for efficient neural network inference. In: Low-Power Computer Vision. Boca Raton, FL, USA: CRC Press, 2022: 291-326.
[6] Jacob B, Kligys S, Chen B, et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018: 2704-2713.
[7] Krishnamoorthi R. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv:1806.08342, 2018.
[8] Nagel M, Fournarakis M, Amjad R A, et al. A white paper on neural network quantization. arXiv:2106.08295, 2021.
[9] Li H, Kadav A, Durdanovic I, et al. Pruning filters for efficient ConvNets. In: Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[10] Fang G, Ma X, Song M, et al. DepGraph: Towards any structural pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023: 16091-16101.
[11] Han S, Mao H, Dally W J. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In: Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[12] Wang K, Liu Z, Lin Y, et al. HAQ: Hardware-aware automated quantization with mixed precision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019: 8612-8620.
[13] Dong Z, Yao Z, Gholami A, et al. HAWQ: Hessian aware quantization of neural networks with mixed-precision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019: 293-302.
[14] Yao Z, Dong Z, Zheng Z, et al. HAWQ-V3: Dyadic neural network quantization. In: Proceedings of the International Conference on Machine Learning (ICML), 2021: 11875-11886.
[15] Cai Y, Yao Z, Dong Z, et al. ZeroQ: A novel zero shot quantization framework. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 13169-13178.
[16] Shen M, Yin H, Molchanov P, et al. Structural pruning via latency-saliency knapsack. In: Advances in Neural Information Processing Systems (NeurIPS), 2022.
[17] Frantar E, Alistarh D. SPDY: Accurate pruning with speedup guarantees. In: Proceedings of the International Conference on Machine Learning (ICML), 2022: 6726-6743.
[18] Frantar E, Singh S P, Alistarh D. Optimal brain compression: A framework for accurate post-training quantization and pruning. In: Advances in Neural Information Processing Systems (NeurIPS), 2022.
[19] Nagel M, Amjad R A, van Baalen M, et al. Up or down? Adaptive rounding for post-training quantization. In: Proceedings of the International Conference on Machine Learning (ICML), 2020: 7197-7206.
[20] Li Y, Gong R, Tan X, et al. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In: Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[21] Wei X, Gong R, Li Y, et al. QDrop: Randomly dropping quantization for extremely low-bit post-training quantization. In: Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[22] Liu J, Niu L, Yuan Z, et al. PD-Quant: Post-training quantization based on prediction difference metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023: 24427-24437.
[23] He Y, Zhang X, Sun J. Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017: 1389-1397.
[24] Luo J H, Wu J, Lin W. ThiNet: A filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017: 5058-5066.
[25] He Y, Lin J, Liu Z, et al. AMC: AutoML for model compression and acceleration on mobile devices. In: Proceedings of the European Conference on Computer Vision (ECCV), 2018: 784-800.
[26] Wang T, Wang K, Cai H, et al. APQ: Joint search for network architecture, pruning and quantization policy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 2078-2087.
[27] Wang Y, Lu Y, Blankevoort T. Differentiable joint pruning and quantization for hardware efficiency. In: Proceedings of the European Conference on Computer Vision (ECCV), 2020: 259-277.
[28] Dai X, Zhang P, Wu B, et al. ChamNet: Towards efficient network design through platform-aware model adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019: 11398-11407.
[29] Everingham M, Van Gool L, Williams C K I, et al. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 2010, 88(2): 303-338.
[30] Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common objects in context. In: Proceedings of the European Conference on Computer Vision (ECCV), 2014: 740-755.