EXECUTION-FEEDBACK-GUIDED CONTEXT SELECTION FOR LARGE LANGUAGE MODEL-BASED SOFTWARE TESTING AND DEBUGGING
Keywords:
Large language models, Automated debugging, Program repair, Fault localization, Context selection, Software testingAbstract
Large language models (LLMs) are now routinely used to generate unit tests, localize faults and repair programs, yet their effectiveness on real projects hinges on a step that is rarely engineered explicitly: deciding which code enters the prompt. Whole files exceed context budgets, lexical retrieval misses the helper functions in which faults hide, and stack traces are silent for assertion failures. This paper presents EFCS, an execution-feedback-guided context selection approach that turns the by-products of test execution—test outcomes, exception traces, statement-level spectra and dynamic call edges—into a relevance model over function-level code units. EFCS fuses spectrum and traceback evidence, assembles a token-budgeted context from execution-trimmed function bodies and interface skeletons of runtime callees, and re-selects the context across debugging iterations through an evidence memory and a cost-aware feedback-escalation policy. We evaluate EFCS on 897 single-fault and 210 two-fault instances seeded into seven real-world Python projects with 4,182 developer-written tests, counting tokens with the cl100k tokenizer of GPT-3.5 and GPT-4. On held-out faults, EFCS places the faulty statement inside a 1K-token context for 94.6% of the faults, compared with 38.2% for BM25 retrieval, 58.7% for stack-trace selection and 9.6% for focal-file prompting; under a 2K budget it delivers interface-complete contexts for 95.3% of the faults while using 10.9% fewer tokens than spectrum-based selection. In a replay of iterative debugging, EFCS resolves 97.2% of two-fault instances within five model calls, lowers the share of wasted calls to 8.6%, and its escalation policy reduces traced test executions by 64.8% at a comparable resolution rate.References
[1] Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021.
[2] Xia C S, Zhang L. Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv:2304.00385, 2023.
[3] Chen X, Lin M, Schärli N, et al. Teaching large language models to self-debug. arXiv:2304.05128, 2023.
[4] Xia C S, Wei Y, Zhang L. Automated program repair in the era of large pre-trained language models. In: Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023. DOI: 10.1109/ICSE48619.2023.00129.
[5] Jiang N, Liu K, Lutellier T, et al. Impact of code language models on automated program repair. In: Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023. DOI: 10.1109/ICSE48619.2023.00125.
[6] Fan Z, Gao X, Mirchev M, et al. Automated repair of programs from large language models. In: Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023. DOI: 10.1109/ICSE48619.2023.00128.
[7] Lemieux C, Inala J P, Lahiri S K, et al. CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models. In: Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023.
[8] Schäfer M, Nadi S, Eghbali A, et al. An empirical evaluation of using large language models for automated unit test generation. arXiv:2302.06527, 2023.
[9] Yuan Z, Lou Y, Liu M, et al. No more manual tests? Evaluating and improving ChatGPT for unit test generation. arXiv:2305.04207, 2023.
[10] Kang S, Yoon J, Yoo S. Large language models are few-shot testers: Exploring LLM-based general bug reproduction. In: Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023.
[11] Kang S, An G, Yoo S. A preliminary evaluation of LLM-based fault localization. arXiv:2308.05487, 2023.
[12] Hou X, Zhao Y, Liu Y, et al. Large language models for software engineering: A systematic literature review. arXiv:2308.10620, 2023.
[13] Wang J, Huang Y, Chen C, et al. Software testing with large language model: Survey, landscape, and vision. arXiv:2307.07221, 2023.
[14] Liu K, Koyuncu A, Bissyandé T F, et al. You cannot fix what you cannot find! An investigation of fault localization bias in benchmarking automated program repair systems. In: Proceedings of the 12th IEEE International Conference on Software Testing, Verification and Validation (ICST), 2019: 102-113.
[15] Jimenez C E, Yang J, Wettig A, et al. SWE-bench: Can language models resolve real-world GitHub issues? arXiv:2310.06770, 2023.
[16] Zhang F, Chen B, Zhang Y, et al. RepoCoder: Repository-level code completion through iterative retrieval and generation. arXiv:2303.12570, 2023.
[17] Ding Y, Wang Z, Ahmad W U, et al. CrossCodeEval: A diverse and multilingual benchmark for cross-file code completion. In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.
[18] Liu N F, Lin K, Hewitt J, et al. Lost in the middle: How language models use long contexts. arXiv:2307.03172, 2023.
[19] Robertson S, Zaragoza H. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 2009, 3(4): 333-389.
[20] Abreu R, Zoeteweij P, van Gemund A J C. On the accuracy of spectrum-based fault localization. In: Proceedings of Testing: Academic and Industrial Conference Practice and Research Techniques (TAICPART-MUTATION), 2007: 89-98.
[21] Madaan A, Tandon N, Gupta P, et al. Self-Refine: Iterative refinement with self-feedback. arXiv:2303.17651, 2023.
[22] Shinn N, Cassano F, Berman E, et al. Reflexion: Language agents with verbal reinforcement learning. arXiv:2303.11366, 2023.
[23] Wu Y, Li Z, Zhang J M, et al. ConDefects: A new dataset to address the data leakage concern for LLM-based fault localization and program repair. arXiv:2310.16253, 2023.
[24] Olausson T X, Inala J P, Wang C, et al. Is self-repair a silver bullet for code generation? arXiv:2306.09896, 2023.
[25] Just R, Jalali D, Ernst M D. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In: Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), 2014: 437-440.
[26] Just R, Jalali D, Inozemtseva L, et al. Are mutants a valid substitute for real faults in software testing? In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 2014: 654-665.