UNCERTAINTY-AWARE LLM-AS-A-JUDGE EVALUATION FOR RELIABLE SEARCH AND RECOMMENDATION SYSTEMS: CALIBRATED CONFIDENCE AND IMPACT-AWARE SELECTIVE HUMAN REVIEW
Keywords:
LLM-as-a-judge, Relevance assessment, Uncertainty calibration, Selective human review, Test collections, Search and recommendation evaluationAbstract
Large language models (LLMs) are increasingly used as judges that produce relevance labels for evaluating search and recommendation systems, yet their graded judgments fluctuate with the prompt, the model and the assessment criterion, disagree with human assessors in a topic-dependent way, and come without trustworthy confidence. This paper presents CARE (Consistency-Aware calibration with Risk- and Evaluation-impact-driven review), an uncertainty-aware framework that turns a panel of LLM judgments into calibrated relevance distributions and spends a limited human budget where it matters most for evaluation outcomes. CARE encodes multi-dimensional consistency signals across repeated runs, prompts, LLMs, criteria and fine-tuning; calibrates them with a hierarchical multinomial model whose topic-specific grade biases receive an empirical-Bayes prior and a Laplace posterior; and routes pairs to assessors by expected ordinal error weighted by their impact on system comparisons, complemented by online posterior updates, soft expected-gain qrels and risk-targeted auto-acceptance. On the TREC DL 2023-based LLMJudge benchmark with 33 real LLM label sets (GPT-4o, Llama-3-8B/70B and FLAN-T5 judges using zero-shot, chain-of-thought, few-shot, criteria-decomposed and fine-tuned pipelines), changing only the prompt flips 22–46% of graded labels, whereas repeated sampling flips fewer than 0.3%. With ten human labels per topic, CARE raises Cohen’s κ with TREC assessors from 0.273 (majority vote) to 0.412 and reduces the expected calibration error from 0.150 to 0.036. Reviewing 10% of the pairs selected by CARE lifts the Kendall’s τ between system rankings from 0.642 (random review) to 0.895, and τ≥0.85 is reached with 7.7% of the judging effort.References
[1] Thomas P, Spielman S, Craswell N, et al. Large language models can accurately predict searcher preferences. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2024. DOI: 10.1145/3626772.3657707.
[2] Faggioli G, Dietz L, Clarke C L A, et al. Perspectives on large language models for relevance judgment. In: Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR), 2023: 39-50.
[3] MacAvaney S, Soldaini L. One-shot labeling for automatic relevance estimation. In: Proceedings of the 46th International ACM SIGIR Conference, 2023: 2230-2235.
[4] Upadhyay S, Pradeep R, Thakur N, et al. UMBRELA: UMbrela is the (open-source reproduction of the) Bing RELevance assessor. arXiv:2406.06519, 2024.
[5] Mehrdad N, Mohapatra H, Bagdouri M, et al. Large language models for relevance judgment in product search. In: Proceedings of the LLM4Eval Workshop at SIGIR, 2024. arXiv:2406.00247.
[6] Zhang X, Li Y, Wang J, et al. Large language models as evaluators for recommendation explanations. arXiv:2406.03248, 2024.
[7] Rahmani H A, Yilmaz E, Craswell N, et al. LLMJudge: LLMs for relevance judgments. In: Proceedings of the 1st Workshop on Large Language Models for Evaluation in IR (LLM4Eval) at SIGIR, 2024.
[8] Rahmani H A, Siro C, Aliannejadi M, et al. LLM4Eval: Large language model for evaluation in IR. In: Proceedings of the 47th International ACM SIGIR Conference, 2024: 3040-3043.
[9] Craswell N, Mitra B, Yilmaz E, et al. Overview of the TREC 2023 deep learning track. In: Proceedings of the 32nd Text REtrieval Conference (TREC 2023), NIST Special Publication 1328, 2023.
[10] Sclar M, Choi Y, Tsvetkov Y, et al. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In: Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[11] Wang P, Li L, Chen L, et al. Large language models are not fair evaluators. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024: 9440-9450.
[12] Alaofi M, Thomas P, Scholer F, et al. LLMs can be fooled into labelling a document as relevant. In: Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in IR in the Asia Pacific Region (SIGIR-AP), 2024: 32-41.
[13] Clarke C L A, Dietz L. LLM-based relevance assessment still can’t replace human relevance assessment. arXiv:2412.17156, 2024.
[14] Soboroff I. Don’t use LLMs to make relevance judgments. arXiv:2409.15133, 2024.
[15] Rahmani H A, Yilmaz E, Craswell N, et al. JudgeBlender: Ensembling judgments for automatic relevance assessment. arXiv:2412.13268, 2024.
[16] Verga P, Hofstätter S, Althammer S, et al. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv:2404.18796, 2024.
[17] Jung J, Brahman F, Choi Y. Trust or escalate: LLM judges with provable guarantees for human agreement. arXiv:2407.18370, 2024.
[18] Angelopoulos A N, Bates S, Fannjiang C, et al. Prediction-powered inference. Science, 2023, 382(6671): 669-674.
[19] Saad-Falcon J, Khattab O, Potts C, et al. ARES: An automated evaluation framework for retrieval-augmented generation systems. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024.
[20] Farzi N, Dietz L. Pencils down! Automatic rubric-based evaluation of retrieve/generate systems. In: Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR), 2024: 175-184.
[21] Meng C, Arabzadeh N, Askari A, et al. Query performance prediction using relevance judgments generated by large language models. arXiv:2404.01012, 2024.
[22] Zheng L, Chiang W L, Sheng Y, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In: Advances in Neural Information Processing Systems 36 (NeurIPS), Datasets and Benchmarks Track, 2023.
[23] Xiong M, Hu Z, Lu X, et al. Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In: Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[24] Guo C, Pleiss G, Sun Y, et al. On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML), 2017: 1321-1330.
[25] Dawid A P, Skene A M. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 1979, 28(1): 20-28.
[26] Kim H C, Ghahramani Z. Bayesian classifier combination. In: Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS), 2012: 619-627.
[27] Saerens M, Latinne P, Decaestecker C. Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Computation, 2002, 14(1): 21-41.
[28] Carterette B, Allan J, Sitaraman R. Minimal test collections for retrieval evaluation. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2006: 268-275.
[29] Yilmaz E, Aslam J A, Robertson S. A new rank correlation coefficient for information retrieval. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2008: 587-594.
[30] Geifman Y, El-Yaniv R. Selective classification for deep neural networks. In: Advances in Neural Information Processing Systems 30 (NeurIPS), 2017.