A Survey of Calibration Process for Black-Box LLMs

In recent years, Confidence Estimation and Calibration have frequently been discussed together, as the estimation of confidence is often influenced by the uncertainty in the model or data, and calibration methods help the model recognize its own knowledge limitations (Jiang et al., 2021; Shrivastava et al., 2023). Calibration allows LLMs to adjust their confidence to more accurately reflect the quality of their outputs (Kuhn et al., 2023; Duan et al., 2023). For example, in the context of diagnosing rare diseases, LLMs may estimate a 95% confidence score to an incorrect response, while the accurate confidence, due to a lack of domain knowledge, should be closer to 40%. Calibration can identify this discrepancy and adjust the model’s confidence to more accurately reflect

2.1 Confidence Estimation

For black-box LLMs, Confidence Estimation depends exclusively on input-output information, leading researchers to develop methods that extract reference information from model outputs through designed interactions and multiple queries. While some studies term this process ”Confidence Elicitation” (Xiong et al., 2023; Huang et al., 2024b; Shrivastava et al., 2023), we maintain the term ”Confidence Estimation”.

Two main approaches exist: Consistency and Self-Reflection, both post-processing techniques that enable near-well-calibrated estimation without accessing model parameters. These methods can be employed individually or integrated into hybrid approaches, sometimes leveraging third-party proxy models.