Detecting Correct Answers to Open Questions and Its Impact on Language Models’ Confidence Scores
Published in LatinX in AI (LXAI) Research Workshop at NAACL 2024, Mexico City, 2024
Assessing confidence and calibration in open-ended generative QA is challenging due to lexical variation and surface-level mismatches. This paper investigates automatic equivalence detection between generated answers and reference answers, analyzing how evaluation fidelity impacts confidence scoring and calibration metrics.