Selectively Answering Visual Questions
Published in Findings of the Association for Computational Linguistics: ACL 2024, 2024
Large multi-modal models (LMMs) often hallucinate or answer incorrectly with high confidence. We propose selective answering strategies for visual question answering (VQA), allowing models to withhold answers under high uncertainty. We evaluate calibration metrics, risk-coverage tradeoffs, and effective prompting techniques across standard VQA benchmarks.