Abstract
Object hallucination in large vision-language models (LVLMs) remains a major obstacle to trustworthy generation. An intuitive mitigation strategy is to suppress components of hidden representations associated with hallucinations. However, since these components may also contain useful visual-semantic information, directly suppressing them can impair the model’s multimodal capabilities. We therefore propose ResOT, a training-free inference-time method that repairs representations through localized distribution alignment. Specifically, ResOT confines the intervention to a low-dimensional residual subspace that captures hallucination-associated structures not explained by the dominant faithful subspace. Within this subspace, ResOT uses Gaussian optimal transport (OT) to define repair targets that align the hallucinated distribution with the faithful one while minimizing changes to the original representations. During inference, ResOT adaptively controls how far each token state moves toward its OT target. Across multiple benchmarks and three representative LVLMs, ResOT achieves the best overall performance in hallucination mitigation and captioning among the compared methods. It also improves multimodal performance over the corresponding vanilla LVLMs.