Judge Before Answer:
Can MLLM Discern the False Premise in Question?

Jidong Li, Lingyong Fang, Haodong Zhao, Sufeng Duan, Gongshen Liu

📋 Overview

The original code of constructing JBA dataset is given in this repository, which is an evaluation set of false premise problems for MLLM.

🤗Find our dataset on Huggingface

🚀 Quick Start

Find our JBA dataset in dataset/Judge_Before_Answer.json or on Huggingface, image ids are from Visual Genome. Also, you can run main.py to constrcut your own JBA dataset.

Run test.py to generate test results for MLLM.

Run evaluate.py to evaluate the results and get metrics.