A model for multimodal information retrieval

Finding useful information from large multimodal document collections such as the WWW without encountering numerous false positives poses a challenge to multimodal information retrieval systems (MMIR). A general model for multimodal information retrieval is proposed by which a user's information need is expressed through composite, multimodal queries, and the most appropriate weighted combination of indexing techniques is determined by a machine learning approach in order to best satisfy the information need. The focus is on improving precision and recall in a MMIR system by optimally combining text and image similarity. Experiments are presented which demonstrate the utility of individual indexing systems in improving overall average precision.