Comparing user-assisted and automatic query translation

For the 2002 Cross-Language Evaluation Forum Interactive Track, the University of Maryland team focused on query formulation and reformulation. Twelve people performed a total of forty eight searches in the German document collection using English queries. Half of the searches were with user-assisted query translation, and half with fully automatic query translation. For the user-assisted query translation condition, participants were provided two types of cues about the meaning of each translation: a list of other terms with the same translation (potential synonyms), and a sentence in which the word was used in a translation-appropriate context. Four searchers performed the official iCLEF task, the other eight searched a smaller collection. Searchers performing the official task were able to make more accurate relevance judgments with user-assisted query translation for three of the four topics. We observed that the number of query iterations seems to vary systematically with topic, system, and collection, and we are analyzing query content and ranked retrieval measures to obtain further insight into these variations in search behavior.