Building an information retrieval test collection for spontaneous conversational speech

Douglas W. Oard; Dagobert Soergel; David Doermann; Xiaoli Huang; G. Craig Murray; Jianqiang Wang; Bhuvana Ramabhadran; Martin Franz; Samuel Gustman; James Mayfield; Liliya Kharevych; Stephanie Strassel

Building an information retrieval test collection for spontaneous conversational speech

Douglas W. Oard
, Dagobert Soergel
, David Doermann
, Xiaoli Huang
, G. Craig Murray
, Jianqiang Wang
, Bhuvana Ramabhadran
, Martin Franz
, Samuel Gustman
, James Mayfield
, Liliya Kharevych
, Stephanie Strassel

University of Maryland, College Park
IBM
Visual History Foundation
Johns Hopkins University Applied Physics Laboratory
California Institute of Technology
University of Pennsylvania

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

52 Scopus citations

Abstract

Test collections model use cases in ways that facilitate evaluation of information retrieval systems. This paper describes the use of search-guided relevance assessment to create a test collection for retrieval of spontaneous conversational speech. Approximately 10,000 thematically coherent segments were manually identified in 625 hours of oral history interviews with 246 individuals. Automatic speech recognition results, manually prepared summaries, controlled vocabulary indexing, and name authority control are available for every segment. Those features were leveraged by a team of four relevance assessors to identify topically relevant segments for 28 topics developed from actual user requests. Search-guided assessment yielded sufficient inter-annotator agreement to support formative evaluation during system development. Baseline results for ranked retrieval are presented to illustrate use of the collection.

Original language	English
Title of host publication	Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
Editors	K. Jarvelin, J. Allen, P. Bruza, M. Sanderson
Pages	41-48
Number of pages	8
State	Published - 2004
Event	Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - Sheffield, United Kingdom Duration: Jul 25 2004 → Jul 29 2004

Publication series

Name	Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Conference

Conference	Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
Country/Territory	United Kingdom
City	Sheffield
Period	07/25/04 → 07/29/04

Keywords

Automatic Speech Recognition
Oral History
Search-Guided Relevance Assessment

Cite this

Oard, D. W., Soergel, D., Doermann, D., Huang, X., Murray, G. C., Wang, J., Ramabhadran, B., Franz, M., Gustman, S., Mayfield, J., Kharevych, L., & Strassel, S. (2004). Building an information retrieval test collection for spontaneous conversational speech. In K. Jarvelin, J. Allen, P. Bruza, & M. Sanderson (Eds.), Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 41-48). (Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval).

Oard, Douglas W. ; Soergel, Dagobert ; Doermann, David et al. / Building an information retrieval test collection for spontaneous conversational speech. Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. editor / K. Jarvelin ; J. Allen ; P. Bruza ; M. Sanderson. 2004. pp. 41-48 (Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval).

@inproceedings{41031c213b7a4e4894c4063e9239336f,

title = "Building an information retrieval test collection for spontaneous conversational speech",

abstract = "Test collections model use cases in ways that facilitate evaluation of information retrieval systems. This paper describes the use of search-guided relevance assessment to create a test collection for retrieval of spontaneous conversational speech. Approximately 10,000 thematically coherent segments were manually identified in 625 hours of oral history interviews with 246 individuals. Automatic speech recognition results, manually prepared summaries, controlled vocabulary indexing, and name authority control are available for every segment. Those features were leveraged by a team of four relevance assessors to identify topically relevant segments for 28 topics developed from actual user requests. Search-guided assessment yielded sufficient inter-annotator agreement to support formative evaluation during system development. Baseline results for ranked retrieval are presented to illustrate use of the collection.",

keywords = "Automatic Speech Recognition, Oral History, Search-Guided Relevance Assessment",

author = "Oard, \{Douglas W.\} and Dagobert Soergel and David Doermann and Xiaoli Huang and Murray, \{G. Craig\} and Jianqiang Wang and Bhuvana Ramabhadran and Martin Franz and Samuel Gustman and James Mayfield and Liliya Kharevych and Stephanie Strassel",

year = "2004",

language = "English",

isbn = "1581138814",

series = "Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval",

pages = "41--48",

editor = "K. Jarvelin and J. Allen and P. Bruza and M. Sanderson",

booktitle = "Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval",

note = "Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval ; Conference date: 25-07-2004 Through 29-07-2004",

}

Oard, DW, Soergel, D , Doermann, D, Huang, X, Murray, GC, Wang, J, Ramabhadran, B, Franz, M, Gustman, S, Mayfield, J, Kharevych, L & Strassel, S 2004, Building an information retrieval test collection for spontaneous conversational speech. in K Jarvelin, J Allen, P Bruza & M Sanderson (eds), Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 41-48, Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, United Kingdom, 07/25/04.

Building an information retrieval test collection for spontaneous conversational speech. / Oard, Douglas W.; Soergel, Dagobert ; Doermann, David et al.
Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ed. / K. Jarvelin; J. Allen; P. Bruza; M. Sanderson. 2004. p. 41-48 (Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Building an information retrieval test collection for spontaneous conversational speech

AU - Oard, Douglas W.

AU - Soergel, Dagobert

AU - Doermann, David

AU - Huang, Xiaoli

AU - Murray, G. Craig

AU - Wang, Jianqiang

AU - Ramabhadran, Bhuvana

AU - Franz, Martin

AU - Gustman, Samuel

AU - Mayfield, James

AU - Kharevych, Liliya

AU - Strassel, Stephanie

PY - 2004

Y1 - 2004

N2 - Test collections model use cases in ways that facilitate evaluation of information retrieval systems. This paper describes the use of search-guided relevance assessment to create a test collection for retrieval of spontaneous conversational speech. Approximately 10,000 thematically coherent segments were manually identified in 625 hours of oral history interviews with 246 individuals. Automatic speech recognition results, manually prepared summaries, controlled vocabulary indexing, and name authority control are available for every segment. Those features were leveraged by a team of four relevance assessors to identify topically relevant segments for 28 topics developed from actual user requests. Search-guided assessment yielded sufficient inter-annotator agreement to support formative evaluation during system development. Baseline results for ranked retrieval are presented to illustrate use of the collection.

AB - Test collections model use cases in ways that facilitate evaluation of information retrieval systems. This paper describes the use of search-guided relevance assessment to create a test collection for retrieval of spontaneous conversational speech. Approximately 10,000 thematically coherent segments were manually identified in 625 hours of oral history interviews with 246 individuals. Automatic speech recognition results, manually prepared summaries, controlled vocabulary indexing, and name authority control are available for every segment. Those features were leveraged by a team of four relevance assessors to identify topically relevant segments for 28 topics developed from actual user requests. Search-guided assessment yielded sufficient inter-annotator agreement to support formative evaluation during system development. Baseline results for ranked retrieval are presented to illustrate use of the collection.

KW - Automatic Speech Recognition

KW - Oral History

KW - Search-Guided Relevance Assessment

UR - https://www.scopus.com/pages/publications/8644291021

M3 - Conference contribution

AN - SCOPUS:8644291021

SN - 1581138814

SN - 9781581138818

T3 - Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

SP - 41

EP - 48

BT - Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

A2 - Jarvelin, K.

A2 - Allen, J.

A2 - Bruza, P.

A2 - Sanderson, M.

T2 - Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Y2 - 25 July 2004 through 29 July 2004

ER -

Oard DW, Soergel D , Doermann D, Huang X, Murray GC, Wang J et al. Building an information retrieval test collection for spontaneous conversational speech. In Jarvelin K, Allen J, Bruza P, Sanderson M, editors, Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2004. p. 41-48. (Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval).

Building an information retrieval test collection for spontaneous conversational speech

Abstract

Publication series

Conference

Keywords

Other files and links

Fingerprint

Cite this