Tangent-V: Math formula image search using line-of-sight graphs

Kenny Davila; Ritvik Joshi; Srirangaraj Setlur; Venu Govindaraju; Richard Zanibbi

doi:10.1007/978-3-030-15712-8_44

Tangent-V: Math formula image search using line-of-sight graphs

Kenny Davila
, Ritvik Joshi
, Srirangaraj Setlur
, Venu Govindaraju
, Richard Zanibbi

SUNY Buffalo
Rochester Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

We present a visual search engine for graphics such as math, chemical diagrams, and figures. Graphics are represented using Line-of-Sight (LOS) graphs, with symbols connected only when they can ‘see’ each other along an unobstructed line. Symbol identities may be provided (e.g., in PDF) or taken from Optical Character Recognition applied to images. Graphics are indexed by pairs of symbols that ‘see’ each other using their labels, spatial displacement, and size ratio. Retrieval has two layers: the first matches query symbol pairs in an inverted index, while the second aligns candidates with the query and scores the resulting matches using the identity and relative position of symbols. For PDFs, we also introduce a new tool that quickly extracts characters and their locations. We have applied our model to the NTCIR-12 Wikipedia Formula Browsing Task, and found that the method can locate relevant matches without unification of symbols or using a math expression grammar. In the future, one might index LOS graphs for entire pages and search for text and graphics. Our source code has been made publicly available.

Original language	English
Title of host publication	Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings
Editors	Djoerd Hiemstra, Philipp Mayr, Norbert Fuhr, Claudia Hauff, Benno Stein, Leif Azzopardi
Publisher	Springer Verlag
Pages	681-695
Number of pages	15
ISBN (Print)	9783030157111
DOIs	https://doi.org/10.1007/978-3-030-15712-8_44
State	Published - 2019
Event	41st European Conference on Information Retrieval, ECIR 2019 - Cologne, Germany Duration: Apr 14 2019 → Apr 18 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11437 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	41st European Conference on Information Retrieval, ECIR 2019
Country/Territory	Germany
City	Cologne
Period	04/14/19 → 04/18/19

Keywords

Graphics search
Image search
Mathematical Information Retrieval (MIR)
PDF symbol extraction

Access to Document

10.1007/978-3-030-15712-8_44

Cite this

Davila, K., Joshi, R., Setlur, S., Govindaraju, V., & Zanibbi, R. (2019). Tangent-V: Math formula image search using line-of-sight graphs. In D. Hiemstra, P. Mayr, N. Fuhr, C. Hauff, B. Stein, & L. Azzopardi (Eds.), Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings (pp. 681-695). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11437 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-15712-8_44

Davila, Kenny ; Joshi, Ritvik ; Setlur, Srirangaraj et al. / Tangent-V : Math formula image search using line-of-sight graphs. Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings. editor / Djoerd Hiemstra ; Philipp Mayr ; Norbert Fuhr ; Claudia Hauff ; Benno Stein ; Leif Azzopardi. Springer Verlag, 2019. pp. 681-695 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{76935a69aa154eba94e4af72b399f519,

title = "Tangent-V: Math formula image search using line-of-sight graphs",

abstract = "We present a visual search engine for graphics such as math, chemical diagrams, and figures. Graphics are represented using Line-of-Sight (LOS) graphs, with symbols connected only when they can {\textquoteleft}see{\textquoteright} each other along an unobstructed line. Symbol identities may be provided (e.g., in PDF) or taken from Optical Character Recognition applied to images. Graphics are indexed by pairs of symbols that {\textquoteleft}see{\textquoteright} each other using their labels, spatial displacement, and size ratio. Retrieval has two layers: the first matches query symbol pairs in an inverted index, while the second aligns candidates with the query and scores the resulting matches using the identity and relative position of symbols. For PDFs, we also introduce a new tool that quickly extracts characters and their locations. We have applied our model to the NTCIR-12 Wikipedia Formula Browsing Task, and found that the method can locate relevant matches without unification of symbols or using a math expression grammar. In the future, one might index LOS graphs for entire pages and search for text and graphics. Our source code has been made publicly available.",

keywords = "Graphics search, Image search, Mathematical Information Retrieval (MIR), PDF symbol extraction",

author = "Kenny Davila and Ritvik Joshi and Srirangaraj Setlur and Venu Govindaraju and Richard Zanibbi",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2019.; 41st European Conference on Information Retrieval, ECIR 2019 ; Conference date: 14-04-2019 Through 18-04-2019",

year = "2019",

doi = "10.1007/978-3-030-15712-8\_44",

language = "English",

isbn = "9783030157111",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "681--695",

editor = "Djoerd Hiemstra and Philipp Mayr and Norbert Fuhr and Claudia Hauff and Benno Stein and Leif Azzopardi",

booktitle = "Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings",

address = "Germany",

}

Davila, K, Joshi, R, Setlur, S , Govindaraju, V & Zanibbi, R 2019, Tangent-V: Math formula image search using line-of-sight graphs. in D Hiemstra, P Mayr, N Fuhr, C Hauff, B Stein & L Azzopardi (eds), Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11437 LNCS, Springer Verlag, pp. 681-695, 41st European Conference on Information Retrieval, ECIR 2019, Cologne, Germany, 04/14/19. https://doi.org/10.1007/978-3-030-15712-8_44

Tangent-V: Math formula image search using line-of-sight graphs. / Davila, Kenny; Joshi, Ritvik; Setlur, Srirangaraj et al.
Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings. ed. / Djoerd Hiemstra; Philipp Mayr; Norbert Fuhr; Claudia Hauff; Benno Stein; Leif Azzopardi. Springer Verlag, 2019. p. 681-695 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11437 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Tangent-V

T2 - 41st European Conference on Information Retrieval, ECIR 2019

AU - Davila, Kenny

AU - Joshi, Ritvik

AU - Setlur, Srirangaraj

AU - Govindaraju, Venu

AU - Zanibbi, Richard

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2019.

PY - 2019

Y1 - 2019

N2 - We present a visual search engine for graphics such as math, chemical diagrams, and figures. Graphics are represented using Line-of-Sight (LOS) graphs, with symbols connected only when they can ‘see’ each other along an unobstructed line. Symbol identities may be provided (e.g., in PDF) or taken from Optical Character Recognition applied to images. Graphics are indexed by pairs of symbols that ‘see’ each other using their labels, spatial displacement, and size ratio. Retrieval has two layers: the first matches query symbol pairs in an inverted index, while the second aligns candidates with the query and scores the resulting matches using the identity and relative position of symbols. For PDFs, we also introduce a new tool that quickly extracts characters and their locations. We have applied our model to the NTCIR-12 Wikipedia Formula Browsing Task, and found that the method can locate relevant matches without unification of symbols or using a math expression grammar. In the future, one might index LOS graphs for entire pages and search for text and graphics. Our source code has been made publicly available.

AB - We present a visual search engine for graphics such as math, chemical diagrams, and figures. Graphics are represented using Line-of-Sight (LOS) graphs, with symbols connected only when they can ‘see’ each other along an unobstructed line. Symbol identities may be provided (e.g., in PDF) or taken from Optical Character Recognition applied to images. Graphics are indexed by pairs of symbols that ‘see’ each other using their labels, spatial displacement, and size ratio. Retrieval has two layers: the first matches query symbol pairs in an inverted index, while the second aligns candidates with the query and scores the resulting matches using the identity and relative position of symbols. For PDFs, we also introduce a new tool that quickly extracts characters and their locations. We have applied our model to the NTCIR-12 Wikipedia Formula Browsing Task, and found that the method can locate relevant matches without unification of symbols or using a math expression grammar. In the future, one might index LOS graphs for entire pages and search for text and graphics. Our source code has been made publicly available.

KW - Graphics search

KW - Image search

KW - Mathematical Information Retrieval (MIR)

KW - PDF symbol extraction

UR - https://www.scopus.com/pages/publications/85064875388

U2 - 10.1007/978-3-030-15712-8_44

DO - 10.1007/978-3-030-15712-8_44

M3 - Conference contribution

AN - SCOPUS:85064875388

SN - 9783030157111

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 681

EP - 695

BT - Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings

A2 - Hiemstra, Djoerd

A2 - Mayr, Philipp

A2 - Fuhr, Norbert

A2 - Hauff, Claudia

A2 - Stein, Benno

A2 - Azzopardi, Leif

PB - Springer Verlag

Y2 - 14 April 2019 through 18 April 2019

ER -

Davila K, Joshi R, Setlur S , Govindaraju V, Zanibbi R. Tangent-V: Math formula image search using line-of-sight graphs. In Hiemstra D, Mayr P, Fuhr N, Hauff C, Stein B, Azzopardi L, editors, Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings. Springer Verlag. 2019. p. 681-695. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-15712-8_44

Tangent-V: Math formula image search using line-of-sight graphs

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this