Constraints on Future Analysis Metadata Systems in High Energy Physics

T. J. Khoo; A. Reinsvold Hall; N. Skidmore; S. Alderweireldt; J. Anders; C. Burr; W. Buttinger; P. David; L. Gouskos; L. Gray; S. Hageböck; A. Krasznahorkay; P. Laycock; A. Lister; Z. Marshall; A. B. Meyer; T. Novak; S. Rappoccio; M. Ritter; E. Rodrigues; J. Rumsevicius; L. Sexton-Kennedy; N. Smith; G. A. Stewart; S. Wertz

doi:10.1007/s41781-022-00086-2

Constraints on Future Analysis Metadata Systems in High Energy Physics

T. J. Khoo
, A. Reinsvold Hall
, N. Skidmore
, S. Alderweireldt
, J. Anders
, C. Burr
, W. Buttinger
, P. David
, L. Gouskos
, L. Gray
, S. Hageböck
, A. Krasznahorkay
, P. Laycock
, A. Lister
, Z. Marshall
, A. B. Meyer
, T. Novak
, S. Rappoccio
, M. Ritter
, E. Rodrigues

J. Rumsevicius, L. Sexton-Kennedy, N. Smith, G. A. Stewart, S. Wertz

Physics

Humboldt University of Berlin
United States Naval Academy
University of Manchester
University of Edinburgh
University of Bern
CERN
Rutherford Appleton Laboratory
Université catholique de Louvain
Fermi National Accelerator Laboratory
United States Department of Energy
University of British Columbia
Lawrence Berkeley National Laboratory
German Electron Synchrotron
Ludwig Maximilian University of Munich
University of Liverpool

Research output: Contribution to journal › Article › peer-review

Abstract

In high energy physics (HEP), analysis metadata comes in many forms—from theoretical cross-sections, to calibration corrections, to details about file processing. Correctly applying metadata is a crucial and often time-consuming step in an analysis, but designing analysis metadata systems has historically received little direct attention. Among other considerations, an ideal metadata tool should be easy to use by new analysers, should scale to large data volumes and diverse processing paradigms, and should enable future analysis reinterpretation. This document, which is the product of community discussions organised by the HEP Software Foundation, categorises types of metadata by scope and format and gives examples of current metadata solutions. Important design considerations for metadata systems, including sociological factors, analysis preservation efforts, and technical factors, are discussed. A list of best practices and technical requirements for future analysis metadata systems is presented. These best practices could guide the development of a future cross-experimental effort for analysis metadata tools.

Original language	English
Article number	13
Journal	Computing and Software for Big Science
Volume	6
Issue number	1
DOIs	https://doi.org/10.1007/s41781-022-00086-2
State	Published - Dec 2022

Keywords

Data analysis
Databases
High energy physics
Metadata
Scientific computing

Access to Document

10.1007/s41781-022-00086-2

Cite this

Khoo, T. J., Reinsvold Hall, A., Skidmore, N., Alderweireldt, S., Anders, J., Burr, C., Buttinger, W., David, P., Gouskos, L., Gray, L., Hageböck, S., Krasznahorkay, A., Laycock, P., Lister, A., Marshall, Z., Meyer, A. B., Novak, T., Rappoccio, S., Ritter, M., ... Wertz, S. (2022). Constraints on Future Analysis Metadata Systems in High Energy Physics. Computing and Software for Big Science, 6(1), Article 13. https://doi.org/10.1007/s41781-022-00086-2

@article{df7e40ec63804e8ca0c6ee25b03ed31b,

title = "Constraints on Future Analysis Metadata Systems in High Energy Physics",

abstract = "In high energy physics (HEP), analysis metadata comes in many forms—from theoretical cross-sections, to calibration corrections, to details about file processing. Correctly applying metadata is a crucial and often time-consuming step in an analysis, but designing analysis metadata systems has historically received little direct attention. Among other considerations, an ideal metadata tool should be easy to use by new analysers, should scale to large data volumes and diverse processing paradigms, and should enable future analysis reinterpretation. This document, which is the product of community discussions organised by the HEP Software Foundation, categorises types of metadata by scope and format and gives examples of current metadata solutions. Important design considerations for metadata systems, including sociological factors, analysis preservation efforts, and technical factors, are discussed. A list of best practices and technical requirements for future analysis metadata systems is presented. These best practices could guide the development of a future cross-experimental effort for analysis metadata tools.",

keywords = "Data analysis, Databases, High energy physics, Metadata, Scientific computing",

author = "Khoo, \{T. J.\} and \{Reinsvold Hall\}, A. and N. Skidmore and S. Alderweireldt and J. Anders and C. Burr and W. Buttinger and P. David and L. Gouskos and L. Gray and S. Hageb{\"o}ck and A. Krasznahorkay and P. Laycock and A. Lister and Z. Marshall and Meyer, \{A. B.\} and T. Novak and S. Rappoccio and M. Ritter and E. Rodrigues and J. Rumsevicius and L. Sexton-Kennedy and N. Smith and Stewart, \{G. A.\} and S. Wertz",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s).",

year = "2022",

month = dec,

doi = "10.1007/s41781-022-00086-2",

language = "English",

volume = "6",

journal = "Computing and Software for Big Science",

issn = "2510-2044",

publisher = "Springer Nature",

number = "1",

}

Khoo, TJ, Reinsvold Hall, A, Skidmore, N, Alderweireldt, S, Anders, J, Burr, C, Buttinger, W, David, P, Gouskos, L, Gray, L, Hageböck, S, Krasznahorkay, A, Laycock, P, Lister, A, Marshall, Z, Meyer, AB, Novak, T, Rappoccio, S, Ritter, M, Rodrigues, E, Rumsevicius, J, Sexton-Kennedy, L, Smith, N, Stewart, GA & Wertz, S 2022, 'Constraints on Future Analysis Metadata Systems in High Energy Physics', Computing and Software for Big Science, vol. 6, no. 1, 13. https://doi.org/10.1007/s41781-022-00086-2

TY - JOUR

T1 - Constraints on Future Analysis Metadata Systems in High Energy Physics

AU - Khoo, T. J.

AU - Reinsvold Hall, A.

AU - Skidmore, N.

AU - Alderweireldt, S.

AU - Anders, J.

AU - Burr, C.

AU - Buttinger, W.

AU - David, P.

AU - Gouskos, L.

AU - Gray, L.

AU - Hageböck, S.

AU - Krasznahorkay, A.

AU - Laycock, P.

AU - Lister, A.

AU - Marshall, Z.

AU - Meyer, A. B.

AU - Novak, T.

AU - Rappoccio, S.

AU - Ritter, M.

AU - Rodrigues, E.

AU - Rumsevicius, J.

AU - Sexton-Kennedy, L.

AU - Smith, N.

AU - Stewart, G. A.

AU - Wertz, S.

PY - 2022/12

Y1 - 2022/12

N2 - In high energy physics (HEP), analysis metadata comes in many forms—from theoretical cross-sections, to calibration corrections, to details about file processing. Correctly applying metadata is a crucial and often time-consuming step in an analysis, but designing analysis metadata systems has historically received little direct attention. Among other considerations, an ideal metadata tool should be easy to use by new analysers, should scale to large data volumes and diverse processing paradigms, and should enable future analysis reinterpretation. This document, which is the product of community discussions organised by the HEP Software Foundation, categorises types of metadata by scope and format and gives examples of current metadata solutions. Important design considerations for metadata systems, including sociological factors, analysis preservation efforts, and technical factors, are discussed. A list of best practices and technical requirements for future analysis metadata systems is presented. These best practices could guide the development of a future cross-experimental effort for analysis metadata tools.

AB - In high energy physics (HEP), analysis metadata comes in many forms—from theoretical cross-sections, to calibration corrections, to details about file processing. Correctly applying metadata is a crucial and often time-consuming step in an analysis, but designing analysis metadata systems has historically received little direct attention. Among other considerations, an ideal metadata tool should be easy to use by new analysers, should scale to large data volumes and diverse processing paradigms, and should enable future analysis reinterpretation. This document, which is the product of community discussions organised by the HEP Software Foundation, categorises types of metadata by scope and format and gives examples of current metadata solutions. Important design considerations for metadata systems, including sociological factors, analysis preservation efforts, and technical factors, are discussed. A list of best practices and technical requirements for future analysis metadata systems is presented. These best practices could guide the development of a future cross-experimental effort for analysis metadata tools.

KW - Data analysis

KW - Databases

KW - High energy physics

KW - Metadata

KW - Scientific computing

UR - https://www.scopus.com/pages/publications/85135164039

U2 - 10.1007/s41781-022-00086-2

DO - 10.1007/s41781-022-00086-2

M3 - Article

AN - SCOPUS:85135164039

SN - 2510-2044

VL - 6

JO - Computing and Software for Big Science

JF - Computing and Software for Big Science

IS - 1

M1 - 13

ER -

Constraints on Future Analysis Metadata Systems in High Energy Physics

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this