TY - GEN
T1 - University at Buffalo at SemEval-2023 Task 11
T2 - 17th International Workshop on Semantic Evaluation, SemEval 2023, co-located with the 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023
AU - Sullivan, Michael J.
AU - Yasin, Mohammed Nasheed
AU - Jacobs, Cassandra L.
N1 - Publisher Copyright:
© 2023 Association for Computational Linguistics.
PY - 2023
Y1 - 2023
N2 - Modeling the most likely label when an annotation task is perspective-dependent discards relevant sources of variation that come from the annotators themselves. We present three approaches to modeling the controversiality of a particular text. First, we explicitly represented annotators using annotator embeddings to predict the training signals of each annotator’s selections in addition to a majority class label. This method leads to reduction in error relative to models without these features, allowing the overall result to influence the weights of each annotator on the final prediction. In a second set of experiments, annotators were not modeled individually but instead annotator judgments were combined in a pairwise fashion that allowed us to implicitly combine annotators. Overall, we found that aggregating and explicitly comparing annotators’ responses to a static document representation produced high-quality predictions in all datasets, though some systems struggle to account for large or variable numbers of annotators.
AB - Modeling the most likely label when an annotation task is perspective-dependent discards relevant sources of variation that come from the annotators themselves. We present three approaches to modeling the controversiality of a particular text. First, we explicitly represented annotators using annotator embeddings to predict the training signals of each annotator’s selections in addition to a majority class label. This method leads to reduction in error relative to models without these features, allowing the overall result to influence the weights of each annotator on the final prediction. In a second set of experiments, annotators were not modeled individually but instead annotator judgments were combined in a pairwise fashion that allowed us to implicitly combine annotators. Overall, we found that aggregating and explicitly comparing annotators’ responses to a static document representation produced high-quality predictions in all datasets, though some systems struggle to account for large or variable numbers of annotators.
UR - https://www.scopus.com/pages/publications/85160942425
U2 - 10.18653/v1/2023.semeval-1.135
DO - 10.18653/v1/2023.semeval-1.135
M3 - Conference contribution
AN - SCOPUS:85160942425
T3 - 17th International Workshop on Semantic Evaluation, SemEval 2023 - Proceedings of the Workshop
SP - 978
EP - 985
BT - 17th International Workshop on Semantic Evaluation, SemEval 2023 - Proceedings of the Workshop
A2 - Ojha, Atul Kr.
A2 - Dogruoz, A. Seza
A2 - Da San Martino, Giovanni
A2 - Madabushi, Harish Tayyar
A2 - Kumar, Ritesh
A2 - Sartori, Elisa
PB - Association for Computational Linguistics
Y2 - 13 July 2023 through 14 July 2023
ER -