Uncovering patterns of semantic predictability in sentence processing

Psycholinguistic researchers often collect cloze probabilities in order to measure the predictability of upcoming words but have largely discarded the variability in the structure of responses people provide. This variability in the semantic structure of responses may be important for understanding selection during language production; however, it has proven difficult to model the semantic variability of participants’ responses, and thus upcoming semantic uncertainty. Recent advances in large language models (LLMs) permit us to approximate the degree of semantic variability in cloze responses, but most methods are restricted to symbolic or hand-crafted meaning representations. We show in two studies that Bayesian Gaussian mixture models can cluster LLM representations of participants’ responses and produce coherent, taxonomically similar clusters. We apply these clustering algorithms to response time data in a serial cloze task and show that the semantic structure of cloze responses influences how quickly people are able to provide a response. We show clear effects of semantic competition on production speed. In addition to providing novel operationalizations of what semantic competition might look like in the cloze task, we explain how this clustering method is extensible to other datasets and applications of interest to researchers of semantic processing in psycholinguistics.