CAREER: A Computational Approach to the Study of Behavior and Social Interactions

Face-to-face meetings of groups of people with shared interests or beliefs are still one of the most effective ways to capture the attention of participants, engage them in conversation, and drive productive collaborations. Such meetings could include review panels, academic study groups, classrooms, or board meetings. The ability to understand, manage and react to the social processes occurring within such a group is a core aspect of social intelligence, which is important for collaboration in many contexts. This project is aimed at developing an integrated computational framework useful for analyzing the behaviors of individuals in a group; evaluating the interactions occurring within the group; and measuring any collective behaviors that might emerge from the group. This will be accomplished by exploring novel machine learning techniques to understand human behavior in face-to-face meetings. Outcomes of the study will advance understanding of collaboration and interactions in areas such as personalized health and STEM teaching and learning. The investigator will involve students and faculty who will be trained in the use of computational methods for teaching and learning, as well as cross-cultural interactions. The investigator will apply social dynamics and computational methods to expand understanding of social interactions in face-to-face engagement using a level of analysis that reveals subtle signaling, personality, affect, and individual and collective influence. Testbeds will be developed from video data of students learning in a summer research program and new data sets will be developed based on faculty learning groups and students participating in a learning group at a University abroad. The first step will involve codifying individual behaviors and creating a model to generate data clusters representing prototypical behaviors in group meetings. Facial action units will serve as inputs into the model. The next step will explore interactivity of codified clusters representing individuals. Coupled recurrent networks will be used to understand interaction and influence by linking chains of sequenced data and applying recurrent neural networks. This variant of recurrent neural networks will be validated using artificially generated data sets. A third step will explore macro-emotions in groups by mapping individual behavior and personality types to influences on collective group emotion. Observations and features from an existing data set comprising galvanic skin response, facial expression, action units and gestures will be used. Methods to fuse these multimodal inputs will be explored and will consist of early fusion techniques (clustering) and later fusion techniques (regressors and classifiers). The testbeds for this research are diverse, including individuals with sensory disabilities and underrepresented minority groups. Methods emerging from the research could be used to assess learning and collaboration in classroom settings, meetings requiring strategic decision making, and study groups. Students participating in the research will gain important computational skills and knowledge requiring an integration of psychology and sociology. Students will also have an opportunity to participate in collaboration abroad. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.