LLMs as UXR Participants?: A How-to Guide and Comparative Analysis
This talk explores the potential and limitations of using Large Language Models (LLMs) as surrogate research participants through a series of simulated choice-based survey experiments. The first half details an open-source Python program I built that runs Maximum Difference Scaling (MaxDiff) experiments—a survey method where participants choose the most and least important items from sets of options—using LLM users, including customizable personas and comprehensive analytics reporting. The talk will walk through the AI-assisted development process, laying out best practices for AI-assisted software development, covering key considerations like building in stages, implementing unit tests, enforcing structured LLM outputs, and managing API costs effectively.

The second half describes the methods and findings of an experiment using this application. By comparing a large sample of LLM-generated personas against real data from humans, I demonstrate that LLMs can achieve moderate alignment with aggregate human preferences but fundamentally fail to capture human variability, even at maximum temperature settings. Most strikingly, removing a single seemingly-innocuous sentence from the system prompt completely reshuffled individual model-human alignment while leaving aggregate alignment relatively unchanged. These findings reveal the stark and often unpredictable sensitivity of LLM models to prompt engineering, an effect that may be moderated by model temperature. These findings have important implications for responsible AI and user research applications. As we increasingly rely on AI for understanding human needs and preferences, it is critical to recognize that subtle prompt variations can alter research outcomes in unpredictable ways, with the potential to amplify or obscure bias baked into LLMs and underscoring the need for rigorous prompt testing and evaluation.

About our speaker
Dr. Aaron Gardony was a Cognitive Scientist at the DEVCOM Soldier Center and a Visiting Scientist at the Center for Applied Brain and Cognitive Sciences (CABCS) at the time of this work. He received his joint doctorate in Psychology and Cognitive Science from Tufts University in 2016, a Master of Science from Tufts University in 2014, and a BA from Tufts University in 2009. His current work focuses on Responsible AI and Safety Evaluation.

Naviagation: Enter the building through this gate and take left.

Expanding the Design Space for Explainable AI in Human-AI Interactions 

Explainable AI (XAI) has largely been designed and evaluated through the lens of four recurring metrics: Trust, Reliance, Acceptance, and Performance (TRAP). While these metrics are essential for developing safe and responsible AI, they can also trap us in a constrained design space for how explanations provide value in human-AI interactions. Furthermore, mixed results on whether XAI actually helps calibrate reliance or foster appropriate trust raise the question of whether we are designing XAI with the right goals in mind. This talk explores how we can expand the design space for XAI by moving beyond the TRAP goals. I will discuss how domain experts appropriate AI explanations for purposes unanticipated by designers, how AI explanations can mediate understanding between physicians and other stakeholders, and how we can repurpose generative AI as an explanation tool to support various goals. By reframing XAI as a practical tool for reasoning and human–human interaction, rather than solely as a transparency mechanism, this talk invites us to consider what’s next for explainable AI

About our speaker
Katelyn Morrison is a 5th-year Ph.D. candidate in the Human-Computer Interaction Institute at Carnegie Mellon University’s School of Computer Science, advised by Adam Perer. Her research bridges technical machine learning approaches and human-centered methods to design and evaluate human-centered explainable AI (XAI) systems in high-stakes contexts, such as healthcare. In recognition of her work at the intersection of AI and health, she was awarded a Digital Health Innovations Fellowship from the Center for Machine Learning and Health at Carnegie Mellon University. Her research experience spans industry, government, and non-profit organizations, including the Software Engineering Institute, Microsoft Research, and IBM Research. Before joining Carnegie Mellon University, Katelyn earned her bachelor’s degree in Computer Science with a certificate in Sustainability from the University of Pittsburgh. She is currently on the job market for faculty, postdoc, and research scientist positions.

Naviagation: Enter the building through this gate and take left.

AI-Supported Multitasking in Human-Computer Interaction

In the future, humans will cooperate with a wide range of AI-based systems in both working (i.e., decision and recommender systems, language models, or industry robots) and private (i.e., fully- or semi-automated vehicles, smart home applications, or ubiquitous computing systems) environments. Cooperation with these systems involves shared (i.e., concurrent multitasking) and traded (i.e., task switching) interaction. As it is known that frequently changing attention can yield decreased performance as well as higher error rates and stress, future systems must consider human attention as a limited resource to be perceived as valuable and trustworthy. This talk addresses the emerging problems that occur when users frequently switch their attention between multiple systems or activities and proposes to develop a new class of AI-based interactive systems that integrally manage user attention. Therefore, we designed a software architecture that utilizes reinforcement learning and principles of computational rationality to optimize task switching. While computational rationality allows the system to simulate and adapt to different types of users, reinforcement learning does not require labeled training data, so that the concept can be applied to a wide range of tasks. The architecture has demonstrated its potential in laboratory studies and is currently being extended to support various multitasking situations. The talk concludes with a critical assessment of the underlying concepts while providing a research agenda to improve cooperation with computer systems.

About our speaker
Philipp Wintersberger is a Full Professor of Intelligent User Interfaces at IT:U Linz, as well as an external lecturer at TU Wien and FH Hagenberg. He leads an interdisciplinary team of scientists on FWF, FFG, and industry-funded research projects focusing on human-machine cooperation in safety-critical AI-based systems. He has (co)authored various works published at major journals and conferences (such as ACM CHI, IUI, AutomotiveUI, or Human Factors), and his contributions have won several awards. Further, he is a member of the ACM AutomotiveUI steering committee, has contributed to HCI conferences in various roles in the past (Technical Program Chair AutomotiveUI’21, Workshop Chair MuM’23, Diversity and Inclusion Chair Muc’22), and is one of the main organizers of the CHI workshop on Explainable Artificial Intelligence (XAI).

Naviagation: Enter the building through this gate and take left.