Thesis Proposal Defense - Anika Sayara

Name: Anika Sayara
Date: January 22, 2025
Time: 12 PM
Location: ICCS 304
Supervisor: Dongwook Yoon

Title: Bridging Gulfs of Execution and Evaluation in AI-Assisted VR Programming by Enabling User Interaction with Externalized AI Knowledge

Abstract:
Generative AI can assist VR programming by translating users' natural-language prompts into code. However, conveying complex VR interactions with spatial intricacies often requires more than just words. Embodied prompting—combining speech with bodily gestures—offers a more natural way for users to express their programming intent in VR. Despite its intuitiveness, users still struggle to articulate their programming intent (gulf of execution) and assess AI-generated outputs (gulf of evaluation) due to knowledge asymmetry: the AI lacks access to users' implicit knowledge and evolving mental models, while users cannot observe the AI's internal knowledge that shape its interpretation of prompts. Externalizing the AI’s knowledge makes its reasoning processes visible, helping users align their intent with the AI’s interpretation and improving their ability to evaluate AI-generated outputs. Additionally, enabling users to interact with these representations by pointing or directly manipulating them helps users articulate or clarify their programming intent more effectively. This approach addresses both the gulfs of evaluation and execution, advancing human-AI collaboration for programming in VR.