Integration of AI Agents for Semantic Processing and Procedural Animation in Virtual Reality OSCE Simulations

Abstract

The research corresponds to a technological development study with a descriptive approach, focused on the architectural development of a low-cost interactive virtual reality (VR) framework designed to facilitate the configuration of Objective Structured Clinical Examination (OSCE) training scenarios for medical students. The platform presents the user with a virtual patient in a 3D environment, managing its behavior through Oculus LipSync and Unity's procedural animation and audio components to enable speaking, gesticulating, and moving. As the cognitive core, the Wit.ai platform was integrated for speech recognition, and its transcription is processed by the Gemini 1.5 Flash API to detect clinical intents. The model returns a structured response in JSON format, which is verbally expressed through Google TTS. For clinical simulation and patient personality, a JSON file containing history and physical (H&P) data, symptoms, and temperament was designed to act as a restrictive system prompt. The main contribution of this work lies in proposing an accessible infrastructure that eliminates reliance on rigid dialogue trees, allowing educators to configure dynamic scenarios without evaluating clinical performance or pedagogical efficacy with end-users yet.