Fusing Motion Capture with Generative Artificial Intelligence for Creating Pedagogical 3D Animated Stories

Motion capture (hereafter mocap) is gaining significance in educational research. Mocap is generally used in the movie industry but little attention is paid to it in academic research primarily because mocap technologies are perceived as costly and challenging to apply in education. For example, the studio for mocap needs a lot of space. Setting up the mocap suit is time-consuming, and the wiring is complex. The interface of bundled mocap software applications is not user-friendly. Mocap data is also tricky to integrate into the 3D modelling application for animated characters. Meanwhile, using 3D modelling software applications is another hurdle, such as the cost concern and steep learning curve before integrating the motion data for the 3D animation. Although 3D animation is fascinating, using 3D animation in education is always discouraging. This project aims to design instructional materials for pre-service teachers or non-educational professionals to make 3D animated stories with mocap technologies and generative artificial intelligence (AI). The project team will produce a manageable and sustainable workflow, including AI-assisted storyboarding, 3D character design, 3D modelling and texturing, 3D scene creation, mocap data generation, audio recording and editing, and final integration as the 3D animation for pedagogical use for different subjects or non-educational scenarios with diverse learning outcomes. This project supports the transformation of pedagogies, curriculum, assessment, and student development as the 6-P pedagogical approach promoted at the EdUHK. The instructional materials will include videos, which play an essential role in flipped teaching and learning. The materials also incorporate our project team’s mocap research study finding, which involved using a markerless visual-motor tracking system for assessing fine and gross motor function in children with developmental coordination disorder (DCD).

Funding Source: UGC - Other Specific Funds/Earmarked Grants^