A handwriting evaluation system with multi-modal sensors

BACKGROUND: Handwriting is an important and common activity for students, which may occupy around 50% time in school life. Handwriting is a kind of fine movement, which requires the coordination of eyes, hands and body. Traditional assessment of handwriting are more dependent on the final handwriting output, while the coordination skills of eye, hand and posture are rarely considered and quantitated, due to the limitations of clinical observation. The rapid development of sensor and computer vision technology provides sharp tools to record, track and recognize the movement of eyes, hands and body, which makes it possible to build an automated system for handwriting evaluation with various sensors. In this paper, we proposed a multi-modal sensing system to record the gazes, strikes and limb body movement, and developed data processing and analysis algorithms.
METHODS: First, we built a data acquisition device including an eye tracker, a tablet and a camera, which is used to record the gaze, writing and body movement respectively. Second, we used both traditional data processing and machine learning method to process the collected data, including statistical feature extraction, automatic time stamp matching, automatic segmentation of characters, feature extraction and rule-based classification. Finally, we recruited a small group of subjects to test the system and the algorithms.
RESULTS: In this preliminary study, five participants were recruited, including two kids and three adults. We used the proposed data acquisition device and processing method to process the data collected by multi-modal sensors. The experimental results show that there are noticeable differences in the features of head up frequency, pen tip pressure and head movement between kids and adults, when they are writing Chinese characters. The effectiveness of the handwriting evaluation system with multi-modal sensors is verified by the experiments designed in this study.
CONCLUSIONS: The handwriting evaluation system with multi-modal sensors based on artificial intelligence can evaluate individual performance in the process of handwriting. The system has the advantages of convenient data acquisition, distinguishing subtle actions, and objective and fine granular analysis results. In addition, the system can be applied to other studies related to attention control or fine motor. Copyright © 2020 The Education University of Hong Kong (EdUHK).