A handwriting evaluation system with multi-modal sensors

Wendi WANG, Hong FU

Research output: Contribution to conferencePapers


BACKGROUND: Handwriting is an important and common activity for students, which may occupy around 50% time in school life. Handwriting is a kind of fine movement, which requires the coordination of eyes, hands and body. Traditional assessment of handwriting are more dependent on the final handwriting output, while the coordination skills of eye, hand and posture are rarely considered and quantitated, due to the limitations of clinical observation. The rapid development of sensor and computer vision technology provides sharp tools to record, track and recognize the movement of eyes, hands and body, which makes it possible to build an automated system for handwriting evaluation with various sensors. In this paper, we proposed a multi-modal sensing system to record the gazes, strikes and limb body movement, and developed data processing and analysis algorithms.
METHODS: First, we built a data acquisition device including an eye tracker, a tablet and a camera, which is used to record the gaze, writing and body movement respectively. Second, we used both traditional data processing and machine learning method to process the collected data, including statistical feature extraction, automatic time stamp matching, automatic segmentation of characters, feature extraction and rule-based classification. Finally, we recruited a small group of subjects to test the system and the algorithms.
RESULTS: In this preliminary study, five participants were recruited, including two kids and three adults. We used the proposed data acquisition device and processing method to process the data collected by multi-modal sensors. The experimental results show that there are noticeable differences in the features of head up frequency, pen tip pressure and head movement between kids and adults, when they are writing Chinese characters. The effectiveness of the handwriting evaluation system with multi-modal sensors is verified by the experiments designed in this study.
CONCLUSIONS: The handwriting evaluation system with multi-modal sensors based on artificial intelligence can evaluate individual performance in the process of handwriting. The system has the advantages of convenient data acquisition, distinguishing subtle actions, and objective and fine granular analysis results. In addition, the system can be applied to other studies related to attention control or fine motor. Copyright © 2020 The Education University of Hong Kong (EdUHK).
Original languageEnglish
Publication statusPublished - Nov 2020
EventThe International Conference on Education and Artificial Intelligence 2020 (ICEAI 2020) - , Hong Kong
Duration: 09 Nov 202011 Nov 2020


ConferenceThe International Conference on Education and Artificial Intelligence 2020 (ICEAI 2020)
Abbreviated titleICEAI 2020
Country/TerritoryHong Kong
Internet address


Wang, W., & Fu, H. (2020, November). A handwriting evaluation system with multi-modal sensors. Paper presented at The International Conference on Education and Artificial Intelligence 2020 (ICEAI 2020), Hong Kong, China.


  • Artificial intelligence
  • Handwriting evaluation system
  • Multi-modal sensors
  • Fine motion


Dive into the research topics of 'A handwriting evaluation system with multi-modal sensors'. Together they form a unique fingerprint.