A new framework with multiple tasks for detecting and locating pain events in video

Junkai CHEN, Zheru CHI, Hong FU

Research output: Contribution to journalArticlespeer-review

1 Citation (Scopus)

Abstract

Automatically detecting and locating pain events in video is an important task in medical assessment. It is a challenging problem in facial expression analysis due to spontaneous faces, head movements and pose variations. In this paper, we explore the role of facial information at various time scales (frame, segment and sequence) and propose a new framework for pain event detection and locating in video. We introduce a feature-level fusion method for pain event detection and a multiple-task fusion method for locating pain events, respectively. Both spatial and spatial–temporal features are utilized in our study. At first, we employ the histogram of oriented gradients (HOG) of fiducial points (P-HOG) to extract spatial features from each video frame and train an SVM as a frame-based pain event detector. Secondly, HOG from Three Orthogonal Planes (named as HOG-TOP) is used to characterize the dynamic textures of a video segment, a segment-based classifier (SVM) is then trained for segment-level detection. We further apply a max pooling strategy to obtain the global P-HOG and HOG-TOP to represent the whole video sequence and a multiple kernel fusion is employed to find an optimal feature-level fusion. An SVM with multiple kernels is trained to perform sequence-level (pain event) detection. Finally, an effective probabilistic fusion method is proposed to integrate the detection results of the three different tasks (frame-level, segment-level and sequence-level detection) to locate pain events in video. Extensive experiments conducted on the UNBC-McMaster Shoulder Pain database show that our proposed method outperforms other state-of-the-art methods both in pain event detection and locating in video. Our sequence-level event detection method has also been applied to facial expression recognition in video with good results. Copyright © 2016 Elsevier Inc. All rights reserved.
Original languageEnglish
Pages (from-to)113-123
JournalComputer Vision and Image Understanding
Volume155
Early online date04 Nov 2016
DOIs
Publication statusPublished - Feb 2017

Citation

Chen, J., Chi, Z., & Fu, H. (2017). A new framework with multiple tasks for detecting and locating pain events in video. Computer Vision and Image Understanding, 155, 113-123. doi: 10.1016/j.cviu.2016.11.003

Keywords

  • Pain event detection
  • Pain event locating
  • P-HOG
  • HOG-TOP
  • Probabilistic fusion
  • Multiple kernel fusion

Fingerprint Dive into the research topics of 'A new framework with multiple tasks for detecting and locating pain events in video'. Together they form a unique fingerprint.