#NHS70 The NHS highlights its use of digital innovation through MIRA and other solutions. Watch the video here

An Objective Evaluation Method for Rehabilitation Exergames at ICL

by Reza Haghighi Osgouei, Dept. of Surgery and Cancer, Imperial College London, London, UK, r.haghighi-osgouei@imperial.ac.uk
David Soulsby, Acute Team Lead Paediatric Physiotherapist, Chelsea and Westminster Hospital, London, UK, david.soulsby@chelwest.nhs.uk
Fernando Bello, Dept. of Surgery and Cancer, Imperial College London, London, UK, f.bello@imperial.ac.uk


Full article published in Osgouei, R. H., Soulsbv, D., & Bello, F. (2018, August). An objective evaluation method for rehabilitation exergames. In 2018 IEEE Games, Entertainment, Media Conference (GEM) (pp. 28-34). IEEE.



The aim of this work is to objectively evaluate the performance of patients using a virtual rehabilitation system
called MIRA. MIRA is a software platform which converts conventional therapeutic exercises into games, enabling the
user to practice the given exercise by playing a game. The system includes a motion sensor to track and capture user’s
movements. Our assessment of the performance quality is based on the recorded trajectories of the human skeleton joints. We employ two different machine learning approaches, dynamic time warping (DTW) and hidden Markov modeling (HMM), both widely used for gesture recognition, to compare the user’s performance with that of a reference as ground truth.

Data collection and results

For this initial study, we have chosen shoulder abduction of the left arm as the exercise. The correct or reference execution requires keeping the arm fully stretched while moving from 0 to 180 as shown in Fig. 3(b). We also devised two incorrect executions to objectively compare with the reference. The first one keeps the arm stretched without making the full range of motion (Fig. 3(c)), whilst the second one does not keep the arm stretched and does not make the full range of motion (Fig. 3(d)).


Fig. 3: The 25 skeleton joints tracked by the Kinect V2 (a). Cardinal joints involved in shoulder abduction of the left
hand: 1) spin-shoulder, 2) shoulder-left, 3) elbow-left, and 4) wrist-left. From the variant position data, two invariant features are extracted: θ2 and θ3. Shoulder abduction is executed in three different ways: a reference with fully stretched arm and full range of motion (b), an incorrect execution with fully stretched arm but half range of motion (c), and a second incorrect execution with a closed arm and half range of motion (d).


The four cardinal joints involved in this exercise are spin-shoulder (X1), shoulder-left (X2), elbow-left (X3), and
wrist-left (X4) as shown in Fig. 3(a). The 3D position of joint j (1 <= j <= 4) at time stamp t is denoted by vector
[xj(t); yj(t); zj(t)]. As position coordinates are dependent on the user size and location in front of the camera (Kinect), we extract two invariant features, namely shoulder angle (θ2) and arm angle (θ3) as shown in Fig. 3(a).

These two scalar features are sufficient to represent the three executions since θ2 reflects the range of motion and θ3 indicates if the arm is being stretched or not. For each execution, a motion trajectory T(l) is formed by the sequence of feature values within the time frame 0 <= t <= l, with l being the execution time.

For simplicity, the reference trajectory is denoted by T0 and the two incorrect trajectories by T1 and T2, with execution times l0, l1, and l2, respectively. The data of a single user was collected repeating each exercise five times.

HMM and DTW algorithms were trained, obtaining several results. Variations between the scores obtained for different trials by DTW are smaller than for HMM. For example, the standard deviation for D00 is significantly smaller than L00 (0.66 vs. 4.46). The lowest score reported between reference trials by DTW is 98.3% while it is 88.3% by HMM. It would appear that HMM tends to be more sensitive to small-scale differences compared to DTW. This different level of sensitivity can be advantageous. Assuming a patient just started a rehabilitation process, highlighting small deviations might not be a good idea. So, in the early phases, scores reported by DTW might be
more helpful and encouraging. Once the patient’s performance has improved, reporting small deviations could better assess his/her performance.

More detailed results are presented in the full article.


We presented the results of applying two machine learning approaches, DTW and HMM, to objectively evaluate a patient’s performance with respect to a reference conducting therapeutic exercise using the MIRA rehabilitation system.

The motivation behind this study was to introduce a more clinically relevant measure than the currently used game scores. Tested on a shoulder abduction exercise, we have reported that the similarity scores obtained by both approaches, well reflect the level of inconsistency between the correct and incorrect performances. The scores obtained by each technique for the same task were different, indicating their different level of sensitivity. For example, for the given exercise, DTW is less sensitive than HMM to deviations on range of motion
than on the arm not being fully stretched. In addition, HMM is more sensitive than DTW to subtle variations from the
reference. This suggests that each method might be more suitable at certain stages of rehabilitation or indeed for
certain exercises. In early phases of a rehabilitation process, DTW-based evaluation might be more effective not focusing on the details, whereas later on, HMM-based evaluation could be advantageous to highlight subtle inconsistencies.

We intend to improve the current work in two aspects. First, by conducting a more substantial human user study with actual patients. Second, by correlating a physician’s evaluation of the performances with the proposed objective similarity score.



Subscribe to our Newsletter

We'd love to stay in touch and if you're interested to hear about some of our most important news and achievements, please consider subscribing to our periodic newsletter below.