Technology-based balance performance assessment can eliminate floor and ceiling effects – Scientific Reports

Participants

We recruited 49 adult participants from July 2017 to December 2018, with ages ranging between 19 and 66 years (15 female and 34 male). The participants were recruited from a tertiary rehabilitation hospital and through word of mouth. 36 of the participants were able-bodied (AB) (13 females, 23 males, mean age 33 ± 10.1 years) and 13 with impaired mobility (IMP) (2 females, 11 males, mean age 45 ± 15.6 years). Inclusion criteria for those with impaired mobility were: (1) major single lower limb amputation (ankle or more proximal) and wearing a prosthesis for community ambulation; (2) lower limb impairment with permanent musculoskeletal deficiency (i.e., fused joint, peripheral nerve injury) and cleared for full weight bearing with or without bracing support; and (3) diagnosis of traumatic brain injury at least 6 months ago, ambulating in the community. Exclusion criteria for this group were: non-weight bearing on any extremity, unable to walk without walker support, unable to speak English, insufficient capacity for informed consent, or the existence of medical conditions precluding exercise. The resulting IMP cohort was composed of transtibial (n = 7) and transfemoral prosthesis users (n = 4), an individual with a lower limb impairment due to hemophilic arthropathy of the knee (n = 1), and an individual with mild traumatic brain injury (n = 1).

Written informed consent was obtained from all participants, and the experimental procedures were approved by the University of Alberta’s Health Research Ethics Board-Health Panel (Pro00066076) and conducted according to the ethical guidelines. Researchers could identify individual participants during the data collection sessions, but not after the data was collected as each participant was assigned a unique de-identified participant number and only this number was linked to the data.

Study design

A cross-sectional validation study took place in a tertiary rehabilitation hospital. In addition to one visit for completing the PAT, participants were invited for a second visit to perform standard functional assessment tests in the gymnasium (GYM). Participants from the AB cohort were also invited for a third visit to repeat the PAT on the CAREN. 38 of the 49 total participants completed the GYM testing (29 of the 36 AB and 9 or the 13 IMP), and 13 of the 36 AB participants completed the retest component on the CAREN. The reasons for participants not attending the additional session were related to insufficient time to participate or the inability to contact.

Experimental procedures

The PAT testing on the CAREN consisted of a series of tasks, repeated for increasing levels of difficulty, grouped within 3 different modules. Completing all the tasks for all levels of difficulty took participants an average of 90 min, ranging between 70 and 120 min. Participants took 3-min breaks in between modules, corresponding to the time it takes the computer to load the application for the next module on the CAREN. Although additional breaks were offered if needed, none of the participants required a longer break. The GYM tests lasted less than 30 min, and participants took 1-min breaks between tasks.

For tests on the CAREN, participants donned a safety harness as per the CAREN protocol and had 3 motion capture marker plates placed: one on each foot, and one on the back at the level of the waist (Supplementary Fig. S4). Before each module started, one of the experimenters explained the process for the module. The participants were informed that a set of instructions would be presented on the screen, describing the goal of the module and the tasks that would follow. Each module consisted of 4 stages: first, the participant was presented with a set of instructions, explaining the task for the module. In some cases, a virtual avatar would demonstrate the task in addition to the visual text instructions. Second, the participant would be asked to assume a comfortable standing position that would be used to calibrate the system. Following calibration, when the participant indicated they were ready, the CAREN operator initialized the first task of the module. At the end of the module, the participant would see a message on the screen to inform them that the module had been completed and the next module will be loaded. At the end of the PAT testing, the participant was guided off the platform, and the motion capture marker plates and safety harness were taken off by an experimenter.

Task selection

To ensure content validity, we identified a selection of relevant tasks to be implemented on the CAREN for the PAT after examining currently available assessment tools for balance performance in the clinic^7,21. By deconstructing the tools, we obtained a series of activities that are widely used in balance assessments. Different tasks associated with such activities were then summarized and grouped into 3 separate PAT modules to assess balance performance during: (1) single leg support; (2) stepping in different directions; and (3) standing on a moving platform (perturbations). Each of these modules was presented in the PAT as “games” on the CAREN.

Task implementation

The 3 different modules developed for the PAT are presented in detail in the Supplementary Information. Module S1 (Supplementary Fig. S1), named “The Blocks”, consists of a game where the participant standing in a virtual field is required to clear the path for an oncoming train of blocks by lifting the foot, thereby challenging single leg stance (single leg stance). Module S2 (Supplementary Fig. S2), named “The Targets”, consists of a game where the participant standing in a virtual field is required to use their feet to step on targets appearing in a semicircle in front of them. To succeed, participants had to step with either one or both feet (challenging stepping). Module S3 (Supplementary Fig. S3), named “The Bus”, consists of a game where the participant standing inside a virtual bus is required to maintain balance after sudden shifts in the position of the bus, i.e., platform (perturbation challenge).

Outcome measures

Different measures were defined for each module of the PAT according to the task biomechanics within the module:

1. 1.

   Balance was assessed during standing by measures of the centre of pressure (CoP) displacement²². Specifically, the mean velocity of the CoP (mvCoP) has been shown to be a reliable measure of balance when the test duration is longer than 20 s²³. Thus, for “S1: The Blocks”, we computed the outcome measure mvCoP from the CoP displacement measured starting the moment the participant’s foot left the ground and ending 30 s later or when the foot was lowered again, whichever occurred first.

2. 2.

   Foot clearance has been shown to be correlated with stability during walking²⁴, and is minimized to save energy at the expense of an increase in the risk of tripping and possibly falling²⁵. Conversely, if a risk of falling is perceived, foot clearance usually increases, and a more cautious step will be taken. Under the assumption that a larger foot clearance indicates more cautious stepping, we used the position of the marker clusters on the feet to measure the maximum lift (peakLift), defined along the trajectory of the leading foot when stepping onto the targets, as our outcome measure for “S2: The Targets”.

3. 3.

   Maintaining balance under perturbation conditions relies on different postural strategies depending on the perceived challenge. There are two different classes of postural strategies: (1) “fixed-support” strategies, in which the feet remain in place when responding to the perturbation and balance is maintained using ankle and/or hip motion; and (2) “change-in-support” reactions, where rapid stepping or reaching movements are executed to maintain balance²⁶. In general, change-in-support strategies are used when fixed-support responses fail to maintain balance in response to a perturbation. We determined the presence of a stepping response based on the position of the marker clusters on the feet and defined the ratio of the number of trials in which a change-in-support strategy (i.e., stepping response) was necessary over the total number of trials (ratioStepping) as our outcome measure for “S3: The Bus”.

PAT scoring

The scoring for the PAT was defined using data collected on the first visit of participants in the AB cohort. Details on the PAT scoring structure are presented in Supplementary Information.

We defined a series of scoring functions to transform outcome measures into scores. Separate scoring functions were developed for each module based on the module’s corresponding outcome measure. Scoring functions were calculated to fit the data collected from each of the modules. If available, data from both legs were combined for the calculation of the scoring function. Thus, for “S1: The Blocks”, a single scoring function was defined based on values for the measure of mvCOP obtained from all participants using the data collected from both legs.

Although, for “S2: The Targets”, measures from both legs were combined, the scoring function was calculated based on additional criteria. First, all but the measures from the diagonal directions (± 45°) were excluded because: (1) for the 0° direction (forward), participants always stepped on the target with the same leg, hence, creating an unbalanced set of data that limits the ability of the measure to characterize the general performance independent of the stepping leg; and (2) the direction of ± 90° (sideways) required a movement that largely limits the movement of the knee; as such, it is not a good measure of general performance. Separate scoring functions were calculated for each combination of level and stepping type (i.e., single foot, double foot). If a target was not reached, then a score of zero was assigned. The scoring functions for “S2: The Targets” were defined based on values for the measure of peakLift obtained from all participants for each combination of level and stepping type.

For “S3: The Bus”, separate scoring functions were calculated for each level of difficulty regardless of the direction of the perturbation. The scoring functions for “S3: The Bus” were defined based on values for the measure of ratioStepping obtained from each level.

To calculate the combined PAT score, it is necessary to first calculate the scores for each module separately, applying the scoring functions to the measures obtained from the PAT. The scores from S1 and S2 were calculated separately for each leg and the general score was based on the lowest scoring leg. This allowed characterization of the true performance of any given participant. The score for S1 would then be calculated as the lowest score obtained from the individual tests for each leg. For S2, we first calculated the sub-scores for all combinations of level and stepping type separately for each leg, and then calculated, for each leg, the average sub-score. The lowest of these results gives the score for S2. The score for S3 would be calculated as the average of the sub-scores obtained for each level. Finally, the combined PAT score was calculated as the average score across the three modules’ scores.

Statistical analysis

Test–retest reliability

Test–retest reliability was assessed using the subset of participants from the AB cohort that were tested on the CAREN on two separate visits (n = 13). The average time (mean ± SD) between visits was 126 ± 84 days, the minimum being 7 days and the maximum 385 days. Time between sessions was difficult to control due to the challenge of synchronizing the schedule of a given participant with that of CAREN access at the rehabilitation hospital. For the repeat cohort, the Limits of Agreement (LoA) introduced by Bland and Altman^27,28 were calculated to determine the agreement between the scores (both the individual module scores and the combined PAT scores) from the two different sessions. The LoA also allowed inspecting the results for learning effects between sessions and flagging outlier candidates in the data set²⁹. Subsequently, the test–retest reliability of the scores was assessed using both the Intraclass Correlation Coefficient ICC_3,1^30,31 and the Concordance Correlation Coefficient CCC, along with its associated bias correction factor C_b^32,33.

Concurrent convergent validity

Concurrent convergent validity was assessed between the PAT scores from the CAREN and the standard functional assessment tests performed in a gymnasium (GYM). The standard tests used for assessing balance performance in the GYM testing were the Dynamic Gait Index (DGI)³⁴, the Berg Balance Scale (BBS)^35,36, the Amputee Mobility Predictor (AMP)³⁷, and the Comprehensive High-Level Activity Mobility Predictor (CHAMP), composed of the Single-Leg Support Test (SLS), the Edgren Side Step Test (ESST), the T-Test (TT), and the Illinois Agility Test (IAT)³⁸. For the AMP, only impaired participants were included given the nature of the test (n = 9). For all other tests, able-bodied participants were included in addition to the impaired participants (n = 38). We calculated the Spearman Correlation Coefficient (ρ) to evaluate bivariate correlations between the PAT score and the scores obtained from the DGI, BBS, AMP, SLS, ESST, TT, IAT, and the overall CHAMP.

Construct validity

Construct validity of the PAT was examined by inspecting: (1) the ranking of IMP participants compared to AB participants; and (2) the correlation between scores and age. We included data from all cohorts in this study (n = 49). We expected that: (1) participants in the IMP cohort would be ranked at lower levels than participants in the AB cohort; and (2) older participants would be ranked lower than their younger counterparts. We evaluated the Spearman Correlation Coefficient (ρ) between the PAT scores and the participants’ age separately for able-bodied and impaired participants, to account for the inherent difference in performance between both groups.

Internal consistency

The internal consistency of the modules of the PAT was inspected to determine the level of correlation across them. Since the three modules evaluate different features of balance performance, we expected them to have some level of agreement, and testing their underlying structure would allow us to determine overlap and redundancy.

We used Cronbach α to determine the internal consistency of the PAT and performed a Principal Component Analysis (PCA) to examine the contribution of each module’s score to the total score.

Statistics performed on the PAT scores

The Standard Error of Measurement (SEM) of the combined PAT score was calculated using the intra-class correlation coefficient (ICC_3,1) and standard deviation (SD) of the PAT score as \(SEM=SD\cdot \sqrt{1-{ICC}_{\mathrm{3,1}}}\). The Minimum Detectable Change (MDC) was calculated as \(MDC=SEM\cdot z\cdot \sqrt{2}\), with z (z-score) representing the confidence interval for the normal distribution (z = 1.96 for 95% confidence interval)³¹. Differences in the scores for the AB and IMP cohorts were compared by means of a one-way ANOVA. Comparisons were performed on the individual scores for each module as well as on the final combined PAT score.