Criteria for assessing the tools of disability outcomes research

doi:10.1053/apmr.2000.20619

Archives of Physical Medicine and Rehabilitation

Volume 81, Supplement 2, December 2000, Pages S15-S20

https://doi.org/10.1053/apmr.2000.20619 Get rights and content

Abstract

Andresen EM. Criteria for assessing the tools of disability outcomes research. Arch Phys Med Rehabil 2000;81 Suppl 2:S15-S20. To recommend instrument assessment criteria, deriving from psychometric textbooks and articles and disability and research experts, for reviewing and assessing surveys and questionnaires for disability outcomes research. Traditional criteria are recommended, including psychometric properties of validity, reliability, and sensitivity to change, as are newer statistical methods for assessing scaling properties, such as Rasch analysis. Special consideration is needed for generic instruments that may be poorly scaled for disability research. Pragmatic aspects of data collection, including acceptability and disability accommodation, also are recommended. © 2000 by the American Congress of Rehabilitation Medicine

Section snippets

Organization

Each article reviewing the “tools” of disability outcomes research begins with a general review of the construct, its domains, and the available measures. This is followed by in-depth reviews of selected instruments. Authors offer recommendations from their lists of selected instruments, and provide summary report cards of their assessments. The criteria for this report card are listed in table 1. Each review includes a selected reference list of the developmental and testing literature of the

General considerations

A number of articles offer guidelines for the scientific criteria and practical attributes that should be considered in selecting outcomes research tools. Especially relevant are the criteria used to decide among tools measuring HRQOL.8, 9, 10, 11, 12, 13, 14, 15, 16 Guidelines from the Medical Outcomes Trust⁹ are perhaps the best, recent example and these were previously adapted for SCI outcomes research.¹⁷ These adaptations form the framework of the present reviews. A summary of the elements

Mode of administration and alternate forms

Different studies may require that surveys be given as interviews, self-administered forms (different modes), or in large-print, Braille, or other special formats. Reviewers use the concept of accessibility³⁴ in critiquing their selected instruments. To rate an A in this regard, measures should have been tested and found free of mode effects (differences by the type of administration) and be useful for various appropriate respondents. For example, if a survey is generally given as a

Cultural and language adaptations

The final attribute to be considered in grading outcomes research tools is the availability of appropriate cultural or language adaptations. Direct translation of measures is a minimum standard, and evidence of more comprehensive cultural and linguistic adaptation is required for higher grades. Ideally, the proper outcomes measures selected for use in a multicultural or multilingual project would have been developed with input from the groups themselves, and not require translation or

Conclusion

Users of disability outcomes measures will need to balance among these recommended criteria, just as the authors of reviews have.1, 2, 3, 4, 5 Balancing traditional psychometric criteria and the pragmatic issues of these tools, like disability accommodation, should continue to be explored and reported by researchers to add to our ability to choose the best.

Acknowledgements

The author thanks Dr. Kathleen Wyrwich at Saint Louis University for her strong work on reviewing measurement criteria for HRQOL measures, without which this work would have been substantially less thorough. Dr. Kristofer Hagglund, University of Missouri, Columbia, assisted greatly by reviewing and commenting on this manuscript. Patricia McLendon was invaluable in the preparation of the manuscript.

References (58)

ME Cohen et al.
The tools of disability outcomes research functional status measures
Arch Phys Med Rehabil
(2000)
EM Andresen et al.
Health-related quality of life outcomes measures
Arch Phys Med Rehabil
(2000)
DJ Lollar et al.
Measures of outcomes for children and youth
Arch Phys Med Rehabil
(2000)
VJ Vahle et al.
Depression measures in outcomes research
Arch Phys Med Rehabil
(2000)
MPJM Dijkers et al.
Measures of social outcomes in disability research
Arch Phys Med Rehabil
(2000)
EM Andresen et al.
Disability outcomes research: why this supplement, on this topic, at this time?
Arch Phys Med Rehabil
(2000)
AR Meyers et al.
A model of outcomes research: spinal cord injury
Arch Phys Med Rehabil
(2000)
KN Lohr et al.
Evaluating quality-of-life and health status instruments: development of scientific review criteria
Clin Ther
(1996)
DB Gray et al.
The ICIDH-2: developments for a new era of outcomes research
Arch Phys Med Rehabil
(2000)
EM Andresen et al.
Performance of health-related quality-of-life instruments in a spinal cord injured population
Arch Phys Med Rehabil
(1999)

AR Meyers et al.

Enabling our instruments: accommodation, universal design, and access to participation in research

Arch Phys Med Rehabil

(2000)

CA Lantz et al.

Behavior and interpretation of the K statistic: resolution of the two paradoxes

J Clin Epidemiol

(1996)

M Bullinger et al.

Translating health status questionnaires and evaluating their quality: the IQOLA Project approach. International Quality of Life Assessment

J Clin Epidemiol

(1998)

JE Ware et al.

Overview of the SF-36 Health Survey and the International Quality of Life Assessment (IQOLA) Project

J Clin Epidemiol

(1998)

M Bergner et al.

Health status measures: an overview and guide for selection

Annu Rev Public Health

(1987)

DL Patrick et al.

Generic and disease-specific measures in assessing health status and quality of life

Med Care

(1989)

DL Patrick et al.

Measurement of health status in the 1990s

Annu Rev Public Health

(1990)

ML Rothman et al.

Issues in the measurement of health status in asthma research

Med Care

(1993)

JM Richards et al.

Measures of life quality, role performance, and functional status in asthma research

Am J Respir Crit Care Med

(1994)

EF. Juniper

Quality-of-life considerations in the treatment of asthma

Pharmacoeconomics

(1995)

EF Juniper et al.

Measuring quality of life in asthma

Am Rev Respir Dis

(1993)

I McDowell et al.

Measuring health: a guide to rating scales and questionnaires

(1996)

Andresen EM, Fouts BS. Choosing a tool for assessing outcomes: an example of health-related quality of life in a spinal...

S. Nagi

Some conceptual issues in disability and rehabilitation

World Health Organization

International classification of impairments, disabilities, and handicaps: a manual of classification relating to the consequences of disease

(1980)

DL Patrick et al.

Health status and health policy: quality of life in health care evaluation and resource allocation

(1993)

GA Morgan et al.

Easy use and interpretation of SPSS for Windows: answering research questions with statistics

(1998)

CA McHorney et al.

Individual-patient monitoring in clinical practice: are available health status surveys adequate?

Qual Life Res

(1995)

Cited by (688)

The burden of face affected questionnaire in patients with systemic sclerosis: Translation, cross-cultural adaptation and psychometric properties in the Turkish version
2024, Egyptian Rheumatologist
Aim of the work: Facial skin involvement is one of the most important features in systemic sclerosis (SSc) patients with physical, emotional and social effects. A newly developed burden of face affected (BoFA) questionnaire is available to quantify the disability associated with facial involvement in SSc patients. This study aimed to translate the BoFA questionnaire into Turkish (BoFA-T) and investigate its psychometric properties. Patients and Methods: Forty-nine SSc patients were included in the study. The translation and cross-cultural adaptation of the BoFA-T were conducted. The BoFA-T, the scleroderma mouth handicap in SSc scale (MHISS), the Rosenberg self-esteem scale (RSS), and the perceived stress scale (PSS) were applied to construct validity. The patients were re-evaluated 7 days later to assess the test–retest reliability of the BoFA-T. Results:The mean age of the 49 patients was 48.5 ± 9.97 years, age at diagnosis 40.1 ± 11.5 years,48 females and 1 male. 24 had diffuse cutaneous and 25 limited cutaneous SSc. The mean BoFA-T was 16.81 ± 22.37, MHISS was 16.93 ± 12.19, RSS was 2.04 ± 1.92 and PSS was 24.89 ± 12.31. BoFA-T has excellent internal consistency (Cronbach’s α = 0.97) and test–retest reliability (ICC = 0.94, 95 %CI = 0.88–0.97). The correlation of BoFA-T with MHISS (r = 0.56, p < 0.001), RSS (r = 0.51, p < 0.001), and PSS (r = 0.57, p < 0.001) was moderate. The BoFA-T questionnaire consists of three-factor subgroups (''self-esteem'', ''future anxiety'', and ''sexuality''), and no floor or ceiling effects were observed in relation to BoFA-T. Conclusion: The BoFA-T demonstrates appropriate psychometric properties and may be effectively utilized to assess facial involvement and disability in Turkish SSc patients.
Immersive Virtual Reality to Assess Arm Kinematics among Older Adults with and without Major Neurocognitive Disorder – An Exploratory Cross-Sectional Study
2024, Neuroscience
Despite the recommendation of improving assessment objectivity and frequency, the use of immersive virtual reality to measure and quantify movement quality remains underexplored. In this study, we aimed to evaluate the reliability, validity and usability of an immersive virtual reality application, KinematicsVR, to assess upper limb kinematics among older adults with and without major neurocognitive disorder. The KinematicsVR involves the drawing of three-dimensional straight lines, circles and squares using a controller in a virtual environment. Twenty-eight older adults with or without major neurocognitive disorder were recruited. Reliability was evaluated through correlations on test–retest and validity through correlations between KinematicsVR variables and other functional tests (TEMPA, BBT-VR and Finger-Nose Test). The usability of the KinematicsVR was assessed with the System Usability Scale questionnaire. Kinematic indexes were compared between eight adults with major neurocognitive disorder and eight matched controls. Results indicated that most variables provided by the KinematicsVR had excellent reliability for tasks involving the drawing of straight lines and circles, but moderate reliability for tasks involving the drawing of squares. Secondary analyses showed that the usability of the application was excellent but few significant and strong correlations were observed between variables of the KinematicsVR and the scores of the TEMPA scale, Finger-Nose Test and BBT-VR. Adults with major neurocognitive disorder, when compared to other older adults, made larger and less linear hand movements. These findings provide perspectives for the use of immersive virtual reality to improve assessment frequency and objectivity through the autonomous measure of upper limb kinematics in older adults.
Psychometric Properties of the Spinal Cord Injury-Quality of Life (SCI-QOL) Resilience Short Form in a Sample With Spinal Cord Injury
2024, Archives of Physical Medicine and Rehabilitation
To explore the psychometric properties (eg, data distribution characteristics, convergent or discriminant validity, internal consistency reliability) of the Spinal Cord Injury-Quality of Life measurement system (SCI-QOL) Resilience 8-item short form (SF) in comparison to the criterion standard resilience measure, Connor Davidson Resilience Scale (CD-RISC) in a sample of individuals with spinal cord injury (SCI).
Descriptive statistics were calculated to examine variable data distribution characteristics. Correlation analyses were conducted for convergent and discriminant validity. Reliability statistics were calculated for resilience and other validity measures.
General community.
Individuals with SCI (N=202; 51.5% male, 48% female).
Not applicable.
Patient-Reported Outcomes Measurement Information System measures (depression, anxiety, ability to participate in social roles and activities, pain intensity, fatigue, sleep disturbance), SCI-QOL short forms (SF) (resilience, positive affect and well-being, mobility), CD-RISC, National Institutes of Health Toolbox for Assessment of Neurological and Behavioral Function - perceived stress (NIH Toolbox-perceived stress), and the Satisfaction with Life Scale were administered.
The mean and SD for the SCI-QOL Resilience SF (mean=48.60; SD=8.20) approximated the normative mean (mean=50, SD=10). The SCI-QOL Resilience SF scores were essentially normally distributed though somewhat kurtotic, with skew=-0.17 and excess kurtosis=1.4; internal consistency reliability was good (Cronbach's alpha=0.89). Convergent validity was supported by significant moderate correlations in expected directions between the SCI-QOL Resilience SF and measures of CD-RISC resilience, depressive symptoms, anxiety, social participation, positive affect and well-being, stress, and satisfaction with life. Discriminant validity was supported by small non-significant correlations between the SCI-QOL Resilience SF and age, sex, injury level, time since injury, pain intensity, mobility, sleep disturbance, and fatigue.
The SCI-QOL Resilience SF demonstrated good convergent and discriminant validity. Our study showed that the SCI-QOL Resilience SF is a psychometrically valid tool that can reliably estimate levels of resilience in the SCI population.
Validation of voice scales into Spanish: Systematic review of the psychometric properties
2024, Revista de Logopedia, Foniatria y Audiologia
To use a test in a language or culture other than the original it is necessary to carry out, in addition to its adaptation, a psychometric validation. This systematic review assesses the validation studies of the voice self-report scales in Spanish.
A systematic review was performed searching ten databases. The assessment was carried out following the criteria proposed by Terwee et al. (2007) together with some specifically proposed for this study. Validation studies in Spanish of self-report voice scales published in indexed journals were included and the search was updated on February 2nd, 2023.
15 studies that evaluated 12 scales were reviewed. It was verified that not all the validations were adjusted to the criteria used and that the properties to verify the metric strength of the validations were, in general, few.
This systematic review shows that the included studies do not report much evidence of metric quality. It should be considered that different strategies have currently been developed to obtain more and better evidence of reliability and validity. Our contribution is to reflect on the usual practice of validation of self-report scales in Spanish language. The most important weakness is the possibility of using broader and more current evaluation protocols. We also propose to continue this work, completing it with a meta-analytic study.
Para utilizar una prueba en una lengua o cultura distinta de la original es preciso realizar, además de su adaptación, una validación psicométrica. Esta revisión sistemática valora los estudios de validación de las escalas de autoinforme de voz en español.
Se realizó una revisión sistemática buscando en diez bases de datos. La valoración se llevó a cabo siguiendo los criterios propuestos por Terwee et al. (2007) junto con algunos específicamente propuestos para este trabajo. Se incluyeron estudios de validación en español de escalas de autoinforme publicados en revistas indexadas. La última búsqueda fue realizada el 2 de febrero de 2023.
Se revisaron 15 trabajos que evaluaron 12 escalas. Se comprobó que no todas las validaciones se ajustaron a los criterios utilizados y que las propiedades para comprobar la robustez métrica de estas fueron, por lo general, pocas.
Esta revisión sistemática muestra que los estudios incluidos no reportan demasiada evidencia de calidad métrica. Debería considerarse que en la actualidad se han desarrollado diferentes estrategias para obtener más y mejor evidencia de fiabilidad y validez. Nuestra contribución ha sido valorar la práctica de la validación de las escalas de autoinforme en lengua española. La más importante debilidad es la posibilidad de usar algún protocolo más amplio y actual. También proponemos continuar este trabajo con un estudio metaanalítico.
Item Banks for Measuring the Effect of Blood Pressure Dysregulation on Health-Related Quality of Life in Persons With Spinal Cord Injury
2023, Archives of Physical Medicine and Rehabilitation
To report on the development and calibration of the new Blood Pressure Dysregulation Measurement System (BPD-MS) item banks that assess the effect of BPD on health-related quality of life (HRQOL) and the daily activities of Veterans and non-Veterans with spinal cord injury (SCI).
Cross-sectional survey study.
Two Veteran Affairs medical centers and a SCI model system site.
454 respondents with SCI (n=262 American Veterans and n=192 non-Veterans; N=454).
Not applicable
The BPD-MS item banks.
BPD item pools were developed and refined using literature reviews, qualitative data from focus groups, and cognitive debriefing of persons with SCI and professional caregivers. The item banks then underwent expert review, reading level assessment, and translatability review prior to field testing. The items pools consisted of 180 unique questions (items). Exploratory and confirmatory factor analyses, item response theory modeling, and differential item function investigations resulted in item banks that included a total of 150 items: 75 describing the effect of autonomic dysreflexia on HRQOL, 55 describing the effect of low blood pressure (LBP) on HRQOL, and 20 describing the effect of LBP on daily activities. In addition, 10-item short forms were constructed based on item response theory-derived item information values and the clinical relevance of item content.
The new BPD-MS item banks and corresponding 10-item short forms were developed using established rigorous measurement development standards, which represents the first BPD-specific patient-reported outcomes measurement system unique for use in the SCI population.
Development of a Short Form Assessment Combining the Fugl-Meyer Assessment–Upper Extremity and the Wolf Motor Function Test for Evaluating Stroke Recovery
2023, Archives of Physical Medicine and Rehabilitation
To use Rasch methodologies to combine the items of the Fugl-Meyer Assessment–Upper Extremity (FMA-UE, motor skill) and the Wolf Motor Function Test (WMFT, motor function) onto a single measurement metric and create an FMA-UE+WMFT short form.
Secondary analysis of preintervention data from 2 upper extremity stroke rehabilitation trials. Confirmatory factor analysis and Rasch rating scale analysis were first applied to examine the properties of the pooled item bank and then item response theory methodologies were used to develop the short form. Confirmatory factor analysis and Rasch analysis were then applied to the short form to examine the dimensionality and measurement properties.
Outpatient academic medical research center.
Data from 167 participants who completed the FMA-UE and WMFT (rating scale score) were pooled (N=167). Participants were eligible if they had a stroke ≥3 months prior and had upper extremity (UE) hemiparesis and excluded if they had severe UE hemiparesis, severe UE spasticity, or UE pain.
Not applicable.
The dimensionality and measurement properties of the pooled 30-item FMA-UE and the 15-item WMFT and short form were examined.
Five items from the pool of 45 items were misfit and were removed. The 40-item pool demonstrated adequate measurement properties. A 15-item short form was then developed and met rating diagnostic scale criteria. All items on the 15-item short form met the Rasch fit criteria, and the assessment met criteria for reliability (Cronbach alpha=.94), separation (person separation = 3.7), and strata (number of strata = 5).
Items from the FMA-UE and WMFT can be pooled to create a psychometrically sound 15-item short form.

View all citing articles on Scopus

^☆: Funded, in part, by the Centers for Disease Control and Prevention through the Saint Louis University Prevention Research Center (grants no. U48/CCU710806, R13/CCR717040-01), and by the American Association of Spinal Cord Injury Psychologists and Social Workers.

^☆☆: No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit upon the author or upon any organizations with which the author is associated.

^★: Reprint requests to Elena M. Andresen, Dept of Community Health, Saint Louis University School of Public Health, 3663 Lindell Blvd, St. Louis, MO 63108, e-mail: [email protected].

View full text

Tools of Disability Outcomes ResearchCriteria for assessing the tools of disability outcomes research☆,☆☆,★

Abstract

Section snippets

Organization

General considerations

Mode of administration and alternate forms

Cultural and language adaptations

Conclusion

Acknowledgements

Arch Phys Med Rehabil

Arch Phys Med Rehabil

Arch Phys Med Rehabil

Arch Phys Med Rehabil

Arch Phys Med Rehabil

Arch Phys Med Rehabil

Arch Phys Med Rehabil

Clin Ther

Arch Phys Med Rehabil

Arch Phys Med Rehabil

Arch Phys Med Rehabil

J Clin Epidemiol

J Clin Epidemiol

J Clin Epidemiol

Health status measures: an overview and guide for selection

Annu Rev Public Health

Generic and disease-specific measures in assessing health status and quality of life

Med Care

Measurement of health status in the 1990s

Annu Rev Public Health

Issues in the measurement of health status in asthma research

Med Care

Measures of life quality, role performance, and functional status in asthma research

Am J Respir Crit Care Med

Quality-of-life considerations in the treatment of asthma

Pharmacoeconomics

Measuring quality of life in asthma

Am Rev Respir Dis

Measuring health: a guide to rating scales and questionnaires

Some conceptual issues in disability and rehabilitation

International classification of impairments, disabilities, and handicaps: a manual of classification relating to the consequences of disease

Health status and health policy: quality of life in health care evaluation and resource allocation

Easy use and interpretation of SPSS for Windows: answering research questions with statistics

Individual-patient monitoring in clinical practice: are available health status surveys adequate?

Qual Life Res

Tools of Disability Outcomes Research
Criteria for assessing the tools of disability outcomes research☆,☆☆,★