Development and validation of an Intelligent Cognitive Assessment System (ICAS) for persons with cerebral vascular accident (CVA)

Abstract

Background: Advancing technology and the rapidly increasing use of personal computers have speeded up the development of innovative assessment procedures and at a lower and affordable cost. One of the possible applications in rehabilitation is the use of computer-adaptive testing (CAT) for administering testing items that are adaptable to the patient’s ability level. CAT was proposed as an effective means to provide accurate and quick screening of cognitive deficits in persons with stroke in the present study. It was further developed into an Intelligence Cognitive Assessment System (ICAS) that enhances outcome prediction by adding artificial intelligence. In this project, ICAS was designed to be embedded with three special features that are not all found in typical cognitive assessments. Firstly, ICAS is a CAT designed for a comprehensive assessment of cognitive abilities for stroke survivors. Secondly, ICAS’s scoring system has been developed by modern psychometrics, using the Rasch model, in arriving at a linear ratio scale on which scores can be compared directly at different time points or between different patients. Thirdly, artificial neural networking (ANN), an artificial intelligence approach, was used to reinforce ICAS’s predictive ability with regard to functional outcomes in stroke survivors. The aim of this project was thus to develop and validate this newly developed ICAS for stroke rehabilitation. Method: Three operational phases of study were conducted to achieve specific objectives of the project. Phase I was to investigate the content validity of the assessment items of ICAS. An expert panel review of the ICAS software and its trial run among 14 stroke survivors were initially carried out. This phase also served as a pilot study to provide preliminary data for implementation of Phases II and III. In Phase II, the test item difficulty measures, item structure (construct validity), and item stability of the ICAS were investigated. Cognitive functions of another group of 30 stroke subjects were assessed by the ICAS and by the Chinese versions of the Mini-Mental Status Examination (MMSE-CV) and the Neurobehavioral Cognitive Status Examination (NCSE-CV) respectively. Phase III investigated the psychometric properties of the ICAS and built up an ANN model for predicting functional outcomes of stroke survivors which was based on the ICAS results and other demographical characteristics. The cognitive functions of a third batch of 66 subjects were assessed by both the ICAS and MMSE-CV. Demographics and clinical data such as age, gender, types of stroke, lesion side, residual upper limb function, and residual self-care function (as indicated by the initial post-stroke Modified Barthel Index or MBI) were collected together with the ICAS score. They served as predictors to forecast the MBI value at discharge stage using a specific ANN model. Result: In the Phase I study, the content validity of the ICAS was established and the Intraclass Correlation Coefficient (ICC(2,k)) among the panel members for the agreement with content relevance was 0.972 (p < 0.01). In addition, 58 out of 65 ICAS testing items got good to excellent rating in the content relevance rated by the panel members. In the Phase II study, the Rasch analysis of the 65 testing items revealed that the item difficulty measures of the ICAS ranged from 4.3 to 5.8. Only 3 items fell outside the INFIT statistics with criteria from 0.6 to 1.3. If the criteria were readjusted to 0.5 to 1.5, all the items fitted the INFIT criteria. For the OUTFIT statistics, 10 of the items fell outside the range at the 0.5 to 1.5 level. However, the principle component analysis of the residual revealed that 66.1% of the variance could be explained by the model. The unexplained variance explaining the first contrast was 3.7%. These findings indicated that the ICAS testing items were unidimensional in nature. The stroke subjects’ abilities were also found to be statistically significant and highly correlated with the MMSE-CV and the NCSE-CV scores. In the Phase III study, the correlation of ICAS with MMSE-CV was 0.757 (p < 0.001). Both the test-retest reliability of ICAS (Cronbach’s alpha = 0.878) and the correlation of the test-retest (0.789; p < 0.001) were satisfactory. Finally, a cutoff score of 3.02 was found to be able to determine the existence of cognitive impairment, indicating sensitivity of 80.5% and a false positive rate of 4%. In the ANN prediction model, there was a high correlation between the observed discharge MBI value and the model predicted discharge MBI value (correlation coefficient = 0.85; p < 0.001). Conclusion: The results suggested that the ICAS items fit the Rasch model and items are unidimensional to measure the cognitive functions for stroke survivors. Moreover, the Rasch based cognitive ability score is a linear ratio scale which might be as valid and useful as MMSE-CV and NCSE-CV in measuring cognitive function in stroke patients. Secondly, the ANN prediction model was found to be effective in predicting the functional outcomes after stroke rehabilitation, based on demographic data and residual cognitive and physical functions. These pieces of information can be useful for treatment planning and to predict a home discharge programme for better recovery and/or reintegration into the community. Thirdly, the psychometric properties of the ICAS were initially established. It can be an efficient alternative tool in determining cognitive impairment in stroke survivors for rehabilitation and related research studies. Lastly, future study can be further validated by increasing the test items in ICAS and among other neuro-disability groups.