A Comparison Of Irt Linking Methods Across Different Conditions

Author : Jiajing Huang
Publisher :
ISBN 13 :
Total Pages : 0 pages
Book Rating : 4.:/5 (135 download)

DOWNLOAD NOW!

Download or read book A Comparison of IRT Linking Approaches Under the Nonequivalent Groups Anchor Test Design written by Jiajing Huang and published by . This book was released on 2021 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: The nonequivalent-groups anchor-test (NEAT) data-collection design is commonly used in large-scale assessments. Under this design, different test groups take different test forms. Each test form has its own unique items and all test forms share a set of common items. If item response theory (IRT) models are applied to analyze the test data, the estimates of item parameters in different test forms and ability scores of different test groups may not be on the same scale because of the IRT property of scale indeterminacy. To compare the items in different forms and examinees in different test groups, a statistical procedure, linking, is needed to convert the item parameters and ability scores to a common scale. There are several IRT linking approaches under the NEAT design. Some of the approaches make use of the common (anchor) items shared by test forms and are referred to as common-item linking approaches. The often used common-item linking approaches under the NEAT design include concurrent calibration, separate calibration with moments-linking approaches (e.g., mean/mean and mean/sigma), and characteristic-curve linking approaches (e.g., Haebara approach and Stocking-Lord approach). All common-item linking approaches hold the same key assumption - that common items perform in the same way across different test forms, which means the common-item parameters are assumed to be invariant. When the common-item parameters are non-invariant, the results of common-item linking might be affected. The other type of linking approach studied in this dissertation, common-population linking, does not assume the invariance of common items. This linking approach makes use of one group's ability distributions derived from different calibration runs to do linking, and assumes that the ability distributions of different test groups are the same in one of the steps involved. Under the NEAT design, the test groups possibly come from different populations, which means the equivalence assumption for group ability distributions might be violated. Violation of this assumption might threaten the accuracy of common-population linking results. Many studies have compared the performance of several popular common-item linking approaches under a variety of conditions. However, no study has compared the common-population linking approach with common-item linking approaches under the NEAT design. This study aimed to compare the performance of common-population linking and common-item linking approaches (specifically, concurrent calibration and separate calibration with Stocking-Lord linking approach) under some simulated conditions. The design factors included differences in ability distributions, parameter drift for some anchor (common) items, sample size, test length, and the numbers of anchor items. The simulated study findings showed that no one approach always outperformed the other two approaches across all conditions in the scenario that the scores of the focal group (taking the new form) were linked to the scale of the reference group (taking the old form). Generally, the type of parameter drift for anchor items impacted the performance of the Stocking-Lord and the concurrent calibration approaches, but even under conditions of drift, these two approaches still performed well. The concurrent calibration was more sensitive to sample size and the number of anchor items than the Stocking-Lord approach. For the common-population approach, when the ability distribution of the focal group had same mean and standard deviation as the reference group, this approach sometimes performed equally well or slightly better than the other two approaches. Once the two groups had different means and standard deviations, its performance was poor. In addition to a simulation study, empirical data from the Florida Standards Assessments (FSA) were used to demonstrate the applications of these approaches in a practical test setting. Florida Department of Education (FLDOE) applies separate calibrations with the Stocking-Lord linking approach. In this empirical study, concurrent calibration with nonequivalent groups and a common-population linking approach were applied and were compared with the original results generated from the FLDOE's separate calibration with Stocking-Lord linking. Results showed that the scale scores from the concurrent calibration and the common-population linking correlated strongly with the original FSA scale scores. The linked item parameters from the two linking approaches were also similar to the corresponding ones on the FSA scale. In conclusion, when some anchor items had parameter drift and the two test groups had different ability distributions, the Stocking-Lord approach generally performed the best; the common-population approach performed the worst in the majority of simulated conditions except when its underlying assumption of equal ability distributions across groups was not violated.