An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning

BALTA, EBRU; ÖMÜR SÜNBÜL, SEÇİL

An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning

Ebru BALTA, (Corresponding author: Ankara University, Faculty of Educational Science, Department of Measurement and Evaluation, TURKEY,)

SEÇİL ÖMÜR SÜNBÜL (Mersin University, Faculty of Educational Science, Department of Measurement and Evaluation, TURKEY,)

Eurasian Journal of Educational Research

8 2

Yıl: 2017 Cilt: 17 Sayı: 72 Sayfa Aralığı: 23 - 41 Metin Dili: İngilizce İndeks Tarihi: 29-07-2022

An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning

Öz:

Purpose : Position effects may influence examinees’ test performances in several ways and trigger other psychometric issues, such as Differential Item Functioning (DIF) .This study aims to supply test forms in which items in the test are ordered differently, depending on their difficulty level (from easy to difficult or difficult to easy), to determine whether the items in the test form result in DIF and whether a consistency exists between the methods for detecting DIF. Research Methods: Methods of Mantel Haenszel (MH) and Logistic Regression (LR) have been taken into consideration to identify whether the items in the tests involve DIF The data of the work includes the answers of 300 students in the focal group and the reference group, who sat for three mathematics achievement tests. The data obtained from the tests have been statistically analyzed by using the R- 3.2.0. software program. Findings: Results of this study can be summarized with the following findings: “ordering the items differently, depending on their difficulty level, affects the probability of individuals in various groups answering the items correctly; also, LR and MH methods produce different results with respect to the items with DIF, which they have identified similar in terms of magnitude order in the amount of DIF. Implications for Research and Practice: In further test-developing studies, in order to identify if DIF emerges when giving the test form which has a different ordering of items, with regard to subjects and cognitive difficulty levels.

Anahtar Kelime:

Konular: Eğitim, Eğitim Araştırmaları

Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık

Barcikovski, R. S., & Olsen, H. (1975). Test item arrangement and adaptation level. The Journal of Psychology, 90(1), 87-93. doi: 10.1080/00223980.1975.9923929.
Bertrand, R., & Boiteau, N. (2003). Comparing the stability of IRT-based and non IRTbased DIF methods in different cultural context using TIMSS data. (EDRS Reports – Research -143 , ED 476 924, TM 034 975). Quebec, Canada: NA. (ERIC Document Reproduction Service No. ED476924).
Bulut, O. (2015). An empirical analysis of gender-based DIF due to test booklet effect. European Journal of Research on Education, 3(1), 7-16. Retrieved from http://iassr2.org/rs/030102.pdf
Camilli, G., & Shepard, L. A. (1994). Methods for identifying biased test items. Hollywood: Sage Publication.
Clauser, B. E., & Mazor, K. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement, Issues and Practice. 17(1), 31–44. doi: 10.1111/j.1745-3992.1998.tb00619.x.
Chiu, P. (2012,April) . The Effect of item position on state mathematics assessment. Paper presented at the Annual Meeting of the American Educational Research Association, Vancouver, Canada.
Gomez‐Benito, J., & Navas‐Ara, M. J. (2000). A comparison of Ki‐kare, RFA and IRT based procedures in the detection of DIF. Quality ve Quantity, 34(1),17–31. doi: 10.1023/A:1004703709442.
Hahne, J. (2008). Analyzing position effects within reasoning items using the LLTM for structurally Incomplete data. Psychology Science Quarterly, 50(3), 379-390. Retrieved from http://journaldatabase.info/articles/analyzing_position_effects_within.ht ml
Hohensinn, C., Kubinger, K. D., Reif, M., Schleicher, E., & Khorramdel, L. (2011). Analysing item position effects due to test booklet design within large-scale assessment. Educational Research and Evaluation, 17(6), 497-509. doi: 10.1080/13803611.2011.632668.
Holland, P. W., & Wainer, H. E. (1993). Differential item functioning. Hillsdale, New Jersey: Lawrence Erlbaum Associates.
Jodoin, M. G., & Gierl, M.J. (2001). Evaluating type I error and power rates using an effect size measure with logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329-349. doi:10.1207/S15324818AME1404_2.
Kingston, N. M., & Dorans, N. J. (1984). Item location effects and their implications for IRT equating and adaptive testing. Applied Psychological Measurement, 8(2), 147–154. Retrieved from doi: 10.1177/014662168400800202.
Kleinke, D. J. (1980). Item order, response location, and examinee sex and handedness on performance on multiple-choice tests. Journal of Educational Research, 73(4), 225–229. doi:10.1080/00220671.1980.10885240.
Klimko, I. P. (1984). Item arrangement, cognitive entry characteristics, sex, and test anxiety as predictors of achievement examination performance. Journal of Experimental Education, 52(4), 214-219. doi: 10.1080/00220973.1984.11011896.
Leary, L. F., & Dorans, N. J. (1985). Implications for altering the context in which test items appear: A Historical Perspective on Immediate Concern. Review of Educational Research, 55(33), 387-413. doi: 10.3102/00346543055003387.
Lord, F. M.,& Novick, M. R. (1968). Statistical theories of mental test scores. Reading, Mass.: Addison-Welsley Publishing Company.
Louisa, N. (2013). Effect of item arrangement on test reliability coefficients: implications for testing. Journal of Research in Education and Society, 4(3), 54- 62. Retrieved from http://journals.sagepub.com/doi/abs/10.1177/001316447303300224?journ alCode=epma
Magis, D., Beland, S., & Raiche, G. (2015). difR: Collection of methods to detect dichotomous differential item functioning (DIF). [Computer software]. Available from http://CRAN.R-project.org/package=difR
Miller, S. K. (1989). Interaction effects of gender and item arrangement on test and item performance (Unpublished doctoral dissertation). University of Nebraska, Lincoln.
Narayanan, P., & Swaminathan, H. (1994). Performance of the Mantel-Haenszel and simultaneous item bias procedures for detecting differential item functioning. Applied Psychological Measurement, 18(4), 315-338. doi: 10.1177/014662169401800403.
Pang, X. L., & et all. (1994,April). Performance of Mantel-Haenszel and Logistic Regression DIF procedures over replications using real data. Paper presented at the Annual Meeting of the American Educational Research Association,New Orleans, LA.
Penfield, R. D., & Camilli, G. (2007). Differential item functioning and item bias. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics psychometrics (Vol.26, pp. 125–167). Amsterdam: Elsevier.
Ryan, K. E. ,& Chiu, S. (2001). An examination of item context effects, DIF, and gender DIF. Applied Measurement in Education, 14 (1), 73–90. doi:10.1207/S15324818AME1401_06.
Wiberg, M. (2007). Measuring and detecting differential item functioning in criterionreferenced licensing test: a theoretic comparison of methods (EM No. 60). Umea, Sweden: Umea University, Department of Educational Measurement.
Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361-370. doi: 10.1111/j.1745-3984.1990.tb00754.x.
Zieky, M. (1993). Practical questions in the use of DIF statistics in test development. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 337- 347). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Yen, W. M. (1980). The extent, causes and importance of context effects on item parameters for two latent trait models. Journal of Educational Measurement, 17(4), 297–311. doi: 10.1111/j.1745-3984.1980.tb00833.x.
Zumbo, B. D., & Thomas, D. R. (1996). A measure of effect size for a model-based approach for studying DIF (Working paper of the Edgeworth Laboratory for Quantitative Behavioral Science). Prince George, Canada: University of Northern British Columbia.

APA	BALTA E, ÖMÜR SÜNBÜL S (2017). An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. , 23 - 41.
Chicago	BALTA EBRU,ÖMÜR SÜNBÜL SEÇİL An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. (2017): 23 - 41.
MLA	BALTA EBRU,ÖMÜR SÜNBÜL SEÇİL An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. , 2017, ss.23 - 41.
AMA	BALTA E,ÖMÜR SÜNBÜL S An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. . 2017; 23 - 41.
Vancouver	BALTA E,ÖMÜR SÜNBÜL S An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. . 2017; 23 - 41.
IEEE	BALTA E,ÖMÜR SÜNBÜL S "An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning." , ss.23 - 41, 2017.
ISNAD	BALTA, EBRU - ÖMÜR SÜNBÜL, SEÇİL. "An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning". (2017), 23-41.

APA	BALTA E, ÖMÜR SÜNBÜL S (2017). An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. Eurasian Journal of Educational Research, 17(72), 23 - 41.
Chicago	BALTA EBRU,ÖMÜR SÜNBÜL SEÇİL An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. Eurasian Journal of Educational Research 17, no.72 (2017): 23 - 41.
MLA	BALTA EBRU,ÖMÜR SÜNBÜL SEÇİL An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. Eurasian Journal of Educational Research, vol.17, no.72, 2017, ss.23 - 41.
AMA	BALTA E,ÖMÜR SÜNBÜL S An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. Eurasian Journal of Educational Research. 2017; 17(72): 23 - 41.
Vancouver	BALTA E,ÖMÜR SÜNBÜL S An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning. Eurasian Journal of Educational Research. 2017; 17(72): 23 - 41.
IEEE	BALTA E,ÖMÜR SÜNBÜL S "An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning." Eurasian Journal of Educational Research, 17, ss.23 - 41, 2017.
ISNAD	BALTA, EBRU - ÖMÜR SÜNBÜL, SEÇİL. "An Investigation of Ordering Test Items Differently Depending on Their Difficulty Level by Differential Item Functioning". Eurasian Journal of Educational Research 17/72 (2017), 23-41.