Abstract
When students respond rapidly to an item during an assessment, it suggests that they may have guessed. Guessing adds error to ability estimates. Treating rapid responses as incorrect answers increases the accuracy of ability estimates for timed high-stakes summative tests like the ACT. There are fewer reasons to guess rapidly in non-timed formative tests, like those used as part of many personalized learning systems. Data from approximately 75 thousand formative assessments, from 777 students at two northern California charter high schools, were analyzed. The accuracy of ability estimates is only slightly improved by treating responses made in less than five seconds as incorrect responses. Simulations show that the advantage is related to: whether guesses are made rapidly, the amount of time required for thoughtful responses, the number of response alternatives, and the preponderance of guessing. An R function is presented to implement this procedure. Consequences of using this procedure are discussed.
References
ACT Inc. (2018). Preparing for the ACT test. Iowa City, IA: ACT, Inc. Retrieved from https://www.act.org/content/dam/act/unsecured/documents/Preparing-for-the-ACT.pdfSearch in Google Scholar
Agresti, A. (2002). Categorical data analysis (2nd ed.). Hoboken, NJ: Wiley Interscience.10.1002/0471249688Search in Google Scholar
Arney, L. (2015). Go blended! A handbook for blending technology in schools. San Francisco, CA: Jossey-Bass.Search in Google Scholar
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. doi: 10.18637/jss.v067.i0110.18637/jss.v067.i01Search in Google Scholar
Benaglia, T., Chauveau, D., Hunter, D. R., & Young, D. (2009). mixtools: An R package for analyzing finite mixture models. Journal of Statistical Software, 32(6), 1–29. Retrieved from http://www.jstatsoft.org/v32/i06/10.18637/jss.v032.i06Search in Google Scholar
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444. doi: 10.1146/annurev-psych-113011-14382310.1146/annurev-psych-113011-143823Search in Google Scholar
Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. doi: 10.18637/jss.v048.i0610.18637/jss.v048.i06Search in Google Scholar
Clark, A. (2016). Surfing uncertainty: Prediction, action, and the embodied mind. New York, NY: Oxford University Press.10.1093/acprof:oso/9780190217013.001.0001Search in Google Scholar
Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning & Verbal Behavior, 11, 671–684. doi: 10.1016/S0022-5371(72)80001-X10.1016/S0022-5371(72)80001-XSearch in Google Scholar
Cuban, L. (2001). Oversold & underused: Computers in the classroom. Cambridge, MA: Harvard University Press.10.4159/9780674030107Search in Google Scholar
De Boeck, P., & Jeon, M. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10, 102. doi: 10.3389/fpsyg.2019.0010210.3389/fpsyg.2019.00102Search in Google Scholar
Feinberg, R. A., & Rubright, J. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35, 36–49.10.1111/emip.12111Search in Google Scholar
Ferster, B. (2014). Teaching machines: Learning from the intersection of education and technology. Baltimore, MD: Johns Hopkins Press.10.1353/book.36140Search in Google Scholar
Fox, J.-P., Klotzke, K., & Entink, R. K. (2019). LNIRT: Lognormal response time item response theory models [Computer software manual]. Retrieved from https://CRAN.R-project.org/package=LNIRT (R package version 0.4.0)Search in Google Scholar
Gentle, J. E. (2009). Computational statistics. New York, NY: Springer.10.1007/978-0-387-98144-4Search in Google Scholar
Goldstein, H. (2011). Multilevel statistical models (4th ed.). Chichester, UK: Wiley.Search in Google Scholar
Griffin, P., & Care, E. (Eds.). (2015). Assessment and teaching of 21st century skills: Methods and approach. New York, NY: Springer.10.1007/978-94-017-9395-7Search in Google Scholar
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). New York: Springer.10.1007/978-0-387-84858-7Search in Google Scholar
Herman, J., & Linn, R. (2014). New assessments, new rigor. Educational Leadership, 71, 34–37.Search in Google Scholar
Horn, M. B., & Staker, H. (2015). Blended: Using disruptive innovation to improve schools. San Francisco, CA: Jossey-Bass.Search in Google Scholar
Kyllonen, P. C., & Zu, J. (2016). Use of response time for measuring cognitive ability. Journal of Intelligence, 4(14). doi: 10.3390/jintelligence404001410.3390/jintelligence4040014Search in Google Scholar
Luce, R. D. (1986). Response times: Their role in inferring elementary mental organization. Oxford University Press. doi: 10.1093/acprof:oso/9780195070019.001.000110.1093/acprof:oso/9780195070019.001.0001Search in Google Scholar
Mair, P. (2018). Modern psychometrics with R. Cham, Switzerland: Springer.10.1007/978-3-319-93177-7Search in Google Scholar
Metcalfe, J. (2017). Learning from errors. Annual Review of Psychology, 68, 465–489. Retrieved from 10.1146/annurev-psych-010416-04402210.1146/annurev-psych-010416-044022Search in Google Scholar
Minsky, M. (2019). Inventive minds: Marvin Minsky on education. Cambridge, MA: The MIT Press.10.7551/mitpress/11558.001.0001Search in Google Scholar
Murphy, M., Redding, S., & Twyman, J. S. (Eds.). (2016). Handbook on personalized learning for states, districts, and schools. Charlotte, NC: Information Age Publishing, Inc.Search in Google Scholar
Palmer, E. M., Horowitz, T. S., Torralba, A., & Wolfe, J. M. (2011). What are the shapes of response time distributions in visual search? Journal of Experimental Psychology: Human Perception and Performance, 37, 58–71. doi: 10.1037/a002074710.1037/a0020747Search in Google Scholar
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. Retrieved from https://www.frontiersin.org/article/10.3389/fpsyg.2017.00422 doi: 10.3389/fpsyg.2017.0042210.3389/fpsyg.2017.00422Search in Google Scholar
Ratcliff, R. (1978). Theory of memory retrieval. Psychological Review, 85, 59–108. doi: 10.1037/0033-295X.85.2.5910.1037/0033-295X.85.2.59Search in Google Scholar
Ratcliff, R., Smith, P. L., & McKoon, G. (2015). Modeling regularities in response time and accuracy data with the diffusion model. Current Directions in Psychology Science, 24, 458–470. doi: 10.1177/096372141559622810.1177/0963721415596228Search in Google Scholar
Roediger, H. L., III, & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1, 181–210. doi: 10.1111/j.1745-6916.2006.00012.x10.1111/j.1745-6916.2006.00012.xSearch in Google Scholar
Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26. doi: 10.1111/1529-1006.00110.1111/1529-1006.001Search in Google Scholar
van der Linden, W. J. (2011). Modeling response times with latent variables: Principles and applications. Psychological Test and Assessment Modeling, 53, 334–358.Search in Google Scholar
Wagenmakers, E.-J., van der Maas, H. L. J., & Grasman, R. P. P. P. (2007). An EZ-diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14, 3–22.10.3758/BF03194023Search in Google Scholar
Wang, T., & Hanson, B. A. (2005). Development and calibration of an item response model that incorporates response time. Applied Psychological Measurement, 29, 323–339.10.1177/0146621605275984Search in Google Scholar
Wickham, H. (2015). Advanced R. Boca Raton, FL: CRC Press.Search in Google Scholar
Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36. doi: 10.1111/emip.1216510.1111/emip.12165Search in Google Scholar
Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19–38.10.1111/j.1745-3984.2006.00002.xSearch in Google Scholar
Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.10.1207/s15324818ame1802_2Search in Google Scholar
Wise, S. L., & Ma, L. (2012). Setting response time thresholds for a cat item pool: The normative threshold method. Paper presented at the annual meeting of the National Council on Measurement in Education, Vancouver, Canada.Search in Google Scholar
Wright, D. B. (2016). Treating All Rapid Responses as Errors (TARRE) improves estimates of ability (slightly). Psychological Test and Assessment Modeling, 58, 15–31.Search in Google Scholar
Wright, D. B. (2018). A framework for research on education with technology. Frontiers in Education, 3. Retrieved from https://www.frontiersin.org/article/10.3389/feduc.2018.00021 doi: 10.3389/feduc.2018.0002110.3389/feduc.2018.00021Search in Google Scholar
Wright, D. B. (in press). Speed gaps: Exploring differences in response latencies among groups. Educational Measurement: Issues and Practice.Search in Google Scholar
Wright, D. B., & London, K. (2009). Multilevel modelling: Beyond the basic applications. British Journal of Mathematical and Statistical Psychology, 62, 439–456.10.1348/000711008X327632Search in Google Scholar
© 2019 Daniel B. Wright, published by De Gruyter Open
This work is licensed under the Creative Commons Attribution 4.0 Public License.
