Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Open Education Studies

Editor-in-Chief: Bastiaens, Theo

Open Access
Online
ISSN
2544-7831
See all formats and pricing
More options …

Treating Rapid Responses as Incorrect for Non-Timed Formative Tests

Daniel B. Wright
  • Corresponding author
  • Dunn Family Endowed Chair and Professor of Educational Assessment at the University of Nevada, Las Vegas, E-mail:daniel.wright@unlv.edu
  • Email
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
Published Online: 2019-09-18 | DOI: https://doi.org/10.1515/edu-2019-0004

Abstract

When students respond rapidly to an item during an assessment, it suggests that they may have guessed. Guessing adds error to ability estimates. Treating rapid responses as incorrect answers increases the accuracy of ability estimates for timed high-stakes summative tests like the ACT. There are fewer reasons to guess rapidly in non-timed formative tests, like those used as part of many personalized learning systems. Data from approximately 75 thousand formative assessments, from 777 students at two northern California charter high schools, were analyzed. The accuracy of ability estimates is only slightly improved by treating responses made in less than five seconds as incorrect responses. Simulations show that the advantage is related to: whether guesses are made rapidly, the amount of time required for thoughtful responses, the number of response alternatives, and the preponderance of guessing. An R function is presented to implement this procedure. Consequences of using this procedure are discussed.

Keywords: response times; ability; personalized learning

References

  • ACT Inc. (2018). Preparing for the ACT test. Iowa City, IA: ACT, Inc. Retrieved from https://www.act.org/content/dam/act/unsecured/documents/Preparing-for-the-ACT.pdf

  • Agresti, A. (2002). Categorical data analysis (2nd ed.). Hoboken, NJ: Wiley Interscience.Google Scholar

  • Arney, L. (2015). Go blended! A handbook for blending technology in schools. San Francisco, CA: Jossey-Bass.Google Scholar

  • Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. doi: 10.18637/jss.v067.i01Google Scholar

  • Benaglia, T., Chauveau, D., Hunter, D. R., & Young, D. (2009). mixtools: An R package for analyzing finite mixture models. Journal of Statistical Software, 32(6), 1–29. Retrieved from http://www.jstatsoft.org/v32/i06/

  • Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444. doi: 10.1146/annurev-psych-113011-143823Google Scholar

  • Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. doi: 10.18637/jss.v048.i06Google Scholar

  • Clark, A. (2016). Surfing uncertainty: Prediction, action, and the embodied mind. New York, NY: Oxford University Press.Google Scholar

  • Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning & Verbal Behavior, 11, 671–684. doi: 10.1016/S0022-5371(72)80001-XGoogle Scholar

  • Cuban, L. (2001). Oversold & underused: Computers in the classroom. Cambridge, MA: Harvard University Press.Google Scholar

  • De Boeck, P., & Jeon, M. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10, 102. doi: 10.3389/fpsyg.2019.00102Google Scholar

  • Feinberg, R. A., & Rubright, J. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35, 36–49.Google Scholar

  • Ferster, B. (2014). Teaching machines: Learning from the intersection of education and technology. Baltimore, MD: Johns Hopkins Press.Google Scholar

  • Fox, J.-P., Klotzke, K., & Entink, R. K. (2019). LNIRT: Lognormal response time item response theory models [Computer software manual]. Retrieved from https://CRAN.R-project.org/package=LNIRT (R package version 0.4.0)

  • Gentle, J. E. (2009). Computational statistics. New York, NY: Springer.Google Scholar

  • Goldstein, H. (2011). Multilevel statistical models (4th ed.). Chichester, UK: Wiley.Google Scholar

  • Griffin, P., & Care, E. (Eds.). (2015). Assessment and teaching of 21st century skills: Methods and approach. New York, NY: Springer.Google Scholar

  • Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). New York: Springer.Google Scholar

  • Herman, J., & Linn, R. (2014). New assessments, new rigor. Educational Leadership, 71, 34–37.Google Scholar

  • Horn, M. B., & Staker, H. (2015). Blended: Using disruptive innovation to improve schools. San Francisco, CA: Jossey-Bass.Google Scholar

  • Kyllonen, P. C., & Zu, J. (2016). Use of response time for measuring cognitive ability. Journal of Intelligence, 4(14). doi: 10.3390/jintelligence4040014Google Scholar

  • Luce, R. D. (1986). Response times: Their role in inferring elementary mental organization. Oxford University Press. doi: 10.1093/acprof:oso/9780195070019.001.0001Google Scholar

  • Mair, P. (2018). Modern psychometrics with R. Cham, Switzerland: Springer.Google Scholar

  • Metcalfe, J. (2017). Learning from errors. Annual Review of Psychology, 68, 465–489. Retrieved from 10.1146/annurev-psych-010416-044022Google Scholar

  • Minsky, M. (2019). Inventive minds: Marvin Minsky on education. Cambridge, MA: The MIT Press.Google Scholar

  • Murphy, M., Redding, S., & Twyman, J. S. (Eds.). (2016). Handbook on personalized learning for states, districts, and schools. Charlotte, NC: Information Age Publishing, Inc.Google Scholar

  • Palmer, E. M., Horowitz, T. S., Torralba, A., & Wolfe, J. M. (2011). What are the shapes of response time distributions in visual search? Journal of Experimental Psychology: Human Perception and Performance, 37, 58–71. doi: 10.1037/a0020747Google Scholar

  • Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. Retrieved from https://www.frontiersin.org/article/10.3389/fpsyg.2017.00422 doi: 10.3389/fpsyg.2017.00422

  • Ratcliff, R. (1978). Theory of memory retrieval. Psychological Review, 85, 59–108. doi: 10.1037/0033-295X.85.2.59Google Scholar

  • Ratcliff, R., Smith, P. L., & McKoon, G. (2015). Modeling regularities in response time and accuracy data with the diffusion model. Current Directions in Psychology Science, 24, 458–470. doi: 10.1177/0963721415596228Google Scholar

  • Roediger, H. L., III, & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1, 181–210. doi: 10.1111/j.1745-6916.2006.00012.xGoogle Scholar

  • Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26. doi: 10.1111/1529-1006.001Google Scholar

  • van der Linden, W. J. (2011). Modeling response times with latent variables: Principles and applications. Psychological Test and Assessment Modeling, 53, 334–358.Google Scholar

  • Wagenmakers, E.-J., van der Maas, H. L. J., & Grasman, R. P. P. P. (2007). An EZ-diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14, 3–22.Google Scholar

  • Wang, T., & Hanson, B. A. (2005). Development and calibration of an item response model that incorporates response time. Applied Psychological Measurement, 29, 323–339.Google Scholar

  • Wickham, H. (2015). Advanced R. Boca Raton, FL: CRC Press.Google Scholar

  • Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36. doi: 10.1111/emip.12165Google Scholar

  • Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19–38.Google Scholar

  • Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.Google Scholar

  • Wise, S. L., & Ma, L. (2012). Setting response time thresholds for a cat item pool: The normative threshold method. Paper presented at the annual meeting of the National Council on Measurement in Education, Vancouver, Canada.Google Scholar

  • Wright, D. B. (2016). Treating All Rapid Responses as Errors (TARRE) improves estimates of ability (slightly). Psychological Test and Assessment Modeling, 58, 15–31.Google Scholar

  • Wright, D. B. (2018). A framework for research on education with technology. Frontiers in Education, 3. Retrieved from https://www.frontiersin.org/article/10.3389/feduc.2018.00021 doi: 10.3389/feduc.2018.00021

  • Wright, D. B. (in press). Speed gaps: Exploring differences in response latencies among groups. Educational Measurement: Issues and Practice.Google Scholar

  • Wright, D. B., & London, K. (2009). Multilevel modelling: Beyond the basic applications. British Journal of Mathematical and Statistical Psychology, 62, 439–456.Google Scholar

About the article

Received: 2018-11-04

Accepted: 2019-06-21

Published Online: 2019-09-18


Citation Information: Open Education Studies, Volume 1, Issue 1, Pages 56–72, ISSN (Online) 2544-7831, DOI: https://doi.org/10.1515/edu-2019-0004.

Export Citation

© 2019 Daniel B. Wright, published by De Gruyter Open. This work is licensed under the Creative Commons Attribution 4.0 Public License. BY 4.0

Comments (0)

Please log in or register to comment.
Log in