Skip to content
BY 4.0 license Open Access Published by De Gruyter Open Access September 18, 2019

Treating Rapid Responses as Incorrect for Non-Timed Formative Tests

  • Daniel B. Wright EMAIL logo
From the journal Open Education Studies


When students respond rapidly to an item during an assessment, it suggests that they may have guessed. Guessing adds error to ability estimates. Treating rapid responses as incorrect answers increases the accuracy of ability estimates for timed high-stakes summative tests like the ACT. There are fewer reasons to guess rapidly in non-timed formative tests, like those used as part of many personalized learning systems. Data from approximately 75 thousand formative assessments, from 777 students at two northern California charter high schools, were analyzed. The accuracy of ability estimates is only slightly improved by treating responses made in less than five seconds as incorrect responses. Simulations show that the advantage is related to: whether guesses are made rapidly, the amount of time required for thoughtful responses, the number of response alternatives, and the preponderance of guessing. An R function is presented to implement this procedure. Consequences of using this procedure are discussed.


ACT Inc. (2018). Preparing for the ACT test. Iowa City, IA: ACT, Inc. Retrieved from in Google Scholar

Agresti, A. (2002). Categorical data analysis (2nd ed.). Hoboken, NJ: Wiley Interscience.10.1002/0471249688Search in Google Scholar

Arney, L. (2015). Go blended! A handbook for blending technology in schools. San Francisco, CA: Jossey-Bass.Search in Google Scholar

Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. doi: 10.18637/jss.v067.i0110.18637/jss.v067.i01Search in Google Scholar

Benaglia, T., Chauveau, D., Hunter, D. R., & Young, D. (2009). mixtools: An R package for analyzing finite mixture models. Journal of Statistical Software, 32(6), 1–29. Retrieved from in Google Scholar

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444. doi: 10.1146/annurev-psych-113011-14382310.1146/annurev-psych-113011-143823Search in Google Scholar

Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. doi: 10.18637/jss.v048.i0610.18637/jss.v048.i06Search in Google Scholar

Clark, A. (2016). Surfing uncertainty: Prediction, action, and the embodied mind. New York, NY: Oxford University Press.10.1093/acprof:oso/9780190217013.001.0001Search in Google Scholar

Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning & Verbal Behavior, 11, 671–684. doi: 10.1016/S0022-5371(72)80001-X10.1016/S0022-5371(72)80001-XSearch in Google Scholar

Cuban, L. (2001). Oversold & underused: Computers in the classroom. Cambridge, MA: Harvard University Press.10.4159/9780674030107Search in Google Scholar

De Boeck, P., & Jeon, M. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10, 102. doi: 10.3389/fpsyg.2019.0010210.3389/fpsyg.2019.00102Search in Google Scholar

Feinberg, R. A., & Rubright, J. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35, 36–49.10.1111/emip.12111Search in Google Scholar

Ferster, B. (2014). Teaching machines: Learning from the intersection of education and technology. Baltimore, MD: Johns Hopkins Press.10.1353/book.36140Search in Google Scholar

Fox, J.-P., Klotzke, K., & Entink, R. K. (2019). LNIRT: Lognormal response time item response theory models [Computer software manual]. Retrieved from (R package version 0.4.0)Search in Google Scholar

Gentle, J. E. (2009). Computational statistics. New York, NY: Springer.10.1007/978-0-387-98144-4Search in Google Scholar

Goldstein, H. (2011). Multilevel statistical models (4th ed.). Chichester, UK: Wiley.Search in Google Scholar

Griffin, P., & Care, E. (Eds.). (2015). Assessment and teaching of 21st century skills: Methods and approach. New York, NY: Springer.10.1007/978-94-017-9395-7Search in Google Scholar

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). New York: Springer.10.1007/978-0-387-84858-7Search in Google Scholar

Herman, J., & Linn, R. (2014). New assessments, new rigor. Educational Leadership, 71, 34–37.Search in Google Scholar

Horn, M. B., & Staker, H. (2015). Blended: Using disruptive innovation to improve schools. San Francisco, CA: Jossey-Bass.Search in Google Scholar

Kyllonen, P. C., & Zu, J. (2016). Use of response time for measuring cognitive ability. Journal of Intelligence, 4(14). doi: 10.3390/jintelligence404001410.3390/jintelligence4040014Search in Google Scholar

Luce, R. D. (1986). Response times: Their role in inferring elementary mental organization. Oxford University Press. doi: 10.1093/acprof:oso/9780195070019.001.000110.1093/acprof:oso/9780195070019.001.0001Search in Google Scholar

Mair, P. (2018). Modern psychometrics with R. Cham, Switzerland: Springer.10.1007/978-3-319-93177-7Search in Google Scholar

Metcalfe, J. (2017). Learning from errors. Annual Review of Psychology, 68, 465–489. Retrieved from 10.1146/annurev-psych-010416-04402210.1146/annurev-psych-010416-044022Search in Google Scholar

Minsky, M. (2019). Inventive minds: Marvin Minsky on education. Cambridge, MA: The MIT Press.10.7551/mitpress/11558.001.0001Search in Google Scholar

Murphy, M., Redding, S., & Twyman, J. S. (Eds.). (2016). Handbook on personalized learning for states, districts, and schools. Charlotte, NC: Information Age Publishing, Inc.Search in Google Scholar

Palmer, E. M., Horowitz, T. S., Torralba, A., & Wolfe, J. M. (2011). What are the shapes of response time distributions in visual search? Journal of Experimental Psychology: Human Perception and Performance, 37, 58–71. doi: 10.1037/a002074710.1037/a0020747Search in Google Scholar

Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. Retrieved from doi: 10.3389/fpsyg.2017.0042210.3389/fpsyg.2017.00422Search in Google Scholar

Ratcliff, R. (1978). Theory of memory retrieval. Psychological Review, 85, 59–108. doi: 10.1037/0033-295X.85.2.5910.1037/0033-295X.85.2.59Search in Google Scholar

Ratcliff, R., Smith, P. L., & McKoon, G. (2015). Modeling regularities in response time and accuracy data with the diffusion model. Current Directions in Psychology Science, 24, 458–470. doi: 10.1177/096372141559622810.1177/0963721415596228Search in Google Scholar

Roediger, H. L., III, & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1, 181–210. doi: 10.1111/j.1745-6916.2006.00012.x10.1111/j.1745-6916.2006.00012.xSearch in Google Scholar

Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26. doi: 10.1111/1529-1006.00110.1111/1529-1006.001Search in Google Scholar

van der Linden, W. J. (2011). Modeling response times with latent variables: Principles and applications. Psychological Test and Assessment Modeling, 53, 334–358.Search in Google Scholar

Wagenmakers, E.-J., van der Maas, H. L. J., & Grasman, R. P. P. P. (2007). An EZ-diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14, 3–22.10.3758/BF03194023Search in Google Scholar

Wang, T., & Hanson, B. A. (2005). Development and calibration of an item response model that incorporates response time. Applied Psychological Measurement, 29, 323–339.10.1177/0146621605275984Search in Google Scholar

Wickham, H. (2015). Advanced R. Boca Raton, FL: CRC Press.Search in Google Scholar

Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36. doi: 10.1111/emip.1216510.1111/emip.12165Search in Google Scholar

Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19–38.10.1111/j.1745-3984.2006.00002.xSearch in Google Scholar

Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.10.1207/s15324818ame1802_2Search in Google Scholar

Wise, S. L., & Ma, L. (2012). Setting response time thresholds for a cat item pool: The normative threshold method. Paper presented at the annual meeting of the National Council on Measurement in Education, Vancouver, Canada.Search in Google Scholar

Wright, D. B. (2016). Treating All Rapid Responses as Errors (TARRE) improves estimates of ability (slightly). Psychological Test and Assessment Modeling, 58, 15–31.Search in Google Scholar

Wright, D. B. (2018). A framework for research on education with technology. Frontiers in Education, 3. Retrieved from doi: 10.3389/feduc.2018.0002110.3389/feduc.2018.00021Search in Google Scholar

Wright, D. B. (in press). Speed gaps: Exploring differences in response latencies among groups. Educational Measurement: Issues and Practice.Search in Google Scholar

Wright, D. B., & London, K. (2009). Multilevel modelling: Beyond the basic applications. British Journal of Mathematical and Statistical Psychology, 62, 439–456.10.1348/000711008X327632Search in Google Scholar

Received: 2018-11-04
Accepted: 2019-06-21
Published Online: 2019-09-18

© 2019 Daniel B. Wright, published by De Gruyter Open

This work is licensed under the Creative Commons Attribution 4.0 Public License.

Downloaded on 7.6.2023 from
Scroll to top button