Modelling and Using Response Times in Online Courses


  • Ilia Rushkin Harvard University
  • Isaac Chuang MIT
  • Dustin Tingley Harvard University



MOOC, response time, lognormal time distribution


Each time a learner in a self-paced online course seeks to answer an assessment question, it takes some time for the student to read the question and arrive at an answer to submit. If multiple attempts are allowed, and the first answer is incorrect, it takes some time to provide a second answer. Here we study the distribution of such “response times.” We find that the log-normal statistical model for such times, previously suggested in the literature, holds for online courses. Users who, according to this model, tend to take longer on submits are more likely to complete the course, have a higher level of engagement, and achieve a higher grade. This finding can be the basis for designing interventions in online courses, such as MOOCs, which would encourage “fast” users to slow down.


Baker, F. B., & Kim, S. H. (2004). Item response theory: Parameter estimation techniques. Boca Raton, FL: CRC Press.

Beck, J. E. (2005). Engagement tracing: Using response times to model student disengagement. Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology, 125, 88.

Bertling, M., & Chuang, I. (2015). Response time as a measure of motivation and applied effort in MOOCs. Unpublished manuscript.

Cetintas, S., Si, L., Xin, Y. P. P., & Hord, C. (2009). Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. IEEE Transactions on Learning Technologies, 3(3), 228–236.

Dai, Y. H., & Yuan, Y. (2001). An efficient hybrid conjugate gradient method for unconstrained optimization. Annals of Operations Research, 103(1–4), 33–47.

Goldstein, H. (2011). Multilevel statistical models (Vol. 922). Hoboken, NJ: John Wiley & Sons.

Grabe, M., & Sigler, E. (2002). Studying online: Evaluation of an online study environment. Computers & Education, 38(4), 375–383.

Grimmett, G., & Stirzaker, D. (2001). Probability and random processes. Oxford, UK: Oxford University Press.

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Thousand Oaks, CA: Sage.

Ho, A., Reich, J., Nesterko, S., Seaton, D., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses, fall 2012–summer 2013. HarvardX and MITx Working Paper No. 1.

Lin, C., Shen, S., & Chi, M. (2016). Incorporating student response time and tutor instructional interventions into student modeling. Proceedings of the 24th Conference on User Modeling, Adaptation and Personalization (UMAP 2016), 13–16 July 2016, Halifax, Nova Scotia, Canada (pp. 157–161). New York: ACM.

Maris, E. (1993). Additive and multiplicative models for gamma distributed random variables, and their application as psychometric models for response times. Psychometrika, 58(3), 445–469. http://dx.doi.org10.1007/BF02294651

Roskam, E. E. (1987). Toward a psychometric theory of intelligence. In E. E. Roskam & R. Suck (Eds.), Progress in mathematical psychology, 1 (pp. 151–174). New York: Elsevier Science.

Roskam, E. E. (1997). Models for speed and time-limit tests. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 187–208). New York: Springer.

Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114.

Scheiblechner, H. (1979). Specifically objective stochastic latency mechanisms. Journal of Mathematical Psychology, 19(1), 18–38.

Scheiblechner, H. (1985). Psychometric models for speed-test construction: The linear exponential model. Test design: Developments in psychology and psychometrics, 219–244.

Schnipke, D. L., & Scrams, D. J. (1999). Representing response-time information in item banks. Law School Admission Council Computerized Testing Report. LSAC Research Report Series.

Thissen, D. (1983). Timed testing: An approach using item response theory. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 179–203). Cambridge, MA: Academic Press.

van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204.

van der Linden, W. J., Scrams, D. J., & Schnipke, D. L. (1999). Using response-time constraints to control for differential speededness in computerized adaptive testing. Applied Psychological Measurement, 23(3), 195–210.

Verhelst, N. D., Verstralen, H. H., & Jansen, M. G. H. (1997). A logistic model for time-limit tests. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 169–185). New York: Springer.

Wang, S., Zhang, S., Douglas, J., & Culpepper, S. (2018). Using response times to assess learning progress: A joint model for responses and response times. Measurement: Interdisciplinary Research and Perspectives, 16(1), 45–58.

Zhan, P., Jiao, H., & Liao, D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286.




How to Cite

Rushkin, I., Chuang, I., & Tingley, D. (2019). Modelling and Using Response Times in Online Courses. Journal of Learning Analytics, 6(3), 76–89.