Modelling and Using Response Times in Online Courses
Keywords:MOOC, response time, lognormal time distribution
Each time a learner in a self-paced online course seeks to answer an assessment question, it takes some time for the student to read the question and arrive at an answer to submit. If multiple attempts are allowed, and the first answer is incorrect, it takes some time to provide a second answer. Here we study the distribution of such “response times.” We find that the log-normal statistical model for such times, previously suggested in the literature, holds for online courses. Users who, according to this model, tend to take longer on submits are more likely to complete the course, have a higher level of engagement, and achieve a higher grade. This finding can be the basis for designing interventions in online courses, such as MOOCs, which would encourage “fast” users to slow down.
Baker, F. B., & Kim, S. H. (2004). Item response theory: Parameter estimation techniques. Boca Raton, FL: CRC Press. http://dx.doi.org/10.1201/9781482276725
Beck, J. E. (2005). Engagement tracing: Using response times to model student disengagement. Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology, 125, 88.
Bertling, M., & Chuang, I. (2015). Response time as a measure of motivation and applied effort in MOOCs. Unpublished manuscript.
Cetintas, S., Si, L., Xin, Y. P. P., & Hord, C. (2009). Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. IEEE Transactions on Learning Technologies, 3(3), 228–236. http://dx.doi.org/10.1109/TLT.2009.44
Dai, Y. H., & Yuan, Y. (2001). An efficient hybrid conjugate gradient method for unconstrained optimization. Annals of Operations Research, 103(1–4), 33–47. http://dx.doi.org/10.1023/A:1012930416777
Goldstein, H. (2011). Multilevel statistical models (Vol. 922). Hoboken, NJ: John Wiley & Sons. http://dx.doi.org/10.1002/9780470973394
Grabe, M., & Sigler, E. (2002). Studying online: Evaluation of an online study environment. Computers & Education, 38(4), 375–383. http://dx.doi.org/10.1016/S0360-1315(02)00020-9
Grimmett, G., & Stirzaker, D. (2001). Probability and random processes. Oxford, UK: Oxford University Press.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Thousand Oaks, CA: Sage.
Ho, A., Reich, J., Nesterko, S., Seaton, D., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses, fall 2012–summer 2013. HarvardX and MITx Working Paper No. 1. http://dx.doi.org/10.2139/ssrn.2381263
Lin, C., Shen, S., & Chi, M. (2016). Incorporating student response time and tutor instructional interventions into student modeling. Proceedings of the 24th Conference on User Modeling, Adaptation and Personalization (UMAP 2016), 13–16 July 2016, Halifax, Nova Scotia, Canada (pp. 157–161). New York: ACM. http://dx.doi.org/10.1145/2930238.2930291
Maris, E. (1993). Additive and multiplicative models for gamma distributed random variables, and their application as psychometric models for response times. Psychometrika, 58(3), 445–469. http://dx.doi.org10.1007/BF02294651
Roskam, E. E. (1987). Toward a psychometric theory of intelligence. In E. E. Roskam & R. Suck (Eds.), Progress in mathematical psychology, 1 (pp. 151–174). New York: Elsevier Science. https://psycnet.apa.org/record/1987-98116-009
Roskam, E. E. (1997). Models for speed and time-limit tests. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 187–208). New York: Springer. http://dx.doi.org/10.1007/978-1-4757-2691-6_11
Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114. http://dx.doi.org/10.2307/3002019
Scheiblechner, H. (1979). Specifically objective stochastic latency mechanisms. Journal of Mathematical Psychology, 19(1), 18–38. http://dx.doi.org/10.1016/0022-2496(79)90003-8
Scheiblechner, H. (1985). Psychometric models for speed-test construction: The linear exponential model. Test design: Developments in psychology and psychometrics, 219–244. http://dx.doi.org/10.1016/B978-0-12-238180-5.50012-4
Schnipke, D. L., & Scrams, D. J. (1999). Representing response-time information in item banks. Law School Admission Council Computerized Testing Report. LSAC Research Report Series.
Thissen, D. (1983). Timed testing: An approach using item response theory. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 179–203). Cambridge, MA: Academic Press. http://dx.doi.org/10.1016/B978-0-12-742780-5.50019-6
van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204. http://dx.doi.org/10.3102/10769986031002181
van der Linden, W. J., Scrams, D. J., & Schnipke, D. L. (1999). Using response-time constraints to control for differential speededness in computerized adaptive testing. Applied Psychological Measurement, 23(3), 195–210. http://dx.doi.org/10.1177/01466219922031329
Verhelst, N. D., Verstralen, H. H., & Jansen, M. G. H. (1997). A logistic model for time-limit tests. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 169–185). New York: Springer. http://dx.doi.org/10.1007/978-1-4757-2691-6_10
Wang, S., Zhang, S., Douglas, J., & Culpepper, S. (2018). Using response times to assess learning progress: A joint model for responses and response times. Measurement: Interdisciplinary Research and Perspectives, 16(1), 45–58. http://dx.doi.org/10.1080/15366367.2018.1435105
Zhan, P., Jiao, H., & Liao, D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286. http://dx.doi.org/10.1111/bmsp.12114
How to Cite
LicenseAuthors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) license that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).