Integrating Option Tracing into Knowledge Tracing
Enhancing Learning Analytics for Mathematics Multiple-Choice Questions
DOI:
https://doi.org/10.18608/jla.2025.8557Keywords:
knowledge tracing, option tracing, mathematics multiple-choice questions, learning pattern analytics, research paperAbstract
Knowledge tracing (KT) is a method to evaluate a student’s knowledge state (KS) based on their historical problem-solving records by predicting the next answer’s binary correctness. Although widely applied to closed-ended questions, it lacks a detailed option tracing (OT) method for assessing multiple-choice questions (MCQs). This paper introduces a general OT method that can be seamlessly integrated into deep knowledge tracing (DKT) methods through data processing techniques and network output modules. Using a million-level assignment record of MCQs from a K–12 math learning platform, which includes two types of knowledge components (KCs), skill and misconception, we converted five different DKT models into deep option tracing (DOT) models. Performance metrics demonstrate that OT enhances KT performance and effectively identifies students’ future option selection tendencies. Furthermore, using the best OT model, we extracted students’ problem-solving sequence features and learning gains to analyze learning patterns. The results reveal that for beginners in middle school mathematics, consecutive errors in the same skill might lead to greater learning gains. Finally, we applied network analysis to reveal connections between skills based on students’ error tendencies. Our work contributes to KT methods and related empirical findings in learning analytics (LA) for knowledge assessment.
References
Abdelrahman, G., & Wang, Q. (2019). Knowledge tracing with sequential key-value memory networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), 21–25 July 2019, Paris, France (pp. 175–184). ACM. https://doi.org/10.1145/3331184.3331195
Abdelrahman, G., & Wang, Q. (2023). Learning data teaching strategies via knowledge tracing. Knowledge-Based Systems, 269, 110511. https://doi.org/10.1016/j.knosys.2023.110511
Abdelrahman, G., Wang, Q., & Nunes, B. (2023). Knowledge tracing: A survey. ACM Computing Surveys, 55(11), article number 93. https://doi.org/10.1145/3578362
Abida, K., Azeem, M., & Gondal, M. B. (2011). Assessing students’ math proficiency using multiple-choice and short constructed response item formats. International Journal of Technology, Knowledge and Society, 7(3), 135. https://doi.org/10.18848/1832-3669/CGP/v07i03/56206
Adams, D. M., McLaren, B. M., Durkin, K., Mayer, R. E., Rittle-Johnson, B., Isotani, S., & Van Velsen, M. (2014). Using erroneous examples to improve mathematics learning with a web-based tutoring system. Computers in Human Behavior, 36, 401–411. https://doi.org/10.1016/j.chb.2014.03.053
Adler, P. S., & Clark, K. B. (1991). Behind the learning curve: A sketch of the learning process. Management Science, 37(3), 267–281. https://doi.org/10.1287/mnsc.37.3.267
An, S., Kim, J., Kim, M., & Park, J. (2022). No task left behind: Multi-task learning of knowledge tracing and option tracing for better student assessment. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4), 4424–4431. https://doi.org/10.1609/aaai.v36i4.20364
Biber, C., Tuna, A., & Korkmaz, S. (2013). The mistakes and the misconceptions of the eighth grade students on the subject of angles. European Journal of Science and Mathematics Education, 1(2), 50–59. https://doi.org/10.30935/scimath/9387
Borasi, R. (1996). Reconceiving mathematics instruction: A focus on errors. Issues in curriculum theory, policy, and research series. Ablex Publishing.
Britton, S. (2002). Are students able to transfer mathematical knowledge? Second International Conference on the Teaching of Mathematics, 1–6 July 2002, Hersonissos, Crete, Greece. http://users.math.uoc.gr/∼ictm2/Proceedings/pap267.pdf
Cai, D., Zhang, Y., & Dai, B. (2019). Learning path recommendation based on knowledge tracing model and reinforcement learning. In Proceedings of the 2019 IEEE 5th International Conference on Computer and Communications (ICCC 2019), 6–9 December 2019, Chengdu, China (pp. 1881–1885). IEEE. https://doi.org/10.1109/ICCC47050.2019.9064104
Chen, Y., Liu, Q., Huang, Z., Wu, L., Chen, E., Wu, R., Su, Y., & Hu, G. (2017). Tracking knowledge proficiency of students with educational priors. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM 2017), 6–10 November 2017, Singapore (pp. 989–998). ACM. https://doi.org/10.1145/3132847.3132929
Chi, M. T. H. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. The Journal of the Learning Sciences, 14(2), 161–199. https://doi.org/10.1207/s15327809jls1402_1
Coderre, S. P., Harasym, P., Mandin, H., & Fick, G. (2004). The impact of two multiple-choice question formats on the problem-solving strategies used by novices and experts. BMC Medical Education, 4, article number 23. https//doi.org/10.1186/1472-6920-4-23
Corbett, A. T., & Anderson, J. R. (1995). Knowledge decomposition and subgoal reification in the ACT programming tutor. In J. Greer (Ed.), Artificial Intelligence and Education, 1995: The Proceedings of World Conference on Artificial Intelligence in Education (AI-ED 1995), 16–19 August 1995, Washington, DC, USA. AACE.
Crooks, N. M., & Alibali, M.W. (2014). Defining and measuring conceptual knowledge in mathematics. Developmental Review, 34(4), 344–377. https://doi.org/10.1016/j.dr.2014.10.001
Cunningham, A. E., Perry, K. E., Stanovich, K. E., & Stanovich, P. J. (2004). Disciplinary knowledge of K–3 teachers and their knowledge calibration in the domain of early literacy. Annals of Dyslexia, 54, 139–167. https://doi.org/10.1007/s11881-004-0007-y
Curtis, D. A., Heller, J. I., Clarke, C., Rabe-Hesketh, S., & Ramirez, A. (2009). The impact of Math Pathways and Pitfalls on students’ mathematics achievement. In Proceedings of the Annual Meeting of the American Educational Research Association, 13–17 April 2009, San Diego, CA. AERA.
DiNapoli, J., & Miller, E. K. (2022). Recognizing, supporting, and improving student perseverance in mathematical problemsolving: The role of conceptual thinking scaffolds. The Journal of Mathematical Behavior, 66, 100965. https://doi.org/10.1016/j.jmathb.2022.100965
Dixon, R. A., & Brown, R. A. (2012). Transfer of learning: Connecting concepts during problem solving. Journal of Technology Education, 24(1), 2–17. https://doi.org/10.21061/jte.v24i1.a.1
Du, H., Xing, W., & Zhang, Y. (2022). Misconception of abstraction: When to use an example and when to use a variable? In J. Vahrenhold, K. F. ad Matthias Hauswirth, & D. Franklin (Eds.), Proceedings of the 2022 ACM Conference on International Computing Education Research (ICER 2022), 7–11 August 2022, Lugano, Switzerland, and online (pp. 28–29, Vol. 2). ACM. https://doi.org/10.1145/3501709.3544276
Durkin, K., & Rittle-Johnson, B. (2012). The effectiveness of using incorrect examples to support learning about decimal magnitude. Learning and Instruction, 22(3), 206–214. https://doi.org/10.1016/j.learninstruc.2011.11.001
Ebbinghaus, H. (2013). Memory: A contribution to experimental psychology. Annals of Neurosciences, 20(4), 155. https://europepmc.org/article/MED/25206041
Farr, M. J. (1987). The long-term retention of knowledge and skills: A cognitive and instructional perspective. Springer Verlag.
Ghosh, A., Heffernan, N., & Lan, A. S. (2020). Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2020), 6–10 July 2020, online (pp. 2330–2339). ACM. https://doi.org/10.1145/3394486.3403282
Ghosh, A., Raspat, J., & Lan, A. (2021). Option tracing: Beyond correctness analysis in knowledge tracing. In I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, & V. Dimitrova (Eds.), International conference on artificial intelligence in education. AIED 2021. Lecture notes in computer science (pp. 137–149, Vol. 12748). Springer. https://doi.org/10.1007/978-3-030-78292-4_12
Gong, Y., Beck, J. E., & Heffernan, N. T. (2010). Comparing knowledge tracing and performance factor analysis by using multiple model fitting procedures. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems. ITS 2010. Lecture notes in computer science (pp. 35–44, Vol. 6094). Springer. https://doi.org/10.1007/978-3-642-13388-6_8
Große, C. S., & Renkl, A. (2007). Finding and fixing errors in worked examples: Can this foster learning outcomes? Learning and Instruction, 17(6), 612–634. https://doi.org/10.1016/j.learninstruc.2007.09.008
Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge. https://doi.org/10.4324/9780203887332
Hayati, R., & Setyaningrum, W. (2019). Identification of misconceptions in middle school mathematics utilizing certainty of response index. Journal of Physics: Conference Series, 1320(1), 012041. https://doi.org/10.1088/1742-6596/1320/1/012041
Huang, S., Liu, Z., Zhao, X., Luo, W., & Weng, J. (2023). Towards robust knowledge tracing models via k-sparse attention. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023), 23–27 July 2023, Taipei, Taiwan (pp. 2441–2445). ACM. https://doi.org/10.1145/3539618.3592073
Huang, Y., Hollstein, J. D. G., & Brusilovsky, P. (2016). Modeling skill combination patterns for deeper knowledge tracing. In Late-Breaking Results, Posters, Demos, Doctoral Consortium and Workshops Proceedings of the 24th ACM Conference on User Modeling, Adaptation and Personalisation (UMAP 2016), 13–16 July 2016, Halifax, Nova Scotia, Canada. https://ceur-ws.org/Vol-1618/PALE4.pdf
Im, Y., Choi, E., Kook, H., & Lee, J. (2023). Forgetting-aware linear bias for attentive knowledge tracing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM 2023), 21–25 October 2023, Birmingham, UK (pp. 3958–3962). ACM. https://doi.org/10.1145/3583780.3615191
Isotani, S., McLaren, B. M., & Altman, M. (2010). Towards intelligent tutoring with erroneous examples: A taxonomy of decimal misconceptions. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems. ITS 2010. Lecture notes in computer science (pp. 346–348, Vol. 6095). Springer. https://doi.org/10.1007/978-3-642-13437-1_66
Kramarski, B. (2004). Making sense of graphs: Does metacognitive instruction make a difference on students’ mathematical conceptions and alternative conceptions? Learning and Instruction, 14(6), 593–619. https://doi.org/10.1016/j.learninstruc.2004.09.003
Li, H., Guo, R., Li, C., & Xing, W. (2024). Automated quality assessment of multimodal mathematical stories generated by generative artificial intelligence. In Proceedings of the Eleventh ACM Conference on Learning@ Scale (L@S 2024), 18–20 July 2024, Atlanta, Georgia, USA (pp. 110–121). ACM. https://doi.org/10.1145/3657604.3662029
Li, H., Li, C., Xing, W., Baral, S., & Heffernan, N. (2024). Automated feedback for student math responses based on multimodality and fine-tuning. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK 2014), 24–28 March 2014, Indianapolis, Indiana, USA (pp. 763–770). ACM. https://doi.org/10.1145/3636555.3636860
Li, H., Xing, W., Li, C., Zhu, W., & Heffernan, N. (2024). Positive affective feedback mechanisms in an online mathematics learning platform. In Proceedings of the Eleventh ACM Conference on Learning@ Scale (L@S 2024), 18–20 July 2024, Atlanta, Georgia, USA (pp. 371–375). ACM. https://doi.org/10.1145/3657604.3664666
Li, H., Xing, W., Li, C., Zhu, W., & Oh, H. (2024). Are simpler math stories better? Automatic readability assessment of GAI-generated multimodal mathematical stories validated by engagement. British Journal of Educational Technology. https://doi.org/10.1111/bjet.13554
Liu, Z., Liu, Q., Chen, J., Huang, S., Gao, B., Luo, W., & Weng, J. (2023). Enhancing deep knowledge tracing with auxiliary tasks. In Y. Ding, J. Tang, J. Sequeda, L. Aroyo, C. Castillo, & G.-J. Houben (Eds.), Proceedings of the ACM Web Conference 2023 (WWW 2023), 30 April–4 May 2023, Austin, Texas, USA (pp. 4178–4187). ACM. https://doi.org/10.1145/3543507.3583866
Liu, Z., Liu, Q., Chen, J., Huang, S., & Luo, W. (2023). SimpleKT: A simple but tough-to-beat baseline for knowledge tracing. arXiv preprint arXiv:2302.06881. https://doi.org/10.48550/arXiv.2302.06881
Lu, Y., Wang, D., Chen, P., Meng, Q., & Yu, S. (2023). Interpreting deep learning models for knowledge tracing. International Journal of Artificial Intelligence in Education, 33(3), 519–542. https://doi.org/10.1007/s40593-022-00297-z
Lyu, B., Li, C., Li, H., & Xing, W. (2024). Roles of joining time, technology use, and social interaction in sustaining student participation in an online mathematics discussion board. In Proceedings of the Eleventh ACM Conference on Learning@ Scale (L@S 2024), 18–20 July 2024, Atlanta, Georgia, USA (pp. 398–402). ACM. https://doi.org/10.1145/3657604.3664672
McLaren, B. M., Adams, D., Durkin, K., Goguadze, G., Mayer, R. E., Rittle-Johnson, B., Sosnovsky, S., Isotani, S., & Van Velsen, M. (2012). To err is human, to explain and correct is divine: A study of interactive erroneous examples with middle school math students. In A. Ravenscroft, S. Lindstaedt, C. Delgado Kloos, & D. Hern´andez-Leo (Eds.), 21st century learning for 21st century skills. EC-TEL 2012. Lecture notes in computer science (pp. 222–235, Vol. 7563). Springer. https://doi.org/10.1007/978-3-642-33263-0 18
McLaren, B. M., Adams, D. M., & Mayer, R. E. (2015). Delayed learning effects with erroneous examples: A study of learning decimals with a web-based tutor. International Journal of Artificial Intelligence in Education, 25(4), 520–542. https://doi.org/10.1007/s40593-015-0064-x
Mishra, L. (2020). Conception and misconception in teaching arithmetic at primary level. Journal of Critical Reviews, 7(5), 936–939. https://www.researchgate.net/publication/344066409_CONCEPTION_AND_MISCONCEPTION_IN_TEACHING_ARITHMETIC_AT_PRIMARY_LEVEL
Ndemo, Z., & Ndemo, O. (2018). Secondary school students’ errors and misconceptions in learning algebra. Journal of Education and Learning, 12(4), 690–701. https://doi.org/10.11591/edulearn.v12i4.9556
Pelanek, R. (2017). Bayesian knowledge tracing, logistic models, and beyond: An overview of learner modeling techniques. User Modeling and User-Adapted Interaction, 27, 313–350. https://doi.org/10.1007/s11257-017-9193-2
Pu, Y., Wu, W., Han, Y., & Chen, D. (2018). Parallelizing Bayesian knowledge tracing tool for large-scale online learning analytics. In Proceedings of the 2018 IEEE International Conference on Big Data (Big Data), 10–13 December 2018, Seattle, Washington, USA (pp. 3245–3254). IEEE. https://doi.org/10.1109/BigData.2018.8622355
Rakes, C. R., & Ronau, R. N. (2019). Rethinking mathematics misconceptions: Using knowledge structures to explain systematic errors within and across content domains. International Journal of Research in Education and Science, 5(1), 1-21. https://www.ijres.net/index.php/ijres/article/view/482
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. arXiv preprint arXiv:1908.10084. https://doi.org/10.48550/arXiv.1908.10084
Resnick, L. B., Nesher, P., Leonard, F., Magone, M., Omanson, S., & Peled, I. (1989). Conceptual bases of arithmetic errors: The case of decimal fractions. Journal for Research in Mathematics Education, 20(1), 8–27. https://doi.org/10.2307/749095
Rochmad, R., Kharis, M., Agoestanto, A., Zahid, M. Z., & Mashuri, M. (2018). Misconception as a critical and creative thinking inhibitor for mathematics education students. Unnes Journal of Mathematics Education, 7(1), 57–62. https://doi.org/10.15294/ujme.v7i1.18078
Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of practice: Common principles in three paradigms suggest new concepts for training. Psychological Science, 3(4), 207–218. https://doi.org/10.1111/j.1467-9280.1992.tb00029.x
Scruggs, R., Baker, R. S., & McLaren, B. M. (2019). Extending deep knowledge tracing: Inferring interpretable knowledge and predicting post-system performance. arXiv preprint arXiv:1910.12597. https://doi.org/10.48550/arXiv.1910.12597
Shen, S., Chen, E., Liu, Q., Huang, Z., Huang, W., Yin, Y., Su, Y., & Wang, S. (2022). Monitoring student progress for learning process-consistent knowledge tracing. IEEE Transactions on Knowledge and Data Engineering, 35(8), 8213–8227. https://doi.org/10.1109/TKDE.2022.3221985
Siegler, R. S. (2002). Microgenetic studies of self-explanation. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning (pp. 31–58). Cambridge University Press.
Stacey, K., Helme, S., Steinle, V., Baturo, A., Irwin, K., & Bana, J. (2001). Preservice teachers’ knowledge of difficulties in decimal numeration. Journal of Mathematics Teacher Education, 4, 205–225. https://doi.org/10.1023/A:1011463205491
Stafylidou, S., & Vosniadou, S. (2004). The development of students’ understanding of the numerical value of fractions. Learning and Instruction, 14(5), 503–518. https://doi.org/10.1016/j.learninstruc.2004.06.015
Stark, R., Kopp, V., & Fischer, M. R. (2011). Case-based learning with worked examples in complex domains: Two experimental studies in undergraduate medical education. Learning and Instruction, 21(1), 22–33. https://doi.org/10.1016/j.learninstruc.2009.10.001
Stout, W., Henson, R., & DiBello, L. (2023). Optimal classification methods for diagnosing latent skills and misconceptions for option-scored multiple-choice item quizzes. Behaviormetrika, 50(1), 177–215. https://doi.org/10.1007/s41237-022-00172-0
Thorndike, E. L., &Woodworth, R. S. (1901). The influence of improvement in one mental function upon the efficiency of other functions. II. The estimation of magnitudes. Psychological Review, 8(4), 553–564. https://doi.org/10.1037/h0071363
Tsovaltzi, D., Melis, E., McLaren, B. M., Meyer, A.-K., Dietrich, M., & Goguadze, G. (2010). Learning from erroneous examples: When and how do students benefit from them? In M. Wolpers, P. Kirschner, M. Scheffel, S. Lindstaedt, & V. Dimitrova (Eds.), Sustaining TEL: From innovation to learning and practice. EC-TEL 2010. Lecture notes in computer science (pp. 357–373, Vol. 6383). Springer. https://doi.org/10.1007/978-3-642-16020-2_24
Turkdogan, A., Baki, A., & Cepni, S. (2013). The anatomy of mistakes: Categorizing students’ mistakes in mathematics within learning theories. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 1(1), 13–26. https://dergipark.org.tr/en/pub/turkbilmat/issue/21560/231415
Van Den Broek, P., & Kendeou, P. (2008). Cognitive processes in comprehension of science texts: The role of co-activation in confronting misconceptions. Applied Cognitive Psychology, 22(3), 335–351. https://doi.org/10.1002/acp.1418
VanLehn, K. (1999). Rule-learning events in the acquisition of a complex skill: An evaluation of Cascade. The Journal of the Learning Sciences, 8(1), 71–125. https://doi.org/10.1207/s15327809jls0801_3
Vosniadou, S. (1994). Capturing and modeling the process of conceptual change. Learning and Instruction, 4(1), 45–69. https://doi.org/10.1016/0959-4752(94)90018-3
Webb, N. L. (2002). Depth-of-knowledge levels for four content areas. Language Arts, 28(March), 1–9. http://ossucurr.pbworks.com/w/file/fetch/49691156/Norm%20web%20dok%20by%20subject%20area.pdf
Yang, S., Liu, X., Su, H., Zhu, M., & Lu, X. (2022). Deep knowledge tracing with learning curves. In K. S. Candan, T. N. Dinh, M. T. Thai, & T. Washio (Eds.), Proceedings of the 2022 IEEE International Conference on Data Mining Workshops (ICDMW 2022), 28 November–1 December 2022, Orlando, Florida (pp. 282–291). IEEE. https://doi.org/10.1109/ICDMW58026.2022.00046
Yeung, C.-K. (2019). Deep-IRT: Make deep learning based knowledge tracing explainable using item response theory. arXiv preprint arXiv:1904.11738. https://doi.org/10.48550/arXiv.1904.11738
Yudelson, M. V., Koedinger, K. R., & Gordon, G. J. (2013). Individualized Bayesian knowledge tracing models. In H. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Artificial intelligence in education. AIED 2013. Lecture notes in computer science (pp. 171–180, Vol. 7926). Springer. https://doi.org/10.1007/978-3-642-39112-5_18
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
