Leveraging Process-Action Epistemic Network Analysis to Illuminate Student Self-Regulated Learning with a Socratic Chatbot
DOI:
https://doi.org/10.18608/jla.2025.8549Keywords:
self-regulated learning, learning analytics, epistemic network analysis, ordered network analysis, generative artificial intelligence, chatbot, research paperAbstract
The growing use of generative AI (GenAI) has sparked discussions regarding integrating these tools into educational settings to enrich the learning experience of teachers and students. Self-regulated learning (SRL) research is pivotal in addressing this inquiry. One prevalent manifestation of GenAI is the large-language model (LLM) chatbot, enabling users to seek information and assistance. This paper aims to showcase how data on student interaction with a chatbot can be used in learning analytics to gain insights into SRL. This is achieved by adapting existing SRL frameworks to comprehend 34 students’ interaction with an educational Socratic chatbot for a statistics class at the introductory undergraduate level. Chatbot conversations from students are categorized into learning actions and processes using the framework’s process-action library. Thereafter, we analyze this data through ordered epistemic network analysis, furnishing valuable insights into how different students interact with the chatbot. Our findings reveal that higher-scoring students engage more frequently in reflective and evaluative activities, while lower-scoring students focus on searching for answers. Furthermore, students should shift from structured problem-solving, such as solving classroom questions, to questioning fundamental concepts with the chatbot and soliciting more examples to improve their learning gains.
References
Artstein, R. (2017). Inter-annotator agreement. In N. Ide & J. Pustejovsky (Eds.), Handbook of linguistic annotation (pp. 297-313). Springer Netherlands. https://doi.org/10.1007/978-94-024-0881-2 11 Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555–596. https://doi.org/10.1162/coli.07-034-r2
Azevedo, R., Guthrie, J. T., & Seibert, D. (2004). The role of self-regulated learning in fostering students’ conceptual understanding of complex systems with hypermedia. Journal of Educational Computing Research, 30(1–2), 87–111. https://doi.org/10.2190/dvwx-gm1t-6thq-5wc7
Azevedo, R., Moos, D. C., Greene, J. A., Winters, F. I., & Cromley, J. G. (2007). Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development, 56(1), 45–72. https://doi.org/10.1007/s11423-007-9067-0
Bakhtiar, A., & Hadwin, A. (2020). Dynamic interplay between modes of regulation during motivationally challenging episodes in collaboration. Frontline Learning Research, 8(2), 1–34. https://doi.org/10.14786/flr.v8i2.561
Bandura, A. (2002). Social foundations of thought and action. In D. F. Marks (Ed.), The health psychology reader (pp. 94-106). SAGE Publications Ltd. https://doi.org/10.4135/9781446221129.n6
Biwer, F., Wiradhany, W., oude Egbrink, M., Hospers, H., Wasenitz, S., Jansen, W., & de Bruin, A. (2021). Changes and adaptations: How university students self-regulate their online learning during the COVID-19 pandemic. Frontiers in Psychology, 12, 642593. https://doi.org/10.3389/fpsyg.2021.642593
Bowman, D., Swiecki, Z., Cai, Z., Wang, Y., Eagan, B., Linderoth, J., & Shaffer, D. W. (2021). The mathematical foundations of epistemic network analysis. In A. R. Ruis & S. B. Lee (Eds.), Advances in quantitative ethnography (pp. 91–105). Springer International Publishing. https://doi.org/10.1007/978-3-030-67788-6_7
Brenner, C. A. (2022). Self-regulated learning, self-determination theory and teacher candidates’ development of competency-based teaching practices. Smart Learning Environments, 9(3). https://doi.org/10.1186/s40561-021-00184-5
Bushuyev, S., Bushuyeva, N., Murzabekova, S., & Khussainova, M. (2023). Innovative development of educational systems in the BANI environment. Scientific Journal of Astana IT University, 14(14), 104–115. https://doi.org/10.37943/14ynsz2227
Cheong, K. H., Lai, J. W., Yap, J. H., Cheong, G. S. W., Budiman, S. V., Ortiz, O., Mishra, A., & Yeo, D. J. (2023). Utilizing Google Cardboard virtual reality for visualization in multivariable calculus. IEEE Access, 11, 75398–75406. https://doi.org/10.1109/access.2023.3281753
Chihab, L., El Mhouti, A., Massar, M., & Hamdane, K. (2023). Learning analytics and big data: Huge potential to improve online education. In S. Motahhir & B. Bossoufi (Eds.), Digital technologies and applications. ICDTA 2023. Lecture notes in networks and systems (pp. 405–411, Vol. 668). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-29857-8 41
Chocarro, R., Cortinas, M., & Marcos-Mat ´as, G. (2021). Teachers’ attitudes towards chatbots in education: A technology acceptance model approach considering the effect of social language, bot proactiveness, and users’ characteristics. Educational Studies, 49(2), 295–313. https://doi.org/10.1080/03055698.2020.1850426
Chung, M. -K. (2000). The development of self-regulated learning. Asia Pacific Education Review, 1(1), 55–66. https://doi.org/10.1007/bf03026146
Clemons, M. L., & Hopkins, T. (2020). Facilitating success: Using self-regulated learning and servant leadership in the college classroom. Journal of Teaching and Learning with Technology, 9(1), 127–132. https://doi.org/10.14434/jotlt.v9i1.29176
Daniela, P. (2015). The relationship between self-regulation, motivation and performance at secondary school students. Procedia—Social and Behavioral Sciences, 191, 2549–2553. https://doi.org/10.1016/j.sbspro.2015.04.410 Dilmegani, C. (2023). Top 6 use cases of generative AI in education [Accessed: 13 June 2023]. https://research.aimultiple.com/generative-ai-in-education/
Dunlosky, J., & Ariel, R. (2011). Chapter four—self-regulated learning and the allocation of study time. In B. H. Ross (Ed.), Advances in research and theory (pp. 103–140, Vol. 54). Academic Press. https://doi.org/10.1016/B978-0-12-385527-5.00004-8
Edisherashvili, N., Saks, K., Pedaste, M., & Leijen, ¨A. (2022). Supporting self-regulated learning in distance learning contexts at higher education level: Systematic literature review. Frontiers in Psychology, 12, 792422. https://doi.org/10.3389/fpsyg.2021.792422
Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Ga ˇsevi ´c, D. (2025). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544
Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317. https://doi.org/10.1504/ijtel.2012.051816
Greene, J. A., & Azevedo, R. (2009). A macro-level analysis of SRL processes and their relations to the acquisition of a sophisticated mental model of a complex system. Contemporary Educational Psychology, 34(1), 18–29. https://doi.org/10.1016/j.cedpsych.2008.05.006
Hartley, K., Hayak, M., & Ko, U. H. (2024). Artificial intelligence supporting independent student learning: An evaluative case study of ChatGPT and learning to code. Education Sciences, 14(2), 120. https://doi.org/10.3390/educsci14020120
Jarvela, S., & Bannert, M. (2021). Temporal and adaptive processes of regulated learning—What can multimodal data tell Learning and Instruction, 72, 101268. https://doi.org/10.1016/j.learninstruc.2019.101268
Krippendorff, K. (1995). On the reliability of unitizing continuous data. Sociological Methodology, 25, 47–76. https://doi.org/10.2307/271061
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2022). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1), 973–1018. https://doi.org/10.1007/s10639-022-11177-3
Lai, J. W. (2024a). Adapting self-regulated learning in an age of generative artificial intelligence chatbots. Future Internet, 16(6), 218. https://doi.org/10.3390/fi16060218
Lai, J. W. (2024b). Dynamic opinion formation in networks: A multi-issue and evidence-based approach. Computers, 13(8), 190. https://doi.org/10.3390/computers13080190
Lai, J. W., & Cheong, K. H. (2024). Winning with losses: The surprising success of negative strategies in social interaction behavior. Physical Review Letters, 133(16), 167401. https://doi.org/10.1103/physrevlett.133.167401
Li, Y., Sha, L., Yan, L., Lin, J., Rakovi ´c, M., Galbraith, K., Lyons, K., Ga ˇsevi ´c, D., & Chen, G. (2023). Can large language models write reflectively. Computers and Education: Artificial Intelligence, 4, 100140. https://doi.org/10.1016/j.caeai.2023.100140
Lim, Z. Y., Munshi, T., Yap, J. H., Mishra, A., Lai, J. W., Yeo, D. J., & Cheong, K. H. (2023). Using opinion dynamics to identify groups for targeted intervention in lifelong learning: A case study of SkillsFuture in Singapore. International Journal of Modern Physics B, 38(20), 2450257. https://doi.org/10.1142/s0217979224502576
Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 31–40. https://er.educause.edu/-/media/files/article-downloads/erm1151.pdf
Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 18(1), 50–60. https://doi.org/10.1214/aoms/1177730491
Meng, J., & Dai, Y. ( (2021). Emotional support from AI chatbots: Should a supportive partner self-disclose or not? Journal of Computer-Mediated Communication, 26(4), 207–222. https://doi.org/10.1093/jcmc/zmab005
Molenaar, I. (2021). Personalisation of learning: Towards hybrid human-AI learning technologies. In A. Schleicher (Ed.), OECD digital education outlook 2021. OECD. https://doi.org/10.1787/2cc25e37-en
Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. https://doi.org/10.3389/fpsyg.2017.00422
Panadero, E., Klug, J., & Jarvela, S. (2015). Third wave of measurement in the self-regulated learning field: When measurement and intervention come hand in hand. Scandinavian Journal of Educational Research, 60(6), 723–735. https://doi.org/10.1080/00313831.2015.1066436
Pintrich, P. R. (2000). Chapter 14—the role of goal orientation in self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). Academic Press. https://doi.org/10.1016/b978-012109890-2/50043-3
Qiu, W., Khong, A. W. H., Supraja, S., & Tang, W. (2024). A dual-mode grade prediction architecture for identifying at-risk students. IEEE Transactions on Learning Technologies, 17, 803–814. https://doi.org/10.1109/tlt.2023.3333029
Qiu, W., Su, C. L., Jamil, N. B., Ng, S. S., Chen, C.-M., & Lim, F. S. (2024). “I am here to guide you”: A detailed examination of late 2023 Gen-AI tutors capabilities in stepwise tutoring in an undergraduate statistics course. In Proceedings of the 18th International Technology, Education and Development Conference (INTED2024), 4–6 March 2024, Valencia, Spain (pp. 3761–3770). IATED. https://doi.org/10.21125/inted.2024.0984
Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nulty, A., Bagley, E., Frank, K., Rupp, A. A., & Mislevy, R. (2009). Epistemic network analysis: A prototype for 21st-century assessment of learning. International Journal of Learning and Media, 1(2), 33–53. https://doi.org/10.1162/ijlm.2009.0013
She, C., Liang, Q., Jiang, W., & Xing, Q. (2023). Learning adaptability facilitates self-regulated learning at school: The chain mediating roles of academic motivation and self-management. Frontiers in Psychology, 14, 1162072. https://doi.org/10.3389/fpsyg.2023.1162072
Siebert-Evenstone, A. L., Arastoopour Irgens, G., Collier, W., Swiecki, Z., Ruis, A. R., & Williamson Shaffer, D. (2017). In search of conversational grain size: Modeling semantic structure using moving stanza windows. Journal of Learning Analytics, 4(3), 123–139. https://doi.org/10.18608/jla.2017.43.7
Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. https://doi.org/10.1177/0002764213498851
Stoeger, H., & Ziegler, A. (2010). Do pupils with differing cognitive abilities benefit similarly from a self-regulated learning training program? Gifted Education International, 26(1), 110–123. https://doi.org/10.1177/026142941002600113
Tan, Y., Ruis, A., Marquart, C., Cai, Z., Knowles, M., & Shaffer, D. (2023). Ordered network analysis. In C. Dams¸ a & A. Barany (Eds.), Advances in quantitative ethnography. ICQE 2022. Communications in computer and information science (pp. 108–120, Vol. 1785). Springer. https://doi.org/10.1007/978-3-031-31726-2_8
Tegos, S., Demetriadis, S., Papadopoulos, P. M., & Weinberger, A. (2016). Conversational agents for academically productive talk: A comparison of directed and undirected agent interventions. International Journal of Computer-Supported Collaborative Learning, 11(4), 417–440. https://doi.org/10.1007/s11412-016-9246-2
Tsai, Y. - S., Perrotta, C., & Gasevic, D. (2019). Empowering learners with personalised learning approaches? Agency, equity and transparency in the context of learning analytics. Assessment & Evaluation in Higher Education, 45(4), 554–567. https://doi.org/10.1080/02602938.2019.1676396
Williamson, G. (2015). Self-regulated learning: An overview of metacognition, motivation and behaviour. Journal of Initial Teacher Inquiry, 1, 25–27. https://doi.org/10.26021/851
Winne, P. H. (2011). A cognitive and metacognitive analysis of self-regulated learning. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 15–32). Routledge. https://doi.org/10.4324/9780203839010
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. Hacker, J. Dunlosky, & A. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Taylor & Francis. https://psycnet.apa.org/record/1998-07283-011
Wisconsin Center for Education Research. (2024). ENA [Accessed: 13 April 2024]. https://www.epistemicnetwork.org/
Wollny, S., Schneider, J., Mitri, D. D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are we there yet? A systematic literature review on chatbots in education. Frontiers in Artificial Intelligence, 4, 654924. https://doi.org/10.3389/frai.2021.654924
Wright, D. B. (2020). Gain scores, ANCOVA, and propensity matching procedures for evaluating treatments in education. Open Education Studies, 2(1), 45–65. https://doi.org/10.1515/edu-2020-0107
Zhai, C., Wibowo, S., & Li, L. D. (2024). The effects of over-reliance on AI dialogue systems on students’ cognitive abilities: A systematic review. Smart Learning Environments, 11(1). https://doi.org/10.1186/s40561-024-00316-7
Zhang, Y., Tian, Y., Yao, L., Duan, C., Sun, X., & Niu, G. (2022). Individual differences matter in the effect of teaching presence on perceived learning: From the social cognitive perspective of self-regulated learning. Computers & Education, 179, 104427. https://doi.org/10.1016/j.compedu.2021.104427
Zimmerman, B. J. (1989). A social cognitive view of self-regulated academic learning. Journal of Educational Psychology, 81(3), 329–339. https://doi.org/10.1037/0022-0663.81.3.329
Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1), 166–183. https://doi.org/10.3102/0002831207312909
Zimmerman, B. J. (2015). Self-regulated learning: Theories, measures, and outcomes. In J. D. Wright (Ed.), International encyclopedia of the social & behavioral sciences (2nd edition, pp. 541–546). Elsevier. https://doi.org/10.1016/B978-0-08-097086-8.26060-1
Zimmerman, B. J., & Campillo, M. (2003). Motivating self-regulated problem solvers. In J. E. Davidson & R. J. Sternberg (Eds.), The psychology of problem solving (pp. 233–262). Cambridge University Press. https://doi.org/10.1017/CBO9780511615771.009
Zimmerman, B. J., & Pons, M. M. (1986). Development of a structured interview for assessing student use of self-regulated learning strategies. American Educational Research Journal, 23(4), 614–628. https://doi.org/10.3102/00028312023004614
Zimmerman, B. J., & Schunk, D. H. (Eds.). (2011). Handbook of self-regulation of learning and performance. Routledge. https://doi.org/10.4324/9780203839010
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
