Exploring the Potential of Generative AI to Support Non-experts in Learning Analytics Practice
DOI:
https://doi.org/10.18608/jla.2025.8615Keywords:
genAI, scaling learning analytics, data analysis expertise, research paperAbstract
Generative AI (GenAI) has the potential to revolutionize the analysis of educational data, significantly impacting learning analytics (LA). This study explores the capability of non-experts, including administrators, instructors, and students, to effectively use GenAI for descriptive LA tasks without requiring specialized knowledge in data processing, visualization, or programming. Through a laboratory experiment, participants with varying levels of expertise in data analysis engaged in three tasks with different levels of difficulty using ChatGPT. The findings reveal that while there is a small effect of previous expertise on performance, novices and experts achieved remarkably similar scores. Additionally, the study identifies that action sequence variables, such as the sequence’s complexity and the presence of specific actions such as evaluating and checking results, significantly predict performance. These results suggest that while current GenAI technologies are not yet ready to fully support non-experts, they hold the promise of supporting stakeholders, regardless of their technical background, to perform descriptive data analysis in the context of LA practice. This research seeks to start a discussion within the LA community about leveraging AI to scale and expand LA practices, potentially transforming how educational stakeholders engage with and benefit from LA.
References
Ahn, S. (2024). Data science through natural language with ChatGPT’s Code Interpreter. Translational and Clinical Pharmacology, 32(2), 73–82. https://doi.org/10.12793/tcp.2024.32.e8
Campos, F. C., Ahn, J., DiGiacomo, D. K., Nguyen, H., & Hays, M. (2021). Making sense of sensemaking: Understanding how K–12 teachers and coaches react to visual analytics. Journal of Learning Analytics, 8(3), 60–80. https://doi.org/10.18608/jla.2021.7113
Carlisle, S. (2018). Software: Tableau and Microsoft Power BI. Technology∥Architecture+Design, 2(2), 256–259. https://doi.org/10.1080/24751448.2018.1497381
Castiglioni, I., Rundo, L., Codari, M., Di Leo, G., Salvatore, C., Interlenghi, M., Gallivanone, F., Cozzi, A., D’Amico, N. C., & Sardanelli, F. (2021). AI applications to medical images: From machine learning to deep learning. Physica Medica, 83, 9–24. https://doi.org/10.1016/j.ejmp.2021.02.006
Cecchini, D., Nazir, A., Chakravarthy, K., & Kocaman, V. (2024). Holistic evaluation of large language models: Assessing robustness, accuracy, and toxicity for real-world applications. In A. Ovalle, K.-W. Chang, Y. T. Cao, N. Mehrabi, J. Zhao, A. Galstyan, J. Dhamala, A. Kumar, & R. Gupta (Eds.), Proceedings of the Fourth Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), 21 June 2024, Mexico City, Mexico (pp. 109–117). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.trustnlp-1.11
Ceruzzi, P. (1996). From scientific instrument to everyday appliance: The emergence of personal computers, 1970–77. History and Technology, an International Journal, 13(1), 1–31. https://doi.org/10.1080/07341519608581893
Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In Proceedings of the Second International Conference on Learning Analytics and Knowledge (LAK 2012), 29 April–2 May 2012, Vancouver, British Columbia, Canada (pp. 134–138). ACM. https://doi.org/10.1145/2330601.2330636
Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213. https://doi.org/10.1037/h0026256
Daele, S. V., & Janssenswillen, G. (2022). Identifying the steps in an exploratory data analysis: A process-oriented approach. In M. Montali, A. Senderovich, & M. Weidlich (Eds.), Process mining workshops. ICPM 2022. Lecture notes in business information processing (pp. 526–538, Vol. 468). Springer. https://doi.org/10.1007/978-3-031-27815-0 38
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024). Impact of AI assistance on student agency. Computers & Education, 210, 104967. https://doi.org/10.1016/j.compedu.2023.104967
Dourado, R. A., Rodrigues, R. L., Ferreira, N., Mello, R. F., Gomes, A. S., & Verbert, K. (2021). A teacher-facing learning analytics dashboard for process-oriented feedback in online learning. In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK 2021), 12–16 April 2021, Irvine, California, USA (pp. 482–489). ACM. https://doi.org/10.1145/3448139.3448187
Echeverria, V., Martinez-Maldonado, R., Buckingham Shum, S., Chiluiza, K., Granda, R., & Conati, C. (2018). Exploratory versus explanatory visual learning analytics: Driving teachers’ attention through educational data storytelling. Journal of Learning Analytics, 5(3), 73–97. https://doi.org/10.18608/jla.2018.53.6
Echeverria, V., Yan, L., Zhao, L., Abel, S., Alfredo, R., Dix, S., Jaggard, H., Wotherspoon, R., Osborne, A., Buckingham Shum, S., Gašević, D., & Martinez-Maldonado, R. (2024). Teamslides: A multimodal teamwork analytics dashboard for teacher-guided reflection in a physical learning space. In Proceedings of the 14th International Conference on Learning Analytics and Knowledge (LAK 2024), 18–22 March 2024, Tokyo, Japan (pp. 112–122). ACM. https://doi.org/10.1145/3636555.3636857
Estrellado, R., Freer, E., Rosenberg, J., & Velasquez, I. (2020). Data science in education using R. Routledge. https://doi.org/10.4324/9780367822842
Fernandez Nieto, G. M., Kitto, K., Buckingham Shum, S., & Martinez-Maldonado, R. (2022). Beyond the learning analytics dashboard: Alternative ways to communicate student data insights combining visualisation, narrative and storytelling. In Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK 2022), 21–25 March 2022, online (pp. 219–229). ACM. https://doi.org/10.1145/3506860.3506895
Field, A. P. (2005). Kendall’s coefficient of concordance. In B. Everitt & D. Powell (Eds.), Encyclopedia of statistics in behavioral science (pp. 1010–1011, Vol. 2). Wiley. https://doi.org/10.1002/0470013192.bsa327
Flanagan, B., Wasson, B., & Gašević, D. (Eds.). (2024). Proceedings of the 14th International Conference on Learning Analytics and Knowledge (LAK 2024), 18–22 March 2024, Tokyo, Japan. ACM. https://doi.org/10.1145/3636555
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378. https://doi.org/10.1037/h0031619
Furche, T., Gottlob, G., Libkin, L., Orsi, G., & Paton, N. W. (2016). Data wrangling for big data: Challenges and opportunities. In W. Martens & T. Zeume (Eds.), Proceedings of the 19th International Conference on Extending Database Technology (ICDT 2016), 15–18 March 2016, Bordeaux, France (pp. 473–478). Schloss Dagstuhl—Leibniz-Zentrum fur Informatik. https://doi.org/10.5441/002/edbt.2016.44
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. https://doi.org/10.1136/amiajnl-2011-000089
Goswami, A., Ramakrishna, G., & Sethi, R. (2021). Review on smile detection. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 7(2), 577–583. https://doi.org/10.32628/CSEIT2172134
Gundlach, E., & Ward, M. D. (2021). The data mine: Enabling data science across the curriculum. Journal of Statistics and Data Science Education, 29(sup1), S74–S82. https://doi.org/10.1080/10691898.2020.1848484
Habib, S., Vogel, T., Anli, X., & Thorne, E. (2024). How does generative artificial intelligence impact student creativity? Journal of Creativity, 34(1), 100072. https://doi.org/10.1016/j.yjoc.2023.100072
Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018). The classroom as a dashboard: Co-designing wearable cognitive augmentation for K-12 teachers. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 7–9 March 2018, Sydney, Australia (pp. 79–88). ACM. https://doi.org/10.1145/3170358.3170377
Jaimovitch-Lopez, G., Ferri, C., Hernandez-Orallo, J., Mart´ınez-Plumed, F., & Ramırez-Quintana, M. J. (2023). Can language models automate data wrangling? Machine Learning, 112(6), 2053–2082. https://doi.org/10.1007/s10994-022-06259-9
Kaliisa, R., & Dolonen, J. A. (2023). Cada: A teacher-facing learning analytics dashboard to foster teachers’ awareness of students’ participation and discourse patterns in online discussions. Technology, Knowledge and Learning, 28(3), 937–958. https://doi.org/10.1007/s10758-022-09598-7
Khosravi, H., Viberg, O., Kovanovic, V., & Ferguson, R. (2023). Generative AI and learning analytics. Journal of Learning Analytics, 10(3), 1–6. https://doi.org/10.18608/jla.2023.8333
Klašnja-Milićević, A., Ivanović, M., & Budimac, Z. (2017). Data science in education: Big data and learning analytics. Computer Applications in Engineering Education, 25(6), 1066–1078. https://doi.org/10.1002/cae.21844
Leiner, B. M., Cerf, V. G., Clark, D. D., Kahn, R. E., Kleinrock, L., Lynch, D. C., Postel, J., Roberts, L. G., & Wolff, S. (2009). A brief history of the internet. ACM SIGCOMM Computer Communication Review, 39(5), 22–31. https://doi.org/10.1145/1629607.1629613
Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410
Maddigan, P., & Susnjak, T. (2023). Chat2VIS: Generating data visualizations via natural language using ChatGPT, Codex and GPT-3 large language models. IEEE Access, 11, 45181–45193. https://doi.org/10.1109/ACCESS.2023.3274199
Mandinach, E. B., & Gummer, E. S. (2016). What does it mean for teachers to be data literate: Laying out the skills, knowledge, and dispositions. Teaching and Teacher Education, 60, 366–376. https://doi.org/10.1016/j.tate.2016.07.011
Martinez-Maldonado, R., Echeverria, V., Fernandez Nieto, G., & Buckingham Shum, S. (2020). From data to insights: A layered storytelling approach for multimodal learning analytics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI 2020), 25–30 April 2020, Honolulu, Hawaii, USA (pp. 1–15). ACM. https://doi.org/10.1145/3313831.3376148
Martinez-Maldonado, R., Elliott, D., Axisa, C., Power, T., Echeverria, V., & Buckingham Shum, S. (2022). Designing translucent learning analytics with teachers: An elicitation process. Interactive Learning Environments, 30(6), 1077–1091. https://doi.org/10.1080/10494820.2019.1710541
Martinez-Maldonado, R., Kay, J., Yacef, K., & Schwendimann, B. (2012). An interactive teacher’s dashboard for monitoring groups in a multi-tabletop learning environment. In S. Cerri, W. Clancey, G. Papadourakis, & K. Panourgia (Eds.), Intelligent tutoring systems. ITS 2012. Lecture notes in computer science (pp. 482–492, Vol. 7315). Springer. https://doi.org/10.1007/978-3-642-30950-2 62
Marvin, G., Hellen, N., Jjingo, D., & Nakatumba-Nabende, J. (2023). Prompt engineering in large language models. In I. Jacob, S. Piramuthu, & P. Falkowski-Gilski (Eds.), Data intelligence and cognitive informatics. ICDICI 2023. Algorithms for intelligent systems (pp. 387–402). Springer. https://doi.org/10.1007/978-981-99-7962-2_30
McGrath, A. L. (2014). Content, affective, and behavioral challenges to learning: Students’ experiences learning statistics. International Journal for the Scholarship of Teaching and Learning, 8(2), 6. https://doi.org/10.20429/ijsotl.2014.080206
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://doi.org//10.11613/BM.2012.031
Mulia, A. P., Piri, P. R., & Tho, C. (2023). Usability analysis of text generation by ChatGPT OpenAI using system usability scale method. Procedia Computer Science, 227, 381–388. https://doi.org/10.1016/j.procs.2023.10.537
Ochoa, X. (2022). Multimodal learning analytics: Rationale, process, examples, and direction. In C. Lang, G. Siemens, A. Wise, D. Gasevic, & A. Merceron (Eds.), The handbook of learning analytics (pp. 54–65). SoLAR. https://doi.org/10.18608/hla22.006
Owolabi, A. T., Okunlola, O. O., Adewuyi, E. T., Idowu, J. I., & Oladapo, O. J. (2024). The advent of ChatGPT: Job made easy or job loss to data analysts. WSEAS Transactions on Computers, 23, 24–40. https://doi.org/10.37394/23205.2024.23.3
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024). Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12, 484–506. https://doi.org/10.1162/tacl_a_00660
Park, Y., & Jo, I.-H. (2019). Factors that affect the success of learning analytics dashboards. Educational Technology Research and Development, 67(6), 1547–1571. https://doi.org/10.1007/s11423-019-09693-0
Paulsen, L., & Lindsay, E. (2024). Learning analytics dashboards are increasingly becoming about learning and not just analytics—A systematic review. Education and Information Technologies, 1–30. https://doi.org/10.1007/s10639-023-12401-4
Plakandaras, V., Gogas, P., Papadimitriou, T., & Tsamardinos, I. (2022). Credit card fraud detection with automated machine learning systems. Applied Artificial Intelligence, 36(1), 2086354. https://doi.org/10.1080/08839514.2022.2086354
Pozdniakov, S., Martinez-Maldonado, R., Tsai, Y.-S., Echeverria, V., Srivastava, N., & Gasevic, D. (2023). How do teachers use dashboards enhanced with data storytelling elements according to their data visualisation literacy skills? In Proceedings of the 13th International Conference on Learning Analytics and Knowledge (LAK 2023), 13–17 March2023, Arlington, Texas, USA (pp. 89–99). ACM. https://doi.org/10.1145/3576050.3576063
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023). “It’s weird that it knows what I want”: Usability and interactions with Copilot for novice programmers. ACM Transactions on Computer-Human Interaction, 31(1), 1–31. https://doi.org/10.1145/3617367
Raja, K., & Ramathilagam, S. (2021). Washing machine using fuzzy logic controller to provide wash quality. Soft Computing, 25(15), 9957–9965. https://doi.org/10.1007/s00500-020-05477-4
Ritschard, G. (2023). Measuring the nature of individual sequences. Sociological Methods & Research, 52(4), 2016–2049. https://doi.org/10.1177/00491241211036156
Sa, D., Guimaraes, T., Abelha, A., & Santos, M. F. (2024). Low code approach for business analytics. Procedia Computer Science, 231, 421–426. https://doi.org/10.1016/j.procs.2023.12.228
Santoni de Sio, F., & Mecacci, G. (2021). Four responsibility gaps with artificial intelligence: Why they matter and how to address them. Philosophy & Technology, 34(4), 1057–1084. https://doi.org/10.1007/s13347-021-00450-x
Sarmiento, J. P., & Wise, A. F. (2022). Participatory and co-design of learning analytics: An initial review of the literature. In Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK 2022), 21–25 March 2022, online (pp. 535–541). ACM. https://doi.org/10.1145/3506860.3506910
Sharma, A. K., Sharma, D. M., Purohit, N., Rout, S. K., & Sharma, S. A. (2022). Analytics techniques: Descriptive analytics, predictive analytics, and prescriptive analytics. In P. Jeyanthi, T. Choudhury, D. Hack-Polay, T. Singh, & S. Abujar (Eds.), Decision intelligence analytics and the implementation of strategic business management. EAI/Springer innovations in communication and computing (pp. 1–14). Springer. https://doi.org/10.1007/978-3-030-82763-2_1
Shen, Y., Ai, X., Soosai Raj, A. G., Leo John, R. J., & Syamkumar, M. (2024). Implications of ChatGPT for data science education. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2024), 20–23 March 2024, Portland, Oregon, USA (pp. 1230–1236). ACM. https://doi.org/10.1145/3626252.3630874
Shreiner, T. L., & Dykes, B. (2021). Visualizing the teaching of data visualizations in social studies: A study of teachers’ data literacy practices, beliefs, and knowledge. Theory & Research in Social Education, 49(2), 262–306. https://doi.org/10.1080/00933104.2020.1850382
Singh, A., Singh, S., & Rathee, J. (2023). Basic principles of data wrangling. In M. Niranjanamurthy, K. Sheoran, G. Dhand, & P. Kaur (Eds.), Data wrangling: Concepts, applications and tools (pp. 1–18). Wiley. https://doi.org/10.1002/9781119879862.ch1
Stokel-Walker, C., & Van Noorden, R. (2023). What ChatGPT and generative AI mean for science. Nature, 614(1), 214–216. https://doi.org/10.1038/d41586-023-00340-6
Sullivan, A. (2014). Determining an inter-rater agreement metric for researchers evaluating student pathways in problem solving. [Master’s thesis, Iowa State University, Iowa]. https://doi.org/10.31274/etd-180810-2820
Toosi, A., Bottino, A. G., Saboury, B., Siegel, E., & Rahmim, A. (2021). A brief history of AI: how to prevent another winter (a critical review). PET Clinics, 16(4), 449–469. https://doi.org/10.1016/j.cpet.2021.07.001
van Leeuwen, A., Knoop-van Campen, C. A., Molenaar, I., & Rummel, N. (2021). How teacher characteristics relate to how teachers use dashboards: Results from two case studies in K-12. Journal of Learning Analytics, 8(2), 6–21. https://doi.org/10.18608/jla.2021.7325
van Leeuwen, A., Teasley, S. D., & Wise, A. F. (2022). Teacher and student facing learning analytics. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), Handbook of learning analytics (2nd edition, pp. 130–140). SoLAR. https://doi.org/10.18608/hla22.013
Verbert, K., Ochoa, X., De Croon, R., Dourado, R. A., & De Laet, T. (2020). Learning analytics dashboards: The past, the present and the future. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 35–40). ACM. https://doi.org/10.1145/3375462.3375504
Wallace, D., & Green, S. B. (2013). Analysis of repeated measures designs with linear mixed models. In D. S. Moskowitz & S. L. Hershberger (Eds.), Modeling intraindividual variability with repeated measures data (pp. 103–134). Psychology Press. https://psycnet.apa.org/record/2001-05300-005
Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK 2014), 24–28 March 2014, Indianapolis, Indiana, USA (pp. 203–211). ACM. https://doi.org/10.1145/2567574.2567588
Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://doi.org/10.18608/jla.2019.62.4
Yan, L., Martinez-Maldonado, R., & Gašević, D. (2024). Generative artificial intelligence in learning analytics: Contextualising opportunities and challenges through the learning analytics cycle. In Proceedings of the 14th International Conference on Learning Analytics and Knowledge (LAK 2024), 18–22 March 2024, Tokyo, Japan (pp. 101–111). ACM. https://doi.org/10.1145/3636555.3636856
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024). A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, 4(2), 100211. https://doi.org/10.1016/j.hcc.2024.100211
Zheng, Y. (2023). ChatGPT for teaching and learning: An experience from data science education. In Proceedings of the 24th Annual Conference on Information Technology Education (SIGITE 2023), 11–14 October, Marietta, Georgia, USA (pp. 66–72). ACM. https://doi.org/10.1145/3585059.3611431
Zhu, J.-P., Cai, P., Niu, B., Ni, Z., Xu, K., Huang, J., Wan, J., Ma, S., Wang, B., Zhang, D., Tang, L., & Liu, Q. (2024). Chat2Query: A zero-shot automatic exploratory data analysis system with large language models. In 2024 IEEE 40th International Conference on Data Engineering (ICDE 2024), 13–16 May 2024, Utrecht, Netherlands (pp. 5429–5432). IEEE. https://doi.org/10.1109/ICDE60146.2024.00420
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
