A LAK of Direction

Misalignment Between the Goals of Learning Analytics and its Research Scholarship

Authors

DOI:

https://doi.org/10.18608/jla.2023.7913

Keywords:

commentary paper, systematic review, learning analytics

Abstract

Learning analytics defines itself with a focus on data from learners and learning environments, with corresponding goals of understanding and optimizing student learning.  In this regard, learning analytics research, ideally, should be characterized by studies that make use of data from learners engaged in education systems, should measure student learning, and should make efforts to intervene and improve these learning environments. However, a common concern among members of the learning analytics research community is that these standards are not being met.  In two analysis waves, we review a large and comprehensive sample of research articles from the proceedings of the three most recent Learning Analytics and Knowledge conferences, the premier conference venue for learning analytics research, and from articles published during the same time in the Journal of Learning Analytics (over the years of 2020, 2021, and 2022).  We find that 37.4% of articles do not analyze data from learners in an education system, 71.1% do not include any measure of learning, and 89.0% of articles do not attempt to intervene in the learning environment.  We contrast these findings with the stated definition of learning analytics and infer, like others before us, that scholarship in learning analytics research presently lacks clear direction toward its stated goals.  We invite critical discussion of these findings from the learning analytics community, through open peer commentary.

References

Ahn, J., Nguyen, H., Campos, F., & Young, W. (2021). Transforming everyday information into practical analytics with crowdsourced assessment tasks. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 66–76). ACM Press. https://doi.org/10.1145/3448139.3448146

Baek, C., & Doleck, T. (2021). Educational data mining versus learning analytics: A review of publications from 2015 to 2019. Interactive Learning Environments, 1–23. https://doi.org/10.1080/10494820.2021.1943689

Baker, R. S., & Inventado, P. S. (2014). Educational data mining and learning analytics. In J. A. Larusson & B. White (Eds.), Learning analytics (pp. 61–75). Springer. https://doi.org/10.1007/978-1-4614-3305-7_4

Bergner, Y., Gray, G., & Lang, C. (2018). What does methodology mean for learning analytics? Journal of Learning Analytics, 5(2), 1–8. https://doi.org/10.18608/jla.2018.52.1

Bruyat, C., & Julien, P.-A. (2001). Defining the field of research in entrepreneurship. Journal of Business Venturing, 16(2), 165–180. https://doi.org/10.1016/S0883-9026(99)00043-9

Canning, E. A., Priniski, S. J., & Harackiewicz, J. M. (2019). Unintended consequences of framing a utility-value intervention in two-year colleges. Learning and Instruction, 62, 37–48. https://doi.org/10.1016/j.learninstruc.2019.05.001

Carpenter, D., Cloude, E., Rowe, J., Azevedo, R., & Lester, J. (2021). Investigating student reflection during game-based learning in middle grades science. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 280–291). ACM Press. https://doi.org/10.1145/3448139.3448166

Cerratto Pargman, T., & McGrath, C. (2021). Mapping the ethics of learning analytics in higher education: A systematic literature review of empirical research. Journal of Learning Analytics, 8(2), 123–139. https://doi.org/10.18608/jla.2021.1

Chen, B., Chen, X., & Xing, W. (2015). “Twitter archeology” of learning analytics and knowledge conferences. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 340–349). ACM Press. https://doi.org/10.1145/2723576.2723584

Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445–459. https://doi.org/10.3102/00346543053004445

Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 134–138). ACM Press. https://doi.org/10.1145/2330601.2330636

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6), 683–695. https://doi.org/10.1080/13562517.2013.827653

Dawson, S., Joksimović, S., Poquet, O., & Siemens, G. (2019). Increasing the impact of learning analytics. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 446–455). ACM Press. https://doi.org/10.1145/3303772.3303784

Dowell, N., Joksimović, S., Scheffel, M., & Siemens, G. (2021). Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA. ACM Press. https://doi.org/10.1145/3448139

Emara, M., Hutchins, N., Grover, S., Snyder, C., & Biswas, G. (2021). Examining student regulation of collaborative, computational, problem-solving processes in open-ended learning environments. Journal of Learning Analytics, 8(1), 49–74. http://doi.org/10.18608/jla.2021.7230

Ferguson, R., & Clow, D. (2017). Where is the evidence? A call to action for learning analytics. Proceedings of the 7th International Conference on Learning Analytics and Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 56–65). ACM Press. https://doi.org/10.1145/3027385.3027396

Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., Rienties, B., Ullmann, T., & Vuorikari, R. (2016). Research evidence on the use of learning analytics: Implications for education policy. In R. Vuorikari, J. Castaño Muñoz (Eds.), Joint Research Centre Science for Policy Report (EUR 28294 EN). https://doi.org/10.2791/955210

Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59, 64–71. https://doi.org/10.1007/s11528-014-0822-x

Gray, G., & Bergner, Y. (2022). A practitioner’s guide to measurement in learning analytics: Decisions, opportunities, and challenges. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), Handbook of learning analytics, 2nd. ed. (pp. 20–28). Society for Learning Analytics Research. https://doi.org/10.18608/hla22.002

Ionita, R. F., Corlatescu, D. G., Dascalu, M., & McNamara, D. S. (2021). How has the LAK community evolved in its first decade? A detailed modeling through interactive CNA sociograms. Proceedings of the 17th International Scientific Conference on eLearning and Software for Education (eLSE 2021), 22–23 April 2016, Bucharest, Romania (pp. 52–68).

Joksimović, S., Siemens, G., Wang, Y. E., San Pedro, M. O. Z., & Way, J. (2020). Editorial: Beyond cognitive ability. Journal of Learning Analytics, 7(1), 1–4. https://doi.org/10.18608/jla.2020.71.1

Jørnø, R. L., & Gynther, K. (2018). What constitutes an “actionable insight” in learning analytics? Journal of Learning Analytics, 5(3), 198–221. https://doi.org/10.18608/jla.2018.53.13

Kitto, K., Whitmer, J., Silvers, A. E., & Webb, M. (2020). Creating data for learning analytics ecosystems. Society for Learning Analytics Research (SoLAR). https://www.solaresearch.org/core/creating-data-for-learning-analytics-ecosystems/

Klebanov, B. B., Loukina, A., Lockwood, J., Liceralde, V. R. T., Sabatini, J., Madnani, N., Gyawali, B., Wang, Z., & Lentini, J. (2020). Detecting learning in noisy data: The case of oral reading fluency. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 490–495). ACM Press. https://doi.org/10.1145/3375462.3375490

Knight, S., Buckingham Shum, S., & Littleton, K. (2014). Epistemology, assessment, pedagogy: Where learning meets analytics in the middle space. Journal of Learning Analytics, 1(2), 23–47. https://doi.org/10.18608/jla.2014.12.3

Kornell, N., & Bjork, R. A. (2007). The promise and perils of self-regulated study. Psychonomic Bulletin & Review, 14(2), 219–224. https://doi.org/10.3758/BF03194055

Kovanović, V., Scheffel, M., Pinkwart, N., & Verbert, K. (2020). Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany. ACM Press. https://doi.org/10.1145/3375462

Larrabee Sønderlund, A., Hughes, E., & Smith, J. (2019). The efficacy of learning analytics interventions in higher education: A systematic review. British Journal of Educational Technology, 50(5), 2594–2618. https://doi.org/10.1111/bjet.12720

Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education — A literature review. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends. Studies in Systems, Decision and Control, vol. 94. Springer. https://doi.org/10.1007/978-3-319-52977-6_1

Liu, L. T., Wang, S., Britton, T., & Abebe, R. (2022). Lost in translation: Reimagining the machine learning life cycle in education. https://doi.org/10.48550/arXiv.2209.03929

Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30–40.

Ma, Y., Celepkolu, M., & Boyer, K. E. (2022). Detecting impasse during collaborative problem solving with multimodal learning analytics. Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK ’22), 21–25 March 2022, Online (pp. 45–55). ACM Press. https://doi.org/10.1145/3506860.3506865

Motz, B. A., Carvalho, P. F., de Leeuw, J. R., & Goldstone, R. L. (2018). Embedding experiments: Staking causal inference in authentic educational contexts. Journal of Learning Analytics, 5(2), 47–59. https://doi.org/10.18608/jla.2018.52.4

Motz, B., Brooks, C., Quick, J., Bergner, Y., Gray, G., Lang, C., Li, W., & Marmolejo-Ramos, F. (2022). A baseline measure of open research practices in learning analytics. Pre-conference Workshop of the 12th International Conference on Learning Analytics & Knowledge. https://doi.org/10.35542/osf.io/325d7

Newell, A. (1973). You can’t play 20 questions with nature and win: Projective comments on the papers of this symposium. In W. G. Chase (Ed.), Visual information processing (pp. 283–308). Academic Press. https://doi.org/10.1016/B978-0-12-170150-5.50012-3

Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. https://doi.org/10.1037/0033-295X.84.3.231

Núñez, R., Allen, M., Gao, R., Miller Rigoli, C., Relaford-Doyle, J., & Semenuks, A. (2019). What happened to cognitive science? Nature Human Behaviour, 3, 782–791. https://doi.org/10.1038/s41562-019-0626-2

Ochoa, X., & Merceron, A. (2018). Quantitative and qualitative analysis of the learning analytics and knowledge conference 2018. Journal of Learning Analytics, 5(3), 154–166. https://doi.org/10.18608/jla.2018.53.10

Papamitsiou, Z., & Economides, A. A. (2014). Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence. Educational Technology & Society, 17(4), 49–64. https://www.jstor.org/stable/jeductechsoci.17.4.49

Papamitsiou, Z., Giannakos, M. N., & Ochoa, X. (2020). From childhood to maturity: Are we there yet? Mapping the intellectual progress in learning analytics during the past decade. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 559–568). ACM Press. https://doi.org/10.1145/3375462.3375519

Pardo, A., Poquet, O., Martínez-Maldonado, R., & Dawson, S. (2017). Provision of data-driven student feedback in LA & EDM. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), Handbook of learning analytics (pp. 163–174). Society for Learning Analytics Research. https://doi.org/10.18608/hla17.014

Petrosino, A., Turpin‐Petrosino, C., Hollis‐Peel, M. E., & Lavenberg, J. G. (2013). Scared Straight and other juvenile awareness programs for preventing juvenile delinquency: A systematic review. Campbell Systematic Reviews, 9(1), 1–55. https://doi.org/10.4073/csr.2013.5

Pressley, M., Goodchild, F., Fleet, J., Zajchowski, R., & Evans, E. D. (1989). The challenges of classroom strategy instruction. The Elementary School Journal, 89(3), 301–342. http://doi.org/10.1086/461578

Reeves, T. C., & Lin, L. (2020). The research we have is not the research we need. Educational Technology Research and Development, 68, 1991–2001. https://doi.org/10.1007/s11423-020-09811-3

Reinhart, A. L., Haring, S. H., Levin, J. R., Patall, E. A., & Robinson, D. H. (2013). Models of not-so-good behavior: Yet another way to squeeze causality and recommendations for practice out of correlational data. Journal of Educational Psychology, 105(1), 241–247. https://doi.org/10.1037/a0030368

Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated survey. WIREs Data Mining and Knowledge Discovery, 10(3), e1355. https://doi.org/10.1002/widm.1355

Shibani, A., Knight, S., & Buckingham Shum, S. (2022). Questioning learning analytics? Cultivating critical engagement as student automated feedback literacy. Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK ’22), 21–25 March 2022, Online (pp. 326–335). ACM Press. https://doi.org/10.1145/3506860.3506912

Si Na, K., & Tasir, Z. (2017). A systematic review of learning analytics intervention contributing to student success in online learning. Proceedings of the 2017 International Conference on Learning and Teaching in Computing and Engineering (LaTICE ’17), 20–23 April 2017, Hong Kong, China (pp. 62–68). ACM Press. https://doi.org/10.1109/LaTICE.2017.18

Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. https://doi.org/10.1177/0002764213498851

Siemens, G. (2020, July 15). Learning engineering, learning analytics, educational data mining. [Online discussion]. Learning Analytics Google Group. https://bookdown.org/cl3584/book/Learning%20engineering,%20learning%20analytics,%20educational%20data%20mining.pdf

Society for Learning Analytics Research. (2021). What is learning analytics? https://www.solaresearch.org/about/what-is-learning-analytics/

Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018). The current landscape of learning analytics in higher education. Computers in Human Behavior, 89, 98–110. https://doi.org/10.1016/j.chb.2018.07.027

Viberg, O., Khalil, M., & Baars, M. (2020). Self-regulated learning and learning analytics in online learning environments: A review of empirical research. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 524–533). ACM Press. https://doi.org/10.1145/3375462.3375483

Werch, C. E., & Owen, D. (2002). Iatrogenic effects of alcohol and drug prevention programs. Journal of Studies on Alcohol, 63(5), 581–590. https://doi.org/10.15288/jsa.2002.63.581

Wise, A. F., & Shaffer, D. W. (2015). Why theory matters more than ever in the age of big data. Journal of Learning Analytics, 2(2), 5–13. https://doi.org/10.18608/jla.2015.22.2

Wise, A. F., Knight, S., & Ochoa, X. (2021). What makes learning analytics research matter. Journal of Learning Analytics, 8(3), 1–9. https://doi.org/10.18608/jla.2021.7647

Wise, A. F., Martinez-Maldonado, R., & Hilliger, I. (2022). Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK ’22), 21–25 March 2022, Online. ACM Press. https://doi.org/10.1145/3506860

Wong, B. T.-M., & Li, K. C. (2020). A review of learning analytics intervention in higher education (2011–2018). Journal of Computers in Education, 7, 7–28. https://doi.org/10.1007/s40692-019-00143-7

Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. https://doi.org/10.1016/j.learninstruc.2012.03.004

Downloads

Published

2023-03-09

How to Cite

Motz, B. A., Bergner, Y., Brooks, C. A., Gladden, A., Gray, G., Lang, C., Li, W., Marmolejo-Ramos, F., & Quick, J. D. (2023). A LAK of Direction: Misalignment Between the Goals of Learning Analytics and its Research Scholarship. Journal of Learning Analytics, 1-13. https://doi.org/10.18608/jla.2023.7913

Issue

Section

Open Peer Commentary