Design Analytics for Mobile Learning

Scaling up the Classification of Learning Designs Based on Cognitive and Contextual Elements




supervised machine learning, learning design, learning analytics, mobile learning, contextual learning, analytics for learning design, research paper


This research was triggered by the identified need in literature for large-scale studies about the kinds of designs that teachers create for mobile learning (m-learning). These studies require analyses of large datasets of learning designs. The common approach followed by researchers when analyzing designs has been to manually classify them following high-level pedagogically guided coding strategies, which demands extensive work. Therefore, the first goal of this paper is to explore the use of supervised machine learning (SML) to automatically classify the textual content of m-learning designs using pedagogically relevant classifications, such as the cognitive level demanded by students to carry out specific designed tasks, the phases of inquiry learning represented in the designs, or the role that the situated environment has in the designs. Because not all SML models are transparent, but researchers often need to understand their behaviour, the second goal of this paper is to consider the trade-off between models’ performance and interpretability in the context of design analytics for m-learning. To achieve these goals, we compiled a dataset of designs deployed using two tools, Avastusrada and Smartzoos. With this dataset, we trained and compared different models and feature extraction techniques. We further optimized and compared the best performing and most interpretable algorithms (EstBERT and Logistic Regression) to consider the second goal with an illustrative case. We found that SML can reliably classify designs with accuracy > 0.86 and Cohen’s kappa > 0.69.


Bulathwela, S., Pe ́rez-Ortiz, M., Lipani, A., Yilmaz, E., & Shawe-Taylor, J. (2020). Predicting engagement in video lectures. In A. N. Rafferty, J. Whitehill, C. Romero, & V. Cavalli-Sforza (Eds.), Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020), 10–13 July 2020, online (pp. 50–60). 62.pdf

Chen, F., & Cui, Y. (2020). Utilizing student time series behaviour in learning management systems for early prediction of course performance. Journal of Learning Analytics, 7(2), 1–17.

Conati, C., Porayska-Pomsta, K., & Mavrikis, M. (2018). AI in education needs interpretable machine learning: Lessons from open learner modelling. In B. Kim, K. R. Varshney, & A. Weller (Eds.), Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018), 14 July 2018, Stockholm, Sweden (pp. 21–27).

Dalziel, J., Conole, G., Wills, S., Walker, S., Bennett, S., Dobozy, E., Cameron, L., Badilescu-Buga, E., & Bower, M. (2016). The Larnaca Declaration on Learning Design. Journal of Interactive Media in Education, (1).

De Jong, T., Gillet, D., Rodríguez-Triana, M. J., Hovardas, T., Dikke, D., Doran, R., Dziabenko, O., Koslowsky, J., Korventausta, M., Law, E., Pedaste, M., Tasiopoulou, E., Vidal, G., & Zacharia, Z. C. (2021). Understanding teacher design practices for digital inquiry-based science learning: The case of Go-Lab. Educational Technology Research and Development, 1–28.

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Eagan, B. R., Rogers, B., Serlin, R., Ruis, A. R., Arastoopour Irgens, G., & Shaffer, D. W. (2017). Can we rely on IRR? Testing the assumptions of inter-rater reliability. Proceedings of the 12th International Conference on Computer Supported Collaborative Learning (CSCL 2017), 18–22 June 2017, Philadelphia, PA, USA.

Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5-6), 304–317.

Gervet, T., Koedinger, K., Schneider, J., & Mitchell, T. (2020). When is deep learning the best approach to knowledge tracing? Journal of Educational Data Mining, 12(3), 31–54.

Hernández-Leo, D., Asensio-Pérez, J. I., Derntl, M., Prieto, L. P., & Chacón, J. (2014). ILDE: Community environment for conceptualizing, authoring and deploying learning activities. In C. Rensing, S. de Freitas, T. Ley, & P. J. Muñoz-Merino (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2014), 16–19 September 2014, Graz, Austria (pp. 490–493). Springer.

Hernández-Leo, D., Martinez-Maldonado, R., Pardo, A., Muñoz-Cristóbal, J. A., & Rodríguez-Triana, M. J. (2019). Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, 50(1), 139–152.

Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice, 41(4), 212–218.

Macfadyen, L. P., Lockyer, L., & Rienties, B. (2020). Learning design and learning analytics: Snapshot 2020. Journal of Learning Analytics, 7(3), 6–12.

McDonald, N., Schoenebeck, S., & Forte, A. (2019). Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1–23.

Melero, J., Herna ́ndez-Leo, D., Sun, J., Santos, P., & Blat, J. (2015). How was the activity? A visualization support for a case of location-based learning design. British Journal of Educational Technology, 46(2), 317–329.

Mettis, K., & Väljataga, T. (2021). Designing learning experiences for outdoor hybrid learning spaces. British Journal of Educational Technology, 52(1), 498–513.

Mihaescu, M. C., & Popescu, P. S. (2021). Review on publicly available datasets for educational data mining. Wiley Interdisci- plinary Reviews: Data Mining and Knowledge Discovery, 11(3), e1403.

Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., & Gao, J. (2021). Deep learning–based text classification: A comprehensive review. ACM Computing Surveys (CSUR), 54(3), 1–40.

Mun ̃oz-Cristo ́bal, J. A., Herna ́ndez-Leo, D., Carvalho, L., Martinez-Maldonado, R., Thompson, K., Wardak, D., & Goodyear, P. (2018). 4FAD: A framework for mapping the evolution of artefacts in the learning design process. Australasian Journal of Educational Technology, 34(2).

Mun ̃oz-Cristo ́bal, J. A., Rodr ́ıguez-Triana, M. J., Gallego-Lema, V., Arribas-Cubero, H. F., Asensio-Pe ́rez, J. I., & Mart ́ınez- Mone ́s, A. (2018). Monitoring for awareness and reflection in ubiquitous learning environments. International Journal of Human–Computer Interaction, 34(2), 146–165.

Pedaste, M., Ma ̈eots, M., Siiman, L. A., De Jong, T., Van Riesen, S. A., Kamp, E. T., Manoli, C. C., Zacharia, Z. C., & Tsourlidaki, E. (2015). Phases of inquiry-based learning: Definitions and the inquiry cycle. Educational Research Review, 14, 47–61.

Pérez–Sanagustín, M., Martínez, A., & Delgado-Kloos, C. (2013). Etiquetar: Tagging learning experiences. In D. Hernández- Leo, T. Ley, R. Klamma, & A. Harrer (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2013), 17–21 September 2013, Paphos, Cyprus (pp. 573–576). Springer.

Persico, D., & Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2), 230–248.

Pishtari, G., Rodríguez-Triana, M. J., Sarmiento-Márquez, E. M., Pérez-Sanagustín, M., Ruiz-Calleja, A., Santos, P., P. Prieto, L., Serrano-Iglesias, S., & Väljataga, T. (2020). Learning design and learning analytics in mobile and ubiquitous learning: A systematic review. British Journal of Educational Technology, 51(4), 1078–1100.

Pishtari, G., & Rodríguez-Triana, M. J. (2022). An analysis of mobile learning tools in terms of pedagogical affordances and support to the learning activity life cycle. In E. Gil, Y. Mor, Y. Dimitriadis, & C. Köppe (Eds.), Hybrid learning spaces (pp. 167–183). Springer.

Pishtari, G., Rodríguez-Triana, M. J., & Väljataga, T. (2021). A multi-stakeholder perspective of analytics for learning design in location-based learning. International Journal of Mobile and Blended Learning (IJMBL), 13(1), 1–17.

Pishtari, G., Väljataga, T., Tammets, P., Savitski, P., Rodríguez-Triana, M. J., & Ley, T. (2017). Smartzoos: Modular open educational resources for location-based games. In É. Lavoué, H. Drachsler, K. Verbert, J. Broisin, & M. Pérez- Sanagustín (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 513–516). Springer.

Pozzi, F., Asensio-Pérezc, J. I., & Persico, D. (2016). The case for multiple representations in the learning design life cycle. In B. Gros, Kinshuk, & M. Maina (Eds.), The future of ubiquitous learning (pp. 171–196). Springer.

Prieto, L. P., Pishtari, G., Rodríguez-Triana, M. J., & Eagan, B. (2021). Comparing natural language processing approaches to scale up the automated coding of diaries in single-case learning analytics. In A. R. Ruis & S. B. Lee (Eds.), Second International Conference on Quantitative Ethnography: Conference Proceedings Supplement (ICQE 2020), 1–3 February 2021, online (pp. 39–42).

Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2016), 13–17 August 2016, San Francisco, CA, USA (pp. 1135–1144). ACM.

Rodríguez-Triana, M. J., Prieto, L. P., & Pishtari, G. (2021). What do learning designs show about pedagogical adoption? An analysis approach and a case study on inquiry-based learning. In T. D. Laet, R. Klemke, C. Alario-Hoyos, I. Hilliger, & A. Ortega-Arranz (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2021), 20–24 September 2021, Bolzano, Italy (pp. 275–288). Springer.

Rudin, C., & Radin, J. (2019). Why are we using black box models in AI when we don’t need to? A lesson from an explainable AI competition. Harvard Data Science Review, 1(2).

Santos, P., Pe ́rez-Sanagustín, M., Herna ́ndez-Leo, D., & Blat, J. (2011). QuesTInSitu: From tests to routes for assessment in situ activities. Computers & Education, 57(4), 2517–2534.

Shaffer, D. W. (2017). Quantitative ethnography. Cathcart Press.

Sharples, M. (2015). Making sense of context for mobile learning. In J. Traxler & A. Kukulska-Hulme (Eds.), Mobile learning: The next generation (pp. 140–153). Taylor and Francis.

Silva, V. A., Bittencourt, I. I., & Maldonado, J. C. (2019). Automatic question classifiers: A systematic review. IEEE Transactions on Learning Technologies, 12(4), 485–502.

Tanvir, H., Kittask, C., & Sirts, K. (2021). EstBERT: A pretrained language-specific BERT for Estonian. arXiv preprint arXiv:2011.04784.

Toetenel, L., & Rienties, B. (2016). Analysing 157 learning designs using learning analytic approaches as a means to evaluate the impact of pedagogical decision making. British Journal of Educational Technology, 47(5), 981–992.

Viera, A. J., & Garrett, J. M. (2005). Understanding interobserver agreement: The kappa statistic. Family Medicine, 37(5), 360–363.

Xu, X., Wang, J., Peng, H., & Wu, R. (2019). Prediction of academic performance associated with internet usage behaviors using machine learning algorithms. Computers in Human Behavior, 98, 166–173.




How to Cite

Pishtari, G., Prieto, L. P., Rodríguez-Triana, M. J., & Martinez-Maldonado, R. (2022). Design Analytics for Mobile Learning: Scaling up the Classification of Learning Designs Based on Cognitive and Contextual Elements. Journal of Learning Analytics, 9(2), 236-252.



Research Papers

Most read articles by the same author(s)

<< < 1 2