Using HeuristicsMiner to Analyze Problem-Solving Processes

Exemplary Use Case of a Productive-Failure Study


  • Christian Hartmann Technical University of Munich
  • Nikol Rummel Institute of Educational Research, Ruhr University Bochum, Germany
  • Maria Bannert TUM School of Education, Technical University of Munich, Germany



productive-failure approach, classroom setting, process mining, HeuristicsMiner, problem-solving strategies, research paper


This paper presents a fine-grained process analysis of 22 students in a classroom-based learning setting. The students engaged (and failed) in problem-solving attempts prior to instruction (i.e., the Productive-Failure approach). We used the HeuristicsMiner algorithm to analyze the data of a quasi-experimental study. The applied algorithm allowed us to investigate temporally structured think-aloud data, to outline productive and unproductive problem-solving strategies. Our analyses and findings demonstrated that HeuristicsMiner enables researchers to effectively mine problem-solving processes and sequences, even for smaller sample sizes, which cannot be done with traditional code-and-count strategies. The limitations of the algorithm, as well as further implications for educational research and practice, are also discussed.


Azevedo, R. (2014). Issues in dealing with sequential and temporal characteristics of self- and socially-regulated learning. Metacognition and Learning, 9(2), 217–228.

Bannert, M., Reimann, P., & Sonnenberg, C. (2014). Process mining techniques for analysing patterns and strategies in students’ self-regulated learning. Metacognition and Learning, 9(2), 161–185.

Bannert, M., Sonnenberg, C., Mengelkamp, C., & Pieger, E. (2015). Short- and long-term effects of students’ self-directed metacognitive prompts on navigation behavior and learning performance. Computers in Human Behavior, 52, 293–306.

Berardi-Coletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger, E. R. (1995). Metacognition and problem solving: A process-oriented approach. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(1), 205–223.

Bingham, C. B., & Haleblian, J. (2012). How firms learn heuristics: Uncovering missing components of organizational learning. Strategic Entrepreneurship Journal, 6(2), 152–177.

Bogarín, A., Romero, C., Cerezo, R., & Sánchez-Santillán, M. (2014, March). Clustering for improving educational process mining. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 11–15). ACM Press.

Brand, C., Hartmann, C., & Rummel, N. (2018). Exploring relevant problem-solving processes in learning from productive failure. International Society of the Learning Sciences.

Chi, M. T. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 6(3), 271–315.

Csanadi, A., Eagan, B., Kollar, I., Shaffer, D. W., & Fischer, F. (2018). When coding-and-counting is not enough: Using epistemic network analysis (ENA) to analyze verbal data in CSCL research. International Journal of Computer-Supported Collaborative Learning, 13(4), 419–438.

DeCaro, M. S., & Rittle-Johnson, B. (2012). Exploring mathematics problems prepares children to learn from instruction. Journal of Experimental Child Psychology, 113(4), 552–568.

Granberg, C. (2016). Discovering and addressing errors during mathematics problem-solving: A productive struggle? The Journal of Mathematical Behavior, 42, 33–48.

Hartmann, C., van Gog, T., & Rummel, N. (2020). Do examples of failure effectively prepare students for learning from subsequent instruction? Applied Cognitive Psychology, 34, 879–889.

Hartmann, C., van Gog, T., & Rummel, N. (2021). Preparatory effects of problem solving versus studying examples prior to instruction. Instructional Science, 49(1), 1—21.

Heirweg, S., De Smul, M., Devos, G., & Van Keer, H. (2019). Profiling upper primary school students’ self-regulated learning through self-report questionnaires and think-aloud protocol analysis. Learning and Individual Differences, 70, 155–168.

Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266.

Järvelä, S., & Bannert, M. (2021). Temporal and adaptive processes of regulated learning: What can multimodal data tell? Learning and Instruction, 72.

Kapur, M. (2012). Productive failure in learning the concept of variance. Instructional Science, 40(4), 651–672.

Kapur, M. (2014). Productive failure in learning math. Cognitive Science, 38, 1008–1022.

Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239–264.

Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research. Computers in Human Behavior, 19(3), 335–353.

Loibl, K., & Rummel, N. (2014a). Knowing what you don’t know makes failure productive. Learning and Instruction, 42(3), 305–326.

Loibl, K., & Rummel, N. (2014b). The impact of guidance during problem solving prior to instruction on students’ inventions and learning outcomes. Instructional Science, 42(3), 305–326.

Loibl, K., Roll, I., & Rummel, N. (2017). Towards a theory of when and how problem solving followed by instruction supports learning. Educational Psychology Review, 29(4), 693–715.

Martin, T., & Sherin, B. (2013). Learning analytics and computational techniques for detecting and evaluating patterns in learning: An introduction to the special issue. Journal of the Learning Sciences, 22(4), 511–520.

Molenaar, I., & Järvelä, S. (2014). Sequential and temporal characteristics of self and socially regulated learning. Metacognition and Learning, 9(2), 75–85.

Neuman, Y., & Schwarz, B. (1998). Is self‐explanation while solving problems helpful? The case of analogical problem‐solving. British Journal of Educational Psychology, 68(1), 15–24.

Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422.

Ramachandran, A., Huang, C. M., Gartland, E., & Scassellati, B. (2018, February). Thinking aloud with a tutoring robot to enhance learning. Proceedings of the ACM/IEEE International Conference on Human–Robot Interaction (HRI ’18), 5–8 March 2018, Chicago, IL, USA (pp. 59–68). ACM Press.

Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239–257.

Reimann, P., Markauskaite, L., & Bannert, M. (2014). e-Research and learning theory: What do sequence and process mining methods contribute? British Journal of Educational Technology, 45(3), 528–540.

Rogiers, A., Merchie, E., & Van Keer, H. (2020). What they say is what they do? Comparing task-specific self-reports, think-aloud protocols, and study traces for measuring secondary school students’ text-learning strategies. European Journal of Psychology of Education, 35, 315–332.

Schoenfeld, A. H. (1985). Making sense of “out loud” problem-solving protocols. The Journal of Mathematical Behavior, 4(2), 171–191.

Schoenfeld, A. H. (2016). Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics (Reprint). Journal of Education, 196(2), 1–38.

Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics, 3(3), 9–45.

Siemens, G., Gašević, D., Haythornthwaite, C., Dawson, S., Buckingham Shum, S., Ferguson, R., Duval, E., et al. (2011). Open learning analytics: An integrated and modularized platform. Society for Learning Analytics Research.

Sobocinski, M., Malmberg, J., & Järvelä, S. (2017). Exploring temporal sequences of regulatory phases and associated interactions in low- and high-challenge collaborative learning sessions. Metacognition and Learning, 12(2), 275–294.

Sonnenberg, C., & Bannert, M. (2015). Discovering the effects of metacognitive prompts on the sequential structure of SRL-processes using process mining techniques. Journal of Learning Analytics, 2(1), 72–100.

Sonnenberg, C., & Bannert, M. (2018). Using process mining to examine the sustainability of instructional support: How stable are the effects of metacognitive prompting on self-regulatory behavior? Computers in Human Behavior, 96, 159–272.

Van der Aalst, W., Adriansyah, A., & van Dongen, B. (2012). Replaying history on process models for conformance checking and performance analysis. WIREs Data Mining and Knowledge Discovery, 2(2), 182–192.

Van Laer, S., & Elen, J. (2018). Towards a methodological framework for sequence analysis in the field of self-regulated learning. Frontline Learning Research, 6(3), 228–249.

Weber, P., Bordbar, B., & Tino, P. (2013). A principled approach to mining from noisy logs using HeuristicsMiner. Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM 2013), 16–19 April 2013, Singapore (pp. 119–126). IEEE Computer Society.

Weijters, A. J. M. M., Aalst, van der, W. M. P., & Alves De Medeiros, A. K. (2006). Process mining with the HeuristicsMiner algorithm. (BETA publicatie: working papers; Vol. 166). Technische Universiteit Eindhoven.

Winne, P. H. (2014). Issues in researching self-regulated learning as patterns of events. Metacognition and Learning, 9(2), 229–237.

Young, K. A. (2005). Direct from the source: The value of “think-aloud” data in understanding learning. Journal of Educational Enquiry, 6(1), 19–33.




How to Cite

Hartmann, C., Rummel, N., & Bannert, M. (2022). Using HeuristicsMiner to Analyze Problem-Solving Processes: Exemplary Use Case of a Productive-Failure Study. Journal of Learning Analytics, 9(2), 66-86.



Research Papers