Using HeuristicsMiner to Analyze Problem-Solving Processes
Exemplary Use Case of a Productive-Failure Study
DOI:
https://doi.org/10.18608/jla.2022.7363Keywords:
productive-failure approach, classroom setting, process mining, HeuristicsMiner, problem-solving strategies, research paperAbstract
This paper presents a fine-grained process analysis of 22 students in a classroom-based learning setting. The students engaged (and failed) in problem-solving attempts prior to instruction (i.e., the Productive-Failure approach). We used the HeuristicsMiner algorithm to analyze the data of a quasi-experimental study. The applied algorithm allowed us to investigate temporally structured think-aloud data, to outline productive and unproductive problem-solving strategies. Our analyses and findings demonstrated that HeuristicsMiner enables researchers to effectively mine problem-solving processes and sequences, even for smaller sample sizes, which cannot be done with traditional code-and-count strategies. The limitations of the algorithm, as well as further implications for educational research and practice, are also discussed.
References
Azevedo, R. (2014). Issues in dealing with sequential and temporal characteristics of self- and socially-regulated learning. Metacognition and Learning, 9(2), 217–228. https://doi.org/10.1007/s11409-014-9123-1
Bannert, M., Reimann, P., & Sonnenberg, C. (2014). Process mining techniques for analysing patterns and strategies in students’ self-regulated learning. Metacognition and Learning, 9(2), 161–185. https://doi.org/10.1007/s11409-013-9107-6
Bannert, M., Sonnenberg, C., Mengelkamp, C., & Pieger, E. (2015). Short- and long-term effects of students’ self-directed metacognitive prompts on navigation behavior and learning performance. Computers in Human Behavior, 52, 293–306. https://doi.org/10.1016/j.chb.2015.05.038
Berardi-Coletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger, E. R. (1995). Metacognition and problem solving: A process-oriented approach. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(1), 205–223. https://doi.org/10.1037/0278-7393.21.1.205
Bingham, C. B., & Haleblian, J. (2012). How firms learn heuristics: Uncovering missing components of organizational learning. Strategic Entrepreneurship Journal, 6(2), 152–177. https://doi.org/10.1002/sej.1132
Bogarín, A., Romero, C., Cerezo, R., & Sánchez-Santillán, M. (2014, March). Clustering for improving educational process mining. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 11–15). ACM Press. https://doi.org/10.1145/2567574.2567604
Brand, C., Hartmann, C., & Rummel, N. (2018). Exploring relevant problem-solving processes in learning from productive failure. International Society of the Learning Sciences. https://repository.isls.org//handle/1/576
Chi, M. T. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 6(3), 271–315. https://doi.org/10.1207/s15327809jls0603_1
Csanadi, A., Eagan, B., Kollar, I., Shaffer, D. W., & Fischer, F. (2018). When coding-and-counting is not enough: Using epistemic network analysis (ENA) to analyze verbal data in CSCL research. International Journal of Computer-Supported Collaborative Learning, 13(4), 419–438. https://doi.org/10.1007/s11412-018-9292-z
DeCaro, M. S., & Rittle-Johnson, B. (2012). Exploring mathematics problems prepares children to learn from instruction. Journal of Experimental Child Psychology, 113(4), 552–568. https://doi.org/10.1016/j.jecp.2012.06.009
Granberg, C. (2016). Discovering and addressing errors during mathematics problem-solving: A productive struggle? The Journal of Mathematical Behavior, 42, 33–48. https://doi.org/10.1016/j.jmathb.2016.02.002
Hartmann, C., van Gog, T., & Rummel, N. (2020). Do examples of failure effectively prepare students for learning from subsequent instruction? Applied Cognitive Psychology, 34, 879–889. https://doi.org/10.1002/acp.3651
Hartmann, C., van Gog, T., & Rummel, N. (2021). Preparatory effects of problem solving versus studying examples prior to instruction. Instructional Science, 49(1), 1—21. https://doi.org/10.1007/s11251-020-09528-z
Heirweg, S., De Smul, M., Devos, G., & Van Keer, H. (2019). Profiling upper primary school students’ self-regulated learning through self-report questionnaires and think-aloud protocol analysis. Learning and Individual Differences, 70, 155–168. https://doi.org/10.1016/j.lindif.2019.02.001
Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3
Järvelä, S., & Bannert, M. (2021). Temporal and adaptive processes of regulated learning: What can multimodal data tell? Learning and Instruction, 72. https://doi.org/10.1016/j.learninstruc.2019.101268
Kapur, M. (2012). Productive failure in learning the concept of variance. Instructional Science, 40(4), 651–672. https://doi.org/10.1007/s11251-012-9209-6
Kapur, M. (2014). Productive failure in learning math. Cognitive Science, 38, 1008–1022. https://doi.org/10.1111/cogs.12107
Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239–264. https://doi.org/10.1007/s10648-007-9049-0
Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research. Computers in Human Behavior, 19(3), 335–353. https://doi.org/10.1016/S0747-5632(02)00057-2
Loibl, K., & Rummel, N. (2014a). Knowing what you don’t know makes failure productive. Learning and Instruction, 42(3), 305–326. http://dx.doi.org/10.1016/j.learninstruc.2014.08.004
Loibl, K., & Rummel, N. (2014b). The impact of guidance during problem solving prior to instruction on students’ inventions and learning outcomes. Instructional Science, 42(3), 305–326. http://dx.doi.org/10.1007/s11251-013-9282-5
Loibl, K., Roll, I., & Rummel, N. (2017). Towards a theory of when and how problem solving followed by instruction supports learning. Educational Psychology Review, 29(4), 693–715. http://dx.doi.org/10.1007/s10648-016-9379-x
Martin, T., & Sherin, B. (2013). Learning analytics and computational techniques for detecting and evaluating patterns in learning: An introduction to the special issue. Journal of the Learning Sciences, 22(4), 511–520. https://doi.org/10.1080/10508406.2013.840466
Molenaar, I., & Järvelä, S. (2014). Sequential and temporal characteristics of self and socially regulated learning. Metacognition and Learning, 9(2), 75–85. https://doi.org/10.1007/s11409-014-9114-2
Neuman, Y., & Schwarz, B. (1998). Is self‐explanation while solving problems helpful? The case of analogical problem‐solving. British Journal of Educational Psychology, 68(1), 15–24. https://doi.org/10.1111/j.2044-8279.1998.tb01271.x
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. https://doi.org/10.3389/fpsyg.2017.00422
Ramachandran, A., Huang, C. M., Gartland, E., & Scassellati, B. (2018, February). Thinking aloud with a tutoring robot to enhance learning. Proceedings of the ACM/IEEE International Conference on Human–Robot Interaction (HRI ’18), 5–8 March 2018, Chicago, IL, USA (pp. 59–68). ACM Press. https://doi.org/10.1145/3171221.3171250
Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239–257. https://doi.org/10.1007/s11412-009-9070-z
Reimann, P., Markauskaite, L., & Bannert, M. (2014). e-Research and learning theory: What do sequence and process mining methods contribute? British Journal of Educational Technology, 45(3), 528–540. https://doi.org/10.1111/bjet.12146
Rogiers, A., Merchie, E., & Van Keer, H. (2020). What they say is what they do? Comparing task-specific self-reports, think-aloud protocols, and study traces for measuring secondary school students’ text-learning strategies. European Journal of Psychology of Education, 35, 315–332. https://doi.org/10.1007/s10212-019-00429-5
Schoenfeld, A. H. (1985). Making sense of “out loud” problem-solving protocols. The Journal of Mathematical Behavior, 4(2), 171–191. https://psycnet.apa.org/record/1986-28488-001
Schoenfeld, A. H. (2016). Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics (Reprint). Journal of Education, 196(2), 1–38. https://doi.org/10.1177/002205741619600202
Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics, 3(3), 9–45. https://doi.org/10.18608/jla.2016.33.3
Siemens, G., Gašević, D., Haythornthwaite, C., Dawson, S., Buckingham Shum, S., Ferguson, R., Duval, E., et al. (2011). Open learning analytics: An integrated and modularized platform. Society for Learning Analytics Research. https://solaresearch.org/wp-content/uploads/2011/12/OpenLearningAnalytics.pdf
Sobocinski, M., Malmberg, J., & Järvelä, S. (2017). Exploring temporal sequences of regulatory phases and associated interactions in low- and high-challenge collaborative learning sessions. Metacognition and Learning, 12(2), 275–294. https://doi.org/10.1007/s11409-016-9167-5
Sonnenberg, C., & Bannert, M. (2015). Discovering the effects of metacognitive prompts on the sequential structure of SRL-processes using process mining techniques. Journal of Learning Analytics, 2(1), 72–100. https://doi.org/10.18608/jla.2015.21.5
Sonnenberg, C., & Bannert, M. (2018). Using process mining to examine the sustainability of instructional support: How stable are the effects of metacognitive prompting on self-regulatory behavior? Computers in Human Behavior, 96, 159–272. https://doi.org/10.1016/j.chb.2018.06.003
Van der Aalst, W., Adriansyah, A., & van Dongen, B. (2012). Replaying history on process models for conformance checking and performance analysis. WIREs Data Mining and Knowledge Discovery, 2(2), 182–192. https://doi.org/10.1002/widm.1045
Van Laer, S., & Elen, J. (2018). Towards a methodological framework for sequence analysis in the field of self-regulated learning. Frontline Learning Research, 6(3), 228–249. https://doi.org/10.14786/flr.v6i3.367
Weber, P., Bordbar, B., & Tino, P. (2013). A principled approach to mining from noisy logs using HeuristicsMiner. Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM 2013), 16–19 April 2013, Singapore (pp. 119–126). IEEE Computer Society. https://doi.org/10.1109/CIDM.2013.6597226
Weijters, A. J. M. M., Aalst, van der, W. M. P., & Alves De Medeiros, A. K. (2006). Process mining with the HeuristicsMiner algorithm. (BETA publicatie: working papers; Vol. 166). Technische Universiteit Eindhoven.
Winne, P. H. (2014). Issues in researching self-regulated learning as patterns of events. Metacognition and Learning, 9(2), 229–237. https://doi.org/10.1007/s11409-014-9113-3
Young, K. A. (2005). Direct from the source: The value of “think-aloud” data in understanding learning. Journal of Educational Enquiry, 6(1), 19–33. http://hdl.handle.net/10453/6348
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) license that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
