Moments of Confusion in Simulation-Based Learning Environments




Confusion, Confidence, Hyper-correction, Simulation-based learning, Predict-Observe-Explain, POE, Metacognitive mismatch, Frustration, Affect


Confusion is an important epistemic emotion because it can help students focus their attention and effort when solving complex learning tasks. However, unresolved confusion can be detrimental because it may result in students’ disengagement. This is especially concerning in simulation environments using discovery-based learning, which puts more of the onus for learning on the students. Thus, students with misconceptions may become confused.

In this study, the possible moments of confusion in a simulation-based predict-observe-explain (POE) environment were investigated. Log-based interaction patterns of undergraduate students from a fully online course were analyzed. It was found that POE environments can offer a level of difficulty that potentially triggers some confusion, and a likely moment of students’ confusion was the observe task. It was also found that confidence in prior knowledge is an important factor that can contribute to students’ confusion. Students mostly struggled when they discovered a mismatch between the subjective and objective correctness of their responses. The effects of such a mismatch were more pronounced when confusion markers were analyzed than when students’ learning outcomes were observed. These findings may guide future works to bridge the knowledge gaps that lead to confusion in POE environments.


Anderson, R. C., Kulhavy, R. W., & Andre, T. (1971). Feedback procedures in programmed instruction. Journal of Educational Psychology, 62(2), 148–156.

Arguel, A., & Lane, R. (2015). Fostering deep understanding in geography by inducing and managing confusion: an online learning approach. In T. Reiners et al. (Eds.), Globally connected, digitally enabled. Proceedings of the 32nd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2015), 29 November–2 December 2015, Perth, Western Australia (pp. 373–377). Australasian Society for Computers in Learning in Tertiary Education. Retrieved from

Arguel, A., Lockyer, L., Kennedy, G., Lodge, J., & Pachman, M. (2019). Seeking optimal confusion: A review on epistemic emotion management in interactive digital learning environments. Interactive Learning Environments, 27(2), 200–210.

Arguel, A., Lockyer, L., Lipp, O. V., Lodge, J. M., & Kennedy, G. (2017). Inside out: Detecting learners’ confusion to improve interactive digital learning environments. Journal of Educational Computing Research, 55(4), 526–551.

Baker, R., D’Mello, S., Rodrigo, M. M. T., & Graesser, A. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241.

Baker, R., Kalka, J., Aleven, V., Rossi, L., Gowda, S. M., Wagner, A. Z., …, Ocumpaugh, J. (2012). Towards sensor-free affect detection in cognitive tutor algebra. In K. Yacef et al. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining (EDM2012), 19–21 June 2012, Chania, Greece (pp. 126–133). International Educational Data Mining Society. Retrieved from

Baker, R., Moore, G. R., Wagner, A. Z., Kalka, J., Salvi, A., Karabinos, M., & Yaron, D. (2011). The dynamics between student affect and behavior occurring outside of educational software. In Proceedings of the 2011 Conference on Affective Computing and Intelligent Interaction (ACII ’11), 9–12 October 2011, Memphis, TN, USA (pp. 14–24). Springer.

Ballantyne, R., & Bain, J. (1995). Enhancing environmental conceptions: An evaluation of cognitive conflict and structured controversy learning units. Studies in Higher Education, 20(3), 293–303.

Barker, M., & Pinard, M. (2014). Closing the feedback loop? Iterative feedback between tutor and student in coursework assessments. Assessment & Evaluation in Higher Education, 39(8), 899–915.

Bezdek, J. C., & Hathaway, R. J. (2002). VAT: A tool for visual assessment of (cluster) tendency. In Proceedings of the 2002 International Joint Conference on Neural Networks (IJCNN 2002), 12–17 May 2002, Honolulu, HI, USA (pp. 2225–2230). IEEE Computer Society.

Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M. Hough, J. R. Pomerantz (Eds.), & FABBS Foundation, Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56–64). Worth Publishers. Retrieved from

Bosch, N., D’Mello, S., & Mills, C. (2013). What emotions do novices experience during their first computer programming learning session? In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Proceedings of the 16th International Conference on Artificial Intelligence in Education (AIED ʼ13), 9–13 July 2013, Memphis, TN, USA (pp. 11–20). Springer.

Botelho, A. F., Baker, R., Ocumpaugh, J., & Heffernan, N. T. (2018). Studying affect dynamics and chronometry using sensor-free detectors. In K. E. Boyer & M. Yudelson (Eds.), Proceedings of the 11th International Conference on Educational Data Mining (EDM2018), 15–18 July 2018, Buffalo, NY, USA (pp. 157–166). International Educational Data Mining Society. Retrieved from

Boud, D., & Molloy, E. (2012). Feedback in higher and professional education: Understanding it and doing it well (1st Ed.): London, UK: Routledge. Retrieved from

Bowen, M., & Haysom, J. (2014). Predict, observe, explain: Activities enhancing science understanding. Arlington, VA: NSTA Press. Retrieved from

Brown, J., & VanLehn, K. (1980). Repair theory: A generative theory of bugs in procedural skills. In Cognitive Science, 4(4), 379–426.

Bunderson, V. C., Inouye, D. K., & Olsen, J. B. (1989). The four generations of computerized educational measurement. In R. L. Linn (Ed.), Educational measurement (Vol. 3). New York, NY: Macmillan.

Butler, A. C., Karpicke, J. D., & Roediger, H. L. (2008). Correcting a metacognitive error: Feedback increases retention of low confidence correct responses. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(4), 918–928.

Butterfield, B., & Metcalfe, J. (2001). Errors committed with high confidence are hypercorrected. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 1491–1494.

Butterfield, B., & Metcalfe, J. (2006). The correction of errors committed with high confidence. Metacognition and Learning, 1(1), 69–84.

Chairam, S., Somsook, E., & Coll, R. K. (2009). Enhancing Thai students’ learning of chemical kinetics. Research in Science & Technological Education, 27(1), 95–115.

Chen, Y. L., Pan, P. R., Sung, Y. T., & Chang, K. E. (2013). Correcting misconceptions on electronics: Effects of a simulation-based learning environment backed by a conceptual change model. Journal of Educational Technology & Society, 16(2), 212–227. Retrieved from

Chiu, C. Y., Hong, Y. Y., & Dweck, C. S. (1997). Lay dispositionism and implicit theories of personality. Journal of Personality and Social Psychology, 73, 19–30.

Clore, G. L. (1992). Cognitive phenomenology: Feelings and the construction of judgment. In L. L. Martin & A. Tesser (Eds.), The construction of social judgments (pp. 133–163). Hillsdale, NJ: Lawrence Erlbaum Associates. Retrieved from

Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 134–138). New York, NY: ACM.

Clow, D. (2014). Data wranglers: Human interpreters to help close the feedback loop. In Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 49–53). New York, NY: ACM.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillside, NJ, USA: Lawrence Erlbaum Associates.

Colthorpe, K., Zimbardi, K., Ainscough, L., & Anderson, S. (2015). Know thy student! Combining learning analytics and critical reflections to increase understanding of students’ self-regulated learning in an authentic setting. Journal of Learning Analytics, 2(1), 134–155.

Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241–250.

Dalziel, J. (2010). Practical eTeaching strategies for predict—observe—explain, problem-based learning and role plays. Sydney, Australia: LAMS International. Retrieve from

del Valle, R., & Duffy, T. M. (2009). Online learning: Learner characteristics and their approaches to managing learning. Instructional Science, 37(2), 129–149.

Dienes, Z. (2012). Conscious versus unconscious learning of structure. In P. Rebuschat & J. N. Williams (Eds.), Statistical learning and language acquisition, Vol. 1 (pp. 337–364). Berlin, Germany: De Gruyter Mouton.

D’Mello, S., Craig, S. D., Gholson, B., Franklin, S., Picard, R., & Graesser, A. (2005). Integrating affect sensors in an intelligent tutoring system. In Affective interactions: The computer in the affective loop workshop at 2005 International Conference on Intelligent User Interfaces (pp. 7–13). New York, NY: ACM Press.

D’Mello, S., & Graesser, A. (2010). Modeling cognitive-affective dynamics with hidden Markov models. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Meeting of the Cognitive Science Society, 11–14 August 2010, Portland, OR, USA (pp. 2721–2726). Austin, TX: Cognitive Science Society. Retrieved from

D’Mello, S., & Graesser, A. (2011). The half-life of cognitive-affective states during complex learning. Cognition and Emotion, 25(7), 1299–1308.

D’Mello, S., & Graesser, A. (2012a). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157.

D’Mello, S., & Graesser, A. (2012b). Emotions during learning with AutoTutor. In P. Durlach & A. Lesgold (Eds.), Adaptive technologies for training and education (pp. 117–139). Cambridge, UK: Cambridge University Press.

D’Mello, S., & Graesser, A. (2014a). Confusion. In R. Pekrun & L. G. Lisa (Eds.), International handbook of emotions in education (pp. 289–310). Hoboken, NJ, USA: Taylor and Francis.

D’Mello, S., & Graesser, A. (2014b). Confusion and its dynamics during device comprehension with breakdown scenarios. Acta Psychologica, 151, 106–116.

D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153–170.

D’Mello, S., Person, N., & Lehman, B. (2009). Antecedent-consequent relationships and cyclical patterns between affective states and problem solving outcomes. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (AIED ʼ09), 6–10 July 2009, Brighton, UK (pp. 57–64). IOS Press.

D’Mello, S., Picard, R. W., & Graesser, A. (2007). Toward an affect-sensitive AutoTutor. IEEE Intelligent Systems, 22(4), 53–61.

D’Mello, S., Taylor, R. S., & Graesser, A. (2007). Monitoring affective trajectories during complex learning. In D. S. McNamara & G. Trafton (Eds.), Proceedings of the 29th Annual Conference of the Cognitive Science Society (CogSci 2007), 1–4 August 2007, Nashville, TN, USA (pp. 197–202). Austin, TX: Cognitive Science Society.

Driver, R. (1983). The pupil as scientist? Milton Keynes, UK: Open University Press.

Duit, R., & Treagust, D. F. (2003). Conceptual change: A powerful framework for improving science teaching and learning. International Journal of Science Education, 25(6), 671–688.

Ellis, R. A., Han, F., & Pardo, A., (2017). Improving learning analytics—Combining observational and self-report data on student learning. Educational Technology & Society, 20(3), 158–169. Retrieved from

Engelbrecht, J., Harding, A., & Potgieter, M. (2005). Undergraduate students’ performance and confidence in procedural and conceptual mathematics. International Journal of Mathematical Education in Science and Technology, 36(7), 701–712.

Fazio, L. K., & Marsh, E. J. (2009). Surprising feedback improves later memory. Psychonomic Bulletin & Review, 16(1), 88–92.

Fazio, L. K., & Marsh, E. J. (2010). Correcting false memories. Psychological Science, 21(6), 801–803.

Fensham, P. J., & Kass, H. (1988). Inconsistent or discrepant events in science instruction. Studies in Science Education, 15(1), 1–16.

Festinger, L. (1957). A theory of cognitive dissonance. Stanford, CA: Stanford University Press. Retrieved from

Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting learning success. The Internet and Higher Education, 28, 68–84.

Gašević, D., Dawson, S., & Siemens, G. (2014). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71.

Gašević, D., Jovanovic, J., Pardo, A., & Dawson, S. (2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113–128.

Graesser, A. (2011). Learning, thinking, and emoting with discourse technologies. American Psychologist, 66(8), 746–757.

Graesser, A., Baggett, W., & Williams, K. (1996). Question-driven explanatory reasoning. Applied Cognitive Psychology, 10(7), 17–31.<17::AID-ACP435>3.0.CO;2-7

Graesser, A., & D’Mello, S. (2012). Emotions during the learning of difficult material. Psychology of Learning and Motivation, 57, 183–225.

Graesser, A., Lu, S., Olde, B. A., Cooper-Pye, E., & Whitten, S. (2005). Question asking and eye tracking during cognitive disequilibrium: Comprehending illustrated texts on devices when the devices break down. Memory & Cognition, 33(7), 1235–1247.

Graesser, A., & Olde, B. A. (2003). How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. Journal of Educational Psychology, 95(3), 524–536.

Graesser, A., Ozuru, Y., & Sullins, J. (2010). What is a good question? In M. G. McKeown & L. Kucan (Eds.), Bringing reading research to life (pp. 112–141). New York, NY: Guilford. Retreived from

Greiff, S., Wüstenberg, S., Csapó, B., Demetriou, A., Hautamäki, J., Graesser, A., & Martin, R. (2014). Domain-general problem solving skills and education in the 21st century. Educational Research Review, 13, 74–83.

Gunstone, R. F. (1990). Children’s science: A decade of developments in constructivist views of science teaching and learning. The Australian Science Teachers Journal, 36(4).

Gunstone, R. F., & White, R. T. (1980). A matter of gravity. Research in Science Education, 10(1), 35–44.

Horodyskyj, L. B., Mead, C., Belinson, Z., Buxner, S., Semken, S., & Anbar, A. D. (2018). Habitable Worlds: Delivering on the promises of online education. Astrobiology, 18(1), 86–99.

Huang, H. M. (2002). Toward constructivism for adult learners in online learning environments. British Journal of Educational Technology, 33(1), 27–37.

Kang, S. H., Pashler, H., Cepeda, N. J., Rohrer, D., Carpenter, S. K., & Mozer, M. C. (2011). Does incorrect guessing impair fact learning? Journal of Educational Psychology, 103(1), 48–59.

Kapur, M. (2016). Examining productive failure, productive success, unproductive failure and unproductive success in learning. Educational Psychologist, 51(2), 289–299.

Karamustafaoğlu, S., & Mamlok-Naaman, R. (2015). Understanding electrochemistry concepts using the predict-observe-explain strategy. Eurasia Journal of Mathematics, Science and Technology Education, 11(5), 923–936.

Karumbaiah, S., Andres, J. M. A. L., Botelho, A. F., Baker, R., & Ocumpaugh, J. (2018). The implications of a subtle difference in the calculation of affect dynamics. In Proceedings of the 26th International Conference on Computers in Education (ICCE ’18), 26–30 November 2018, Manila, Philippines (pp. 29–38). Philippines: Asia-Pacific Society for Computers in Education. Retrieved from

Kassambara, A. (2017). Practical guide to cluster analysis in R—Unsupervised machine learning. STHDA. Retredfddfived from

Keltner, D., & Shiota, M. N. (2003). New displays and new emotions: A commentary on Rozin and Cohen (2003). Emotion, 3, 86–91.

Kendeou, P., & Broek, P. V. D. (2005). The effects of readers’ misconceptions on comprehension of scientific text. Journal of Educational Psychology, 97(2).

Kennedy, G., & Lodge, J. (2016). All roads lead to Rome: Tracking students’ affect as they overcome misconceptions. In S. Barker, S. Dawson, A. Pardo, & C. Colvin (Eds.), Show Me The Learning: Proceedings of the 33rd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2016). 28–30 November 2016, Wellington, New Zealand (pp. 318–328). Australasian Society for Computers in Learning in Tertiary Education. Retrieved from

Kibirige, I., Osodo, J., & Tlala, K. M. (2014). The effect of predict-observe-explain strategy on learners’ misconceptions about dissolved salts. Mediterranean Journal of Social Sciences, 5(4), 300–310.

Kostyuk, V., Almeda, M., & Baker, R. (2018). Correlating affect and behavior in reasoning mind with state test achievement. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, Australia (pp. 26–30). New York, NY: ACM.

Kovanovic, V., Gašević, D., Dawson, S., Joksimovic, S., Baker, R., & Hatala, M. (2015). Does time-on-task estimation matter? Implications for the validity of learning analytics findings. Journal of Learning Analytics, 2(3), 81–110.

Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567.

Kulhavy, R. W. (1977). Feedback in written instruction. Review of Educational Research, 47(2), 211–232.

Kulhavy, R. W., & Stock, W. A. (1989). Feedback in written instruction: The place of response certitude. Educational Psychology Review, 1(4), 279–308.

Kulhavy, R. W., Yekovich, F. R., & Dyer, J. W. (1976). Feedback and response confidence. Journal of Educational Psychology, 68(5), 522–528.

Lagud, M. C. V., & Rodrigo, M. M. T. (2010). The affective and learning profiles of students using an intelligent tutoring system for algebra. In V. Aleven, J. Kay, & J. Mostow (Eds.), Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS 2010), 14–18 June 2010, Pittsburgh, PA, USA, (pp. 255–263). Berlin, Germany: Springer.

Lazarus, R. S. (1991). Cognition and motivation in emotion. American Psychologist, 46, 352–367.

Lee, D. M. C., Rodrigo, M. M. T., Baker, R., Sugay, J. O., & Coronel, A. (2011). Exploring the relationship between novice programmer confusion and achievement. In S. K. D’Mello, A. C. Graesser, B. W. Schuller, & Jean-Claude Martin (Eds.), Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII ’11), 9–12 October 2011, Memphis, TN, USA (pp. 175–184). Berlin, Germany: Springer.

Lehman, B., D’Mello, S., Chauncey, A., Gross, M., Dobbins, A., Wallace, P., …, Graesser, A. (2011). Inducing and tracking confusion with contradictions during critical thinking and scientific reasoning. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education (AIED ʼ11), 28 June–2 July 2011, Auckland, New Zealand (pp. 171–178). Berlin, Germany: Springer.

Lehman, B., D’Mello, S., & Graesser, A. (2012). Confusion and complex learning during interactions with computer learning environments. The Internet and Higher Education, 15(3), 184–194.

Lehman, B., & Graesser, A. (2015). To resolve or not to resolve? That is the big question about confusion. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED ʼ15), 22–26 June 2015, Madrid, Spain (pp. 216–225). Berlin, Germany: Springer.

Liew, C. W., & Treagust, D. F. (1995). A predict-observe-explain teaching sequence for learning about students’ understanding of heat and expansion of liquids. Australian Science Teachers’ Journal, 41(1), 68–71. Retrieved from'_Understanding_of_Heat_and_Expansion_Liquids

Liew, C. W., & Treagust, D. F. (1998). The effectiveness of predict-observe-explain tasks in diagnosing students’ understanding of science and in identifying their levels of achievement. Retrieved from

Limón, M. (2001). On the cognitive conflict as an instructional strategy for conceptual change: A critical appraisal. Learning and Instruction, 11(4-5), 357–380.

Liu, Z., Pataranutaporn, V., Ocumpaugh, J., & Baker, R. (2013). Sequences of frustration and confusion, and learning. In S. K. DʼMello et al. (Eds.), Proceedings of the 6th International Conference on Educational Data Mining (EDM2013), 6–9 July 2013, Memphis, TN, USA (pp. 114–120). International Educational Data Mining Society/Springer. Retrieved from

Lodge, J., Kennedy, G., Lockyer, L., Arguel, A., & Pachman, M. (2018). Understanding difficulties and resulting confusion in learning: An integrative review. Frontiers in Education, 3, 49.

MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 1 January 1967, Oakland, CA, USA (pp. 281–297).

Mandler, G. (1984). Mind and body: Psychology of emotion and stress. New York, NY: Norton.

Mansour, B. E., & Mupinga, D. M. (2007). Students’ positive and negative experiences in hybrid and online classes. College Student Journal, 41(1). Retrieved from

Markant, D., & Gureckis, T. M. (2014a). A preference for the unpredictable over the informative during self-directed learning. In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th Annual Conference of the Cognitive Science Society (CogSci 2014), 23–26 July 2014, Quebec City, QC, Canada (pp. 958–963). Austin, TX: Cognitive Science Society. Retrieved from

Markant, D. B., & Gureckis, T. M. (2014b). Is it better to select or to receive? Learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1), 94–122.

Markant, D. B., Settles, B., & Gureckis, T. M. (2016). Self-directed learning favors local, rather than global, uncertainty. Cognitive Science, 40(1), 100–120.

McLeod, D. B. (1989). The role of affect in mathematical problem solving. In D. B. McLeod & V. M. Adams (Eds.), Affect and Mathematical Problem Solving (pp. 20–36). New York, NY: Springer.

McQuiggan, S. W., Robison, J. L., & Lester, J. C. (2010). Affective transitions in narrative-centered learning environments. Educational Technology & Society, 13(1), 40–53. Retrieved from

Metcalfe, J., & Finn, B. (2011). People’s hypercorrection of high-confidence errors: Did they know it all along? Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(2).

Miller, W. L., Baker, R., Labrum, M. J., Petsche, K., Liu, Y. H., & Wagner, A. Z. (2015). Automated detection of proactive remediation by teachers in Reasoning Mind classrooms. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 290–294). New York, NY: ACM.

Nawaz, S., Kennedy, G., Bailey, J., Mead, C., & Horodyskyj, L. (2018). Struggle town? Developing profiles of student confusion in simulation-based learning environments. Proceedings of the 35th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2018), 25–28 November 2018, Geelong, Australia (pp. 224–233). Australasian Society for Computers in Learning in Tertiary Education. Retrieved from

Nawaz, S., Srivastava, N., Yu, J. H., Baker, R., Kennedy, G., & Bailey, J. (2020). Analysis of task difficulty sequences in a simulation-based POE environment. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán (Eds.), Proceedings of the 21st International Conference on Artificial Intelligence in Education (AIED ʼ20), 6–10 July 2020, Ifrane, Morocco (virtual conference due to COVID-19) (pp. 423–436). Berlin, Germany: Springer.

Nawaz, S., Srivastava, N., Yu, J. H., Khan, A. A., Kennedy, G., Bailey, J., & Baker, R. (2020). How difficult is the task for you? Modelling and analysis of students’ task difficulty sequences. International Journal of Artificial Intelligence in Education (IJAIED), submitted.

Ocumpaugh, J., Andres, J. M., Baker, R., DeFalco, J., Paquette, L., Rowe, J., … , Sottilare, R. (2017). Affect dynamics in military trainees using vMedic: From engaged concentration to boredom to confusion. In E. André, R. S. Baker, X. Hu, M. M. T. Rodrigo, & B. du Boulay (Eds.), Proceedings of the 18th International Conference on Artificial Intelligence in Education (AIED 2017), 28 June–1 July 2017, Wuhan, China (pp. 238–249). Berlin, Germany: Springer.

Pachman, M., Arguel, A., & Lockyer, L. (2015). Learners’ confusion: Faulty prior knowledge or a metacognitive monitoring error? In T. Reiners et al. (Eds.), Globally connected, digitally enabled. Proceedings of the 32nd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2015), 29 November–2 December 2015, Perth, Western Australia (pp. 522–526). Australasian Society for Computers in Learning in Tertiary Education. Retrieved from

Pardos, Z. A., Baker, R. S. J. D., Pedro, M. S., Gowda, S. M., & Gowda, S. M. (2014). Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. Journal of Learning Analytics, 1(1), 107–128.

Pardos, Z. A., & Horodyskyj, L. (2019). Analysis of student behaviour in Habitable Worlds using continuous representation visualization. Journal of Learning Analytics, 6(1), 1–15.

Paulhaus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of social psychological attitudes, Vol. 1. Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA: Academic Press.

Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of quantitative and qualitative research. Educational Psychologist, 37(2), 91–106.

Pekrun, R., & Stephens, E. J. (2012). Academic emotions. In K. R. Harris, S. Graham, T. Urdan, S. Graham, J. M. Royer, & M. Zeidner (Eds.), APA handbooks in psychology®. APA educational psychology handbook, Vol. 2. Individual differences and cultural and contextual factors (pp. 3–31). Washington, DC: American Psychological Association.

Piaget, J. (1952a). The origins of intelligence. New York, NY: International University Press.

Piaget, J. (1952b). The origins of intelligence in children (Vol. 8). New York, NY: International University Press.

Piaget, J. (1985). The equilibration of cognitive structures: The central problem of intellectual development. Chicago, IL: University of Chicago Press.

Rangel, R. H., Möller, L., Sitter, H., Stibane, T., & Strzelczyk, A. (2017). Sure, or unsure? Measuring students’ confidence and the potential impact on patient safety in multiple-choice questions. Medical Teacher, 39(11), 1189–1194.

Rodrigo, M. M. T., Baker, R. S., Jadud, M. C., Amarra, A. C. M., Dy, T., Espejo-Lanoz, M. B. V., …, Tabanao, E. S. (2009). Affective and behavioral predictors of novice programmer achievement. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE ’09), 6–9 July 2009, Paris, France (pp. 156–160). New York, NY: ACM.

Rodrigo, M. M. T., Baker, R., Lagud, M. C. V., Lim, S. A. L., Macapanpan, A. F., Pascua, S. A. M. S., …, Viehland, N. J. B. (2007). Affect and usage choices in simulation problem solving environments. In Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED 2007), 9–13 July 2007, Los Angeles, CA, USA (pp. 145–152). The Netherlands: IOS Press. Retrieved from

Rodriguez, F., Yu, R., Park, J., Rivas, M. J., Warschauer, M., & Sato, B. K. (2019). Utilizing learning analytics to map students’ self-reported study strategies to click behaviors in STEM courses. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2018, Tempe, AZ, USA (pp. 456–460). New York, NY: ACM.

Rosenthal, J. A. (1996). Qualitative descriptors of strength of association and effect size. Journal of Social Service Research, 21(4), 37–59.

Rosenthal, R., & Rosnow, R. L. (1984). Essentials of behavioral research: Methods and data analysis. New York, NY: McGraw-Hill.

Rozin, P., & Cohen, A. B. (2003). High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of naturally occurring facial expressions of Americans. Emotion, 3, 68–75.

Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.

Silvia, P. J. (2009). Looking past pleasure: Anger, confusion, disgust, pride, surprise, and other unusual aesthetic emotions. Psychology of Aesthetics, Creativity, and the Arts, 3(1), 48–51.

Silvia, P. J. (2010). Confusion and interest: The role of knowledge emotions in aesthetic experience. Psychology of Aesthetics, Creativity, and the Arts, 4(2), 75–80.

Sreerekha, S., Arun, R. R., & Sankar, S. (2016). Effect of predict-observe-explain strategy on achievement in chemistry of secondary school students. International Journal of Education & Teaching Analytics, 1(1). Retrieved from

Storm, C., & Storm, T. (1987). A taxonomic study of the vocabulary of emotions. Journal of Personality and Social Psychology, 53, 805–816.

Sun, P. (2006). Outlier detection in high dimensional, spatial and sequential data sets (PhD thesis). Sydney University.

Tao, P. K., & Gunstone, R. F. (1999). The process of conceptual change in force and motion during computer-supported physics instruction. Journal of Research in Science Teaching, 36(7), 859–882.<859::AID-TEA7>3.0.CO;2-J

Tippett, C. D. (2010). Refutation text in science education: A review of two decades of research. International Journal of Science and Mathematics Education, 8(6), 951–970.

VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 21(3), 209–249.

White, P. A. (1989). Evidence for the use of information about internal events to improve the accuracy of causal reports. British Journal of Psychology, 80, 375–382.

White, R. F., & Gunstone, R. T. (1992). Probing understanding. London, UK: Routledge.

Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students’ calibration of self reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551–572.

Wise, A., Speer, J., Marbouti, F., & Hsiao, Y. T. (2013). Broadening the notion of participation in online discussions: Examining patterns in learners’ online listening behaviors. Instructional Science, 41, 323–343.

Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419.




How to Cite

Nawaz, S., Kennedy, G., Bailey, J., & Mead, C. (2020). Moments of Confusion in Simulation-Based Learning Environments. Journal of Learning Analytics, 7(3), 118-137.