Moments of Confusion in Simulation-Based Learning Environments
Keywords:Confusion, Confidence, Hyper-correction, Simulation-based learning, Predict-Observe-Explain, POE, Metacognitive mismatch, Frustration, Affect
Confusion is an important epistemic emotion because it can help students focus their attention and effort when solving complex learning tasks. However, unresolved confusion can be detrimental because it may result in students’ disengagement. This is especially concerning in simulation environments using discovery-based learning, which puts more of the onus for learning on the students. Thus, students with misconceptions may become confused.
In this study, the possible moments of confusion in a simulation-based predict-observe-explain (POE) environment were investigated. Log-based interaction patterns of undergraduate students from a fully online course were analyzed. It was found that POE environments can offer a level of difficulty that potentially triggers some confusion, and a likely moment of students’ confusion was the observe task. It was also found that confidence in prior knowledge is an important factor that can contribute to students’ confusion. Students mostly struggled when they discovered a mismatch between the subjective and objective correctness of their responses. The effects of such a mismatch were more pronounced when confusion markers were analyzed than when students’ learning outcomes were observed. These findings may guide future works to bridge the knowledge gaps that lead to confusion in POE environments.
Anderson, R. C., Kulhavy, R. W., & Andre, T. (1971). Feedback procedures in programmed instruction. Journal of Educational Psychology, 62(2), 148–156. https://dx.doi.org/10.1037/h0030766
Arguel, A., & Lane, R. (2015). Fostering deep understanding in geography by inducing and managing confusion: an online learning approach. In T. Reiners et al. (Eds.), Globally connected, digitally enabled. Proceedings of the 32nd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2015), 29 November–2 December 2015, Perth, Western Australia (pp. 373–377). Australasian Society for Computers in Learning in Tertiary Education. Retrieved from http://www.2015conference.ascilite.org/wp-content/uploads/2015/11/ascilite-2015-proceedings.pdf
Arguel, A., Lockyer, L., Kennedy, G., Lodge, J., & Pachman, M. (2019). Seeking optimal confusion: A review on epistemic emotion management in interactive digital learning environments. Interactive Learning Environments, 27(2), 200–210. https://dx.doi.org/10.1080/10494820.2018.1457544
Arguel, A., Lockyer, L., Lipp, O. V., Lodge, J. M., & Kennedy, G. (2017). Inside out: Detecting learners’ confusion to improve interactive digital learning environments. Journal of Educational Computing Research, 55(4), 526–551. https://dx.doi.org/10.1177%2F0735633116674732
Baker, R., D’Mello, S., Rodrigo, M. M. T., & Graesser, A. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241. https://dx.doi.org/10.1016/j.ijhcs.2009.12.003
Baker, R., Kalka, J., Aleven, V., Rossi, L., Gowda, S. M., Wagner, A. Z., …, Ocumpaugh, J. (2012). Towards sensor-free affect detection in cognitive tutor algebra. In K. Yacef et al. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining (EDM2012), 19–21 June 2012, Chania, Greece (pp. 126–133). International Educational Data Mining Society. Retrieved from https://educationaldatamining.org/EDM2012/uploads/procs/EDM_2012_proceedings.pdf
Baker, R., Moore, G. R., Wagner, A. Z., Kalka, J., Salvi, A., Karabinos, M., & Yaron, D. (2011). The dynamics between student affect and behavior occurring outside of educational software. In Proceedings of the 2011 Conference on Affective Computing and Intelligent Interaction (ACII ’11), 9–12 October 2011, Memphis, TN, USA (pp. 14–24). Springer. https://dx.doi.org/10.1007/978-3-642-24600-5_5
Ballantyne, R., & Bain, J. (1995). Enhancing environmental conceptions: An evaluation of cognitive conflict and structured controversy learning units. Studies in Higher Education, 20(3), 293–303. https://dx.doi.org/10.1080/03075079512331381565
Barker, M., & Pinard, M. (2014). Closing the feedback loop? Iterative feedback between tutor and student in coursework assessments. Assessment & Evaluation in Higher Education, 39(8), 899–915. https://dx.doi.org/10.1080/02602938.2013.875985
Bezdek, J. C., & Hathaway, R. J. (2002). VAT: A tool for visual assessment of (cluster) tendency. In Proceedings of the 2002 International Joint Conference on Neural Networks (IJCNN 2002), 12–17 May 2002, Honolulu, HI, USA (pp. 2225–2230). IEEE Computer Society. https://dx.doi.org/10.1109/IJCNN.2002.1007487
Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M. Hough, J. R. Pomerantz (Eds.), & FABBS Foundation, Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56–64). Worth Publishers. Retrieved from https://psycnet.apa.org/record/2011-19926-008
Bosch, N., D’Mello, S., & Mills, C. (2013). What emotions do novices experience during their first computer programming learning session? In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Proceedings of the 16th International Conference on Artificial Intelligence in Education (AIED ʼ13), 9–13 July 2013, Memphis, TN, USA (pp. 11–20). Springer. https://dx.doi.org/10.1007/978-3-642-39112-5_2
Botelho, A. F., Baker, R., Ocumpaugh, J., & Heffernan, N. T. (2018). Studying affect dynamics and chronometry using sensor-free detectors. In K. E. Boyer & M. Yudelson (Eds.), Proceedings of the 11th International Conference on Educational Data Mining (EDM2018), 15–18 July 2018, Buffalo, NY, USA (pp. 157–166). International Educational Data Mining Society. Retrieved from https://educationaldatamining.org/files/conferences/EDM2018/EDM2018_Preface_TOC_Proceedings.pdf
Boud, D., & Molloy, E. (2012). Feedback in higher and professional education: Understanding it and doing it well (1st Ed.): London, UK: Routledge. Retrieved from https://www.routledge.com/Feedback-in-Higher-and-Professional-Education-Understanding-it-and-doing/Boud-Molloy/p/book/9780415692298 https://dx.doi.org/10.4324/9780203074336
Bowen, M., & Haysom, J. (2014). Predict, observe, explain: Activities enhancing science understanding. Arlington, VA: NSTA Press. Retrieved from https://my.nsta.org/resource/2625
Brown, J., & VanLehn, K. (1980). Repair theory: A generative theory of bugs in procedural skills. In Cognitive Science, 4(4), 379–426. https://dx.doi.org/10.1016/S0364-0213(80)80010-3
Bunderson, V. C., Inouye, D. K., & Olsen, J. B. (1989). The four generations of computerized educational measurement. In R. L. Linn (Ed.), Educational measurement (Vol. 3). New York, NY: Macmillan. https://dx.doi.org/10.1002/j.2330-8516.1988.tb00291.x
Butler, A. C., Karpicke, J. D., & Roediger, H. L. (2008). Correcting a metacognitive error: Feedback increases retention of low confidence correct responses. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(4), 918–928. https://dx.doi.org/10.1037/0278-7322.214.171.1248
Butterfield, B., & Metcalfe, J. (2001). Errors committed with high confidence are hypercorrected. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 1491–1494. https://dx.doi.org/10.1037/0278-73126.96.36.1991
Butterfield, B., & Metcalfe, J. (2006). The correction of errors committed with high confidence. Metacognition and Learning, 1(1), 69–84. https://dx.doi.org/10.1007/s11409-006-6894-z
Chairam, S., Somsook, E., & Coll, R. K. (2009). Enhancing Thai students’ learning of chemical kinetics. Research in Science & Technological Education, 27(1), 95–115. https://dx.doi.org/10.1080/02635140802658933
Chen, Y. L., Pan, P. R., Sung, Y. T., & Chang, K. E. (2013). Correcting misconceptions on electronics: Effects of a simulation-based learning environment backed by a conceptual change model. Journal of Educational Technology & Society, 16(2), 212–227. Retrieved from https://drive.google.com/file/d/1dgiywBcc1xq6fKyGmHUzxw6DzzxsD2Z1/view
Chiu, C. Y., Hong, Y. Y., & Dweck, C. S. (1997). Lay dispositionism and implicit theories of personality. Journal of Personality and Social Psychology, 73, 19–30. https://dx.doi.org/10.1037/0022-35188.8.131.52
Clore, G. L. (1992). Cognitive phenomenology: Feelings and the construction of judgment. In L. L. Martin & A. Tesser (Eds.), The construction of social judgments (pp. 133–163). Hillsdale, NJ: Lawrence Erlbaum Associates. Retrieved from https://psycnet.apa.org/record/1992-98414-005
Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 134–138). New York, NY: ACM. https://dx.doi.org/10.1145/2330601.2330636
Clow, D. (2014). Data wranglers: Human interpreters to help close the feedback loop. In Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 49–53). New York, NY: ACM. https://dx.doi.org/10.1145/2567574.2567603
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillside, NJ, USA: Lawrence Erlbaum Associates. https://dx.doi.org/10.4324/9780203771587
Colthorpe, K., Zimbardi, K., Ainscough, L., & Anderson, S. (2015). Know thy student! Combining learning analytics and critical reflections to increase understanding of students’ self-regulated learning in an authentic setting. Journal of Learning Analytics, 2(1), 134–155. https://dx.doi.org/10.18608/jla.2015.21.7
Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241–250. https://dx.doi.org/10.1080/1358165042000283101
Dalziel, J. (2010). Practical eTeaching strategies for predict—observe—explain, problem-based learning and role plays. Sydney, Australia: LAMS International. Retrieve from https://practicaleteachingstrategies.com/files/LAMSbook2010.Final.pdf
del Valle, R., & Duffy, T. M. (2009). Online learning: Learner characteristics and their approaches to managing learning. Instructional Science, 37(2), 129–149. https://dx.doi.org/10.1007/s11251-007-9039-0
Dienes, Z. (2012). Conscious versus unconscious learning of structure. In P. Rebuschat & J. N. Williams (Eds.), Statistical learning and language acquisition, Vol. 1 (pp. 337–364). Berlin, Germany: De Gruyter Mouton. https://dx.doi.org/10.1515/9781934078242.337
D’Mello, S., Craig, S. D., Gholson, B., Franklin, S., Picard, R., & Graesser, A. (2005). Integrating affect sensors in an intelligent tutoring system. In Affective interactions: The computer in the affective loop workshop at 2005 International Conference on Intelligent User Interfaces (pp. 7–13). New York, NY: ACM Press.
D’Mello, S., & Graesser, A. (2010). Modeling cognitive-affective dynamics with hidden Markov models. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Meeting of the Cognitive Science Society, 11–14 August 2010, Portland, OR, USA (pp. 2721–2726). Austin, TX: Cognitive Science Society. Retrieved from https://cognitivesciencesociety.org/wp-content/uploads/2019/01/cogsci10_proceedings.pdf
D’Mello, S., & Graesser, A. (2011). The half-life of cognitive-affective states during complex learning. Cognition and Emotion, 25(7), 1299–1308. https://dx.doi.org/10.1080/02699931.2011.613668
D’Mello, S., & Graesser, A. (2012a). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. https://dx.doi.org/10.1016/j.learninstruc.2011.10.001
D’Mello, S., & Graesser, A. (2012b). Emotions during learning with AutoTutor. In P. Durlach & A. Lesgold (Eds.), Adaptive technologies for training and education (pp. 117–139). Cambridge, UK: Cambridge University Press. https://dx.doi.org/10.1017/CBO9781139049580.010
D’Mello, S., & Graesser, A. (2014a). Confusion. In R. Pekrun & L. G. Lisa (Eds.), International handbook of emotions in education (pp. 289–310). Hoboken, NJ, USA: Taylor and Francis.
D’Mello, S., & Graesser, A. (2014b). Confusion and its dynamics during device comprehension with breakdown scenarios. Acta Psychologica, 151, 106–116. https://dx.doi.org/10.1016/j.actpsy.2014.06.005
D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153–170. https://dx.doi.org/10.1016/j.learninstruc.2012.05.003
D’Mello, S., Person, N., & Lehman, B. (2009). Antecedent-consequent relationships and cyclical patterns between affective states and problem solving outcomes. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (AIED ʼ09), 6–10 July 2009, Brighton, UK (pp. 57–64). IOS Press. https://dl.acm.org/doi/10.5555/1659450.1659463
D’Mello, S., Picard, R. W., & Graesser, A. (2007). Toward an affect-sensitive AutoTutor. IEEE Intelligent Systems, 22(4), 53–61. https://dx.doi.org/10.1109/MIS.2007.79
D’Mello, S., Taylor, R. S., & Graesser, A. (2007). Monitoring affective trajectories during complex learning. In D. S. McNamara & G. Trafton (Eds.), Proceedings of the 29th Annual Conference of the Cognitive Science Society (CogSci 2007), 1–4 August 2007, Nashville, TN, USA (pp. 197–202). Austin, TX: Cognitive Science Society.
Driver, R. (1983). The pupil as scientist? Milton Keynes, UK: Open University Press.
Duit, R., & Treagust, D. F. (2003). Conceptual change: A powerful framework for improving science teaching and learning. International Journal of Science Education, 25(6), 671–688. https://dx.doi.org/10.1080/09500690305016
Ellis, R. A., Han, F., & Pardo, A., (2017). Improving learning analytics—Combining observational and self-report data on student learning. Educational Technology & Society, 20(3), 158–169. Retrieved from https://www.jstor.org/stable/26196127
Engelbrecht, J., Harding, A., & Potgieter, M. (2005). Undergraduate students’ performance and confidence in procedural and conceptual mathematics. International Journal of Mathematical Education in Science and Technology, 36(7), 701–712. https://dx.doi.org/10.1080/00207390500271107
Fazio, L. K., & Marsh, E. J. (2009). Surprising feedback improves later memory. Psychonomic Bulletin & Review, 16(1), 88–92. https://dx.doi.org/10.3758/PBR.16.1.88
Fazio, L. K., & Marsh, E. J. (2010). Correcting false memories. Psychological Science, 21(6), 801–803. https://dx.doi.org/10.1177/0956797610371341
Fensham, P. J., & Kass, H. (1988). Inconsistent or discrepant events in science instruction. Studies in Science Education, 15(1), 1–16. https://dx.doi.org/10.1080/03057268808559946
Festinger, L. (1957). A theory of cognitive dissonance. Stanford, CA: Stanford University Press. Retrieved from https://psycnet.apa.org/record/1993-97948-000
Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting learning success. The Internet and Higher Education, 28, 68–84. https://dx.doi.org/10.1016/j.iheduc.2015.10.002
Gašević, D., Dawson, S., & Siemens, G. (2014). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://dx.doi.org/10.1007/s11528-014-0822-x
Gašević, D., Jovanovic, J., Pardo, A., & Dawson, S. (2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113–128. https://dx.doi.org/10.18608/jla.2017.42.10
Graesser, A. (2011). Learning, thinking, and emoting with discourse technologies. American Psychologist, 66(8), 746–757. https://dx.doi.org/10.1037/a0024974
Graesser, A., Baggett, W., & Williams, K. (1996). Question-driven explanatory reasoning. Applied Cognitive Psychology, 10(7), 17–31. https://dx.doi.org/10.1002/(SICI)1099-0720(199611)10:7<17::AID-ACP435>3.0.CO;2-7
Graesser, A., & D’Mello, S. (2012). Emotions during the learning of difficult material. Psychology of Learning and Motivation, 57, 183–225. https://dx.doi.org/10.1016/B978-0-12-394293-7.00005-4
Graesser, A., Lu, S., Olde, B. A., Cooper-Pye, E., & Whitten, S. (2005). Question asking and eye tracking during cognitive disequilibrium: Comprehending illustrated texts on devices when the devices break down. Memory & Cognition, 33(7), 1235–1247. https://dx.doi.org/10.3758/BF03193225
Graesser, A., & Olde, B. A. (2003). How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. Journal of Educational Psychology, 95(3), 524–536. https://dx.doi.org/10.1037/0022-06184.108.40.2064
Graesser, A., Ozuru, Y., & Sullins, J. (2010). What is a good question? In M. G. McKeown & L. Kucan (Eds.), Bringing reading research to life (pp. 112–141). New York, NY: Guilford. Retreived from https://psycnet.apa.org/record/2010-03564-007
Greiff, S., Wüstenberg, S., Csapó, B., Demetriou, A., Hautamäki, J., Graesser, A., & Martin, R. (2014). Domain-general problem solving skills and education in the 21st century. Educational Research Review, 13, 74–83. https://dx.doi.org/10.1016/j.edurev.2014.10.002
Gunstone, R. F. (1990). Children’s science: A decade of developments in constructivist views of science teaching and learning. The Australian Science Teachers Journal, 36(4).
Gunstone, R. F., & White, R. T. (1980). A matter of gravity. Research in Science Education, 10(1), 35–44. https://dx.doi.org/10.1007/BF02356307
Horodyskyj, L. B., Mead, C., Belinson, Z., Buxner, S., Semken, S., & Anbar, A. D. (2018). Habitable Worlds: Delivering on the promises of online education. Astrobiology, 18(1), 86–99. https://dx.doi.org/10.1089/ast.2016.1550
Huang, H. M. (2002). Toward constructivism for adult learners in online learning environments. British Journal of Educational Technology, 33(1), 27–37. https://dx.doi.org/10.1111/1467-8535.00236
Kang, S. H., Pashler, H., Cepeda, N. J., Rohrer, D., Carpenter, S. K., & Mozer, M. C. (2011). Does incorrect guessing impair fact learning? Journal of Educational Psychology, 103(1), 48–59. https://dx.doi.org/10.1037/a0021977
Kapur, M. (2016). Examining productive failure, productive success, unproductive failure and unproductive success in learning. Educational Psychologist, 51(2), 289–299. https://dx.doi.org/10.1080/00461520.2016.1155457
Karamustafaoğlu, S., & Mamlok-Naaman, R. (2015). Understanding electrochemistry concepts using the predict-observe-explain strategy. Eurasia Journal of Mathematics, Science and Technology Education, 11(5), 923–936. https://dx.doi.org/10.12973/eurasia.2015.1364a
Karumbaiah, S., Andres, J. M. A. L., Botelho, A. F., Baker, R., & Ocumpaugh, J. (2018). The implications of a subtle difference in the calculation of affect dynamics. In Proceedings of the 26th International Conference on Computers in Education (ICCE ’18), 26–30 November 2018, Manila, Philippines (pp. 29–38). Philippines: Asia-Pacific Society for Computers in Education. Retrieved from http://icce2018.ateneo.edu/wp-content/uploads/2018/12/C1-04.pdf
Kassambara, A. (2017). Practical guide to cluster analysis in R—Unsupervised machine learning. STHDA. Retredfddfived from https://www.sthda.com
Keltner, D., & Shiota, M. N. (2003). New displays and new emotions: A commentary on Rozin and Cohen (2003). Emotion, 3, 86–91. https://dx.doi.org/10.1037/1528-35220.127.116.11
Kendeou, P., & Broek, P. V. D. (2005). The effects of readers’ misconceptions on comprehension of scientific text. Journal of Educational Psychology, 97(2). https://dx.doi.org/10.1037/0022-0618.104.22.168
Kennedy, G., & Lodge, J. (2016). All roads lead to Rome: Tracking students’ affect as they overcome misconceptions. In S. Barker, S. Dawson, A. Pardo, & C. Colvin (Eds.), Show Me The Learning: Proceedings of the 33rd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2016). 28–30 November 2016, Wellington, New Zealand (pp. 318–328). Australasian Society for Computers in Learning in Tertiary Education. Retrieved from https://2016conference.ascilite.org/wp-content/uploads/ascilite2016_kennedy_full.pdf
Kibirige, I., Osodo, J., & Tlala, K. M. (2014). The effect of predict-observe-explain strategy on learners’ misconceptions about dissolved salts. Mediterranean Journal of Social Sciences, 5(4), 300–310. https://dx.doi.org/10.5901/mjss.2014.v5n4p300
Kostyuk, V., Almeda, M., & Baker, R. (2018). Correlating affect and behavior in reasoning mind with state test achievement. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, Australia (pp. 26–30). New York, NY: ACM. https://dx.doi.org/10.1145/3170358.3170378
Kovanovic, V., Gašević, D., Dawson, S., Joksimovic, S., Baker, R., & Hatala, M. (2015). Does time-on-task estimation matter? Implications for the validity of learning analytics findings. Journal of Learning Analytics, 2(3), 81–110. https://dx.doi.org/10.18608/jla.2015.23.6
Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567. https://dx.doi.org/10.1146/annurev.psych.50.1.537
Kulhavy, R. W. (1977). Feedback in written instruction. Review of Educational Research, 47(2), 211–232. https://dx.doi.org/10.3102/00346543047002211
Kulhavy, R. W., & Stock, W. A. (1989). Feedback in written instruction: The place of response certitude. Educational Psychology Review, 1(4), 279–308. https://dx.doi.org/10.1007/BF01320096
Kulhavy, R. W., Yekovich, F. R., & Dyer, J. W. (1976). Feedback and response confidence. Journal of Educational Psychology, 68(5), 522–528. https://dx.doi.org/10.1037/0022-0622.214.171.1242
Lagud, M. C. V., & Rodrigo, M. M. T. (2010). The affective and learning profiles of students using an intelligent tutoring system for algebra. In V. Aleven, J. Kay, & J. Mostow (Eds.), Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS 2010), 14–18 June 2010, Pittsburgh, PA, USA, (pp. 255–263). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-642-13388-6_30
Lazarus, R. S. (1991). Cognition and motivation in emotion. American Psychologist, 46, 352–367. https://dx.doi.org/10.1037/0003-066X.46.4.352
Lee, D. M. C., Rodrigo, M. M. T., Baker, R., Sugay, J. O., & Coronel, A. (2011). Exploring the relationship between novice programmer confusion and achievement. In S. K. D’Mello, A. C. Graesser, B. W. Schuller, & Jean-Claude Martin (Eds.), Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII ’11), 9–12 October 2011, Memphis, TN, USA (pp. 175–184). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-642-24600-5_21
Lehman, B., D’Mello, S., Chauncey, A., Gross, M., Dobbins, A., Wallace, P., …, Graesser, A. (2011). Inducing and tracking confusion with contradictions during critical thinking and scientific reasoning. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education (AIED ʼ11), 28 June–2 July 2011, Auckland, New Zealand (pp. 171–178). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-642-21869-9_24
Lehman, B., D’Mello, S., & Graesser, A. (2012). Confusion and complex learning during interactions with computer learning environments. The Internet and Higher Education, 15(3), 184–194. https://dx.doi.org/10.1016/j.iheduc.2012.01.002
Lehman, B., & Graesser, A. (2015). To resolve or not to resolve? That is the big question about confusion. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED ʼ15), 22–26 June 2015, Madrid, Spain (pp. 216–225). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-319-19773-9_22
Liew, C. W., & Treagust, D. F. (1995). A predict-observe-explain teaching sequence for learning about students’ understanding of heat and expansion of liquids. Australian Science Teachers’ Journal, 41(1), 68–71. Retrieved from https://www.researchgate.net/publication/234752631_A_Predict-Observe-Explain_Teaching_Sequence_for_Learning_about_Students'_Understanding_of_Heat_and_Expansion_Liquids
Liew, C. W., & Treagust, D. F. (1998). The effectiveness of predict-observe-explain tasks in diagnosing students’ understanding of science and in identifying their levels of achievement. Retrieved from https://eric.ed.gov/?id=ED420715
Limón, M. (2001). On the cognitive conflict as an instructional strategy for conceptual change: A critical appraisal. Learning and Instruction, 11(4-5), 357–380. https://dx.doi.org/10.1016/S0959-4752(00)00037-2
Liu, Z., Pataranutaporn, V., Ocumpaugh, J., & Baker, R. (2013). Sequences of frustration and confusion, and learning. In S. K. DʼMello et al. (Eds.), Proceedings of the 6th International Conference on Educational Data Mining (EDM2013), 6–9 July 2013, Memphis, TN, USA (pp. 114–120). International Educational Data Mining Society/Springer. Retrieved from https://files.eric.ed.gov/fulltext/ED558215.pdf
Lodge, J., Kennedy, G., Lockyer, L., Arguel, A., & Pachman, M. (2018). Understanding difficulties and resulting confusion in learning: An integrative review. Frontiers in Education, 3, 49. https://dx.doi.org/10.3389/feduc.2018.00049
MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 1 January 1967, Oakland, CA, USA (pp. 281–297).
Mandler, G. (1984). Mind and body: Psychology of emotion and stress. New York, NY: Norton.
Mansour, B. E., & Mupinga, D. M. (2007). Students’ positive and negative experiences in hybrid and online classes. College Student Journal, 41(1). Retrieved from https://eric.ed.gov/?id=EJ765422
Markant, D., & Gureckis, T. M. (2014a). A preference for the unpredictable over the informative during self-directed learning. In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th Annual Conference of the Cognitive Science Society (CogSci 2014), 23–26 July 2014, Quebec City, QC, Canada (pp. 958–963). Austin, TX: Cognitive Science Society. Retrieved from https://cogsci.mindmodeling.org/2014/papers/172/paper172.pdf
Markant, D. B., & Gureckis, T. M. (2014b). Is it better to select or to receive? Learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1), 94–122. https://dx.doi.org/10.1037/a0032108
Markant, D. B., Settles, B., & Gureckis, T. M. (2016). Self-directed learning favors local, rather than global, uncertainty. Cognitive Science, 40(1), 100–120. https://dx.doi.org/10.1111/cogs.12220
McLeod, D. B. (1989). The role of affect in mathematical problem solving. In D. B. McLeod & V. M. Adams (Eds.), Affect and Mathematical Problem Solving (pp. 20–36). New York, NY: Springer. https://dx.doi.org/10.1007/978-1-4612-3614-6_2
McQuiggan, S. W., Robison, J. L., & Lester, J. C. (2010). Affective transitions in narrative-centered learning environments. Educational Technology & Society, 13(1), 40–53. Retrieved from https://www.jstor.org/stable/jeductechsoci.13.1.40
Metcalfe, J., & Finn, B. (2011). People’s hypercorrection of high-confidence errors: Did they know it all along? Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(2). https://dx.doi.org/10.1037/a0021962
Miller, W. L., Baker, R., Labrum, M. J., Petsche, K., Liu, Y. H., & Wagner, A. Z. (2015). Automated detection of proactive remediation by teachers in Reasoning Mind classrooms. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 290–294). New York, NY: ACM. https://dx.doi.org/10.1145/2723576.2723607
Nawaz, S., Kennedy, G., Bailey, J., Mead, C., & Horodyskyj, L. (2018). Struggle town? Developing profiles of student confusion in simulation-based learning environments. Proceedings of the 35th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2018), 25–28 November 2018, Geelong, Australia (pp. 224–233). Australasian Society for Computers in Learning in Tertiary Education. Retrieved from https://2018conference.ascilite.org/wp-content/uploads/2018/12/ASCILITE-2018-Proceedings-Final.pdf
Nawaz, S., Srivastava, N., Yu, J. H., Baker, R., Kennedy, G., & Bailey, J. (2020). Analysis of task difficulty sequences in a simulation-based POE environment. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán (Eds.), Proceedings of the 21st International Conference on Artificial Intelligence in Education (AIED ʼ20), 6–10 July 2020, Ifrane, Morocco (virtual conference due to COVID-19) (pp. 423–436). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-030-52237-7_34
Nawaz, S., Srivastava, N., Yu, J. H., Khan, A. A., Kennedy, G., Bailey, J., & Baker, R. (2020). How difficult is the task for you? Modelling and analysis of students’ task difficulty sequences. International Journal of Artificial Intelligence in Education (IJAIED), submitted.
Ocumpaugh, J., Andres, J. M., Baker, R., DeFalco, J., Paquette, L., Rowe, J., … , Sottilare, R. (2017). Affect dynamics in military trainees using vMedic: From engaged concentration to boredom to confusion. In E. André, R. S. Baker, X. Hu, M. M. T. Rodrigo, & B. du Boulay (Eds.), Proceedings of the 18th International Conference on Artificial Intelligence in Education (AIED 2017), 28 June–1 July 2017, Wuhan, China (pp. 238–249). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-319-61425-0_20
Pachman, M., Arguel, A., & Lockyer, L. (2015). Learners’ confusion: Faulty prior knowledge or a metacognitive monitoring error? In T. Reiners et al. (Eds.), Globally connected, digitally enabled. Proceedings of the 32nd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2015), 29 November–2 December 2015, Perth, Western Australia (pp. 522–526). Australasian Society for Computers in Learning in Tertiary Education. Retrieved from http://www.2015conference.ascilite.org/wp-content/uploads/2015/11/ascilite-2015-proceedings.pdf
Pardos, Z. A., Baker, R. S. J. D., Pedro, M. S., Gowda, S. M., & Gowda, S. M. (2014). Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. Journal of Learning Analytics, 1(1), 107–128. https://dx.doi.org/10.18608/jla.2014.11.6
Pardos, Z. A., & Horodyskyj, L. (2019). Analysis of student behaviour in Habitable Worlds using continuous representation visualization. Journal of Learning Analytics, 6(1), 1–15. https://dx.doi.org/10.18608/jla.2019.61.1
Paulhaus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of social psychological attitudes, Vol. 1. Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA: Academic Press. https://dx.doi.org/10.1016/B978-0-12-590241-0.50006-X
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of quantitative and qualitative research. Educational Psychologist, 37(2), 91–106. https://dx.doi.org/10.1207/S15326985EP3702_4
Pekrun, R., & Stephens, E. J. (2012). Academic emotions. In K. R. Harris, S. Graham, T. Urdan, S. Graham, J. M. Royer, & M. Zeidner (Eds.), APA handbooks in psychology®. APA educational psychology handbook, Vol. 2. Individual differences and cultural and contextual factors (pp. 3–31). Washington, DC: American Psychological Association. https://dx.doi.org/10.1037/13274-001
Piaget, J. (1952a). The origins of intelligence. New York, NY: International University Press.
Piaget, J. (1952b). The origins of intelligence in children (Vol. 8). New York, NY: International University Press.
Piaget, J. (1985). The equilibration of cognitive structures: The central problem of intellectual development. Chicago, IL: University of Chicago Press.
Rangel, R. H., Möller, L., Sitter, H., Stibane, T., & Strzelczyk, A. (2017). Sure, or unsure? Measuring students’ confidence and the potential impact on patient safety in multiple-choice questions. Medical Teacher, 39(11), 1189–1194. https://dx.doi.org/10.1080/0142159X.2017.1362103
Rodrigo, M. M. T., Baker, R. S., Jadud, M. C., Amarra, A. C. M., Dy, T., Espejo-Lanoz, M. B. V., …, Tabanao, E. S. (2009). Affective and behavioral predictors of novice programmer achievement. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE ’09), 6–9 July 2009, Paris, France (pp. 156–160). New York, NY: ACM. https://dx.doi.org/10.1145/1562877.1562929
Rodrigo, M. M. T., Baker, R., Lagud, M. C. V., Lim, S. A. L., Macapanpan, A. F., Pascua, S. A. M. S., …, Viehland, N. J. B. (2007). Affect and usage choices in simulation problem solving environments. In Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED 2007), 9–13 July 2007, Los Angeles, CA, USA (pp. 145–152). The Netherlands: IOS Press. Retrieved from https://dl.acm.org/doi/10.5555/1563601.1563629
Rodriguez, F., Yu, R., Park, J., Rivas, M. J., Warschauer, M., & Sato, B. K. (2019). Utilizing learning analytics to map students’ self-reported study strategies to click behaviors in STEM courses. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2018, Tempe, AZ, USA (pp. 456–460). New York, NY: ACM. https://dx.doi.org/10.1145/3303772.3303841
Rosenthal, J. A. (1996). Qualitative descriptors of strength of association and effect size. Journal of Social Service Research, 21(4), 37–59. https://dx.doi.org/10.1300/J079v21n04_02
Rosenthal, R., & Rosnow, R. L. (1984). Essentials of behavioral research: Methods and data analysis. New York, NY: McGraw-Hill.
Rozin, P., & Cohen, A. B. (2003). High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of naturally occurring facial expressions of Americans. Emotion, 3, 68–75. https://dx.doi.org/10.1037/1528-35126.96.36.199
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://dx.doi.org/10.3102/0034654307313795
Silvia, P. J. (2009). Looking past pleasure: Anger, confusion, disgust, pride, surprise, and other unusual aesthetic emotions. Psychology of Aesthetics, Creativity, and the Arts, 3(1), 48–51. https://dx.doi.org/10.1037/a0014632
Silvia, P. J. (2010). Confusion and interest: The role of knowledge emotions in aesthetic experience. Psychology of Aesthetics, Creativity, and the Arts, 4(2), 75–80. https://dx.doi.org/10.1037/a0017081
Sreerekha, S., Arun, R. R., & Sankar, S. (2016). Effect of predict-observe-explain strategy on achievement in chemistry of secondary school students. International Journal of Education & Teaching Analytics, 1(1). Retrieved from http://www.innoriginal.com/index.php/ijeta/article/view/35
Storm, C., & Storm, T. (1987). A taxonomic study of the vocabulary of emotions. Journal of Personality and Social Psychology, 53, 805–816. https://dx.doi.org/10.1037/0022-35188.8.131.525
Sun, P. (2006). Outlier detection in high dimensional, spatial and sequential data sets (PhD thesis). Sydney University.
Tao, P. K., & Gunstone, R. F. (1999). The process of conceptual change in force and motion during computer-supported physics instruction. Journal of Research in Science Teaching, 36(7), 859–882. https://dx.doi.org/10.1002/(SICI)1098-2736(199909)36:7<859::AID-TEA7>3.0.CO;2-J
Tippett, C. D. (2010). Refutation text in science education: A review of two decades of research. International Journal of Science and Mathematics Education, 8(6), 951–970. https://dx.doi.org/10.1007/s10763-010-9203-x
VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 21(3), 209–249. https://dx.doi.org/10.1207/S1532690XCI2103_01
White, P. A. (1989). Evidence for the use of information about internal events to improve the accuracy of causal reports. British Journal of Psychology, 80, 375–382. https://dx.doi.org/10.1111/j.2044-8295.1989.tb02327.x
White, R. F., & Gunstone, R. T. (1992). Probing understanding. London, UK: Routledge. https://dx.doi.org/10.4324/9780203761342
Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students’ calibration of self reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551–572. https://dx.doi.org/10.1016/S0361-476X(02)00006-1
Wise, A., Speer, J., Marbouti, F., & Hsiao, Y. T. (2013). Broadening the notion of participation in online discussions: Examining patterns in learners’ online listening behaviors. Instructional Science, 41, 323–343. https://dx.doi.org/10.1007/s11251-012-9230-9
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. https://dx.doi.org/10.1016/j.learninstruc.2012.03.004
How to Cite
Copyright (c) 2020 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) license that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).