A Method for Automatically Analyzing Intelligent Tutoring System Dialogues with Coh-Metrix
We developed a method for using Coh-Metrix to automatically analyze tutorial dialogues. Coh-Metrix, a web-based tool for automatically evaluating text, is freely available to researchers. We applied the method to 190 tutorial dialogues between women and BRCA Gist from two experiments. BRCA Gist is an intelligent tutoring system (ITS) to help women make decisions about genetic testing for breast cancer risk. Tutorial dialogues scored high on measures of textual cohesion (deep cohesion, referential cohesion, and the composite variable formality). They also scored high on measures of the situation model (LSA verb overlap and causal verb and causal particle). However, there was mixed support for the hypothesis that higher scores on Coh-Metrix variables would predict subsequent comprehension. A Coh-Metrix principle is that the observable cohesion of a text is a reliable guide to the coherence of the reader’s mental representation of that text. Thus it appears that interacting with BRCA Gist helped people form coherent mental representations of complex medical materials. We conclude that Coh-Metrix can be used to reliably assess tutorial dialogues and make inferences about the mental representations of people engaged in conversation with an ITS based on observable characteristics of the statements people make.
Blalock, S. J. & Reyna, V. F. (2016). Using Fuzzy-Trace Theory to understand and improve health judgments, decisions, and behaviors: A literature review. Health Psychology, 35, 781-792.
Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4-16.
Bussmann, H. (1996). Routledge dictionary of language and linguistics. London: Routledge.
Chi, M. T. H., Siler, S. A., Jeong, H., Yamauchi, T., &, Hausmann, R. G. (2001). Learning from human tutoring. Cognitive Science, 25(4), :471-533.
Crossley, S. A., Kyle, K., & McNamara, D. S. (2016). The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods, 48, 1227-1237.
Dowell, N. M. M., Graesser, A. C., & Cai, Z. (2016). Language and discourse analysis with Coh-Metrix: Applications from educational material to learning environments at scale. Journal of Learning Analytics, 3, 72–95. http://dx.doi.org/10.18608/jla.2016.33.5
Fisher, C. R., Wolfe, C. R., Reyna, V. F., Widmer, C. L., Cedillos, E. M., & Brust-Renck, P. G (2013). A signal detection analysis of gist-based discrimination of genetic breast cancer risk. Behavior Research Methods, 45, 613–622.
Graesser, A. C. (2011). Learning, thinking, and emoting with discourse technologies. American Psychologist, 66, 746-757.
Graesser, A. C., Jeon, M., Yang, Y., & Cai, Z. (2007). Discourse cohesion in text and tutorial dialogue. Information Design Journal, 15, 199-213.
Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A., & Louwerse, M. M. (2004). AutoTutor: A tutor with dialogue in natural language. ￼Behavior Research Methods, Instruments, & Computers, 36, 180-192.
Graesser, A. C., McNamara, D. S., Cai, Z. Conley, M. Li, H., & Pennebaker, J. (2014). Coh-Metrix measures text characteristics at multiple levels of language and discourse. The Elementary School Journal, 115(2), 210-229.
Graesser, A., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371-395.
Grice, H. P. (1975). Logic and conversation. In P. Cole, and J.L. Morgan, (Eds.) Speech Acts (pp. 41–58). New York: Academic Press.
Hu X, Han L, Cai Z. Semantic decomposition of student’s contributions: An implementation of LCC in AutoTutor Lite. Paper presented at the Society for Computers in Psychology; November 2008; Chicago, IL.
Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95(2), 163-182.
Kintsch, W. (1994). Text comprehension, memory, and learning. American Psychologist, 49(4), 294-303.
Kintsch, W., & Van Dijk, T. A. (1978). Toward a model of text comprehension and production. Psychological Review, 85(5), 363-394.
Kintsch, W., & Yarbrough, C. (1982). Role of rhetorical structure in text comprehension. Journal of Educational Psychology, 74(6), 828-834.
Li, H. Y. & Graesser, A. C. (2016). Formality of the Chinese collective leadership. Behavior Research Methods, 48, 922-935.
McNamara, D. S., Graesser, A. C., McCarthy,P. M., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press: Cambridge, UK.
McNamara, D. S., Kintsch, E., Butler Songer, N., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14(1), 1-43.
Miller, J. & Kintsch, W. (1980). Readability and recall of short prose passages: A theoretical analysis. Journal of Experimental Psychology: Human Learning and Memory, 6(4), 335-354.
Nelson, J., Perfetti, C., Liben, D., & Liben, M. (2011). Measures of text difficulty: Testing their predictive value for grade levels and student performance. New York: Student Achievement Partners.
Reyna, V. F., (2004). How people make decisions that involve risk - A dual-processes approach. Current Directions in Psychological Science, 13, 60-66.
Reyna V. F. (2008). Theories of medical decision making and health: An evidence-based approach. Medical Decision Making, 28(6), 829-833.
Reyna, V. F. (2012). A new intuitionism: Meaning, memory, and development in Fuzzy-Trace Theory. Judgment and Decision Making. ,7(3),332–359.
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197-221.
Widmer, C. L., Wolfe, C. R., Reyna, V. F., Cedillos-Whynott, E. M., Brust-Renck, P. G., & Weil, A. M. (2015). Tutorial dialogues and gist explanations of genetic breast cancer risk. Behavior Research Methods, 47, 632-648.
Wolfe, C. R., Reyna, V. F., & Brainerd, C. J. (2005). Fuzzy-Trace Theory: Implications for transfer in teaching and learning. In J. P. Mestre (Ed.) Transfer of Learning from a Modern Multidisciplinary Perspective (p. 53-88). Greenwich, CT: Information Age Press.
Wolfe, C. R., Fisher, C. R., Reyna, V. F., & Hu, X. (2012). Improving internal consistency in conditional probability estimation with an Intelligent Tutoring System and web-based tutorials. International Journal of Internet Science, 7, 38-54.
Wolfe, C. R., Reyna, V. F., Widmer, C. L., Cedillos-Whynott, E. M., Brust-Renck, P. G., Weil, A. M., & Hu, X. (2016). Understanding genetic breast cancer risk: Processing loci of the BRCA Gist intelligent tutoring system. Learning and Individual Differences, 49, 178-189.
Wolfe, C. R., Reyna, V. F., Widmer, C. L., Cedillos, E. M., Fisher, C. R., Brust-Renck, P. G., Chaudhry, S. & Damas Vannucchi, I. (2013). Efficacy of a Web-Based Intelligent Tutoring System on genetic testing for breast cancer risk. Presentation to the 6th Annual Scientific Meeting of the International Society for Research on Internet Interventions, Chicago, IL.
Wolfe, C. R., Reyna, V. F., Widmer, C. L., Cedillos, E. M., Fisher, C. R., Brust-Renck, P. G., & Weil, A. M. (2015). Efficacy of a web-based Intelligent Tutoring System for communicating genetic risk of breast cancer: A Fuzzy-Trace Theory approach. Medical Decision Making, 35, 46-59. doi 0272989X14535983.
Wolfe, C. R., Reyna, V. F., Widmer, C. L., Cedillos, E., Weil, A. M., & Brust-Renck, P. G. (2016). Pumps and prompts for gist explanations in tutorial dialogues about breast cancer. Discourse Processes. DOI: 10.1080/0163853X.2016.1199626
Wolfe, C. R., Widmer, C. L., Reyna, V. F., Hu, X., Cedillos, E. M., Fisher, C. R., Brust-Renck, P. G., Williams, T. C., Damas Vannucchi, I., & Weil, A. M. (2013). The development and analysis of tutorial dialogues in AutoTutor Lite. Behavior Research Methods, 45, 623–636.
Wolfe, C. R., Reyna, V. F., Cedillos, E. M., Widmer, C. L., Fisher, C. R., & Brust-Renck, P. G. (2012). An Intelligent Tutoring System to help women decide about testing for genetic breast cancer risk. Paper presented to the 34th Annual Meeting of the Society for Medical Decision Making, Phoenix, AZ.
Zwaan, R. A. & Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123, 162-185.
Share this article:
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.