A Method for Automatically Analyzing Intelligent Tutoring System Dialogues with Coh-Metrix

Authors

DOI:

https://doi.org/10.18608/jla.2018.53.14

Keywords:

Discourse Technology, Tutoring, Breast Cancer Education, Textual Assessment, Situation Model

Abstract

We developed a method for using Coh-Metrix to automatically analyze tutorial dialogues. Coh-Metrix, a web-based tool for automatically evaluating text, is freely available to researchers. We applied the method to 190 tutorial dialogues between women and BRCA Gist from two experiments. BRCA Gist is an intelligent tutoring system (ITS) to help women make decisions about genetic testing for breast cancer risk. Tutorial dialogues scored high on measures of textual cohesion (deep cohesion, referential cohesion, and the composite variable formality). They also scored high on measures of the situation model (LSA verb overlap and causal verb and causal particle). However, there was mixed support for the hypothesis that higher scores on Coh-Metrix variables would predict subsequent comprehension. A Coh-Metrix principle is that the observable cohesion of a text is a reliable guide to the coherence of the reader’s mental representation of that text. Thus it appears that interacting with BRCA Gist helped people form coherent mental representations of complex medical materials. We conclude that Coh-Metrix can be used to reliably assess tutorial dialogues and make inferences about the mental representations of people engaged in conversation with an ITS based on observable characteristics of the statements people make.

Author Biographies

Chrstopher R. Wolfe, Miami University

Professor of Psychology

Colin L. Widmer, Miami University

Ph.D.

Christine V. Torrese, Miami University

B.A.

Mitchell Dandignac, Miami University

M.A.

Downloads

Published

2018-12-11

How to Cite

Wolfe, C. R., Widmer, C. L., Torrese, C. V., & Dandignac, M. (2018). A Method for Automatically Analyzing Intelligent Tutoring System Dialogues with Coh-Metrix. Journal of Learning Analytics, 5(3), 222–234. https://doi.org/10.18608/jla.2018.53.14