Markers of Cognitive Quality in Student Contributions to Online Course Discussion Forums

Authors

DOI:

https://doi.org/10.18608/jla.2022.7250

Keywords:

discussion forum, participation, engagement, quality, community of inquiry, cognitive presence, ICAP, feature analysis, research paper

Abstract

By participating in asynchronous course discussion forums, students can work together to refine their ideas and construct knowledge collaboratively. Typically, some messages simply repeat or paraphrase course content, while others bring in new material, demonstrate reasoning, integrate concepts, and develop solutions. Through the messages they send, students thus display different levels of intellectual engagement with the topic and the course. We refer to this as cognitive quality. The work presented here used two widely studied frameworks for assessing critical discourse and cognitive engagement: the ICAP and Community of Inquiry frameworks. The constructs of the frameworks were used as proxy measures for cognitive quality. Predictive classifiers were trained for both frameworks on the same data in order to discover which attributes of the dialogue were most informative and how those attributes were correlated with framework constructs. We found that longer and more complex messages were associated with indicators of greater quality in both frameworks, and that the threaded reply structure mattered more than chronological order. By including the framework labels as additional model features, we also assessed the links between frameworks. The empirical results provide evidence that the two frameworks measure different aspects of student behaviour relating to cognitive quality.

Author Biography

Dragan Gašević, Monash University



References

Atapattu, T., Thilakaratne, M., Vivian, R., & Falkner, K. (2019). Detecting cognitive engagement using word embeddings within an online teacher professional development community. Computers & Education. doi: 10.1016/j.compedu.2019.05.020

Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanovi´c, V., & Gaševi´c, D. (2020). Towards automatic cross-language classification of cognitive presence in online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 605–614). New York, NY, USA: ACM. doi: 10.1145/3375462.3375496

Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. doi: 10.1023/A:1010933404324

Chawla, N. V., Bowyer, K.W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321–357. doi: 10.1613/jair.953

Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13(2), 145–182. doi: 10.1207/s15516709cog1302 1

Chi, M. T. H., & Wylie, R. (2014). The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes. Educational Psychologist, 49(4), 219–243. doi: 10.1080/00461520.2014.965823

Corich, S., Kinshuk, & Hunt, L. M. (2006). Measuring critical thinking within discussion forums using a computerised content analysis tool. In Proceedings of the 5th International Conference on Networked Learning (pp. 1–8). Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.102.8598&rep=rep1&type=pdf

Farrow, E., Moore, J., & Gaševi´c, D. (2020). Dialogue attributes that inform depth and quality of participation in course discussion forums. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 129–134). New York, NY, USA: ACM. doi: 10.1145/3375462.3375481

Farrow, E., Moore, J., & Gaševi´c, D. (2019). Analysing discussion forum data: A replication study avoiding data contamination. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 170–179). New York, NY, USA: ACM. doi: 10.1145/3303772.3303779

Ferguson, R., Wei, Z., He, Y., & Buckingham Shum, S. (2013). An evaluation of learning analytics to identify exploratory dialogue in online discussions. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 85–93). New York, New York, USA: ACM Press. doi: 10.1145/2460296.2460313

Ferreira, M., Rolim, V., Mello, R. F., Lins, R. D., Chen, G., & Gaševi´c, D. (2020). Towards automatic content analysis of social presence in transcripts of online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 141–150). New York, NY, USA: ACM. doi: 10.1145/3375462.3375495

Ferreira, R., Kovanovi´c, V., Gaševi´c, D., & Rolim, V. (2018). Towards Combined Network and Text Analytics of Student Discourse in Online Discussions. In Proceedings of the International Conference on Artificial Intelligence in Education (Vol. 10331, pp. 111–126). Springer International Publishing. doi: 10.1007/978-3-319-93843-1 9

Gardner, J., Brooks, C., Andres, J. M., & Baker, R. (2018). Replicating MOOC predictive models at scale. In Proceedings of the 5th Annual ACM Conference on Learning at Scale (pp. 1–10). doi: 10.1145/3231644.3231656

Garrison, D. R. (2011). E-learning in the 21st century: A framework for research and practice (2nd ed.). New York and London: Routledge. doi: 10.4324/9780203838761

Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2), 87–105. doi: 10.1016/S1096-7516(00)00016-6

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7–23. doi: 10.1080/08923640109527071

Gaševi´c, D., Adesope, O., Joksimovi´c, S., & Kovanovi´c, V. (2015). Externally-facilitated regulation scaffolding and role assignment to develop cognitive presence in asynchronous online discussions. The Internet and Higher Education, 24, 53–65.

doi: 10.1016/j.iheduc.2014.09.006

Hu, Y., Donald, C., Giacaman, N., & Zhu, Z. (2020). Towards automated analysis of cognitive presence in MOOC discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 135–140). New York, NY,

USA: ACM. doi: 10.1145/3375462.3375473

Kovanovi´c, V., Joksimovi´c, S., Gaševi´c, D., & Hatala, M. (2014). Automated cognitive presence detection in online discussion transcripts. CEUR Workshop Proceedings, 1137. Retrieved from http://ceur-ws.org/Vol-1137/LA machinelearning submission 1.pdf

Kovanovi´c, V., Joksimovi´c, S., Waters, Z., Gaševi´c, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards Automated Content Analysis of Discussion Transcripts: A Cognitive Presence Case. In Proceedings of the Sixth International Conference

on Learning Analytics & Knowledge (pp. 15–24). New York, NY, USA: ACM. doi: 10.1145/2883851.2883950

Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159–174. doi: 10.2307/2529310

Le, Q. V., & Mikolov, T. (2014). Distributed Representations of Sentences and Documents. In Proceedings of the 31st International Conference on Machine Learning (Vol. 32, p. 9). JMLR.org. Retrieved from http://proceedings.mlr.press/v32/le14.pdf

McKlin, T. (2004). Analyzing Cognitive Presence in Online Courses Using an Artificial Neural Network (Doctoral dissertation, Georgia State University). Retrieved from http://scholarworks.gsu.edu/msit diss/1

McKlin, T., Harmon, S. W., Evans, W., & Jones, M. G. (2001). Cognitive presence in web-based learning: A content analysis of students’ online discussions. In Annual proceedings of selected research and development and practice papers presented

at the National Convention of the Association for Educational Communications and Technology (Vol. 1-2, pp. 272–277). Retrieved from https://files.eric.ed.gov/fulltext/ED470101.pdf

McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated Evaluation of Text and Discourse with Coh-Metrix. New York: Cambridge University Press. doi: 10.1017/CBO9780511894664

Neto, V., Rolim, V., Ferreira, R., Kovanovi´c, V., Gaševi´c, D., Dueire Lins, R., & Lins, R. (2018). Automated Analysis of Cognitive Presence in Online Discussions Written in Portuguese. In Proceedings of the 13th European Conference on Technology Enhanced Learning (pp. 245–261). Cham: Springer International Publishing. doi: 10.1007/978-3-319-98572-5_19

Taskin, Y., Hecking, T., Hoppe, H. U., Dimitrova, V., & Mitrovic, A. (2019). Characterizing comment types and levels of engagement in video-based learning as a basis for adaptive nudging. In Proceedings of the 14th European Conference on Technology Enhanced Learning (pp. 362–376). Cham: Springer International Publishing. doi: 10.1007/978-3-030-29736-7_27

Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. doi: 10.1177/0261927X09351676

Wang, X., Wen, M., & Ros´e, C. P. (2016). Towards triggering higher-order thinking behaviors in MOOCs. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 398–407). New York, NY, USA: ACM. doi: 10.1145/2883851.2883964

Wang, X., Yang, D., Wen, M., Koedinger, K., & Ros´e, C. P. (2015). Investigating how student’s cognitive behavior in MOOC discussion forums affect learning gains. In Proceedings of the 8th International Conference on Educational

Data Mining (pp. 226–233). Massachusetts, USA: International Educational Data Mining Society. Retrieved from http://www.educationaldatamining.org/EDM2015/proceedings/full226-233.pdf

Waters, Z., Kovanovi´c, V., Kitto, K., & Gaševi´c, D. (2015). Structure Matters: Adoption of Structured Classification Approach in the Context of Cognitive Presence Classification. In Information Retrieval Technology (Vol. 9460, pp. 227–238). Cham:

Springer International Publishing. doi: 10.1007/978-3-319-28940-3 18

Wise, A. F., & Cui, Y. (2018). Unpacking the relationship between discussion forum participation and learning in MOOCs. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 330–339). New York, New

York, USA: ACM Press. doi: 10.1145/3170358.3170403

Wise, A. F., Cui, Y., & Vytasek, J. (2016). Bringing order to chaos in MOOC discussion forums with content-related thread identification. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 188–197). New York, New York, USA: ACM Press. doi: 10.1145/2883851.2883916

Yogev, E., Gal, K., Karger, D., Facciotti, M. T., & Igo, M. (2018). Classifying and visualizing students’ cognitive engagement in course readings. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (pp. 1–10). New York, NY, USA: ACM. doi: 10.1145/3231644.3231648

Downloads

Published

2021-08-31

How to Cite

Farrow, E., Moore, J., & Gašević, D. (2021). Markers of Cognitive Quality in Student Contributions to Online Course Discussion Forums. Journal of Learning Analytics, 9(2), 38-65. https://doi.org/10.18608/jla.2022.7250

Issue

Section

Extended Conference Papers

Most read articles by the same author(s)

<< < 1 2 3 > >>