Towards Reflective Writing Analytics: Rationale, Methodology and Preliminary Results


  • Simon Buckingham Shum University of Technology Sydney
  • Ágnes Sándor Xerox Research Centre Europe
  • Rosalie Goldsmith University of Technology Sydney
  • Randall Bass Georgetown University
  • Mindy McWilliams Georgetown University



learning analytics, education, writing analytics, reflection, natural language processing, reflective move, rhetoric


When used effectively, reflective writing tasks can deepen learners’ understanding of key concepts, help them critically appraise their developing professional identity, and build qualities for lifelong learning. As such, reflecting writing is attracting substantial interest from universities concerned with experiential learning, reflective practice, and developing a holistic conception of the learner. However, reflective writing is for many students a novel genre to compose in, and tutors may be inexperienced in its assessment. While these conditions set a challenging context for automated solutions, natural language processing may also help address the challenge of providing real time, formative feedback on draft writing. This paper reports progress in designing a writing analytics application, detailing the methodology by which informally expressed rubrics are modelled as formal rhetorical patterns, a capability delivered by a novel web application. Preliminary tests on an independently human-annotated corpus are encouraging, showing improvements from the first to second version, but with much scope for improvement. We discuss a range of issues: the prevalence of false positives in the tests, areas for futures technical improvements, the risks of gaming the system, and the participatory design process that has enabled work across disciplinary boundaries to develop the prototype to its current state.


Aït-Mokhtar, S., Chanod, J-P., & Roux, C. (2002). Robustness beyond shallowness: Incremental deep parsing. Natural Language Engineering, 8(2/3), 121–144.

Bass, R., & Eynon, B. (2016). Open and integrative: Designing liberal education for the new digital ecosystem. Association of American Colleges and Universities.

Boud, D., Keogh, R., & Walker, D. (1985). Reflection: Turning experience into learning. London: Routledge, Abingdon, Oxon.

Boud, D., & Walker, D. (1998). Promoting reflection in professional courses: The challenge of context. Studies in Higher Education, 23(2), 191–206.

Brun, C., & Hagège, C. (2003). Normalization and paraphrasing using symbolic methods. Proceedings of the 2nd International Workshop on Paraphrasing (PARAPHRASE ’03), Vol. 16, 11 July 2003, Sapporo, Japan (pp. 41–48). Stroudsburg, PA: Association for Computational Linguistics.

Brun, C., & Hagège, C. (2004). Intertwining deep syntactic processing and named entity detection. Advances in natural language processing (pp. 195–206). Berlin: Springer.

Brun, C., Popa, D. N., & Roux, C. (2014). XRCE: Hybrid classification for aspect-based sentiment analysis. Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), 23–24 August 2014, Dublin, Ireland (pp. 838–842). Association for Computational Linguistics.

Buckingham Shum, S., & Deakin Crick, R. (2016). Learning analytics for 21st century competencies. Journal of Learning Analytics, 3(2), 6–21.

Buckingham Shum, S., Sándor, Á, Goldsmith, R., Wang, X., Bass, R., & McWilliams, M. (2016). Reflecting on reflective writing analytics: Assessment challenges and iterative evaluation of a prototype tool. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 213–222). New York: ACM.

Burnett, M. M., & Scaffidi, C. (2011). End-user development. In M. Soegaard & R. F. Dam (Eds.), The encyclopedia of human–computer interaction, 2nd ed. Interaction Design Foundation.

De Liddo, A., Sándor. Á., & Buckingham Shum, S. (2012). Contested collective intelligence: Rationale, technologies, and a human–machine annotation study. Computer Supported Cooperative Work, 21(4–5), 417–448.

Gibson, A., & Kitto, K. (2015). Analysing reflective text for learning analytics: An approach using anomaly recontextualisation. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 275–279). New York: ACM.

Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C., & Knight, S. (2017). Reflective writing analytics for actionable feedback. Proceedings of the 7th International Conference on Learning Analytics and Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 153–162). New York: ACM.

Gibson, A., Kitto, K., & Bruza, P. (2016). Towards the discovery of learner metacognition from reflective writing. Journal of Learning Analytics, 3(2), 22–36.

Hatton N., & Smith, D. (1995, January). Reflection in teacher education: Towards definition and implementation. Teaching & Teacher Education, 11(1), 33–49.

Huang, Z., ten Teije, A., van Harmelen, F., & Aït-Mokhtar, S. (2014). Semantic representation of evidence-based clinical guidelines. Proceedings of the 6th International Workshop on Knowledge Representation for Health Care (KR4HC 2014), 21 July 2014, Vienna, Austria (pp. 78–94).

Hulsman, R. L., & van der Vloodt, J. (2015). Self-evaluation and peer-feedback of medical students’ communication skills using a web-based video annotation system. Exploring content and specificity. Patient Education and Counseling, 98(3), 356–363.

Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (in press). Designing academic writing analytics for civil law student self-assessment. International Journal of Artificial Intelligence in Education (Special Issue on Multidisciplinary Approaches to Reading and Writing Integrated with Disciplinary Education, Eds. D. McNamara, S. Muresan, R. Passonneau & D. Perin).

Knight, S., Martinez-Maldonado, R., Gibson, A., & Buckingham-Shum, S. (2017). Towards mining sequences and dispersion of rhetorical moves in student written texts. Proceedings of the 7th International Conference on Learning Analytics and Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 228–232). New York: ACM.

Lisacek, F., Chichester, C., Kaplan, A., & Sandor, Á. (2005). Discovering paradigm shift patterns in biomedical abstracts: Application to neurodegenerative diseases. Proceedings of the 1st International Symposium on Semantic Mining in Biomedicine (SMBM), 11–13 April 2005, Cambridge, United Kingdom (pp. 41–50).

MacLean, M., Carter, K., Lövstrand, L., & Moran, T. (1990). User-tailorable systems: Pressing the issues with buttons. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’90), 1–5 April 1990, Seattle, WA, USA (pp. 175–182). New York: ACM.

Moon, J. (2004). A handbook of reflective and experiential learning: Theory and practice. London: RoutledgeFalmer.

Moon, J. (2010). Reflective learning workshop (Handout 10/07), University of Worcester, UK.

Reidsema, C., Goldsmith, R., & Mort, P. (2010). Writing to learn: Reflective practice in engineering design. Proceedings of the 9th Annual ASEE Global Colloquium (ASEE 2010), 18–21 October 2010, Singapore. American Society for Engineering Education.

Rodgers, C. (2002a). Defining reflection: Another look at John Dewey and reflective thinking. Teachers College Record, 104(4), 842–866.

Rodgers, C. (2002b). Voices inside schools. Harvard Educational Review, 72(2), 230–254.

Ryan, M. (2010). The 4 R’s model of reflective thinking, version 1.5. Developing Reflective Approaches to Writing (DRAW) Project, Queensland University of Technology.

Ryan, M. (2011). Improving reflective writing in higher education: A social semiotic perspective, Teaching in Higher Education, 16(1), 99–111. Http://

Ryan, M., & Ryan, M. (2012). Developing a systematic, cross-faculty approach to teaching and assessing reflection in higher education. Australian Government, Office of Learning and Teaching.

Sándor, Á. (2007). Modeling metadiscourse conveying the author’s rhetorical strategy in biomedical research abstracts. Revue Française de Linguistique Appliquée, 12(2), 97–108.

Sándor, Á., & Vorndran, A. (2009). Detecting key sentences for automatic assistance in peer reviewing research articles in educational sciences. Proceedings of the Workshop on Text and Citation Analysis for Scholarly Digital Libraries, 47th Annual Meeting of the Association for Computational Linguistics, 2–7 August 2009, Singapore.

Scouller, K. (1998). The influence of assessment methods on students’ learning approaches: Multiple choice question examination versus assignment essay. Higher Education, 35(4), 453–472.

Simsek, D., Buckingham Shum, S., Sándor, Á., De Liddo, A., & Ferguson, R. (2013). XIP dashboard: Visual analytics from automated rhetorical parsing of scientific metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics (DCLA13), 8 April 2013, Leuven, Belgium.

Simsek, D., Sándor, Á., Buckingham Shum, S., Ferguson, R., De Liddo, A., & Whitelock, D. (2015). Correlations between automated rhetorical analysis and tutors’ grades on student essays. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 355–359). New York: ACM.

Sumsion, J., & Fleet, A. (1996). Reflection: Can we assess it? Should we assess it? Assessment & Evaluation in Higher Education, 21(2), 121–130.

Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing automatically detected reflective texts with human judgements. Proceedings of the 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-TEL ʼ12), 18 September 2013, Saarbrucken, Germany (pp. 101–116).

Webster-Wright, A. (2013). The eye of the storm: A mindful inquiry into reflective practices in higher education. Reflective Practice, 14(4), 556–567.




How to Cite

Buckingham Shum, S., Sándor, Ágnes, Goldsmith, R., Bass, R., & McWilliams, M. (2017). Towards Reflective Writing Analytics: Rationale, Methodology and Preliminary Results. Journal of Learning Analytics, 4(1), 58–84.

Most read articles by the same author(s)

1 2 > >>