Are You Being Rhetorical? An Open Corpus of Machine Annotated Rhetorical Moves

Authors

DOI:

https://doi.org/10.18608/jla.2020.73.10

Keywords:

Writing analytics, Corpus analysis, Rhetorical moves, Open data

Abstract

Writing analytics has emerged as a sub-field of learning analytics, with applications including the provision of formative feedback to students in developing their writing capacities. Rhetorical markers in writing have become a key feature in this feedback, with a number of tools being developed across research and teaching contexts. However, there is no shared corpus of texts annotated by these tools, nor is it clear how the tool annotations compare. Thus, resources are scarce for comparing tools for both tool development and pedagogic purposes. In this paper, we conduct such a comparison and introduce a sample corpus of texts representative of the particular genres, a subset of which has been annotated using three rhetorical analysis tools (one of which has two versions). This paper aims to provide both a description of the tools and a shared dataset in order to support extensions of existing analyses and tool design in support of writing skill development. We intend the description of these tools, which share a focus on rhetorical structures, alongside the corpus, to be a preliminary step to enable further research, with regard to both tool development and tool interaction.

Author Biography

Simon Knight, University of Technology Sydney

Lecturer at University of Technology Sydney Connected Intelligence Centre

References

Aït-Mokhtar, S., Chanod, J.-P., & Roux, C. (2002). Robustness beyond shallowness: Incremental deep parsing. Natural Language Engineering, 8(2–3), 121–144. https://dx.doi.org/10.1017/S1351324902002887

Allen, L. K., Jacovina, M. E., & McNamara, D. S. (2016). Computer-based writing instruction. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of Writing Research (pp. 316–329). New York, NY: Guilford Press. Retrieved from https://eric.ed.gov/?id=ED586512

Anthony, L. (1999). Writing research article introductions in software engineering: How accurate is a standard model? IEEE Transactions on Professional Communication, 42(1), 38–46. https://dx.doi.org/10.1109/47.749366

Anthony, L., & Lashkia, G. V. (2003). Mover: A machine learning tool to assist in the reading and writing of technical papers. IEEE Transactions on Professional Communication, 46(3), 185–193. https://dx.doi.org/10.1109/TPC.2003.816789

Buckingham Shum, S., Knight, S., McNamara, D., Allen, L. K., Betik, D., & Crossley, S. (2016). Critical perspectives on writing analytics. In Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ’16), 25–29 April 2016, Edinburgh, UK (pp. 481–483). New York, NY: ACM. https://dx.doi.org/10.1145/2883851.2883854

Buckingham Shum, S., Sándor, Á., Goldsmith, R., Wang, X., Bass, R., & McWilliams, M. (2016). Reflecting on reflective writing analytics: Assessment challenges and iterative evaluation of a prototype tool. Proceedings of the 6th International Conference on Learning Analytics & Knowledge, 25–29 April 2016, Edinburgh, UK (pp. 213–222). New York, NY: ACM. https://dx.doi.org/10.1145/2883851.2883955

Burstein, J., Chodorow, M., & Leacock, C. (2003). CriterionSM online essay evaluation: An application for automated evaluation of student essays. In J. Riedl & R. W. Hill Jr. (Eds.), Proceedings of the Fifteenth Conference on Innovative Applications of Artificial Intelligence (IAAI 2003), 12–14 August 2003, Acapulco, Mexico (pp. 3–10). Retrieved from https://dblp.org/db/conf/iaai/iaai2003.html

Cai, (L.) J. (2016). An exploratory study on an integrated genre-based approach for the instruction of academic lexical phrases. Journal of English for Academic Purposes, 24, 58–74. https://dx.doi.org/10.1016/J.JEAP.2016.09.002

Chang, C.-F., & Kuo, C.-H. (2011). A corpus-based approach to online materials development for writing research articles. English for Specific Purposes, 30(3), 222–234. https://dx.doi.org/10.1016/j.esp.2011.04.001

Cope, B., & Kalantzis, M. (1993). Introduction: How a genre approach to literacy can transform the way writing is taught. In B. Cope & M. Kalantzis (Eds.), The powers of literacy: A genre approach to teaching writing (pp. 1–21). London, UK: Taylor & Francis. Retrieved from https://www.taylorfrancis.com/books/9780203149812/chapters/10.4324/9780203149812-9

Cotos, E. (2011). Potential of automated writing evaluation feedback. CALICO Journal, 28(2), 420–459. https://dx.doi.org/10.11139/cj.28.2.420-459

Cotos, E. (2014). Enhancing writing pedagogy with learner corpus data. ReCALL, 26(2), 202–224. https://dx.doi.org/10.1017/S0958344014000019

Cotos, E. (2016). Computer-assisted research writing in the disciplines. In S. A. Crossley & D. S. McNamara (Eds.), Adaptive educational technologies for literacy instruction (pp. 225–242). London, UK: Taylor & Francis. Retrieved from https://www.taylorfrancis.com/books/9781315647500/chapters/10.4324/9781315647500-15

Cotos, E. (2018). Move analysis. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics (pp. 1–8). Oxford, UK: John Wiley & Sons. https://dx.doi.org/10.1002/9781405198431.wbeal1485

Cotos, E., Huffman, S., & Link, S. (2015). Furthering and applying move/step constructs: Technology-driven marshalling of Swalesian genre theory for EAP pedagogy. Journal of English for Academic Purposes, 19, 52–72. https://dx.doi.org/10.1016/j.jeap.2015.05.004

Cotos, E., Huffman, S., & Link, S. (2017). A move/step model for methods sections: Demonstrating rigour and credibility. English for Specific Purposes, 46, 90–106. https://dx.doi.org/10.1016/j.esp.2017.01.001

Cotos, E., Huffman, S., & Link, S. (2020). Understanding graduate writers’ interaction with and impact of the Research Writing Tutor during revision. Journal of Writing Research, 12(1), 187–232. https://dx.doi.org/10.17239/jowr-2020.12.01.07

Cotos, E., Link, S., & Huffman, S. R. (2016). Studying disciplinary corpora to teach the craft of discussion. Writing and Pedagogy, 87(1), 33-64. https://dx.doi.org/10.1558/wap.v8i1.27661

Cotos, E., Link, S., & Huffman, S. (2017). Effects of DDL technology on genre learning. Language Learning & Technology, 21(3), 104–130. https://dx.doi.org/10125/44623

Cotos, E., & Pendar, N. (2016). Discourse classification into rhetorical functions for AWE feedback. Calico Journal, 33(1), 92-116. https://dx.doi.org/10.1558/cj.v33i1.27047

Cotos, E., & Sándor, Á. (2018). Testing an integrated method for the automated analysis of rhetorical intent in academic genres. Paper presented at AAAL Conference 2018, 24–27 March 2018, Chicago, IL, USA.

De Liddo, A., Sándor, Á., & Buckingham Shum, S. (2012). Contested collective intelligence: Rationale, technologies, and a human-machine annotation study. Computer Supported Cooperative Work (CSCW), 21(4), 417–448. https://dx.doi.org/10.1007/s10606-011-9155-x

Elsevier Labs. (2015). OA STM Corpus. Elsevier Labs Github. Retrieved from http://elsevierlabs.github.io/OA-STM-Corpus/

Flower, L. S., & Hayes, J. R. (1977). Problem-solving strategies and the writing process. College English, 39(4), 449–461. https://dx.doi.org/10.2307/375768

Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C., Knight, S., …, Jarvis, W. (2017). Reflective writing analytics for actionable feedback in authentic assessment. In Proceedings of the 7th International Conference on Learning Analytics & Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 153–162). New York, NY: ACM. https://dx.doi.org/10.1145/3027385.3027436

Hyland, K. (2007). Genre pedagogy: Language, literacy and L2 writing instruction. Journal of Second Language Writing, 16(3), 148–164. https://dx.doi.org/10.1016/j.jslw.2007.07.005

Knight, S., Abel, S., Shibani, A., Yoong Kuan, G., Conijn, R., Gibson, A., …, Buckingham Shum, S. (2020). Are you being rhetorical? An open dataset of machine annotated rhetorical moves. Stash. https://dx.doi.org/10.26195/5f336ead59a43

Knight, S., Allen, L. K., Gibson, A., McNamara, D., & Buckingham Shum, S. (2017). Writing analytics literacy—Bridging from research to practice. In Proceedings of the 7th International Conference on Learning Analytics & Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 496–497). New York, NY: ACM. https://dx.doi.org/10.1145/3027385.3029425

Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (2018). Academic writing analytics for civil law: Participatory design through academic and student engagement. International Journal of Artificial Intelligence in Education, 28(1), 1–28. https://dx.doi.org/10.1007/s40593-016-0121-0

Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., …, Buckingham Shum, S. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12(1), 141–186. https://dx.doi.org/10.17239/jowr-2020.12.01.06

Kuteeva, M., & Negretti, R. (2016). Graduate students’ genre knowledge and perceived disciplinary practices: Creating a research space across disciplines. English for Specific Purposes, 41, 36–49. https://dx.doi.org/10.1016/J.ESP.2015.08.004

Liakata, M., Saha, S., Dobnik, S., Batchelor, C., & Rebholz-Schuhmann, D. (2012). Automatic recognition of conceptualization zones in scientific articles and two life science applications. Bioinformatics, 28(7), 991–1000. https://dx.doi.org/10.1093/bioinformatics/bts071

Lisacek, F., Chichester, C., Kaplan, A., & Sándor, Á. (2005). Discovering paradigm shift patterns in biomedical abstracts: Application to neurodegenerative diseases. In Proceedings of the First International Symposium on Semantic Mining in Biomedicine (SMBM), 11–13 April 2005, Cambridge, UK (pp. 41–50). Retrieved from https://www.researchgate.net/publication/228635443_Discovering_Paradigm_Shift_Patterns_in_Biomedical_Abstracts_Application_to_Neurodegenerative_Diseases

Mann, W. C., & Thompson, S. A. (1987). Rhetorical structure theory: Description and construction of text structures. In G. Kempen (Ed.), Natural language generation (Vol. 135) (pp. 85–95). Dordrecht, The Netherlands: Springer. https://dx.doi.org/10.1007/978-94-009-3645-4_7

McDonald, J., Moskal, A. C. M., Gunn, C., & Donald, C. (2018). Text analytic tools to illuminate student learning. In J. M. Lodge, J. C. Horvath, & L. Corrin (Eds.), Learning analytics in the classroom: Translating learning analytics for teachers. London, UK: Taylor & Francis. https://dx.doi.org/10.4324/9781351113038-11

Mizumoto, A., Hamatani, S., & Imao, Y. (2017). Applying the bundle–move connection approach to the development of an online writing support tool for research articles. Language Learning, 67(4), 885–921. https://dx.doi.org/10.1111/lang.12250

National Commission On Writing. (2003). Report of the National Commission on writing in America’s schools and colleges: The neglected “R,” the need for a writing revolution. The College Board. Retrieved from https://archive.nwp.org/cs/public/print/resource/2523

Nesi, H., Gardner, S., Thompson, P., & Wickens, P. (2004). British academic written English corpus. Oxford Text Archive. Oxford, UK: University of Oxford. Retrieved from http://purl.ox.ac.uk/ota/2539

PMC. (n.d.). Open Access Subset. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/

Sándor, Á. (2007). Modeling metadiscourse conveying the author’s rhetorical strategy in biomedical research abstracts. Revue française de linguistique appliquée, 12(2), 97–108. https://dx.doi.org/10.3917/rfla.122.0097

Sándor, Á., Kaplan, A., & Rondeau, G. (2006). Discourse and citation analysis with concept-matching. In Proceedings of the International Symposium: Discourse and Document (ISDD), 15–16 June 2006, Caen, France (pp. 147–152). Retrieved from https://www.researchgate.net/publication/240828357_Discourse_and_citation_analysis_with_concept-matching

Sándor, Á., & Vorndran, A. (2009). Detecting key sentences for automatic assistance in peer reviewing research articles in educational sciences. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries, 7 August 2009, Singapore (pp. 36–44). Stroudsburg, PA, USA: Association for Computational Linguistics. https://dx.doi.org/10.3115/1699750.1699757

Scardamalia, M., & Bereiter, C. (1987). Knowledge telling and knowledge transforming in written composition. Advances in Applied Psycholinguistics, 2, 142–175.

Shermis, M. D., & Burstein, J. (Eds.). (2013). Handbook of automated essay evaluation: Current applications and new directions. London, UK: Routledge/Taylor & Francis. Retrieved from https://psycnet.apa.org/record/2013-15323-000

Shibani, A. (2018a). AWA-Tutor: A platform to ground automated writing feedback in robust learning design. In S. Buckingham Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Companion Proceedings of the 8th International Learning Analytics and Knowledge Conference (LAK ’18), 5–9 March 2018, Sydney, Australia. Society for Learning Analytics Research (SoLAR). Retrieved from http://bit.ly/lak18-companion-proceedings

Shibani, A. (2018b). Developing a learning analytics intervention design and tool for writing instruction. In S. Buckingham Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Companion Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK ’18), 5–9 March 2018, Sydney, Australia. Society for Learning Analytics Research (SoLAR). Retrieved from http://bit.ly/lak18-companion-proceedings

Shibani, A., Abel, S., Gibson, A., & Knight, S. (2018). Turning the TAP on writing analytics. In S. Buckingham Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Companion Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK ’18), 5–9 March 2018, Sydney, Australia. Society for Learning Analytics Research (SoLAR). Retrieved from http://bit.ly/lak18-companion-proceedings

Shibani, A., Knight, S., & Buckingham Shum, S. (2019). Contextualizable learning analytics design: A generic model and writing analytics evaluations. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2018, Tempe, AZ, USA (pp. 210–219). New York, NY: ACM. https://dx.doi.org/10.1145/3303772.3303785

Shibani, A., Knight, S., & Buckingham Shum, S. (2020). Educator perspectives on learning analytics in classroom practice. Internet and Higher Education, 46, 100730. https://dx.doi.org/10.1016/j.iheduc.2020.100730

Shibani, A., Knight, S., Buckingham Shum, S., & Ryan, P. (2017). Design and implementation of a pedagogic intervention using writing analytics. In W. Chen, J.-C. Yang, A. F. Mohd Ayub, S. L. Wong, & A. Mitrovic (Eds.), Proceedings of the 25th International Conference on Computers in Education, 4–8 December 2017, Christchurch, New Zealand (pp. 306–315). Asia-Pacific Society for Computers in Education. Retrieved from https://www.apsce.net/icce/icce2017/140.115.135.84/icce/icce2017/proceedings_main.html

Simsek, D., Sándor, Á., Buckingham Shum, S., Ferguson, R., De Liddo, A., & Whitelock, D. (2015). Correlations between automated rhetorical analysis and tutors’ grades on student essays. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ’15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 355–359). New York, NY: ACM. https://dx.doi.org/10.1145/2723576.2723603

Swales, J. (1981). Aspects of article introductions (Aston ESP reports No. 1). Birmingham, UK: The University of Aston in Birmingham.

Swales, J. (1990). Genre analysis: English in academic and research settings. Cambridge, UK: Cambridge University Press.

Teufel, S., & Moens, M. (2002). Summarizing scientific articles: Experiments with relevance and rhetorical status. Computational Linguistics, 28(4), 409–445. https://dx.doi.org/10.1162/089120102762671936

Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal, 3(1), 22–36. https://dx.doi.org/10.1080/15544800701771580

Wingate, U. (2012). Using Academic Literacies and genre-based models for academic writing instruction: A “literacy” journey. Journal of English for Academic Purposes, 11(1), 26–37. https://dx.doi.org/10.1016/j.jeap.2011.11.006

Yan, D., Rupp, A. A., & Foltz, P. W. (Eds.). (2020). Handbook of automated scoring: Theory into practice. London, UK: Chapman and Hall/CRC. Retrieved from https://www.routledge.com/Handbook-of-Automated-Scoring-Theory-into-Practice/Yan-Rupp-Foltz/p/book/9781138578272

Downloads

Published

2020-12-17

How to Cite

Knight, S., Abel, S., Shibani, A., Goh, Y. K., Conijn, R., Gibson, A., Vajjala, S., Cotos, E., Sándor, Ágnes, & Buckingham Shum, S. (2020). Are You Being Rhetorical? An Open Corpus of Machine Annotated Rhetorical Moves. Journal of Learning Analytics, 7(3), 138-154. https://doi.org/10.18608/jla.2020.73.10

Most read articles by the same author(s)

1 2 3 > >>