Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Corpus Linguistics and Linguistic Theory

Founded by Gries, Stefan Th. / Stefanowitsch, Anatol

Ed. by Wulff, Stefanie

IMPACT FACTOR 2017: 1.200
5-year IMPACT FACTOR: 1.386

CiteScore 2017: 0.80

SCImago Journal Rank (SJR) 2017: 0.288
Source Normalized Impact per Paper (SNIP) 2017: 0.930

See all formats and pricing
More options …

Coding coherence relations: Reliability and validity

Wilbert Spooren / Liesbeth Degand
Published Online: 2010-11-04 | DOI: https://doi.org/10.1515/cllt.2010.009


This paper tackles the issue of the validity and reliability of coding discourse phenomena in corpus-based analyses. On the basis of a sample analysis of coherence relation annotation that resulted in a poor kappa score, we describe the problem and put it into the context of recent literature from the field of computational linguistics on required intercoder agreement. We describe our view on the consequences of the current state of the art and suggest three routes to follow in the coding of coherence relations: double coding (including discussion of disagreements and explicitation of the coding decisions), single coding (including the risk of coder bias, and a lack of generalizability), and enriched kappa statistics (including observed and specific agreement, and a discussion of the (possible reasons for) disagreement). We end with a plea for complimentary techniques for testing the robustness of our data with the help of automatic (text mining) techniques.

Keywords:: coherence relations; discourse; reliability; interrater agreement; corpus analysis

About the article

Published Online: 2010-11-04

Published in Print: 2010-10-01

Citation Information: Corpus Linguistics and Linguistic Theory, Volume 6, Issue 2, Pages 241–266, ISSN (Online) 1613-7035, ISSN (Print) 1613-7027, DOI: https://doi.org/10.1515/cllt.2010.009.

Export Citation

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Sarah F. V. Eiteljoerge, Nausicaa Pouscoulous, and Elena V. M. Lieven
Frontiers in Psychology, 2018, Volume 9
Andrea Santana, Dorien Nieuwenhuijsen, Wilbert Spooren, and Ted Sanders
Discours, 2017, Number 20
Romualdo Ibáñez O. and Fernando Moncada N.
Spanish in Context, 2017, Volume 14, Number 2, Page 273
Weiying Chen and Jacqueline Evers-Vermeul
Chinese Language and Discourse, 2017, Volume 8, Number 1, Page 95
Christian Burgers, Allison Eden, Robin de Jong, and Sander Buningh
Mobile Media & Communication, 2016, Volume 4, Number 3, Page 327
Catherine T. Bolly, Ludivine Crible, Liesbeth Degand, and Deniz Uygur-Distexhe
Discours, 2015, Number 16
Marianna Bolognesi, Roosmaryn Pilgram, and Romy van den Heerik
Behavior Research Methods, 2016
Marta Andersson and Jennifer Spenader
Lingua, 2014, Volume 148, Page 1
Manfred Stede and Andreas Peldszus
Journal of Pragmatics, 2012, Volume 44, Number 2, Page 214
Liesbeth Degand and Benjamin Fagard
Journal of Pragmatics, 2012, Volume 44, Number 2, Page 154
Jacqueline Evers-Vermeul, Liesbeth Degand, Benjamin Fagard, and Liesbeth Mortier
Linguistics, 2011, Volume 49, Number 2

Comments (0)

Please log in or register to comment.
Log in