Accessible Requires Authentication Published by De Gruyter Mouton June 16, 2021

Observation, experimentation, and replication in linguistics

Jack Grieve
From the journal Linguistics

Abstract

In this paper, I propose that replication failure in linguistics may be due primarily to inherent issues with the application of experimental methods to analyze an inextricably social phenomenon like language, as opposed to poor research practices. Because language use varies across social contexts, and because social context must vary across independent experimental replications, linguists should not be surprised when experimental results fail to replicate at the expected rate. To address issues with replication failure in linguistics, and to increase methodological rigor in our field more generally, I argue that linguists must use experimental methods carefully, keeping in mind their inherent limitations, while acknowledging the scientific value of observational methods, which are often the only way to pursue basic questions in our field.


Corresponding author: Jack Grieve, Department of English Language and Linguistics, University of Birmingham, Frankland Building, Edgbaston, Birmingham B15 2TT, UK, E-mail:

Acknowledgments

I would like to thank Dagmar Divjak, Jason Grafmiller, Nick Groom, Susan Hunston, Gary Lupyan, Akira Murakami, Elliot Murphy, Lukas Sönning, Timo Roettger, Rory Turnbull, Emily Waibel, Valentin Werner, and Bodo Winter for discussing this paper and its contents with me.

References

Abbuhl, Rebekha, Susan Gass & Alison Mackey. 2013. Experimental research design. In Robert J. Podesva & Devyani Sharma (eds.), Research methods in linguistics, 116–134. Cambridge: Cambridge University Press. Search in Google Scholar

Alrabai, Faikieh. 2014. The effects of teachers’ in-class motivational intervention on learners’ EFL achievement. Applied Linguistics 37. 307–333. https://doi.org/10.1093/applin/amu021. Search in Google Scholar

Andor, Józef. 2014. The master and his performance: An interview with Noam Chomsky. Intercultural Pragmatics 1. 93–111. Search in Google Scholar

Arunachalam, S. 2013. Experimental methods for linguists. Language and Linguistics Compass 7(4). 221–232. https://doi.org/10.1111/lnc3.12021. Search in Google Scholar

Asendorpf, Jens B., Mark Conner, Filip De Fruyt, Jan De Houwer, Jaap J. A. Denissen, Klaus Fiedler, Susann Fiedler, David C. Funder, Reinhold Kliegl, Brian A. Nosek, Marco Perugini, Brent W. Roberts, Manfred Schmitt, Marcel A. G. van Aken, Hannelore Weber & Jelte M. Wicherst. 2013. Recommendations for increasing replicability in psychology. European Journal of Personality 27. 108–119. https://doi.org/10.1002/per.1919. Search in Google Scholar

Auer, Edward T., Lynn E. Bernstein & Paula E. Tucker. 2000. Is subjective word familiarity a meter of ambient language? A natural experiment on effects of perceptual experience. Memory & Cognition 28(5). 789–797. https://doi.org/10.3758/bf03198414. Search in Google Scholar

Biber, Douglas. 1993. Representativeness in corpus design. Literary and Linguistic Computing 8. 243–257. https://doi.org/10.1093/llc/8.4.243. Search in Google Scholar

Biber, Douglas & Susan Conrad. 2019. Genre, register and style, 2nd edn. Cambridge: Cambridge University Press. Search in Google Scholar

Charness, Gary, Uri Gneezy & Michael Kuhn. 2012. Experimental methods: Between-subject and within-subject design. Journal of Economic Behavior & Organization 81(1). 1–8. https://doi.org/10.1016/j.jebo.2011.08.009. Search in Google Scholar

Coulthard, Malcolm. 2014. An introduction to discourse analysis. London: Routledge. Search in Google Scholar

Divjak, Dagmar, Tomaž Erjavec & Serge Sharoff. 2017. Slavic computational and corpus linguistics. Journal of Slavic Linguistics 25(2). 171–199. https://doi.org/10.1353/jsl.2017.0008. Search in Google Scholar

Dunning, Thad. 2012. Natural experiments in the social sciences: A design-based approach. Cambridge: Cambridge University Press. Search in Google Scholar

Everett, Daniel L. 2009. Don’t sleep, there are snakes: Life and language in the Amazonian jungle. London: Profile Books. Search in Google Scholar

Finkel, Eli J., Paul W. Eastwick & Harry T. Reis. 2017. Replicability and other features of a high-quality science: Toward a balanced and empirical approach. Journal of Personality and Social Psychology 113. 244–253. https://doi.org/10.1037/pspi0000075. Search in Google Scholar

Gibson, Edward A., Caitllin M. Tan, Richard Futrell, Kyle Mahowald, Lars Konieczny, Barbara Hemforth & Evelina Fedorenko. 2017. Don’t underestimate the benefits of being misunderstood. Psychological Science 28(6). 703–712. https://doi.org/10.1177/0956797617690277. Search in Google Scholar

Glass, David J. & Ned Hall. 2008. A brief history of the hypothesis. Cell 134. 378–381. https://doi.org/10.1016/j.cell.2008.07.033. Search in Google Scholar

Han, Chung-hye, Kyeong-min Kim, Keir Moulton & Jeffrey Lidz. 2020. Null objects in Korean: Experimental evidence for the argument ellipsis analysis. Linguistic Inquiry 51 2. 319–340. https://doi.org/10.1162/ling_a_00342. Search in Google Scholar

Hay, Jennifer, Katie Drager & Paul Warren. 2009. Careful who you talk to: An effect of experimenter identity on the production of the NEAR/SQUARE merger in New Zealand English. Australian Journal of Linguistics 29(2). 269–285. https://doi.org/10.1080/07268600902823128. Search in Google Scholar

Klavan, Jane & Dagmar Divjak. 2016. The cognitive plausibility of statistical classification models: Comparing textual and behavioral evidence. Folia Linguistica 50(2). 355–384. https://doi.org/10.1515/flin-2016-0014. Search in Google Scholar

Krug, Manfred, Julia Schüler & Anette Rosenbach. 2013. Introduction: Investigating language variation and change. In Manfred Krug & Julia Schlüter (eds.), Research methods in language variation and change, 1–13. Cambridge: Cambridge University Press. Search in Google Scholar

Kretzschmar, William A.Jr. 2015. Language and complex systems. Cambridge University Press. Search in Google Scholar

Labov, William. 1972. Sociolinguistic patterns. Philadelphia, PA: University of Pennsylvania Press. Search in Google Scholar

Labov, William. 1984. Field methods of the project on linguistic change and language in use. In John Baugh & Joel Sherzer (eds.), Readings in sociolinguistics, 28–54. Englewood Cliffs, NJ: Prentice-Hall. Search in Google Scholar

McEnery, Tony & Andrew Wilson. 1996. Corpus linguistics. Edinburgh: Edinburgh University Press. Search in Google Scholar

Maxwell, Scott E., Michael Y. Lau & George S Howard. 2015. Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist 70. 487–498. https://doi.org/10.1037/a0039400. Search in Google Scholar

Montrul, Silvina, Rakesh Bhatt & Roxana Girju. 2015. Differential object marking in Spanish, Hindi, and Romanian as heritage languages. Language 564–610. https://doi.org/10.1353/lan.2015.0035. Search in Google Scholar

Nilsen, Erlend B., Diana E. Bowler & John D. C. Linnell. 2020. Exploratory and confirmatory research in the open science era. Journal of Applied Ecology 57(4). 842–847. https://doi.org/10.1111/1365-2664.13571. Search in Google Scholar

Remler, Dahlia K. & Gregg G. Van Ryzin. 2015. Research methods in practice. London: Sage. Search in Google Scholar

Robert J. Podesva & Devyani Sharma (eds.). 2013. Research methods in linguistics. Cambridge: Cambridge University Press. Search in Google Scholar

Roettger, Timo B. 2019. Researcher degrees of freedom in phonetic sciences. Laboratory Phonology: Journal of the Association for Laboratory Phonology 10. 1–27. https://doi.org/10.5334/labphon.147. Search in Google Scholar

Roettger, Timo B., Bodo Winter & Harald Baayen. 2019. Emergent data analysis in phonetic sciences: Towards pluralism and reproducibility. Journal of Phonetics 73. 1–7. https://doi.org/10.1016/j.wocn.2018.12.001. Search in Google Scholar

Rosenbaum, Paul R. 2002. Observational studies, 2nd edn. New York: Springer. Search in Google Scholar

Sadish, William R., Thomas D Cook & Donald T. Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin. Search in Google Scholar

Schütze, Carson T. 2016. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. Berlin: Language Science Press. Search in Google Scholar

Shrout, Patrick E. & Joseph L. Rodgers. 2018. Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual Review of Psychology 69. 487–510. https://doi.org/10.1146/annurev-psych-122216-011845. Search in Google Scholar

Tackett, Jennifer L., Cassandra M. Brandes, Kevin M. King & Kristian E Markon. 2019. Psychology’s replication crisis and clinical psychological science. Annual Review of Clinical Psychology 15. 579–604. https://doi.org/10.1146/annurev-clinpsy-050718-095710. Search in Google Scholar

Tukey, John Wilder. 1977. Exploratory data analysis. Reading, MA: Addison-Wesley. Search in Google Scholar

Van Bavel, Jay J., Peter Mende-Siedleckia, William J. Bradya & Diego A. Reineroa. 2016. Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences 113. 6454–6459. https://doi.org/10.1073/pnas.1521897113. Search in Google Scholar

Vanek, Norbert & Barbara Mertins. 2020. Defying chronology: Crosslinguistic variation in reverse order reports. Linguistics 58(2). 569–603. https://doi.org/10.1515/ling-2019-0006. Search in Google Scholar

Walker, Abby. 2019. The role of dialect experience in topic-based shifts in speech production. Language Variation and Change 31(2). 135–163. https://doi.org/10.1017/s0954394519000152. Search in Google Scholar

Werkmann Horvat, Anna, Marianna Bolognesi & Katrin Kohl. 2021. Creativity is a toaster: Experimental evidence on how multilinguals process novel metaphors. Applied Linguistics. Search in Google Scholar

Yun, Suyeon & Yoonjung Kang. 2019. Variation of the word-initial liquid in North and South Korean dialects under contact. Journal of Phonetics 77. 100918. https://doi.org/10.1016/j.wocn.2019.100918. Search in Google Scholar

Received: 2020-06-26
Accepted: 2021-05-29
Published Online: 2021-06-16
Published in Print: 2021-09-27

© 2021 Walter de Gruyter GmbH, Berlin/Boston