Efficiency in human languages: Corpus evidence for universal principles

1 Aims and background

According to a general functionalist view, languages are efficient in meeting the communicative needs of their users (e.g., Du Bois 1985; MacWhinney et al. 1984; Zipf 1949). The goal of this special collection is to demonstrate how different manifestations of efficiency in the lexicon, grammar and discourse can be tested and explained with novel methods born out of the increasing availability of corpus data from typologically diverse languages. The contributions in this collection focus on different cross-linguistic manifestations of efficiency, written by experts in corpus linguistics, information theory, cognitive science, psycholinguistics, discourse analysis and typology. These papers bring together new insights on efficiency as a universal principle of language.

Corpus evidence has played a major role in the study of linguistic efficiency. Corpora of different languages have been used to detect efficient patterns in numerous areas of grammar, including in the lexicon (e.g., Bentz 2018; Piantadosi et al. 2011; Zipf [1935]1965); in syntax (e.g., Futrell et al. 2015; Hawkins 1994: Ch. 4); and in phonology (e.g., Cohen Priva and Jaeger 2018; Coupé et al. 2019; Fenk-Oczlon and Fenk 2010; Pellegrino et al. 2011). The use of corpora gives researchers two main advantages in comparison with other methods. First, with the emergence of new multilingual collections of comparable and uniformly annotated corpora, we can test if a particular efficient pattern occurs in many diverse languages, which is much more challenging for the experimental approach. Second, many corpora contain language produced in naturalistic settings, in which the communicative pressures leading to the emergence of efficient patterns come into play. As new and diverse corpora with detailed linguistic information are increasingly available, we have more opportunities for studying different types of efficiency cross-linguistically, as demonstrated by the contributions to this collection. With the help of large-scale corpus data, we can disentangle different factors that have been claimed to explain language users’ preferences, to test the links between linguistic and extralinguistic variables, and to fine-tune previous theoretical ideas by making them quantifiable and testable.

This article has the following structure. First, we provide an overview of attested manifestations of efficiency in language (Section 2). Then we focus on the contribution of corpus linguists to this area (Section 3). Finally, we formulate a number of big theoretical and methodological questions that need to be answered in the field and give an overview of how the contributions in this special collection help answer them (Section 4).

2 Efficiency: Main principles, predictions and manifestations

Various linguistic phenomena related to efficiency have been explained in functional linguistics by the principle of economy (Haiman 1983), the Principle of Least Effort (Zipf 1949), Maxims of Quantity (Grice 1975), the form-frequency correspondence universal (Haspelmath in press), and similar principles, but it is only recently that these phenomena have been systematized under one theoretical umbrella (Gibson et al. 2019; Hawkins 2004, 2014; Levshina 2018). And although there remain many theoretical and methodological questions and challenges (see below), we can offer the following working definition of efficiency:

Using concepts from economics, efficiency can also be defined as the maximization of the benefit-to-cost ratio. Although communication can bring diverse benefits (social status, aesthetic pleasure, etc.), the main focus of most current studies on efficiency is on the successful transfer of information through a noisy communication channel, which is studied from an information theoretic perspective (Shannon 1948; see the recent review in Gibson et al. 2019). The costs are related to articulation and diverse cognitive operations, such as extraction of lexemes from long-term memory and processing of long syntactic dependencies. Assuming that the language users’ behavior is efficient, one can formulate at least three falsifiable predictions:

Prediction 1. A communication system should exhibit a positive correlation between costs and benefits. For example, we do not expect a language to exist in which highly informative units are systematically easier to produce than less informative units.

Prediction 2. We also expect to find a negative correlation (or in other words, a trade-off) between the costs of communication for different agents (e.g., the speaker/signer and the recipient) or within different subsystems of one communication system (e.g., overt morphological markers vs. word order), provided the benefits (i.e., amount of information) are kept constant. These expectations are similar to the state of Pareto efficiency in economics,^[1] in which resources cannot be reallocated to make one individual better off without making at least one individual worse off.

Prediction 3. Language users will tend to keep the benefit-to-cost ratio approximately constant per time unit. If the costs are distributed evenly in time, the benefits (transferred information) are also constant. This can explain Fenk and Fenk’s (1980) hypothesis of constant information flow, Levy and Jaeger’s (2007) Uniform Information Density hypothesis, as well as Coupé et al. (2019)’s finding that languages tend to keep the information rate more or less uniform.

Efficiency as management of the amount of the language users’ effort has been observed in many linguistic domains. For example, in phonology, there is ample evidence that words and segments that are more predictable generally, and in a given context, undergo phonetic reduction more frequently than less predictable units (Jurafsky et al. 2001; Seyfarth 2014). For example, the schwa before /r/ and /n/ is absent in high-frequency words, such as every, and present in low-frequency words like artillery (Hooper 1976).

As for the lexicon, according to Zipf’s ([1935]1965, 1949) law of abbreviation, more frequent words tend to be shorter than less frequent ones (see also Kanwal et al. 2017). This can be explained by formal reduction due to clipping (e.g., German Automobil > Auto), as well as by the use of shorter synonyms to express more frequent and accessible concepts, e.g., car instead of automobile (Zipf [1935]1965: 33).

In grammar, efficiency manifests itself in various markedness phenomena. The asymmetries in formal marking (e.g., shorter or zero singular markers and longer plural forms) arise and persist due to the tendency to provide less formal marking to more predictable categories (e.g., singular), and more marking to less predictable ones (e.g., plural) (Haspelmath and Karjus 2018). Another manifestation of efficiency is the well-known interplay between case marking and fixed word order: in order to convey ‘who did what to whom’, languages tend to use either explicit case marking (e.g., Lithuanian) or rigid word order (e.g., English) (Bentz and Christiansen 2013; Blake 2001: 15; Sapir 1921: 66; Sinnemäki 2014). Moreover, it has been argued that some word order patterns are preferred if they facilitate processing. For example, when arranging the constituents that belong to the same head, speakers of VO languages tend to put long constituents after short ones, e.g., longer prepositional phrases after shorter ones (Hawkins 2004). This word order minimizes the domains required for recognizing syntactic structure, reducing simultaneous processing and working memory load. These preferences are known as the principles “Minimize Domains” and “Early Immediate Constituents” (Hawkins 2004, 2014; see also Gibson 1998).

Beyond the sentence level, efficient communication is observed when more accessible referents are expressed by shorter forms than less accessible ones, e.g., nouns for new referents vs. pronouns or zero anaphora for the referents that have already been introduced in discourse (see Ariel’s 1990 Accessibility Theory). Avoidance of high cognitive costs is also associated with restrictions on the introduction of new referents because the integration of new referents in discourse structure involves high processing costs (e.g., Chafe 1987; Gibson 1998).

To summarize, efficient linguistic structures and language use are extremely diverse and can be described by different mathematical relationships. Further examples can be found in Hawkins (2014), Jaeger and Buz (2017), Levshina (2018) and Gibson et al. (2019) and other reviews. Despite these and other new findings, no language is perfectly efficient. Or as Joseph Greenberg once put it, “[a] speaker is like a lousy auto mechanic: every time [s]he fixes something in the language, [s]he screws up something else” (Croft 2002: 5). As such, we also observe other factors, such as analogy and ease of learning, which shape language structure. Therefore, efficiency is a gradual phenomenon, which is best described in terms of probabilistic or so-called ‘soft’ constraints (Bresnan et al. 2001). In order to test these constraints, we need large corpora and quantitative techniques. The contribution of corpus linguistics to the study of efficiency is discussed in the next section.

3 Corpus-based approaches to efficiency

Although ideas about the efficient use of language have a long history (e.g., Gabelentz 1901; Haiman 1983; Zipf [1935]1965), efficiency has remained below the radar of mainstream linguistic theory for most of the 20th century, with the few notable exceptions mentioned above. The interest towards efficiency as a universal principle has only recently re-emerged due to the development of large, richly annotated corpora, which allow new forms of quantitative cross-linguistic investigation. An important role is also played by cutting-edge statistical methods, which include correlation analysis, regression, and computer simulations, as well as by concepts from information theory, such as entropy and information content (e.g., contextual surprisal).

This trend started with corpus-based psycholinguistic research on phonological and grammatical reduction and enhancement, such as hypo- and hyperarticulation in language production, where research has focused mainly on English (e.g., Aylett and Turk 2004; Levy and Jaeger 2007). A more recent trend is to use large-scale cross-linguistic corpora to investigate efficiency due to increasingly easy access to large and parallel electronic corpora, including the Universal Dependencies Corpora (Zeman et al. 2020), the parallel Bible translations (Mayer and Cysouw 2014), and the OPUS corpus (Tiedemann 2012).^[2] Moreover, nowadays there are new projects that allow the investigation of efficiency in spoken data, including less well documented languages, e.g., Multi-CAST (Multilingual Corpus of Annotated Spoken Texts; Haig and Schnell 2016a)^[3] and DoReCo (Language DOcumentation REference COrpus).^[4]

The availability of big data from corpora and the increasingly detailed information they contain (e.g., part-of-speech tags, syntactic dependencies) enables linguists to fit more complex and precise models and to ask more challenging questions than was possible during Zipf’s time and for much of the 20th century. As such, researchers can and will move from investigating basic formal properties of the lexicon to deeper and more complex issues regarding morphosyntax, semantics and discourse pragmatics. For example, newly available corpora have been used to test well-known efficiency effects on large and typologically diverse samples of languages and to challenge previous claims in the literature. Some of the most important findings include:

• Speakers of different languages minimize syntactic dependency lengths in order to optimize processing (Futrell et al. 2015), in line with Gibson’s (1998)Hawkins’ (2004, 2014) theory (see above). At the same time, there is evidence that at least some languages do not follow this principle in some constructions, probably due to conflicting processing pressures (Liu 2020).

• Information conveyed by word order is negatively correlated with information conveyed by internal word structure (Koplenig et al. 2017), which is a generalization of the interplay between case marking and word order mentioned above.

• It has been shown that Zipf’s correlation between word length and frequency is supported by a sample of texts in almost 1,000 languages (Bentz and Ferrer-i-Cancho 2016). Moreover, it has been shown by Piantadosi et al. (2011) that average predictability of words from context is more strongly correlated with word length than simple frequency.

4 New directions of corpus-based research on efficiency: Contributions to this special collection

Since quantitative corpus-based research on efficiency is relatively new, naturally there remain many fundamental theoretical and methodological questions, which are addressed in the contributions in this special collection. These include the four “big” questions below.

Big Question #1. How can we disentangle efficiency from other factors, as well as to identify different pressures related to efficient language use? These issues are related to what has been called competing motivations (Du Bois 1985) and their explanations. For example, linguists have discovered numerous universals and statistical tendencies in both the functional and formalist traditions. But it is still unclear which of them are related to efficiency, and which are not? Specifically, which types of efficient behavior explain which linguistic phenomena?

For example, Zipf’s well-known law of abbreviation is usually explained by the Principle of Least Effort (Zipf 1949). However, a famous argument by Miller (1957) suggests that even a monkey typing randomly would at some point produce strings of characters, such that more frequent strings would be shorter than less frequent ones (similar to what we observe in natural language produced by humans). At the same time, it has been demonstrated that the random typing model does not approximate the frequency distributions found in real corpora (Ferrer-i-Cancho and Elvevåg 2010; see also Piantadosi 2014).

In studies on morphosyntax, there has been a debate on the explanations of efficient patterns in morphological marking, e.g., the tendency of languages in general to have longer marking for plural forms, which are less frequent, with the exception of languages like Welsh in which nouns that occur more frequently in the plural (e.g., birds or strawberries) have longer singulative forms (cf. Haspelmath and Karjus 2018). Some linguists argue that this has nothing to do with communicative pressures – and thus with efficiency. Instead, they explain that longer forms emerge from longer source constructions (e.g., Cristofaro 2019). However, this does not explain why languages overwhelmingly display a bias towards the efficient use of form-frequency correspondences, as in the example with singular and plural marking (see more details in Schmidtke-Bode et al. 2019).

In this collection, the question of efficient patterns in morphological marking is addressed by two case studies. First, it has been suggested that the scarcity of crossing dependencies in syntactic trees is due to the tendency to minimize dependency lengths (Ferrer-i-Cancho 2006). Is this tendency sufficient to explain the observed syntactic fact, or are there other independent factors at play? This question is addressed by Yadav, Husain and Futrell (this collection). Using dependency treebanks from 52 languages, they compare the actual trees against random baselines with the same dependency lengths as the real data. The authors find that the actual trees have fewer crossing dependencies than the random baselines, which suggests that the minimization of dependency lengths alone cannot explain the rarity of crossing dependencies. At the same time, however, the rarity of crossing dependencies and the constraints on dependency lengths can account for gap degree and well-nestedness – two well-studied formal restrictions on dependency trees.

Similarly, the length of a linguistic unit depends on its frequency or predictability (as noted above). However, length can also be thought of as a function of complexity of the superordinate unit, as posited by Menzerath’s Law (Altmann 1980; Menzerath 1954), which states that longer linguistic constructions will contain shorter constituents. When applied to words and morphemes, this means that longer words will be composed of shorter morphemes, and vice versa (shorter and more complex words will contain longer morphemes). Which of these linguistic laws, i.e., Zipf’s vs Menzerath’s, has a stronger effect on the length of linguistic units? This question is addressed by Stave, Paschen, Pellegrino and Seifart in this collection. Their paper is based on the data from nine spoken corpora from the large-scale DoReCo project (see Section 3). Although the Zipfian morpheme frequency is a more powerful predictor of morpheme length, Menzerath’s Law, operationalized as the number of morphemes in a word given its length (in terms of characters), has its independent contribution on the length of morphemes, especially those that occur in morphologically complex words. This study, as well as the study by Yadav et al., provide a good illustration of how one can disentangle competing motivations in language with the help of multilingual corpus data and cutting-edge methods.

Big Question #2. Which old observations and generalizations that have been claimed to be manifestations of efficiency can be corroborated by tests with new data? For example, it has been argued that languages use intransitive subjects as entry points for new discourse referents, serving as a kind of ‘safety valve’ that helps to avoid cognitive overload (Durie 2003; cf. Lambrecht’s [1994: 185] ‘Principle of the Separation of Reference and Role’). This role is particularly salient under conditions of high information pressure, where a relatively high number of referents has to be processed in a given stretch of discourse (Du Bois 1987: 834–836). However, this view has been challenged by Haig and Schnell (2016b) (see also Du Bois 2017). In a paper for this collection, Schiborr, Schnell and Haig investigate these claims against a sample of narrative spoken texts from a variety of different languages taken from the Multi-CAST collection (see Section 3). Their results show that the cognitive challenge of introducing referents is only weakly reflected in the morphosyntax. In other words, intransitive subjects do not specialize in this task. New referents are seamlessly integrated in the narrative using those structures that reflect their semantic role the best; no specialization of predicates is observed.

Big Question #3. While the previous questions focus on the specific communicative and cognitive tasks and linguistic phenomena, we can also “zoom out” and ask, how do global biological and cultural factors shape the language system as a whole, making it more efficient as a cultural tool? For example, Bentz (2018) has demonstrated that a high proportion of L2 speakers decreases the lexical diversity of a language as measured by the unigram-based entropy of a text. This question is also addressed in the paper by Koplenig (this collection), who investigates the efficiency of written language based on a parallel corpus of Bible translations in more than 1,000 languages. Efficiency is measured in information-theoretic terms, as the ratio of information transmission rate to the channel capacity. Put simply, systems that are more efficient have lower predictability of their units from preceding context. Koplenig quantifies this efficiency for every doculect (translated text in a specific language) at the level of characters and words. Notably, languages with a larger number of speakers usually have more efficient written language than languages with a smaller number of speakers. This intriguing result suggests that cultural evolution can shape language structure in many subtle ways, which can be detected only with the help of corpora and statistics.

Big Question #4. How do we formalize and quantify linguistic intuitions about efficiency, and how do we make these theories falsifiable? Previous studies of efficiency contain many inspiring ideas and hypotheses, but often they are not formulated precisely or in testable ways. For example, what exactly do words like ‘predictability’ or ‘expectedness’ or ‘typicality’ actually mean? Which frequency-based measures represent these constructs the best? What is the role of type frequency? These topics are addressed in two case studies in this collection. The focus of the contribution by Levshina is on the cross-linguistic tendencies in differential case marking of the transitive subject and object. Based on data from corpora of spontaneous conversations in five typologically diverse languages, she argues that the use of markers is communicatively efficient because it depends on the predictability of the syntactic roles given specific features of the referents, such as animacy or givenness, rather than on the predictability of the ‘typical’ features given the roles. This suggests that the conditional probability of roles given features is more relevant than the conditional probability of features given roles for explaining differential case marking.

The focus of the contribution by Guzmán Naranjo and Becker is on form-frequency correspondences in nominal inflection. Whereas coding length studies in the Zipfian tradition have focused on token frequency of units, the authors investigate the less well-explored measures that reflect the distribution of inflection markers as types: relative marker frequency in a paradigm, relative cell frequency, flexibility, and entropy of a marker and a paradigm cell. They find that all of these measures help predict the length of inflectional morphemes. Their data come from a cross-linguistic morphological database called UniMorph, which is supplemented by corpus data from several morphologically rich languages. These two contributions demonstrate how one can bring the theory of efficiency forward by trying out different ways of formalizing the theoretical claims.

Thus, the contributions in this special collection provide a thematically unified perspective on efficiency, which has never before been presented in one collection. The authors address the big fundamental questions on linguistic efficiency, which is now possible due to increasing access to corpus data on typologically diverse languages, and due to the careful application of statistical methods. The authors extend and challenge the existing knowledge about universal performance-based principles of human language by bringing to light new insights regarding efficiency as a universal principle of language. They also exemplify how cutting-edge quantitative approaches, coupled with cross-linguistic and multilingual corpora, are being used to investigate the many open theoretical and methodological questions in human language efficiency.

More broadly speaking, the present collection provides new insights to the fundamental question in functional linguistics and in typology: why are human languages the way they are? We believe that convergent linguistic strategies in different languages reported in the case studies emerge due to common cognitive, biological and social pressures shared by language users around the world. Of course, correlations found in corpora do not automatically imply causation. In the future, we expect that evidence from experimental linguistics will support, refine, or possibly even refute, these claims. This collection also provides illustrations of how different explanatory factors and different operationalizations can be disentangled with the help of advanced statistical methods and rich data, which will hopefully resolve the various controversies in present-day functional linguistics (see, e.g., Schmidtke-Bode et al. 2019).