We tested the reactions of native speakers of Russian to Perfective vs. Imperfective aspect using full-sized authentic contexts. An offline survey experiment was undertaken in order to ensure that all of the contextual factors available to all participants responding to the same test item would be the same.

2.2 Procedure

The study was conducted as an online survey using Qualtrics software. ⁴ Each participant completed a “CAPTCHA” task to prove that they were human (not a robot) and was randomly assigned to one of the six texts that served as stimuli. Participants were instructed to rate the acceptability of verb forms as “Excellent”, “Acceptable”, or “Impossible”, and it was possible to give both verbs the same rating. They were told that they must complete the entire task in order to receive a code for a prize lottery, and warned that their code would be eliminated from the lottery if there was evidence that the survey was filled in at random or if they filled out more than one survey. Participants were also told that their participation was voluntary and they could quit the task at any time, and that by participating they were giving their consent. Prior to reading the text and evaluating the verbs, participants were asked to state their age (to confirm that they were 16 or older), their native language (to confirm that they were native speakers of Russian), and their gender identity. No IP addresses or other identifying information was collected.

Figure 1 shows the instructions in blue and the survey text following the horizontal line. The test pairs are in square brackets and highlighted in light blue, with the Perfective verb form followed by a slash and then the corresponding Imperfective verb form. When the participant moves the cursor over a verb in a test pair, the three evaluations pop up as illustrated in Figure 1. Here the participant has moved the cursor over the Imperfective Past tense form prinimal ‘accepted’ (see example [2] above) and the three options are given with iconic coloring: “Excellent” in dark green, “Acceptable” in light green, and “Impossible” in red. The participant needs only to click on one of the options and then move on. The participant is asked to rate both the Perfective and the Imperfective verb forms in each test pair.

Figure 1:

Screenshot illustrating what the task looked like.

There was no time limit, participants were allowed to go back to items as many times as they wanted, and they were prompted to go back and finish any items that they had skipped. Participants were recruited via emails sent to various listservs and individuals in Russia. In the course of one week (13. –20.09.2016), 501 participants successfully completed the task.

The purpose of this analysis is to determine how strongly the context predicts ratings (measured as match to original aspect), and to measure this variable against other possible variables. The input for our analysis is the data from our study, collected in 111,364 lines, each of which recorded values for the variables detailed in Table 3. ⁵

“Rating” is the dependent (response) variable, which consists of ordered categories. “Excellent” was the rating chosen most often, in 57,116 of responses, while “Acceptable” was the rating chosen least often, in only 17,395 of responses. The remaining variables in Table 3 are independent (predictor) variables.

“Matches Original” tells us whether the form being rated is of the same aspect as in the original text or not. See Table 4 below.

“Logarithm of Relative Frequency” is the natural logarithm of the relative frequency of the form being rated vs. the frequency of the corresponding form of the opposite aspect. This measure is called a “logit”. It is customary to logarithmically transform corpus frequency data in order to correct for the extreme skewing of corpus word frequencies, known as “Zipf’s Law” (1949). Logits are logarithmically transformed odds ratios. They have the admirable property of transforming odds ratios (which normally range from zero to 1 on one side, and from 1 to infinity on the other) into a symmetrical distribution. For example, an odds ratio of 1000/1 (relative frequency where one item appears 1000 times and the other only once) = 1000 yields a logit of 6.9, and the reverse relative frequency of 1/1000 = 0.001 yields a logit of −6.9. When the frequencies of two items are the same, the odds ratio is 1, and the logit is 0. The purpose of this measure is to determine whether the relative frequency of the two aspectual forms has any influence on the ratings.

“Text” shows the number of responses for each of the six text stimuli in the study.

“Aspect” indicates the aspect of the form that is rated by the participant, which is either Perfective (p) or Imperfective (i). Table 4 shows what all the combinations of the variables “Matches Original” and “Aspect” mean.

“Subparadigm” records the number of responses for each subparadigm.

“Cue Match” tells us whether there was a cue word present, and, if so, whether the cue is usually associated with the same aspect as in the original text (“True”), with the opposite aspect (“False”), or there was no cue (“None”, the most common value).

“Age” is the age of the participants, which ranged from 16 to 78.

We analyzed the ratings with the mgcv package, using the gam function and setting the family directive to ocat(R = 3), where R specifies the number of ordered categories. The response variable needs to be coded with integers 1 … R.

A main-effects model with by-participant random intercepts representing the variables given in Table 3 is summarized in Table 5.

The central concept underlying this implementation of ordinal regression is the following. We assume there is a latent random variable U that reflects subjects’ intuitions about acceptability. U can assume any value on the real axis. To discretize U into 3 rating categories, the real axis is divided into three bins. For this, we need cut-off points specifying the boundaries between the bins. The first cut-off point is set at −1. For ratings on a three-point scale, a second cut-off point is required that is greater than −1. For the present main-effects model, this cut-off point is estimated at 0.33.

Thus, the mapping of intervals on the real axis to ratings is as follows: The ordinal gam models the latent variable U as a function of the predictors, i.e. the linear predictor η_i is the following:

Given the predicted value of U_i (which is taken to follow a logistic distribution), we inspect which interval on the real axis it falls into, and this in turn determines which rating is predicted (see also Baayen and Divjak 2017 for further discussion).

Table 5 shows that the effect of the variable Matches Original is larger than any other, even when corrections are made for differences in units and numbers of levels. In other words, even when we take into account the fact that there is some tendency to choose a form that is more frequent, and that there are differences associated with various factors relating to the form and context of the token and with the participants, there is a very strong tendency to prefer the original form over the alternative.

As a next step, we considered a model that included two interactions. The first interaction is that of age by frequency ratio, the idea being that experience with language accumulates over the lifetime in such a way that speakers become more proficient as they age, and hence may not need to rely as much on relative frequency of use and more on the “hidden factors” that drive aspect selection (Ramscar et al. 2017). We also considered the interaction of Matches Original by Text, as it is known that the use of aspect can vary substantially between, e.g. scientific texts and fiction. Older speakers appear to rely less on the relative frequency of aspect use, possibly because they are more sensitive to the stylistic/discourse factors that determine aspect use, compared to younger speakers (further details about interactions can be found in the link in Footnote 5).

Word identity is not included as a random effect. The reason for this is that the frequency distribution of verbs is Zipfian, with many verb forms appearing only once and a small number of verbs being used intensively. Including item as a random effect forces the model to find a set of by-item adjustments that follow a normal distribution. Given the Zipfian nature of word probabilities, this is impossible. A model including by-word random intercepts would be misspecified. To see this, consider the large proportion of forms (typically around 50% of the word types) that occur once only. For each of these forms, the model would include not only an intercept adjustment, but also several other item-bound predictors such as Logarithm of Relative Frequency, Matches Original, Aspect, and Subparadigm. Thus, such a model would be overspecified.

In this context, it is worth noting that the logistic GAM is not a Gaussian model, and that there is no error term that should be independently and identically distributed (iid) for p-values in the model summary to be trustworthy. Thus, whereas in a standard linear mixed model for, e.g. reaction times, it would be desirable to include word as random-effect factor to avoid structured errors and violation of the iid model assumption, this issue does not arise in the context of the present ordinal regression model. Furthermore, as this model (for details, see Wood et al. 2016) is not a proportional odds model, assumptions about proportional odds need not be made.

In sum, the statistical analysis brings into relief the importance of the variable Matches Original in determining the rating of a verb form, even when other factors are taken into account. The remainder of our analysis focuses on the rating of original aspect vs. non-original aspect, and how this manifests as an indicator of redundancy vs. open construal in our data.

Our data consists of pairs of ratings for original aspectual verb forms as opposed to non-original aspectual verb forms. The ratings of each verb form can be sorted into two groups: a high rating group where the majority of participants chose “Excellent” or “Acceptable” ratings vs. a low rating group where “Impossible” is the predominant choice. A high rating of the original aspect combined with a low rating of the opposite aspect indicates that aspect is redundant in context, whereas high rating of both aspects indicates that aspect is open to construal.Table 6 shows the distribution of possible combinations between high and low ratings for original and non-original paired aspectual forms.

Most of the data (81%) is represented in the first line of Table 6, where the original aspect receives a high rating, while the corresponding item of the opposite aspect receives a low rating. Here aspect is largely redundant, since native speakers can reliably recover it based only on context. The construal of the event as Perfective or Imperfective is certainly present, but the overt marking of aspect on the verb is unnecessary since aspect can be recovered from context. Examples (3) and (4) from our study illustrate situations in which the Perfective and Imperfective aspect are redundant. Table 7 shows the numbers of ratings that these test items received in our study. Ratings for the original aspect are in shaded boxes.

Example (3) describes a unique punctual event that happened at a specific moment in time, which makes Perfective aspect strongly preferred. The categorical negation (literally ‘never not’) in (4) gives a strong reason to prefer the Imperfective for a situation that is designed to cover all times without limits (rather than specifying a unique event).

Most of the rest of the data (17%) appears in the second line of Table 6, where both the original aspect and the non-original aspect receive high ratings. These items illustrate open construal, since native speakers cannot guess the aspect from context. Examples (5) and (6) illustrate contexts where construal is relatively open, as evidenced in the rating data in Table 8.

The use of aspect in example (5) depends on whether the speaker wishes to emphasize how successful the pickpocket was at snatching money (Perfective) or how continuously he stole money (Imperfective). In example (6) the Perfective construal emphasizes a single event (in this case, ignoring the woman when she accused her son-in-law of stealing from her), whereas the Imperfective emphasizes more a general disregard for the claims of a senile old woman.

For the remainder of the data (2%), most native speakers simply disagree with the authors of the texts. Aspect is largely redundant here as well, even though the authors of the original texts chose the aspect that was less popular among the participants. Examples (7) and (8) illustrate test items of this type, and their ratings are summarized in Table 9.

There are no test items for which native speakers found neither form (original, non-original) to be suitable in the given context, and thus no data in the last line of Table 6.

Table 6 tells an important story about redundancy and open construal. The meanings available for construal have to come from somewhere, so it is reasonable that they are anchored in uses where the construal, though of course present, is redundant. In a usage-based model, this means that the construal associated with, say, the Perfective aspect, is entrenched through repeated exposure to examples where the construal is unmistakable due to the presence of other cues in the context that align with that construal. This entrenchment makes the construal of Perfective robust enough to be capable of expressing Perfective meaning even when there are no other cues to Perfective available in the context. However, these two characterizations, one of a situation where aspectual markers are redundant, and the other of open construal of aspect so that the speaker can choose what to emphasize, are extreme idealizations. In reality, there is a continuum of distribution of ratings in this data. Nearly all examples involve some balance between redundancy and open construal, with most examples reflecting a heavier portion of redundancy than open construal. Redundancy is the norm, and probably sets the standard for what the usually redundant aspectual markers can express in contexts that allow open construal. While it remains to be proven, it is conceivable that this kind of relationship between redundancy and open construal is common among languages of the world.

There is mounting evidence that, instead of converging on a single grammar, native speakers can disagree on what is grammatical in their language and differ widely in their attainment of their native language. The present study contributes to this evidence, and further shows that differences in acceptability ratings are much more pronounced when speakers are presented with unattested language (items that do not match authentic original texts) than with attested language.

Dąbrowska (Dąbrowska 2008, Dąbrowska 2012, Dąbrowska 2013, Dąbrowska 2015; Street and Dąbrowska 2010) has shown, through a series of experiments on native speakers of Polish and English, that native speakers exhibit differences in their grammars. Native speakers can have different strategies for understanding the same grammatical phenomena (for example abstract schematic rules vs. low-level rules vs. memorization of exemplars), yet still produce the same forms, and thus be said to “speak the same language”. Native speakers can also differ in how well they master the grammatical categories of their native language. These differences span various kinds of grammatical phenomena, including morphology and syntax. Dąbrowska (2015: 661–662) attributes such individual differences to both cognitive and environmental factors. In other words, differences can result from differences in inherent ability and also from differences in the language that people are exposed to: the precise input of course varies from person to person, and there are overall differences in the quantity and quality of language exposure that are tied to socio-economic status. Verhagen and Mos (2016) report that both inter-speaker and intra-speaker (measured at an interval of 2–3 weeks) variation in the rating of the familiarity of multiword units can be high, yet when the responses of 86 individuals are averaged, the result is stable.

Variation in our data is influenced by whether the item being rated appeared in the original text or not. When native speakers rated the item that originally appeared in the text (the aspect matching that in the authentic text), they tended to agree that the item was “Excellent”. We see this, for example, in the ratings in Table 7 for examples (3) and (4). In example (3), the original aspect was Perfective, and all 83 participants rated the Perfective form as “Excellent”. In example (4), the original aspect was Imperfective, and all 83 participants rated the Imperfective form as “Excellent”. However, when native speakers rated the non-original aspect (the aspect opposite to the one in the authentic text), they often chose a wide range of ratings, as we see in the ratings of non-matching items in Tables 7 and 8. Example (9) and Table 10 show an extreme example of lack of agreement among participants for the non-original verb form.

While speakers mostly agreed that the original aspect in example (9), the Imperfective, was “Excellent” or “Acceptable”, the rating of the non-original aspect, the Perfective, is completely split across the three options.

One way of measuring the degree of agreement/disagreement across participants is to look at the standard deviation in the rating of each item, since a larger standard deviation will indicate greater diversity of responses. Figure 2 visualizes the distributions of standard deviations for the responses to the original tokens vs. the non-original tokens of the opposite aspect, showing that responses to the originals were different from those to the non-originals also on this measure. The mean standard deviation for original items was 0.286, whereas the mean standard deviation for the non-original items was 0.497, and a paired t-test shows that this difference is significant (test-statistic = −17.630, p=2.9e–57).

Figure 2:

Distributions in standard deviations of ratings for original vs. non-original items.

We find that native speakers can differ in their ratings of Perfective and Imperfective verb forms in Russian. This finding is in line with other experimental evidence showing individual variation in the grammars of native speakers. In addition, we find that native speakers are more consistent in giving positive ratings for the original tokens, whereas they are less consistent in their ratings of the non-original tokens. This may indicate that native speakers are more reliable in reacting to authentic language than in reacting to language that has been manipulated (in this case, by suggesting an aspectual form that does not match the original text). This result may also have implications for how much linguists can rely on the intuitions of native speakers in reaction to constructed “examples” as opposed to authentic ones.

Aspect is one of the most pervasive and characteristic grammatical categories in Russian, requiring its speakers constantly to choose between Perfective and Imperfective. Our survey of the rating of Perfective and Imperfective verb pairs in full, authentic contexts shows that this choice is anything but simple and invariable. We find that in 81% of examples, native speakers can fairly reliably retrieve the original aspect, and that they can do this regardless of whether there is an identifiable “cue” word for aspect in the context. Now that a sample of contexts where aspect is largely redundant has been identified, it is possible to engage in a search for other cues in these contexts beyond the known cues, which have extremely low cue availability. In 17% of contexts, native speakers accept both aspects. The discovery of these distributions and the norming of concrete examples along the scale from categorical grammaticality difference (correct/incorrect) to free variation is a valuable contribution to our knowledge about the behavior of aspect in Russian. These findings can serve as the basis for further research using methods of experimentation and machine learning to ferret out the as-yet unidentified contextual cues to aspect. If such cues can be uncovered, this could have far-reaching implications for both natural language processing and language pedagogy.

The data from our study reveal in a concrete way the relationship between redundancy and open construal, which has not previously been studied empirically. The distribution of ratings described in Section 4 is compatible with the interpretation of redundancy and open construal as co-existing in a continuum. The meanings associated with the two alternative construals offered by Russian aspect are always available, but their salience and independence from context vary. At one end of the continuum, the meanings of Perfective vs. Imperfective are strongly anchored by context, highly redundant, and the choice of aspect is tightly constrained. This end of the continuum is also its center of gravity, the place where most uses are observed and most entrenchment is expected. At the other end of the continuum, construal breaks free from context and operates independently, without the support of redundancy. In these uses, the speaker can deploy aspect to manipulate nuances of meaning, with the option of representing the “same” content in two slightly different ways, emphasizing either the discreteness of the situation as Perfective or its fluidity as Imperfective. Between these two extremes there are varying degrees of redundancy and freedom of construal. It is likely that construal actually needs the redundant uses to empower it to operate on its own when redundancy is reduced. Our data are restricted only to the relationship between redundancy and open construal for Russian aspect; it remains to be seen whether this relationship can be confirmed for other grammatical distinctions and other languages.

Our data also confirm that there are differences among native speakers, revealing a consistent bias toward less variation in response to an attested (naturally occurring) example than to a non-attested (manipulated and possibly unnatural) example. This suggests that native speakers may be more reliable in reacting to authentic language stimuli than in reacting to language stimuli constructed for experimental purposes (even when such language is merely a slight modification of authentic language). This may mean that linguistic experiments involving constructed stimuli could be subject to a certain bias, but this is a topic for future research.

The authors wish to thank the CLEAR group and UiT The Arctic University of Norway for their support of this research, and Adele Goldberg for consultation on psycholinguistics. We are particularly grateful to R. Harald Baayen for supplying the statistical model presented in Section 3.