Experimental filler design influences error correction rates in a word restoration paradigm

1.1 Designing experimental fillers

The design details of linguistic experiments matter. Taking the widespread example of acceptability judgments, a recent paper has found that several task features, from the mode of stimuli presentation, the number of response options, and the use of response labels, significantly influence the sensitivity of an experimental paradigm (Marty et al. 2020; also see Schütze 2016). More generally, it has been shown that the number of both participants and items (Mahowald et al. 2016), and the kind of participants — native speakers versus second-language learners, or linguistics experts versus laypeople — (Dąbrowska 2010; Gross and Culbertson 2011; Schütze and Sprouse 2014; Wasow and Arnold 2005) will likely make a difference in experimental results.

While there is a dense literature on the design of judgment data, recent methods used by (psycho)linguists are not limited to probing linguistic intuitions, but include the measurement of reaction times, eye-movements, or brain activity. However, apart from increasingly popular one-shot tasks (e.g. several experiments in Morgan et al. 2020; von der Malsburg et al. 2020), most of these experimental setups have one design feature in common: the customary inclusion of fillers.

It is all the more surprising, then, that the role of filler items has not been addressed in more detail by the psycholinguistic literature. Here, we start addressing this issue by manipulating the two design decisions which usually influence the creation of filler items: the ratio of critical items to filler items, and the structure of the filler items themselves.

The first question of the correct ratio is most often discussed based on recommendations to include fillers in order to make critical items less visible across the experiment, by reducing the latters’ density, and, in turn, reduce the likelihood that participants detect the critical manipulations and consciously adapt their responses (Cowart 1997; Keller 2000; Schütze and Sprouse 2014). For instance, Marinis and Cunnings (2018: 14), advise a ratio of “at least 1:1”. However, as Table 1 shows, there is large variation in the critical-to-filler ratio, ranging from three critical items for every two fillers to four critical items for every twenty-one fillers.

Some of this variation can be explained by the practice of experimental ‘piggy-backing’: independent experiments are combined, so critical items in one study serve as the next study’s fillers; this usually results in higher ratios. Examples of this strategy are Grant et al.’s (2020) Experiment 2, or Pañeda et al. (2020). The assumption underlying this strategy is that unrelated fillers do not affect a given study’s outcome; by combining multiple experiments into one, more participants can be run, improving each study’s statistical power.

However, what makes this strategy potentially problematic is that it has been observed that the ratio of fillers to critical items affects effect size (Bodner et al. 2006; Green et al. 2020; Harrington Stack et al. 2018), and indeed, some experiments explicitly manipulate the number of their fillers in order to understand the obtained effects better. For instance, Green et al. (2020) increased the number of fillers between Exps. 1 and 2 by a factor of three, and found that their result of interest became weaker.

The other question of interest is that of the filler items’ structure. In the acceptability and grammaticality judgment literature, it has been argued that filler design can, for instance, ensure that the full scale of rating options is used, by including fillers that have certain grammatical or semantic properties (Schütze and Sprouse 2014). Indeed, many experiments are designed with this reasoning in mind, and changing the structure of the filler items has been shown to alter the effects observed (e.g., Garnham et al. 1992).

The picture that emerges is that filler properties can have important effects on the experimental outcome, although their purpose in the first place is to ensure that the critical experimental manipulations can unfold their effect undisturbed by confounds. This presents researchers designing experiments with a problem: how can they ensure that the ratio and structure of their fillers do not introduce a bias of theoretical relevance?

One approach to understanding filler effects is from the larger perspective of context effects: as a participant is building up a representation of the experimental situation, statistical information about the stimuli she encounters plays a major role. One corner of the psycholinguistic literature that has leveraged this insight particularly well is that of error correction.

1.2 Error correction paradigms and statistical information within an experiment

When there is noise in the signal, whether it be background noise, competing auditory signals, or the product of speech errors, listeners often effortlessly correct for it (Levy 2008). This process is necessarily controlled by at least two factors: low-level auditory information, and high-level, top-down expectations derived from contextual information and grammatical knowledge broadly construed (e.g., Poppels and Levy 2016; Warren 1970). Prior work has shown that error correction processes are also sensitive to a third factor, long-term statistical information in the signal (Gibson et al. 2013), and that in some cases listeners adapt their processing strategy based on that information (Schotter et al. 2014). This suggests that the statistical properties of noise throughout an experiment, including filler items, also contributes to the correction process.

This principle is most straightforwardly examined by considering acoustic noise that prevents a listener from hearing the speaker’s full production. For example, if a listener hears “Noah and I can’t decide when to #### the tomatoes,” (with #### representing a segment of the signal where some acoustic noise obscures the identity of the underlying word), they would need to repair the utterance before it can be properly processed – that is, infer what was intended to be in place of the noise to form a Gestalt of a sentence. When doing this, the listener follows top-down expectations: in the utterance above, grammatical expectations would predict that the noise covers a verb, like “plant” or “eat.” Of course, listeners would also be sensitive to bottom-up information, like auditory information gleaned from the noise - does the acoustic information under the noise sound more like /p/ or /i:/?

However, another source of information lies in (subconsciously) taking into account patterns in what we will refer to as the distribution of the noise. Formally, this refers to the probability that a particular word in the original, uncorrupted signal, given its context, becomes noisy before being heard by the listener. More practically, this probability distribution characterizes the listener’s model of the process that generates the noise, and any biases it might have that would help the listener discern what lies under the noisy segments they hear.^[1] In the context of the example above, if most of the noise in a context appears over what is likely to be a word that begins with a vowel, listeners might be more likely to believe that the noisy segment was meant to be “eat” rather than “plant.” This project investigates whether and how this third source of information — the listener’s beliefs about the distribution of the noise in the context — interacts with grammatical expectations when correcting for noise.

A similar question has been investigated by Gibson et al. (2013). In their experiments, participants were asked to read a series of sentences and answer a comprehension question for each. Critical items consisted of sentences that were semantically implausible (i.e., “The girl was kicked by the ball”) paired with a question that probed whether the subject accepted the sentence as written (and adopted an implausible reading) or corrected the sentence to obtain a more plausible reading (i.e., “Did the girl kick something?”). Crucially, the rate of ungrammatical and implausible sentences in the filler items was manipulated between subjects (none vs. a 3:8 ungrammatical to grammatical ratio, and a 1:8 vs. a 5:16 implausible to plausible ratio). Gibson et al. (2013) found both that an increase in syntactic violations in the fillers lowers the rate of literal interpretations of critical items, and that an increase in semantically implausible sentences increased the rate of literal interpretation. These results were taken to indicate that the distribution of potential noise (in this case, in the form of syntactic errors and semantically implausible sentences) affects the error correction process.

However, this paradigm has limitations: first, as the authors mention, participants were allowed to read and re-read each stimulus and corresponding comprehension question as often as desired. Thus, these results cannot distinguish between automatic correction processes and other, conscious corrections. Second, this measure is coarse-grained, as the syntactic errors and semantic implausibilities had to be strong enough to cause listeners to reject the literal reading of a grammatical sentence for a correction to be detected. Finally, Gibson et al. (2013) manipulate the frequency of implausible and ungrammatical sentences in the filler items, which may be unnatural enough to draw the conscious attention of participants to the manipulation.

We aim to reduce some of these shortcomings by adopting a different methodology for investigating the interaction between grammatical knowledge and the distribution of noise in error correction. To do this, we adapt the Speech Restoration paradigm (Warren 1970) to investigate the interaction between grammatical knowledge and automatic error correction processes. In this paradigm, participants listen to stimuli with white noise superimposed over segments of the speech, including one critical region designed such that the addition of noise causes the identity of the underlying segment to be ambiguous. Participants are then instructed to repeat exactly what they had heard. Given this instruction, participants will not consciously correct any errors in the stimuli, and thus any “restoration” of the linguistic material underneath the ambiguous noise in the critical region can be taken as the result of automatic, unconscious error correction processes.

Thus, in order to study the manner in which grammatical knowledge and statistical information interact in error correction, we can simply manipulate the distribution of the superimposed noise across filler items, and measure the resulting effect on error correction rates. This paradigm, and the particular implementation of this paradigm in Mack et al. (2012), offers several advantages over the method used in Gibson et al. (2013): first, the task does not rely on participants’ rejections of literal readings of stimuli, which should lead to a greater sensitivity to factors that influence error correction. In addition, the use of a single-shot auditory stimulus with an “exact” repetition task minimizes the effect of post-comprehension processes on our measure: if participants are asked to report exactly what they believe they heard (which ostensibly includes speech errors) the only repairs from the heard material to the reported material should be those caused by automatic, unconscious processing, though some conscious reconstruction due to memory limitations will inevitably occur. Finally, since the design only requires that statistical information (i.e., the distribution of noise) in the filler items influences the interpretation of the ambiguous noise, the paradigm can be extended to domains in which the literal syntax versus plausible semantics choice provided to participants in Gibson et al. (2013) cannot be used.

We demonstrate the effectiveness of this paradigm by testing a hypothesis about the interaction of the effect of grammatical preference and that of the distribution of noise. Specifically, we suggest that in situations where noise is distributed uniformly at random (hereafter the random noise condition), listeners correct noise based on top-down expectations like grammaticality, since a listener can discover no informative pattern in the noise distribution through which they can recover the segment underneath the noise and thus their best remaining option is to correct the segment using their prior linguistic knowledge. In situations where there is highly systematic distribution of noise (in the above example, a high likelihood that the noise always covers a vowel), a listener relies less on factors like grammaticality, since they now have more direct information about the uncorrupted signal: while our grammatical knowledge can help us determine what we are likely to hear in general, knowledge about the current conversation can more directly inform us as to what a particular speaker was likely to have said in this context. We additionally predict that different grammatical expectations vary in their robustness to this manipulation: stronger constraints will withstand the pressure to rely on distributional information more than weaker rules.

This hypothesis is motivated by the observation that, while Bayesian integration of both factors would be the ideal mechanism to integrate multiple signals into the decision making process (see the Noisy Channel Model (Gibson et al. 2013; Levy 2008; Ryskin et al. 2018)), this neglects the costs associated with tracking and manipulating all available information (Wittenberg and Jackendoff 2018: among many others). An alternative strategy might therefore be to adjust the weight of top-down and bottom-up information based on statistical patterns in the materials (for a similar question, see Schotter et al. 2014). Thus, the listener would rely heavily on top-down, grammatical information more when the inferred noise distribution is uninformative, but would prioritize using the more contextually relevant information from the inferred noise distribution when it is informative.

1.3 Current studies

In three experiments, we test the influence of different distributions of noise in the filler structure on the restoration of a noise covered function word: optional, sentence-initial “it” (Yoon 2001). We gradually increase the ratio of systematically distributed noise to random noise between experiments (see Figure 1) by including both fillers with random and systematic noise (Exp. 1, grey box in Figure 1), and then selectively only fillers with random (Exp. 2, red box in Figure 1) and systematic noise (Exp. 3, green box in Figure 1), while keeping the critical items constant across experiments.

Figure 1:

A diagram of the materials used in each of our experiments. Each dot represents an item used in Experiment 2 of Mack et al. (2012), with yellow dots representing the critical items, dark green dots representing fillers with an optional, sentence initial “that,” and other colored dots representing fillers with a different structure. Exp. 1 (grey box) replicated Mack et al. (2012) and used all of the items. Exps. 2 and 3 used only a subset of the fillers, with Exp. 2 using only the 24 “that”–fillers and Exp. 3 using 24 of the other fillers. Crucially, noise was systematically placed over the sentence-initial “that” in “that”–fillers, while each noise segment was distributed uniformly at random throughout the other filler items.

2.1 Methods

2.1.2 Stimuli and design

Mack et al. (2012) generously provided their original stimuli (as shown in Figure 3), all of which were used without modification in our replication. The 24 critical stimuli (Figure 2) were three-utterance dialogues ending in a potential null-subject construction with the subject position masked by noise. Each critical stimulus also contained a randomly placed additional segment of noise. Each item was manipulated across two factors: tense (past or present) and person (first or third).

Figure 3:

Each dot represents an item used in (Mack et al. 2012)’s Experiment 2, with yellow dots representing the critical items, dark green dots representing fillers with noise over an optional, sentence initial “that,” and other colored dots representing fillers with a different structure and all noise distributed randomly. Exp. 1, like (Mack et al. 2012), used all of the items.

The 60 fillers were either two- or three-utterance dialogues. These stimuli each contained two or three segments of noise distributed throughout the dialogue (see Figure 2).

2.2 Results and discussion

The mean restoration rate (% of expletive subjects produced) was 55.2%, lower than in the original experiment (67%). We replicated the effects of immediacy: participants were more likely to produce “it” in past tense than in present tense sentences, and in third person more than first person sentences (see Figure 4). Just as in Mack et al. (2012), tense was significant (β = −0.73, p < 0.01, χ ²(1) = 11.29), and person was marginally significant (β = −0.44, p = 0.051, χ ²(1) = 3.780) with no significant interaction (β = 0.11, p > 0.80, χ ²(1) = 0.062).

Figure 4:

Results from Exp 1. Participants were more likely to produce expletive subjects in place of noise in past-tense, third person sentences than in present-tense, first person sentences. Error bars are 95% CIs.

We found a lower “it”- restoration rate than the original experiment, and consider this an interesting likely effect of language change, where our participants found null subject constructions more acceptable.

Crucially for our purposes, Exp. 1 successfully replicated the results found in Mack et al. (2012), showing that manipulating tense and person does affect the rate at which listeners restore the “it” at the beginning of critical items in the predicted direction: first person sentences are restored less than third person sentences, and that present tense sentences are restored less than past tense sentences.

This allows us to use this methodology to investigate how grammatical preferences interact with other influences on the correction process in Exps. 2 and 3. We manipulate the level of systematicity in the distribution of the noise in the filler items to test whether softer grammatical violations will cease to be detected with a reduced filler ratio (Exp. 1 vs. Exps. 2 and 3): based on both Mack et al. (2012) and our results, while the effect of tense may weaken slightly, the marginal effect of person should weaken or disappear with a reduced filler-to-critical-item ratio, since, under prediction (2), weaker grammatical effects (like that of person) are liable to weaken or disappear entirely in environments with stronger cues from the distribution of noise. Crucially, reducing the filler ratio necessarily increases the strength of cues from the distribution of noise, since all of the critical items have noise systematically placed over the segment containing the expletive subject

We now test whether this ratio alone, or also the distribution within filler items, influences error correction behavior. In order to do this, we selectively changed the number and structure of fillers in the experiment.

3.1 Methods

3.1.2 Stimuli and design

The stimuli and design were identical to that of (Mack et al. 2012) and Exp. 1, except that we only used a 24-item subset of the fillers in which the noise was distributed randomly (see Figures 2 and 5). Thus, the ratio of structured noise (critical items) to random noise (fillers) was increased to 1:1.

Figure 5:

Each dot represents an item used in Exp. 2, with yellow dots representing the critical items, dark green dots representing fillers with noise over an optional, sentence initial “that,” and other colored dots representing fillers with a different structure and all noise distributed randomly. Exp. 2 used the 24 critical items and 24 fillers with randomly distributed noise.

3.2 Results and discussion

The mean “it” restoration rate was 20.95%. The effect of tense was significant (β = −0.64, p < 0.05, χ ²(1) = 6.21). In contrast to Exp. 1, the effect of person was not significant (β = −0.23, p = 0.30, χ ² = 1.07), and neither was the interaction of tense and person (β = 0.74, p = 0.15, χ ² = 2.06). That is, participants were again more likely to produce expletive subject constructions in past-tense sentences than in present tense sentences, but the marginal effect of person disappeared. As in Exp. 1, the interaction was not significant (Figure 6). Thus, tense seems to be a stronger grammatical preference than person, whose effect was only marginal in Exp. 1, and is not significant with a higher proportion of structured noise in the materials (Exp. 2).

Figure 6:

Results from Exp. 2. Participants were more likely to produce expletive subjects in place of white noise in past-tense, third person sentences than in present-tense, first person sentences.

4.1 Methods

4.2 Results and discussion

The overall restoration rate was 31.3%. As in Exps. 1 and 2, the effect of tense was significant (β = −0.53, p < 0.05, χ ²(1) = 6.06): That is, participants were again more likely to produce expletive subject constructions in past-tense sentences than in present tense sentences. In contrast, the effect of person was not significant (β = −0.13, p = 0.50, χ ²(1) = 0.45), and neither was the interaction of tense and person (β = 0.30, p = 0.41, χ ²(1) = 0.67). After the marginal person effect of Exp. 1, and the non-significant effect in Exp. 2, we take this as supporting evidence that tense generates more robust grammatical top-down expectations (Figure 8).

Figure 8:

Results from Exp. 3. Participants were significantly more likely to produce expletive subject constructions in past tense sentences than in present tense sentences, but we found no significant effect of person. In addition, restoration rates were significantly higher than in Exp. 2