Models of dataset size, question design, and cross-language speech perception for speech crowdsourcing applications

1 Overview

Crowdsourcing can be defined as the purchase of data (labels, speech recordings, etc.), usually on-line, from members of a large, heterogeneous, fluctuating, partially anonymous, and variably skilled labor market. The purchase of speech data and speech labels from crowdsourcing labor markets has the potential to significantly increase the scale of experiments that can be conducted in laboratory phonology, if experiments are designed to overcome the fundamental limitations of crowdsourcing. A few of the key limitations worth discussing include the severely limited control an experimenter has over the accuracy, training, and native language of workers hired to perform any given task. The goal of this article is to describe methods that can be used to compensate for crowd worker errors, for their possible lack of linguistic expertise, and for their possible inability to understand the language they are transcribing.

The experiments described in this paper were designed and conducted with frequent reference to two sources of information: the on-line tutorial guide for the crowdsourcing site used in this work, and the book Crowdsourcing for Speech Processing: Applications to Data Collection, Transcription and Assessment edited by Eskenazi et al. (2013). Material in the on-line tutorial provided the knowledge necessary to set up tasks, and to invite and pay workers. Useful material in Eskenazi et al. (2013) included methods for screening workers, choosing a fair rate of payment, explaining tasks to workers, and evaluating the quality of work already performed. Material in these two key references will not be repeated here, except as necessary to discuss error compensation, compensation for lack of training, and compensation for worker inability to understand the language being transcribed.

Crowdsourcing is partially anonymous, in the sense that tasks are usually connected to workers by way of an intermediate labor broker. The broker may be a company, a consortium of academics (e.g., GalaxyZoo), a non-profit foundation (e.g., SamaSource), or any other institution; it is also possible for an academic research project to solicit crowd workers directly, e.g., by advertising on appropriate mailing lists. When a broker is involved, the broker typically handles payment, so that employers and employees do not see one another’s financial information.

The demographics of crowdsourcing seem to be primarily determined by the ability of brokers to pay workers, which, in turn, is determined by labor and financial regulations in the employer and employee countries of residence. Pavlick et al. (2014) tracked the IP addresses of workers responding to a task, then determined country of residence using reverse address lookup; they also asked each worker to provide a brief description of his or her working conditions. They found that workers were logged in from 86 different countries; the two most frequently reported countries were India and the United States. Workers in more-developed countries reported that they chose to work in crowdsourcing markets as a part-time job with scheduling flexibility. In less-developed countries, crowd workers reported being mostly full-timers, who treat crowdsourcing as a consulting job. Mason and Watts (2009) reported that the payment offered per task affects the speed with which tasks are performed (the number of workers who will perform the task, and the number of such tasks performed by each worker), but not the quality of the work performed. Our own experiments suggest that the result of Mason and Watts (2009) must be qualified. For example, the anonymity of the crowd labor market permits the existence of “spammers”: workers who will enter data at random without any good-faith attempt to perform the assigned task. It is possible to avoid spammers by hiring only those workers with a good reputation (workers who have been assigned high scores by previous employers), but workers with high reputation are usually only willing to work for a reasonable wage.

The remainder of this paper is organized around a series of illustrative examples, which will be periodically made concrete. Suppose that you are trying to describe the sound system of the Betelgeusian language. Perceptual transcription studies, including your own, suggest that the Betelgeusian vowel inventory is characterized by formant frequencies, but also by some other yet-unknown acoustic features (possibly related to the fact that Betelgeusians have two heads; Adams 1979). We consider three approaches that might be used to identify acoustic correlates of the vowel categories in Betelgeusian. First, we consider the most controlled of the three proposed experiments, in which minimal pairs (words that differ only in the vowel) are recorded by cooperative informants. Under controlled conditions, the question of interest is dataset size: how many recorded examples are necessary in order to definitively describe inter-category differences in a certain pre-determined number of acoustic correlates? Second, we consider a somewhat less controlled experimental situation, in which words are recorded without advance knowledge of the vowel category in each word, and therefore some type of waveform labels are necessary. It is assumed that labels are solicited via crowdsourcing, but that there is no way to recruit crowd workers who speak Betelgeusian; therefore it is necessary to simplify the task so that it can be performed by workers who are not native speakers of Betelgeusian.

Finally, we consider the least controlled of the three experiments, in which short recorded phrases in the Betelgeusian language must be transcribed by crowd workers who speak no Betelgeusian. Even in the least controlled experimental situation, it is possible to recover a complete transcription, if one has side knowledge about the vocabulary of Betelgeusian and about the misperception of Betelgeusian vowels by English-speaking crowd workers.

2 Dataset size

As an illustrative example, suppose that you are trying to describe the sound system of the Betelgeusian language. You have decided to plan a trip to Betelgeuse, during which you will acquire recordings of men, women, and children producing each of the vowels in real words matching a list of target consonant contexts. Ultimately, you would like to identify the acoustic correlates that distinguish each vowel from every other. The question answered by this section is: how many examples of each vowel do you need to record?

The question of dataset size is actually two questions: (1) what measurements need to be acquired, and (2) how many training examples are necessary in order to accurately estimate the desired measurements? For example, consider the description of American English vowels by Peterson and Barney (1952). Peterson and Barney proposed that each vowel category, in American English, is characterized by its first and second formant frequencies (F1 and F2). Specifically, they proposed that each vowel category is described by average formant frequencies (µ₁ and µ₂), by the standard deviation of each of the two formants (σ₁ and σ₂), and by the correlation of the two formants (ρ). There are two ways in which a model of this type might be wrong. First, the set of measured parameters (two mean formant frequencies per vowel, two standard deviations per vowel, and one correlation coefficient per vowel) might not be sufficient to characterize the true difference between any pair of vowels. Second, even if the model is correct, the training data (the set of recorded examples from which the parameters are estimated) might be inadequate to estimate the model parameters. Barron (1994) called the first type of error “Approximation Error,” and the second type “Estimation Error.” He demonstrated that, for some types of models, Estimation Error is proportional to n/N, where n is the number of parameters that are being estimated, and N is the number of recorded training examples that are used to perform the estimation.

Figure 1 shows an example of the way in which Estimation Error decreases with N. Suppose that we are trying to distinguish between two types of vowels that are distinguished by some unknown phonological distinctive feature. One of the two vowels is [+feature]; we will call this vowel +1. The other vowel is [−feature]; we will call this vowel −1. Since we do not know, in advance, what the feature is, the best thing we can do is to observe N examples of vowel +1, and compute an average spectrum μ1. Likewise, we will observe N examples of vowel −1, and compute its average spectrum μ−1. But since the training examples are recorded from a randomly selected group of people, there is some variability. In particular, suppose that we now record a vowel that is exactly halfway between +1 and −1, and we use the trained classifier to decide which of the two categories it belongs to. Since the vowel is (by assumption) halfway between the two categories, our classifier should produce the output ‘0’, but since our classifier was trained using too few data, it produces an output that is not exactly ‘0’. Figure 1 shows the actual output of such a classifier, as a function of both N (the number of training examples) and n (the number of trainable parameters, which in this case is equal to D, the dimension of the spectral vector μ1). There are 20 different curves on each set of axes, because the experiment was run 20 times per setting using random simulated training datasets. Notice that, as N gets larger, all of the different random experiments converge to the ideal solution, ‘0.’ The vertical line segment in the middle of each subfigure is drawn when N = n, that is, when the number of training examples is equal to the number of trainable parameters: notice that the experimental conditions have been normalized so that when n = N, the Estimation Error of the classifier is exactly n/N = 1. Further details of the equations underlying Figure 1 are provided in Appendix A.

So how many training examples do you need? The answer is: it depends on how much Estimation Error you are willing to tolerate. Figure 1 is normalized so that the Estimation Error is exactly n/N, but there is usually some multiplier involved; for example, the normalized Gaussian classifier shown in Figure 1 will actually misclassify about 16% of all test tokens when n/N = 1, but only about 2% when n/N = 1/5. There are so many different types of classifiers for which N ≥ 5n is adequate that this ratio has been given a name: the “Rule of 5.” The Rule of 5 says, simply, that in order to train a classifier with n trainable parameters, it is usually adequate to acquire N ≥ 5n training examples.

A Gaussian mixture model (GMM) is a model in which each vowel is allowed to have M different modal spectra, as shown in Figure 2. These different modes might represent different allophones of the same phoneme, or they might represent much smaller sub-categorical differences. For example, in English, the vowel /ɪ/ tends to take on the lip rounding features of the consonants surrounding it: the segment /ɪ/ extracted from the word will sounds (if removed from context) more like /ʊ/ than /ɪ/, yet when perceived in context, it is unquestionably an /ɪ/. In a GMM, each modal production is represented by its own Gaussian, with its own average formant frequency vector (or any other set of features, e.g., Davis and Mermelstein 1980 or Hermansky 1990), as shown in Figure 2. If each Gaussian mean is represented by D frequency samples, and if there are M modes per vowel, then there are a total of n = MD trainable parameters per vowel. As shown in Figure 3, the Estimation Error of a GMM is therefore proportional to n/N = MD/N, which is M times larger than the Estimation Error of a simple Gaussian model. The advantage of a GMM is its flexibility: by using M modes per vowel, it is possible to represent up to M subtly different modal productions of the vowel. The disadvantage is increased Estimation Error. If adequate training data are available, then a GMM is a better model; if the training database is too small, then a Gaussian is a better model.

3 Crowdsourcing with binary error correcting codes

There are, essentially, three ways to get transcribed speech data. First, one can prepare a list of words containing the speech sounds of interest, and ask cooperative informants each to produce an example of each word. Solicited productions can provide enough data to estimate the parameters of a reasonably simple statistical model, e.g., a model with no more than a few dozen trainable parameters (and therefore requiring no more than a few hundred solicited productions). In order to estimate a model with more parameters, it is usually necessary to record spontaneous speech (e.g., news broadcasts, storytelling, interviews, and conversations), and to transcribe it after the fact. The second standard method of acquiring transcriptions is by soliciting them from native speakers of the language being transcribed. Transcription by native speakers is possible if the language has an orthography, and if there are native speakers with computers who are willing to use the orthography to perform transcription. Most of the world’s 7,000 languages do not have a standard orthography, and/or do not have a sufficiently large pool of internet-connected users who are willing and able to perform native-language transcription. When native-language transcription is not possible, all previously published research concludes that transcription is impossible, and that solicited productions are the only available method of inquiry. This paper proposes a method called “mismatched crowdsourcing,” in which transcribers who don’t understand the language are, nevertheless, asked to write what they hear. There are two important obstacles to mismatched crowdsourcing: the transcribers lack discrete phoneme categories matching those of the language they are transcribing, and they lack correct perceptual category boundaries. Incorrect perceptual maps are a problem similar to the problem of second language acquisition, and will be discussed in more detail in a later section. Lack of discrete phoneme categories, on the other hand, can be usefully compared to the lack of expertise in a scientific categorization ontology: category ontologies exist in many areas of science, and most of them are too complicated to be effectively used by non-expert transcribers. This section explores the problem of soliciting partial transcriptions from crowd workers who lack the expertise necessary to provide a full transcription.

For linguistic tasks, the distinction between “easy” and “hard” tasks varies substantially depending on the background and training of the worker. Distinctions learned in elementary school are often considered “easy”, while those learned in graduate school might be considered “hard”. In many situations, age of acquisition has been demonstrated to be a surprisingly useful metric for estimating the difficulty that will be presented by any linguistic task to people who have no formal linguistic training; for example, Kim et al. (2010) found that average age of acquisition was the best predictor of production error rates in dysarthria. In order to obtain useful transcriptions of a language from people who do not speak the language, therefore, it may be useful to factor each vowel or consonant labeling task into several disctinctive feature labeling tasks, and to assign each distinctive feature labeling task only to workers whose native language includes a comparable distinctive feature. Naturally, distinctive feature notation is unfamiliar to most non-linguists; therefore the tasks distributed to non-experts should be mapped into a notation that they have used since elementary school: the standard orthography of their native language.

For example, suppose you have established that the Betelgeusian consonant inventory includes oral plosives with up to eight distinct categories at each place of articulation, apparently categorized as voiced versus unvoiced ([g] versus [k]), unaspirated versus aspirated/breathy ([k,g] versus [k^h,g^h]), and lax versus pharyngealized ([k] versus [k^ʕ]).

Hindi (and other Indic languages) includes a four-way categorization at each place of articulation, among the glottal features voiced versus unvoiced ([g] versus [k]) and unaspirated versus breathy ([k] versus [k^h]); every native speaker of Hindi learns to correctly label these four categories in kindergarten, using the four Devanagari symbols shown in Figure 5(a). Arabic, on the other hand, includes a four-way categorization marked by the features voiced versus unvoiced ([d] vs. [t]) and plain versus pharyngealized ([t] versus [t^ʕ]), and every native speaker learns to perform this labeling task in kindergarten using the Arabic symbols shown in Figure 5(b). In order to transcribe Betelgeusian, therefore, it might be wise to divide the transcription task into several sub-tasks, two of which are shown in Figure 5. Native speakers of Hindi can be solicited to label the glottal features of all plosives, using the Devanagari symbols for ‘ka’, ‘kha’, ‘ga’, and ‘gha’; the experimenter can then interpret each of these labels as a pair of binary distinctive feature labels transcribing the voicing and aspiration of each plosive. Native speakers of Arabic can be solicited to label the voicing and pharyngealization of all plosives, using the Arabic symbols for ‘ta’, ‘da’, ‘t^ʕa’, and ‘d^ʕa’; the experimenter can then interpret each label as a pair of binary distinctive feature labels transcribing voicing and pharyngealization. The final voicing label of each plosive is computed by a majority vote including all transcribers, both Hindi-speaking and Arabic-speaking. The final aspiration label is computed by a majority vote among Hindi-speaking transcribers, while the final pharyngealization label is computed by a majority vote among Arabic-speaking transcribers.

Factoring transcription into sub-tasks is likely to lead to faster, more reliable results, for two reasons. First, we have opened up the labor market for our task: rather than requiring native speakers of Betelgeusian, we are now free to recruit native speakers of Hindi and Arabic, and as shown by Pavlick et al. (2014), the crowd labor market includes many people with native proficiency in at least one of these languages. Second, and equally important, we have converted a difficult labeling task (one that is non-native for all available transcribers) into a series of easy labeling tasks. Each crowd worker listening to the sentence has access to an auditory percept representing the complete transcription, but (if not a native speaker of Betelgeusian) he does not have the expertise to correctly label it. Instead, he has only the expertise to correctly label a sequence of several “easy questions”: binary distinctive features, whose values his transcription can correctly label as either a_j = +1 or a_j = −1.

Errors are introduced because (i) transcriber attention occasionally wanders, and more importantly, because (ii) no two languages use identical implementations of any given distinctive feature. Variability is introduced when two distinctive features appear independently in Betelgeusian, but not in either transcriber language. For example, neither Arabic nor Hindi distinguishes the Betelgeusian phoneme pair [g^h] versus [g^ʕh].

Preliminary experiments suggest that a phoneme that does not exist in the transcriber’s language is misperceived (mapped to symbols in his language) according to a probability distribution that is neither perfectly predictable (with zero error) nor uniformly unpredictable (with maximum error), but somewhere in between. It is therefore possible to talk about the error rate of the jth crowd worker’s phone transcription: he labels binary distinctive features as though he believes any given phone should be given the mth possible phoneme label. Suppose that he has probability 1 − p of guessing the wrong phone label, where 1 − p is presumably rather high, because he is labeling speech in a language he does not know. Instead of asking the jth transcriber to provide a phoneme label, however, suppose that we interpret his transcription as if it provided only one binary distinctive feature label, a_j, which is either a_j = +1 or a_j = −1. The distinctive feature is wrong only if the jth transcriber has picked a phone label with the wrong value of a_j. The probability of this happening is the probability of error, 1 − p, multiplied by the fraction of all errors that are wrong about this particular distinctive feature: thus the probability of any given distinctive feature is less than 1 − p.

Partially correct transcriptions can be accumulated from many different crowd workers by letting c_mj represent the answer the jth worker should have given if hypothesis m were correct (c_j = +1 if the mth possible phoneme label is [+feature_j], c_j = −1 if the mth possible phoneme label is [−feature_j], where feature_j is the distinctive feature we extract from the jth transcription). The best phoneme transcription is then the transcription whose error-correcting code, c_mj, best matches the distinctive feature labels that were actually provided by the labelers. Redundancy in this way permits us to acquire more accurate transcriptions, because even a crowd worker who is wrong about every single phoneme is often, nevertheless, right about many of the distinctive features (Vempaty et al. 2014).

4 Binary coding answers scientific questions

The method of asking easy questions is useful not only because it improves accuracy, but also because it permits the experimenter to ask questions that could not otherwise be asked.

For example, suppose you would like to evaluate the prosodic characteristics of an utterance in a language for which the prosodic system has not yet been studied by linguists. This is a challenging task for two reasons. First, phrase-level prosodic features marking prominences and phrase boundaries are not consistently denoted in standard orthography for most languages, which means that transcribers from any native language background will not have a familiar vocabulary or symbol set with which to identify prosody in a given utterance, regardless of whether the target language is one they speak. A second problem is that the acoustic parameters that encode the prosodic features of a word (e.g., F₀, duration, intensity) vary as a function of many factors other than prosody, including the sex, age, and physiological state of the speaker, the speaker-selected style and rate of speech, and the local phonological and discourse context of the word (Cole 2015). These factors interact with the expression of prosodic features marking prominences and phrase boundaries, with the result that the acoustic cues to prosody are highly variable across utterances, both within and across speakers, making prosodic transcription challenging even for trained transcribers who are native speakers of the language being transcribed. In short, it can be difficult to obtain a prosodic transcription of an utterance using discrete prosodic features when there is imperfect or incomplete knowledge of the feature set and/or of the mapping between prosodic features and acoustic cues.

Linguistic analyses of prosody typically rely on prosodic transcriptions performed by trained experts using a transcription system grounded in phonological analysis. For instance, the Tones and Break Indices (ToBI) transcription system allows for each language or dialect an inventory of tones and break indices as categorical prosodic features (Beckman and Elam 1994; see Figure 6). The ToBI system is based on the Autosegmental-Metrical theory (Pierrehumbert 1981), which proposes discrete prosodic features that encode the syntactic, semantic, and pragmatic properties of the prosodically marked work. While experiments on prosody perception and production can probe the prosodic categories specified in a ToBI transcription for a given language, it is not possible to simply ask crowd workers to transcribe the tones and break indices they hear in speech audio; it takes considerable auditory training, and training in the interpretation of acoustic cues from visual displays of the speech waveform, spectrogram, and pitch track to produce reliable and consistent prosodic transcription. To our knowledge, prosodic transcription of this type is always carried out by transcribers who have native or near-native fluency in the target language. In addition to the requirement of training and native-like fluency, another bottleneck for obtaining prosodic transcription using ToBI or similar systems is transcription time, which can take anywhere from 10 to 100 times the duration of the audio file.

The system of Rapid Prosody Transcription (RPT; Cole et al. 2010a, b) was developed explicitly for the purpose of soliciting judgments of perceived prosodic features from linguistically untrained subjects. RPT expresses the hypothesis that every language user produces and perceives at least two prosodic distinctions: the distinction between prominent versus non-prominent words, and the distinction between phrase-boundary and non-boundary word junctures. In initial experiments, over 100 University of Illinois at Urbana-Champaign undergraduates without significant linguistic training were asked to perform prosodic transcription based only on their auditory impression, without reference to any visual display of the acoustic speech signal (Cole et al. 2010a, b). Transcription was intentionally coarse-grained: transcribers were given only simple definitions of prominence and boundary, and were instructed to mark words where they heard prominence or boundary (Figure 7).

The method is intentionally fast: Transcription is done in real time, with two listening passes per excerpt, based only on auditory impression. Because definitions of prominence and boundary are simplified, the RPT system can be used to solicit prosodic transcriptions from crowd workers. The Language Markup and Experimental Design software (LMEDS) was developed for this purpose, and has been used in recent experiments (Mahrt 2013).

RPT compensates for ambiguity in the definitions of prominence and boundary (because the words prominence and boundary are not understood as precise terms by most language users) through strength in numbers: groups of 15–22 subjects transcribe prosody for the same speech excerpts. The labels assigned by multiple transcribers are aggregated to assign each word in the transcript a Prominence Score (P-Score) or Boundary Score (B-Score) (Figure 8). The B-Score and P-score are each fractions between 0 and 1. The B-score is defined as the proportion of transcribers who marked a boundary following the word. Similarly, each word receives a prominence score (P-score) indicating the proportion of transcribers who marked the word as prominent.

Because the questions asked by RPT can be answered by untrained crowd workers, it is possible for RPT to ask questions of scientific interest that could not otherwise be asked. For example, the question RPT was initially developed to ask: Can untrained transcribers produce reliable and consistent categorical transcriptions of prosody in conversational, spontaneous speech? Substantial evidence now indicates that the answer is yes; categorical prosodic features are perceived in ordinary conversational speech, even by language users with no formal linguistic training (Mahrt et al. 2011, Mahrt et al. 2012; Cole et al. 2014). Having established the validity of prosodic labels produced by untrained listeners, RPT has then been applied in order to probe the acoustic and textual correlates of perceived prosodic prominence and boundary, as heard by untrained listeners (examples of these findings for correlates are given in papers including Cole et al. 2010a, Cole et al. 2010b, Cole et al. 2014; Mahrt et al. 2011, Mahrt et al. 2012).

As another example of the types of scientific questions that can be answered using binary-encoded crowdsourcing, consider the problem of determining whether or not Hindi phrasal stress (prominence of one word within each prosodic phrase) exists, and if so, whether or not it is cued by any intonational correlates. The previously published literature expresses considerable disagreement about this question. While traditional pedagogical references on Hindi grammar (Kellogg 1938) already considered prosody, there have been several more recent studies exclusively focusing on specific aspects of prosody (Moore 1965; Ohala 1983, Ohala 1986; Harnsberger 1994; Nair 2001; Dyrud 2001; Patil et al. 2008; Genzel and Kügler 2010; Féry 2010; Puri 2013). Some of these later studies on Hindi intonation borrow insights from extensive work on the intonational phonology of Bengali (a closely related South-Asian language) (Hayes and Lahiri 1991, Hayes and Lahiri 1992; Fitzpatrick-Cole and Lahiri 1997). These works on Hindi intonation have found that there are systematic prosodic phrasing effects in Hindi. They have also uncovered specific consistent features, like the presence of a rising contour associated with every non-final content word. On the other hand, there is no consensus on several other aspects of Hindi prosody. For instance, though it is widely agreed that there is lexical stress in Hindi (Moore 1965; Ohala 1986; Harnsberger 1994; Nair 2001; Dyrud 2001), there is less agreement on the phonetic correlates of stress and the perceived placement of stress at the word level. As another instance, contradictory theories regarding the relation between prosodic prominence and pitch accents have appeared in prior work (Patil et al. 2008; Féry 2010; Féry and Kentner 2010; Genzel and Kügler 2010). In particular, based on the observation that Hindi speakers do not produce consistent pitch contours on stressed syllables, Féry and colleagues (Patil et al. 2008; Féry 2010; Féry and Kentner 2010) propose that Hindi uses phrasal tones to prosodically structure an utterance and does not have prominence-lending pitch accents.

The question of prominence and boundary perception in Hindi was studied in Jyothi et al. (2014) by recruiting 10 adult speakers of Hindi, and playing to them 10 narrative excerpts in the Hindi language (about 25 seconds each, from the OGI Multi-language Telephone Speech Corpus). Listeners were asked to mark (a) how the speaker breaks up the text into chunks (boundary), and (b) words that are emphasized or stand out relative to other words (prominence). Each listener coded (condition 1) half of the utterances with audio, and (condition 2) half without audio, in order to determine the extent to which responses depended on the text versus the audio (no punctuation was provided in either condition). RPT responses in Hindi were compared to two “reference” ToBI transcriptions. Since there is no standard ToBI transcription system for Hindi, both reference transcriptions were mismatched, but in slightly different ways. Condition 3 consisted of a ToBI transcription performed by a professional linguist, trained in the ToBI transcription of English, who is also a native speaker of Hindi. She assigned prosodic phrase boundary labels based on her knowledge of the acoustic correlates of phrase boundary in Hindi, but because there is no consensus about the status of phrasal prominence in Hindi, she did not similarly seek correlates of phrasal prominence. Instead, she used the “pitch accent” label L+H* to label any acoustically evident pitch rise that she perceived as a phonological gesture, including the pitch rises that are characteristic of most content words in standard Hindi production. Her definition of pitch accent was therefore, by design, incommensurate with the instructions given to subjects in the rapid prosody transcription (RPT) study. Condition 4 was incommensurate with the RPT labels in a different and more severe way: ToBI labels were generated for the same speech data using AuToBI (Rosenberg 2010), a program that is designed to automatically generate ToBI labels for English-language speech.

Agreement of the 10 RPT transcribers with one another, and with each of the two reference transcriptions, was measured using Fleiss’ kappa (Figure 9). Agreements about the coding of phrase boundaries were moderate, supporting the claim that phrase boundary is perceptible in Hindi. Responses to audio stimuli showed moderate agreement with responses to text stimuli, supporting the claim that phrase boundaries can be accurately predicted from text. Responses of the 10 listeners without linguistic training show moderate agreement with ToBI labels produced by the expert linguist, suggesting that the acoustic cues described in the linguistic literature correlate well with the perceptions of untrained listeners. All human Hindi speakers (including both the 10 listeners without linguistic training, and the 1 listener with linguistic training) show moderate agreement with the scores produced by the English-language AuToBI system (which has no information about the Hindi text), suggesting that the acoustic correlates of phrase boundary in Hindi are similar to those used in English.

Kappa scores computed based on descriptions of prominence show a quite different pattern from the boundary scores. Listeners without linguistic training show fair agreement with one another, providing some support for the claim that prominence is perceptible in Hindi. Listener responses to audio stimuli show fair agreement with the responses to text stimuli, supporting the claim that, whatever the Hindi listeners are perceiving, it can be predicted from text without audio (as was also true of phrase boundaries). It is not yet clear, from these results, whether the labels produced by non-expert listeners have any acoustic correlates, or whether they have only textual correlates. Unlike B-scores, P-scores produced by untrained listeners show only slight agreement with the labels produced by the trained linguist, indicating that the instructions given to untrained labelers in the RPT study were effective in achieving the desired outcome: the expert linguist labeled the pitch accent on nearly every content word, but the listeners without linguistic training labeled only a subset of these words. Finally, the English-language AuToBI system showed fair agreement with the expert linguist’s pitch accent labels, and very little (slight) agreement with the non-expert labels, suggesting that the acoustic correlates of pitch accent in English can be used to predict presence of a content word in Hindi, but not phrasal prominence.

5 Crowdsourcing under conditions of language mismatch

In the age of globalization, the act of listening to a language you do not understand is the source of frequent amusement. Transcribing what it sounds like (using words in your own language) is called, in Japanese, soramimi (literally ‘empty ear’), after the “Soramimi Hour” in the popular TV program “Tamori Club”. In English, transcribing speech using words of the wrong language is sometimes called “buffalaxing,” after the screen name of the author of the popular “Benny Lava” video. Buffalax (Mike Sutton) listened to a Tamil love song, “Kalluri vaanil kaayndha nilaavo” (lit. ‘the moon that scorched the college campus,’ danced by Prabhu Deva and Jaya Seal in 2000), and heard the words “My loony bun is fine Benny Lava.” He proceeded to add subtitles to the entire video, providing English lyrics that were phonetically similar to the original Tamil lyrics, but absurd and often outrageous (Phan 2007).

It is important to keep in mind that the Buffalax lyrics represent a single perception: this is the transcription produced by one listener, listening to sung speech with background music, under unknown listening conditions. He is explicitly seeking to map unintelligible Tamil phone sequences into real English words: more than that, he is explicitly searching for an English word sequence that will be funny. Despite its limitations, as a real-world speech perceptual experiment, the Benny Lava video is enlightening. The word “kalluri,” for example, was heard as “my loony,” perhaps in part because the flapped /n/ of “loony” is one of the English phones acoustically most similar to the flapped /r/ of “kalluri.” More enlightening is the title of the spoof: the phrase “fine Benny Lava” is derived from the Tamil words “kayndha nilaavo.” If second-language speech perception tends toward mistakes that minimize distinctive feature substitutions, as proposed by many standard second language (L2) acquisition models (Strange 1995), then the aspirated /dh/ in “kayndha” should have been misperceived as an English /d/, and the title of the spoof should have been “Danny Lava.” Instead, for some reason, the stop was perceived as a /b/. In further experiments described in the remainder of this section, we find that the minimum-feature-distance substitution pattern described by standard L2 acquisition models is approximately sustained, but that L2 phones with no exact first language (L1) match are subject to considerably more variable interpretation than phones with an exact L1 counterpart.

The “Benny Lava” experiment exemplifies a number of the problems that have been addressed in the literature on second language speech perception. Two of the most influential theoretical models of second language speech perception are Flege’s Speech Learning Model (SLM; Flege 1987, Flege 1995, Flege 2007; Flege et al. 2003) and Best’s Perceptual Assimilation Model (PAM) (Best et al. 1988; Best 1994, Best 1995). Flege demonstrated that learners of a second language (L2) create new phonetic categories for the L2 before they are able to reliably distinguish novel phonemes, and that any given L2 category may therefore lump together more than one phoneme in the L2. For example, an American beginning to learn French may create a new French /u/ category that differs from either the American /u/ or the French /u/, because the learner has mistakenly grouped together exemplars of the distinct French vowels /u/ and /y/ (Flege 1987). Best studied L2 phoneme perception during first exposure to the L2, and defined six possible relationships between L2 phonemes, depending on the way in which the L2 phonetic distinction interacts with the L1 (first language) phonological system. A pair of L2 phonemes both mapped to the same L1 phoneme (as in Flege’s /u,y/ example) possess a “Single Category” relationship, which Best demonstrated leads to the greatest difficulty in perceptual learning. In some cases, one of the L2 phonemes may be considered a very good example of the corresponding L1 category, while the other is considered a very poor example; Best defined this relationship to be a “Category Goodness” relationship, and demonstrated that it leads to substantially improved perceptual learning. Two L2 phonemes mapped to different L1 categories possess a “Two Category” relationship, and can be distinguished immediately. Finally, Best defined three different relationships that may occur when one or both of the L2 phonemes are “Uncategorizable,” or for other reasons cannot be mapped to any L1 phonetic category, in which case their distinguishability therefore depends on auditory perceptual acuity rather than phonetic dimensions. The mapping of the Tamil word sequence “kayndha ni laavo” to “fine Benny lava,” for example, may indicate (if it is not simply random noise: remember that this is a single perceptual trial) that the Tamil voiced breathy plosive /dh/ was perceived as “uncategorizable,” rather than being mapped as a poor exemplar of either the English phonemes /d/ or /t^h/.

In creating the “Benny Lava” video, Buffalax explicitly sought English word sequences that were phonetically similar to the Tamil lyrics. Less explicit effects of L1 vocabulary have been demonstrated in every L2 learning situation, even in situations where learners seek to hear nonsense words. Ganong (1980) showed that phonetic category boundaries between L1 phonemes (e.g., voice onset time boundaries between /d/ and /t^h/ in English) shift in a direction that increases the probability of hearing a known English word. Norris et al. (1997) showed that L1 phonotactics also play a role: nonsense phonetic content is perceived in a manner such that every perceived phoneme is part of a phonotactically acceptable L1 “word” (they call this the “possible word constraint”).

Consider the problem of developing speech technology in a language with few internet-connected speakers. Suppose we require that, in order to develop speech technology, it is necessary first to have (1) some amount of recorded speech audio, and (2) some amount of text written in the target language. These two requirements can be met by at least several hundred languages: speech audio can be recorded during weekly minority-language broadcasts on a local radio station, and text can be acquired from printed pamphlets and literacy primers. Recorded speech is, however, not usually transcribed, and the requirement of native language transcription is beyond the economic capabilities of many minority-language communities. We propose a methodology that bypasses the need for native language transcription. Our methodology, which we call “mismatched crowdsourcing,” is essentially formalized soramimi: we propose that speech in a target language should be transcribed by crowd workers who have no knowledge of the target language, and that explicit mathematical models of second language phonetic perception can be used to recover an equivalent transcription in the language of the speaker (Figure 10).

Assume that cross-language phoneme misperception is a finite-memory process, and can therefore be modeled by a finite state transducer (FST). The complete sequence of representations from spoken language to transcribed language can therefore be modeled as a noisy channel represented by the composition of two FSTs (Figure 10): a pronunciation model and a mismatch model. The pronunciation model is an FST representing processes that distort the canonical phoneme string during speech production, including processes of reduction and coarticulation. The mismatch model represents the mapping between the spoken phone string (in symbols matching the phone set of the spoken language) and the transcribed phone string (in symbols matching the orthographic set of the transcribed language).

Of these two FSTs, only the mismatch FST is not a component in any current standard speech technology. The mismatch FST is similar to phone substitution models that are used in computer-assisted language learning (CALL) software; e.g., we have learned CALL models in previous research by initializing weights according to substitution models reported in the second language learning literature, factoring the weights according to a distinctive feature based representation of each phone, and then applying machine learning methods to refine the classifier (Yoon et al. 2009). In research reported in this paper, two different mismatch transducers were tested. The first was an FST implementation of phonetic segment string edit distance, with substitution weights set proportional to the number of phonological distinctive features separating the spoken from the perceived phoneme. The second transducer was identical to the first, but with substitution weights learned from the data resulting from a mismatched crowdsourcing experiment. In order to train the mismatch FST, training data were created by asking crowd workers to transcribe Hindi speech using English-orthography nonsense syllables (producing an English-orthography nonsense transcription of each utterance, i.e., a sequence Ψ=ψ1,ψ2,… of English letters ψ_i), and by then re-transcribing each utterance with a fine phonetic transcription produced by a Hindi-speaking linguist, A=a1,a2,…. Using these transcriptions, the FST substitution costs, deletion costs, and insertion costs can be learned in order to minimize the total alignment cost between mismatched transcriptions and native language or expert transcriptions of a small sub-corpus.

Preliminary experiments in mismatched crowdsourcing were carried out by Jyothi and Hasegawa-Johnson (2015) using Hindi speech excerpts extracted from Special Broadcasting Service (SBS, Australia) radio podcasts. Approximately one hour of speech was extracted from the podcasts (about 10,000 word tokens in total) and phonetically transcribed by a Hindi speaker. The data were then segmented into very short speech clips (1–2 seconds long). The crowd workers were asked to listen to these short clips and provide English text, in the form of nonsense syllables that most closely matched what they heard. The English text (Ψ) was aligned with the Hindi phone transcripts (A) using the mismatch FST illustrated in Figure 11. This FST probabilistically maps each Hindi phone to either a single English letter or a pair of English letters. The FST substitution costs, deletion costs, and insertion costs are learned using the expectation maximization algorithm (EM) (Dempster et al. 1977).

Figure 12 (from Jyothi and Hasegawa-Johnson 2015) visualizes the main probabilistic mappings from Hindi phones (labeled using IPA) to English letter sequences, as learned by EM; the arc costs were initialized using uniform probabilities for all Hindi phone to English letter transitions. Only Hindi phones with 1,000 or more occurrences in the training data are displayed. The plot indicates that the crowd workers predict many of the Hindi sounds correctly, and in cases where they do not, the plot reveals some fairly systematic patterns of mismatch. For example, unaspirated voiceless stops in Hindi such as /p/ and /k/ were sometimes mistaken as their voiced counterparts, /b/ and /g/, respectively. This could be because voiceless stops in Hindi are unaspirated even in word-initial syllables, unlike in English, which causes them to be confused for their voiced counterparts when transcribed by speakers unfamiliar with Hindi.

It is useful to further quantify the variety of transcribed English letters matching each Hindi phone. For example, consider the hypothesis that Hindi phones that also exist in English (phones whose transcriptions, by linguists studying Hindi and/or English speech, commonly use the same IPA symbol) are perceived less variably than phones that do not. This hypothesis can be tested by measuring the equivocation of the English letter transcription (Appendix B), conditioned on the true Hindi phone label. Equivocation computed using standard formulas (Shannon and Weaver 1949; see also Appendix B) is shown in Table 2. The phone class in the last row of Table 2 refers to consonants in Hindi that are commonly transcribed using IPA symbols that do not appear in standard English phonetic transcriptions. The equivocation was lowest for the class of consonants that appeared both in Hindi and English, suggesting that crowd workers were more certain about transcribing these sounds. Conversely, the class of Hindi consonants not appearing in English had the highest equivocation.

Having trained the mismatch FST, we now have a complete and invertible model of the process by which Hindi words are transcribed into English orthography. The language model represents pW, the probability of the Hindi word string W=w1,w2,…. The pronunciation model represents p(A|W), the conditional probability of a Hindi phone string given a Hindi word string. The mismatch transducer shown in Figure 11 represents p(Ψ|A), the probability of the English letters given the Hindi phones. By composing and searching these FSTs, it is possible to estimate p(W|Ψ), the posterior probability of any possible Hindi word string given the available English-orthography transcriptions, and to find the most probable such word string. More than half the time it turns out that the correct Hindi word string is not the one with maximum posterior probability. Instead, these preliminary results suggest that the correct word string is one of the top eight, and that the size of this list can be typically reduced to four by combining the transcriptions from multiple crowd workers. In order to use mismatched crowdsourcing at scale, it will be necessary to apply other methods in order to reduce the size of the N-best list.

Methods derived from communication theory may be applicable. Traditional information theory (Shannon and Weaver 1949) is often concerned with the scenario where the decoder (from the mismatched crowdsourced labels) makes a single “hard” decision about which symbol was produced, but when we have (costly) access to an expert labeler, it is useful for the decoder to produce more than just a single estimate alone (Figure 13). If the decoder puts out more than one estimate, the result is called an N-best list, and the expert can then reduce the remaining equivocation in a feedback loop. We suggest that future work will be able to exploit the typical set in order to permit one informant, who understands the spoken language, to very quickly select the correct spoken-language transcription of each spoken utterance. Thus, for example, utterances might be given to many mismatched crowdsourcers, reducing the size of the N-best list to fewer than 32 words. These 32 words might then be listed to a Hindi-speaking linguist, who works as a sort of high-efficiency “correcting device” (Figure 13). The prompt screen lists N (W|Ψ) + 1 options: The N (W|Ψ) Hindi words that are most probable given the English transcription, and 1 option that says “OTHER” and allows the linguist to type something different.

In order to scale these methods, it will also be necessary to acquire mismatched transcriptions from a large number of internet users, typically users who are interested in language, but who do not speak the language they are transcribing. By setting up mismatched transcription as a code-breaking task, we believe that it can be made enjoyable, increasing the number of internet users who are willing to help us with this task. Figure 14 shows the mockup of an on-line game in which one player (the “coder”) selects and sequences blocks of non-English audio so that they sound like an English language sentence. The second player (the “decoder”) listens to the sequenced audio clips, and tries to guess the English language sentence that was transcribed by the coder. Both players receive points in proportion to the overlap between their transcriptions. A prototype of this game was deployed at the 2015 Beckman Open House at the University of Illinois.

6 Conclusions and future work

In order to model the phonetic correlates of phonology in any language, it is necessary to first (1) define the form of the model, (2) collect relevant training data, and (3) use the data in order to learn the parameters of the model. The number of parameters that can be learned from a dataset is, in general, directly proportional to the number of data. Methods for more precisely estimating the best fit to a dataset include structural risk minimization and cross-validation.

Crowdsourcing is the set of methods whereby labels are solicited from a large temporary labor market, often on line. Crowdsourcing provides labels at much lower cost than traditional professional data transcription, but crowdsourced labels also tend to have higher rates of transcription error. Majority voting reduces error, but triples (or worse) your cost.

Error-correcting codes can be used to reduce the error rate of crowdsourcing: if you factor each hard question into several easy (binary) questions, you can improve accuracy more cheaply, because each crowdsourcer only needs to be partially correct. Error-correcting codes can also be used to divide a task with a relatively constrained labor market (e.g., transcription of a rare language) into many smaller tasks, each of which can be released into a larger labor market (e.g., transcription of binary distinctive features, each of which is easily perceptible by the native speakers of one or more widely-spoken languages).

Besides its economic benefits, factoring a hard problem into many easy questions also has substantial scientific benefits. There are many questions that, phrased precisely, are comprehensible only to people with formal linguistic training. It is often possible to factor the hard question into many easy questions, and to phrase each of the easy questions in a way that is comprehensible to people without formal linguistic training. In this way, the science of easy questions permits us to explore the cognitive representations that are applied to the task of speech perception by people with no formal linguistic training.

Another method for acquiring transcriptions of a rare language is mismatched crowdsourcing, which we define to be the transcription of speech by those who have never learned the language being spoken. Mismatched crowdsourcing generates errors that are biased by the non-native perception of the listener. Biases of this kind have been heavily studied in the second-language learning literature; mismatched crowdsourcing provides a new method for studying these biases, and a new motivation for characterizing them precisely. In particular, by representing second-language speech perception as a noisy channel (a probabilistic finite state transducer), it becomes possible to compute the most likely source-language message based on error-filled mismatched transcriptions.

Future work will include further analysis of the mismatched crowdsourcing model (e.g., using ideas from the communication theoretic literature on guessing with side information; Sundaresan 2006). Some experimental validation of the mismatched crowdsourcing model has already been performed (Jyothi and Hasegawa-Johnson 2015), and more is under way.

The economic goal of this research is to scale the mismatched crowdsourcing approach, using methods including semi-supervised learning (train an automatic speech recognizer when only a few labels are available, and then use it to help acquire more labels) and active learning (use the half-trained speech recognizer to determine which data need human labels), in order to develop speech technology in languages for which the societal benefits of speech technology outweigh the commercial benefits. For example, consider the problem of helping geographically constrained people in remote communities (e.g., mothers with children) to buy and sell on the internet. There are more cell phones than humans in the world today; many people are able to access the voice network who have no reliable access to any other telecommunications modality. People in remote locations are less likely to speak a majority language, however, so if a woman wishes to sell her products on-line, she needs to rely on an educated intermediary to set up the website for her. There is no money to be made by creating speech technology in her language. Of the 6,000 languages spoken in the world today, fewer than 100 are spoken by more than eight million speakers each. Porting speech technology to a language with fewer than one million speakers is unlikely to yield great financial incentives, but if it can be accomplished automatically and almost for free, then we can use this technology to provide freedom and power to those who most need it.