Abstract
The current study explores how the cultural distance of ethnic outgroups relative to the ethnic ingroup is related to stereotypical news representations. It does so by drawing on a sample of more than three million Dutch newspaper articles and uses advanced methods of automated content analysis, namely word embeddings. The results show that distant ethnic outgroup members (i. e., Moroccans) are associated with negative characteristics and issues, while this is not the case for close ethnic outgroup members (i. e., Belgians). The current study demonstrates the usefulness of word embeddings as a tool to study subtle aspects of ethnic bias in mass-mediated content.
News media and racial prejudices
Racial prejudices is argued to be at the heart of the increasingly unfavorable public opinion climate regarding ethnic outgroups in European democracies (Gorodzeisky and Semyonov, 2016). Though the sources of racial prejudice in societies are multifaceted, mass media has been shown to critically contribute to the establishment and re-activation of biased perceptions of outgroup members (e. g., Mastro, 2009). Stereotypes, defined as the shared beliefs about group members’ traits (Greenwald, 1995), can be formed through media exposure – even in the absence of interpersonal contact with outgroup members (Mutz and Goldman, 2016; Ramasubramanian, 2007). Particularly in ethnically segregated western societies, the selective presentation in news messages of specific characteristics, issues, and opinions associated with ethnic outgroup members can have consequences for the development of audiences’ (non-)prejudiced beliefs and stereotypical associations.
Yet, not all ethnic outgroups are alike or portrayed in the same way by the media. Previous research on media portrayals of ethnicity and race documents large variation in the manner in which diverse ethnic outgroups are portrayed in various media environments (Mastro, 2009; Mutz and Goldman, 2016). For instance, in the US, media reports of Latino and Black Americans over-emphasize issues related to illegality and crime, and this tends to align with negative societal predispositions towards these groups (e. g., Mastro, 2009). In Europe, immigrants – especially those from Arabic and Muslim descent – tend to be represented as a threat to the national security and economic welfare of host countries (Ahmed and Matthes, 2017; Eberl et al., 2018). Albeit far less prominent, alternative framing of immigrants has also been documented, representing European immigrants as highly skilled and/or using positive sentiment (Bleich, Stonebraker, Nisar, and Abdelhamid, 2015; Blinder and Jeannet, 2018).
Cultural distance and stereotypical media portrayals
An important factor that could (partially) account for the variation in stereotypicality of media portrayals of outgroups is the cultural distance between the ethnic outgroup relative to the ethnic ingroup (Allport, Clark, and Pettigrew, 1954). Experimental research documents the intervening role of perceived cultural distance in social categorization processes, ultimately affecting the motivation to maintain central divides between those that are like “us” and those that are like “them” (Mahfud, Badea, Verkuyten, and Reynolds, 2018; Taijfel and Turner, 1979; Van Osh and Breukelmans, 2012). Tapping into social identity sentiments, groups that are regarded as more culturally distant are believed to have different norms, values, beliefs, and worldviews, and are often seen as a threat to the host land’s identity, in turn prompting a range of unfavorable intergroup outcomes, such as prejudice and unfavorable attitudes (Van Osh and Breukelmans, 2012). Conversely, groups characterized by a closer cultural distance trigger feelings of similarity and shared social identities, with more favorable intergroup perceptions as a result (Ye, Zhang, Huawen Shen, and Goh, 2014).
Extrapolating these findings to racial prejudice in media coverage, it can be anticipated that negative stereotypical representations in the media pertain especially to more culturally distinct outgroups than those outgroups culturally closer to the national ingroup. Media studies typically suggest that dominant social values and ideologies of ethnic ingroups are placed in the foreground of the media agenda, resulting in the underrepresented and negatively-skewed portrayal of culturally distant minorities (Ahmed and Matthes, 2017; Eberl et al., 2018). Culturally close outgroup members, considered largely equal regarding social norms and ideologies of the majority group, are not likely to stand out as different nor threatening, and may consequently benefit from more favorable media evaluations.
Although explicit, blatant and offensive racial statements may circulate in (social) media environments (Matamoros-Fernández, 2017), subtly communicated and transmitted forms of bias in mainstream media content have been recognized as especially powerful (Mastro, Behm-Morawitz, and Kopacz, 2008; Weisbuch, Pauker, and Ambaday, 2009). In agreement with publicly supported egalitarian norms, news sources and journalists may refrain from blatantly and overtly associating (distant) outgroups with stereotypical attributes (see Gaertner and Dovidio, 2005).
The manifestation of bias in news coverage, then, is more likely to surface in subtle and implicit forms. A promising way to investigate implicit forms of bias in media content is to investigate words that share semantic and syntactic similarity with different outgroups. In particular, one may tap into hidden forms of bias by investigating synonyms and words that often occur in the same linguistic context. Aversive forms of ethnic bias become apparent when stereotypic attributes, such as criminal or offender, are systematically used interchangeably or analogous to particular ethnic groups.
Current study focus and contribution
The investigation of systematic bias in the use of stereotypical attributes analogous to social categories has recently become possible due to the introduction of word embeddings in the field of Artificial Intelligence (AI) (e. g., Mikolov, Corrado, Chen, and Dean, 2013). The current research note explores these NLP-techniques to investigate hidden forms of bias in news coverage of culturally close and distant outgroups in the Dutch news media environment. We trained such embedding models on the total population of news articles that appeared in the five Dutch newspapers with the highest circulation rate (de Volkskrant, NRC Handelsblad, Trouw, Algemeen Dagblad, and De Telegraaf) for the period 2000–2015 (covering more than three million news items). Subsequently, we investigated which words have been ‘learned’ to resemble references to outgroup members. In order to investigate whether distant outgroups also appear more often in stereotypical news contexts, we complemented our analysis with a co-occurrence analysis with multidimensional scaling. Finally, we illustrated our findings using high-dimensional visualization techniques.
The current study makes several contributions to the literature. First, and methodologically, the current study shows how word embeddings can be used to explore and identify hidden ethnic bias in mediated content. This state-of-the-art algorithmic technique mapping relations between words allows for the large-scale identification of bias while maintaining high levels of accuracy (Bolukbasi, Chang, Zou, Saligrama, and Kalai, 2016a; Caliskan, Bryson, and Narayanan, 2017; Garg, Schiebinger, Jurafsky, and Zou, 2018). Second, and on a general theoretical level, the study explores the assumptions about the nature of bias in news coverage across two ethnic categories. In conclusion, the here-reported findings contribute to the formulation of expectations about the stereotypicality of news coverage, which may be generalizable beyond specific ethnic groups.
This research note will continue as follows. First, we will briefly outline possible differences between media outlets with regard to the portrayal of ethnic groups. We will then continue to the Method section, where we explain the general idea behind distributed word embeddings, illustrate their application, and argue why word embedding techniques offer a promising toolkit for researchers interested in media bias. We then discuss the results and reflect on our findings in the Discussion section.
Stereotypicality of news content across outlets
In addition to mapping the implicit nature of stereotypical news coverage, the current study aims to identify sources of variation in these portrayals, so to locate under which circumstances audiences are most likely to be exposed to biased representations of ethnic groups. Scholarship in the domain of media stereotyping has identified outlet type as an influential factor affecting the stereotypicality of news stories. The current study explores differences in the use of ethnic stereotypes in tabloid and quality newspapers. The use of stereotypes, as easily accessible heuristics and oversimplification of complex realities, aligns well with tabloids’ reporting style of simplification and limited word count. In addition, as tabloid newspapers often have close proximity to right-wing populist parties, they may give voice to anti-foreigner sentiments. Empirical evidence supports the notion that ethnic minorities are represented in stereotypical terms in European tabloid newspapers (Arendt, 2010; Kroon, Kluknavská, Vliegenthart, and Boomgaarden, 2016; Van Dijk, 2000). As a consequence, we anticipate stronger bias regarding distant outgroups in popular newspapers compared to quality newspapers.
Method
Classical methods of automated content analysis that measure whether a given word co-occurs with another may give a clear indication of the news contexts in which specific minority groups appear (see, for example, Jacobs, Damstra, Boukes, Swert, and Boukes, 2018; Ruigrok and Atteveldt, 2007). Yet, co-occurrence analyses are less suitable for the detection of stereotypical synonyms and analogies due to the focus on the direct textual context and its inherently deductive nature.
Recently, advancements in natural language processing, and, in particular the use of word embeddings, have made the refined analysis of (hidden) bias in texts possible (Bolukbasi et al., 2016a; Mikolov, Corrado, et al., 2013). Word embeddings are a current state-of-the-art algorithm for capturing, understanding and analyzing aspects of word meaning. The premise of word embeddings is based on the principle of distributional similarity, which Firth (1957) tellingly summarized as “[y]ou shall know a word by the company it keeps” (p. 11). Taking collections of unlabeled sentences as input, word embeddings are trained to learn the meaning of words by analyzing the context in which these appear. The word-embedding algorithm thus transforms human language into meaningful numerical representations in the form of valued vectors. Each word is represented by a vector of – in our case – hundred dimensions. Just as one can easily compute the distance between two objects in a three-dimensional space, one can also compute the distance between two words in a hundred-dimensional space.
We analyze our data using the word2vec algorithm as implemented in the Gensim package for Python (Mikolov, Corrado et al., 2013; Mikolov, Yih, and Zweig, 2013). First, this tool builds a vocabulary using unlabeled sentences from the training dataset. Second, word2vec learns word representations – or, more precisely, embeddings – by training an unsupervised, shallow neural network model. In this training process, embeddings are determined based on the direct context[1] in which specific words occur in the different sentences in the training data. With sufficient instances of contextual similarity, the model will learn that words are associated. In fact, the main idea behind word embeddings is that words closer together in a vector space share semantic meaning. Hence, two synonyms would occupy similar positions in the vector space and have a distance close to zero, whereas very different words should have vector locations further apart. Most similar words can be synonyms, but also represent words that are used in comparable contextual and topical domains (Bolukbasi, Chang, Zou, Saligrama, and Kalai, 2016b). For example, if the word cereal occurs frequently in sentences with the word breakfast, the model will learn that these words share semantic meaning.
Word embeddings have proved particularly useful in modeling diverse lexical tasks, such as information retrieval, as well as the identification and prediction of sentiment and topics (see for an example in the field of communication science: Rudkowsky et al., 2018). Yet, while word embeddings encode relevant semantic information, they also inherently reflect biases and stereotypes if those are present in the training dataset (Bolukbasi et al., 2016a). Of interest to the study’s aim, hidden biases in texts can be accurately detected and analyzed using word embeddings, a claim that is supported by a series of recently published studies in the field of computational sciences (Bolukbasi et al., 2016a; Caliskan et al., 2017; Garg et al., 2018; Tulkens, Hilte, Lodewyckx, Verhoeven, and Daelemans, 2016). These studies show that words close to specific social categories in the vector space may reveal bias. For example, if the vector representation of the word she is close to representations of words such as receptionist, nanny, or housekeeper, while the word he is close to financier, warrior, or broadcaster, this could suggest the presence of gender stereotypes (Bolukbasi et al., 2016a).
Training and data
For the purpose of the current study, we trained embeddings on all news items that appeared in the five Dutch newspapers with the highest circulation rate: de Volkskrant, NRC Handelsblad, Trouw, Algemeen Dagblad, and De Telegraaf for the period January 2000 up to and including December 2015. This resulted in a final sample of 3,316,494 news articles. By training the embeddings on the total population of news articles over this 16-year period, we derived word meanings over a substantial period of time, herewith providing a critical test of our hypotheses of the nature of ethnic stereotypes. In addition, and acknowledging potential differences across newspaper types, we trained two separate sets of embeddings on news items from quality (de Volkskrant, NRC Handelsblad, Trouw, [n = 1,777,024 news items]) and popular (Algemeen Dagblad, and De Telegraaf [n = 1,539,470 news items]) newspapers.
Selection of culturally close and distant ethnic outgroups
The study considers close and distant ethnic outgroups in the case of the Netherlands. Although this country is traditionally seen as highly tolerant towards ethnic outgroups, its political climate towards ‘others’ has toughened significantly over the past two decades (Erisen and Kentmen-Cin, 2017; Vasta, 2007). As the residents from a neighboring country, we consider Belgians as culturally close ethnic outgroups. The Dutch share strong historical, cultural, social and lingual backgrounds with Belgium (Polek, Wöhrle, and van Oudenhoven, 2010). Differences in perceived happiness, life satisfaction, and subjective well-being are relatively small between the neighboring countries (Inglehart and Klingemann, 2000).
For our selection of culturally distant outgroups, the current study focuses on a major ethnic-minority group in the Netherlands: Moroccans. Traditionally this ethnic group does not share historical, cultural or linguistic ties with the Dutch. The vast majority of people from Moroccan descent tend to be Muslims, while the Netherlands is traditionally a Christian country (even though nowadays the share of practicing Christians has declined). Vastly documented negative stereotypes that pertain to Muslims in the Western world are therefore likely to apply to this ethnic group (González, Verkuyten, Weesie, and Poppe, 2008; Richardson, 2004; Savelkoul, Scheepers, Tolsma, and Hagendoorn, 2011; Strabac and Listhaug, 2008). Moreover, the Dutch report experiencing substantial cultural differences with people from Moroccan descent (Van Osh and Breukelmans, 2012).
Analysis word embeddings
We thus explore bias in Dutch newspapers by retrieving the hundred most proximate words to Belgian(s) and the hundred most proximate words to Moroccan(s). A Python script was written to retrieve the hundred nearest neighbors to the selected close and distant outgroups from the embedding models. These words are considered most indicative for coverage about these groups, and therefore powerful in uncovering potential bias (see Agrawal and Awekar, 2018). An example of a neutral attribute close to an ethnic group would be a different nationality; when the word Belgian appears close to the word Italian, this indicates that both words are interchangeably in sentences such as: “The [Belgian / Italian] governments are meeting…” An example of a stereotypic attribute close to an ethnic group is criminal: When used interchangeable with references to Moroccan(s), it suggests bias: “The police have apprehended a [Moroccan / criminal]…”
The authors thoroughly analyzed and categorized these words for the embeddings trained on tabloid newspapers, quality newspapers, and all newspapers simultaneously. For each word, the authors indicated whether or not it was negatively valenced. All the negatively valenced words were compiled into a list of 63 unique words. The words reflect dimensions of criminality, illegality, scam, prostitution, and violence in general. Please consult Appendix B for the complete list.
We visualized the results of our word embedding analysis using a machinelearning algorithm for data visualization, namely t-distributed stochastic neighbor embeddings (t-SNE). This popular method for visualizing high-dimensional data allows data points to be plotted on a two-dimensional map to identify patterns (Van der Maaten and Hinton, 2008). t-SNE uses a Gaussian distribution to create a probability distribution to map relationships between data points in a high-dimensional space. Due to its ability to preserve local structures, t-SNE will tend to plot points near each other that are close in the high-dimensional space.
Analysis of co-occurrence
Although the interchangeability of words to indicate targets (e. g., Moroccans) and negative attributes (e. g., criminals) strongly hints towards implicit bias in news coverage, one may wonder whether negative attributes are also present in news articles that mention close and distant outgroup members. A straightforward manner to assess whether these groups appear in stereotypical news contexts is the analysis of co-occurrences between targets groups and words that indicate stereotypical contexts. Therefore, and in a second step, co-occurrences were calculated between references to the target groups[2] and the compiled list of 63 negatively valenced words based on our embeddings analysis (as included in Appendix B). By investigating whether the target groups are mentioned in news articles that deal with criminality, trafficking and violence, we can draw conclusions about the stereotypicality of the direct news context in which these groups appear. We visualize the results of the co-occurrence analysis using multidimensional scaling, a technique that visualizes the similarity between data points.
Results
Word embeddings. We discuss the results of the embeddings trained on the five largest Dutch newspapers. Both Belgian(s) (N = 45,931) and Moroccan(s) (N = 28,066) figured often in the news, warranting the quality of the embeddings specific to these groups (Schnabel, Labutov, and Mimno, 2015). Tables 1 and 2 display the findings. As can be seen, Belgians are mostly associated in the news with other European nationalities, such as the French, German, Italians, and Spaniards. In addition, references to Belgians were used analogous to sports-related terms in news stories; in particular cyclists, evidenced by proximate words such as cross cyclists (veldrijders), top favorites (topfavorieten), and teammate (teammaat). Notably, none of the hundred most proximate words carry a clear negative connotation.
Turning to our inspection of the hundred most proximate words to Moroccan(s), the data reveal a strikingly different picture. Moroccans often appear in the vicinity of other non-European ethnic minorities, such as Turks, Antilleans, and Surinamese. In addition, we find that around thirty percent of the hundred most proximate words are clearly negatively valenced: Dutch newspapers associate Moroccans with words such as brats (rotjochies), criminals (crimineeltjes), loitering (hangjongeren), crooks (boefjes), and rioters (relschoppers). Please consult Appendix A for a complete list of the hundred most proximate words for both Belgians and Moroccans.
Figure 1 summarizes these findings. As can be seen, Moroccans are often associated with negatively valenced.
The difference in representations of close and distant outgroups across newspaper types are discussed next. The embeddings trained on popular and quality newspapers, respectively, were used to retrieve the hundred most proximate words for both Belgian(s) and Moroccans. The percentages of negatively valenced words among the hundred most proximate words for both outgroups across newspaper types are presented in Table 2. As can be seen, Belgians are neither associated with negative words in popular nor quality newspapers. Moroccans, on the contrary, were used interchangeably with negative terms most often in popular newspapers (40 %) compared to quality newspapers (13 %). Popular newspapers describe Moroccans more often in relation to low status, criminality, and hostility, illustrated by words such as troublemakers, drug addicts, youth gangs, and illegality. These negative associations result in unwarranted and strongly negative representations of Moroccans in popular newspapers. The findings indicate that mainly popular newspapers are responsible for spreading negative associations regarding this distant outgroup category.
Two-dimensional vector representation of the hundred most proximate words to Moroccans.
Note. Empty data points represent neutral words, filled data points represent negatively valenced words. For reasons of readability, only negatively-valenced words are labeled.
Co-occurrences. In a next step, and in order to investigate the stereotypicality of the direct news context in which outgroups appear, co-occurrences were calculated between the target groups and the stereotypical attributes identified in our analysis using word embeddings. The results of the co-occurrence analysis are presented in Table 3. In line with the results of the analysis using word embeddings, we find that Moroccans appear more often in stereotypical news contexts than Belgians. More specifically, the results show that of the 3362 news articles that mention Moroccan(s), 19.2 % also mention at least one of the stereotypical attributes. In contrast, only 5 % of the 11081 news articles that mention Belgians contain stereotypical attributes. Figure 2 displays the results of the multidimensional scaling representation of the co-occurrence matrices. As can be seen, references to Moroccans appear closer to negative attributes than references to Belgians.
Top 20 most proximate words to Belgian(s) and Moroccan(s).
Input | Word | Cosine similarity | Input | Word | Cosine similarity |
Belgian + Belgians | French | 0.75 | Moroccan + Moroccans | Turk | 0.81 |
German | 0.74 | Antilleans | 0.77 | ||
Italians | 0.74 | Turks | 0.73 | ||
Spaniards | 0.74 | Surinamese | 0.72 | ||
Norwegians | 0.73 | Surinamer | 0.72 | ||
Swede | 0.71 | Muslim | 0.69 | ||
Frenchman | 0.71 | Brats | 0.68 | ||
Czechs | 0.70 | Jihadist | 0.67 | ||
Austrian | 0.70 | Criminals | 0.66 | ||
Germans | 0.69 | Algerian | 0.66 | ||
Flames | 0.69 | Moroccan | 0.65 | ||
Danes | 0.68 | Loitering | 0.65 | ||
Swiss | 0.68 | Foreigners | 0.65 | ||
Veteran | 0.68 | Immigrant | 0.65 | ||
Luxembourger | 0.68 | Muslims | 0.65 | ||
Greek | 0.68 | Antillean | 0.65 | ||
Dutchman | 0.67 | Boys | 0.65 | ||
Czech | 0.67 | Crooks | 0.64 | ||
Portuguese | 0.67 | Moluccas | 0.64 | ||
Aussies | 0.67 | Rioters | 0.64 |
Note. Top 20 most proximate words to Belgian(s) and Moroccan(s) according to embeddings trained on de Volkskrant, NRC Handelsblad, Trouw, Algemeen Dagblad, and De Telegraaf, for the period 2000–2015. Words in bold carry a negative connotation.
Percentage of negative associations with close and distant outgroups across newspaper types.
All newspapers | Tabloid newspapers | Quality newspapers | |
Belgian(s) | 0 % | 0 % | 0 % |
Moroccan(s) | 33 % | 40 % | 13 % |
Results of the co-occurrence analysis.
N news articles about target groups | N mentions of the target group | N articles that mention both target group and stereotypical words | N co-occurrences between target group and stereotypical attributes within news articles | % of news articles in which target group appears in stereotypical context | |
Moroccan(s) | 3362 | 5530 | 645 | 899 | 19.2 % |
Belgian(s) | 11081 | 15960 | 558 | 724 | 5.0 % |
Multidimensional scaling representation of the co-occurrences between Moroccan(s), Belgian(s), and negative attributes.
Note. Unlabed datapoints refer to negative attributes (full list is included in Appendix B).
Discussion
This study has investigated representations of culturally close and distant ethnic outgroups in Dutch newspapers. It has done so by drawing on more than three million Dutch newspaper stories and by using word embeddings, an advanced algorithm for capturing, understanding and analyzing aspects of word meaning.
Based on the premises of frameworks of social and group identities (Taijfel and Turner, 1979), it was hypothesized that culturally distant outgroups are more negatively represented by newspapers than culturally close outgroups. The results confirm this expectation: The data show that Dutch newspapers associate Belgians, here considered a close ethnic outgroup, with neutral or sport-related terms. Particularly, Belgians were frequently associated with words such as cyclists, or team mate. On the contrary, the results reveal that Moroccans, considered a distant ethnic outgroup, are depicted in relation to negative issues and characteristics: A considerable number of words frequently used in close proximity to this group carry a clear negative connotation. More specifically, references to this group are replaceable with terms such as criminals, crooks, jihadists, and rioters. Especially popular newspapers prominently use references to Moroccans interchangeably with negative and unwarranted attributes. This finding confirms previous evidence for the stereotypical nature of news stories about ethnic outgroups in these types of newspapers (Arendt, 2010; Kroon et al., 2016; Van Dijk, 2000). Co-occurrence analyses confirm that Moroccans, as compared to Belgians, appear in the news in close proximity to stereotypic attributes.
It is important to note that such news representations are not inconsequential. The linkage between targets (i. e., social groups) and attributes (i. e., issues, characteristics) in media messages, also referred to as mediated associations (Arendt and Karadas, 2017) can cause audience members to see these concepts as related. Van Atteveldt (2008) reasons that “[e]ven if the two are not related or are even explicitly dissociated, this tells us something about the worldview of the source of the messages containing both concepts, and it can cause the receiver of those messages to relate the two concepts” (p. 65). Experimental evidence in the media stereotyping domain supports this argument: Studies consistently find that exposure to stereotypical mediated associations establishes and re-activates cognitive linkages between target groups and attributes, herewith increasing the availability and accessibility of stereotype-congruent associations in the memory (Arendt, 2013; Cho, Gil de Zuniga, Shah, and McLeod, 2006; Verhaeghen, Aikman, and Gulick, 2011).
This study has demonstrated the usefulness of word embeddings in studying the representations of minorities in Dutch news coverage. Following in the footsteps of innovative studies in the field of AI (e. g., Bolukbasi et al., 2016a; Caliskan et al., 2017; Garg et al., 2018), this study has explored the benefits of word embeddings to detect subtle bias in large bodies of news media data. Yet, some limitations should be acknowledged. First, as word embeddings are the outcome of a rather advanced and complex training process, its results might not always be intuitively and straightforwardly interpreted. Second, as we only considered a single close and distant outgroup, the here-reported findings cannot simply be transferred to other nationalities, countries, or newspapers. Future studies should test differences in media representations between culturally similar and distinct groups among a large sample of ethnicities. In addition, due to the static nature of our analysis, it remains unclear to what extent bias in news coverage has become more or less pronounced with time.
The current study has demonstrated that Moroccans, being a culturally distant ethnic minority group in the Netherlands, are unwarrantedly associated in news stories with problems, criminality, and hostility. Changing the media representation of this and other culturally distant minority groups could decrease harmful stereotypes about these groups in society.
References
Agrawal, S., & Awekar, A. (2018). Deep learning for detecting cyberbullying across multiple social media platforms. Lecture notes in Computer Science (Including subseries lecture notes in Artificial Intelligence and lecture notes in Bioinformatics), 10772 LNCS (Table 2), 141–153. doi:10.1007/978-3-319-76941-7_1110.1007/978-3-319-76941-7_11Search in Google Scholar
Ahmed, S., & Matthes, J. (2017). Media representation of Muslims and Islam from 2000 to 2015: A meta-analysis. International Communication Gazette, 79(3), 219–244. doi:10.1177/174804851665630510.1177/1748048516656305Search in Google Scholar
Allport, G. W., Clark, K., & Pettigrew, T. (1954). The nature of prejudice. Cambridge, MA: Addison-Wesley Pub. Co.Search in Google Scholar
Arendt, F. (2010). Cultivation effects of a newspaper on reality estimates and explicit and implicit attitudes. Journal of Media Psychology, 22(4), 147–159. doi:10.1027/1864-1105/a00002010.1027/1864-1105/a000020Search in Google Scholar
Arendt, F. (2013). Dose-dependent media priming effects of stereotypic newspaper articles on implicit and explicit stereotypes. Journal of Communication, 63, 830–851. doi:10.1111/jcom.1205610.1111/jcom.12056Search in Google Scholar
Arendt, F., & Karadas, N. (2017). Content analysis of mediated associations: An automated text-analytic approach. Communication Methods and Measures, 11(2), 105–120. doi:10.1080/19312458.2016.127689410.1080/19312458.2016.1276894Search in Google Scholar
Bleich, E., Stonebraker, H., Nisar, H., & Abdelhamid, R. (2015). Media portrayals of minorities: Muslims in British newspaper headlines, 2001–2012. Journal of Ethnic and Migration Studies, 41(6), 942–962. doi:10.1080/1369183X.2014.100220010.1080/1369183X.2014.1002200Search in Google Scholar
Blinder, S., & Jeannet, A. M. (2018). The “illegal” and the skilled: Effects of media portrayals on perceptions of immigrants in Britain. Journal of Ethnic and Migration Studies, 44(9), 1444–1462. doi:10.1080/1369183X.2017.141225310.1080/1369183X.2017.1412253Search in Google Scholar
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016a). Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. NIPS, 1–9. Retrieved March 20, 2018 from http://arxiv.org/abs/1607.06520.Search in Google Scholar
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016b). Quantifying and reducing stereotypes in word embeddings. doi:10.1016/S0017-9310(97)00074-410.1016/S0017-9310(97)00074-4Search in Google Scholar
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora necessarily contain human biases. Science, 6334(356), 183–186.10.1126/science.aal4230Search in Google Scholar
Cho, J., Gil de Zuniga, H., Shah, D. V., & McLeod, D. M. (2006). Cue convergence: Associative effects on social intolerance. Communication Research, 33(3), 136–154.10.1177/0093650206287075Search in Google Scholar
Eberl, J., Meltzer, C. E., Heidenreich, T., Theorin, N., Lind, F., Berganza, R., …, Schemer, C. (2018). The European media discourse on immigration and its effects: A literature review. Annals of the International Communication Association, 842(3), 207–223. doi:10.1080/23808985.2018.149745210.1080/23808985.2018.1497452Search in Google Scholar
Erisen, C., & Kentmen-Cin, C. (2017). Tolerance and perceived threat toward Muslim immigrants in Germany and the Netherlands. European Union Politics, 18(1), 73–97. doi:10.1177/146511651667597910.1177/1465116516675979Search in Google Scholar
Firth, J. R. (1957). Papers in linguistics, 1934–1951. Oxford University Press.Search in Google Scholar
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. PNAS, 1–33. Retrieved April 20, 2019 from http://arxiv.org/abs/1711.08412.10.1073/pnas.1720347115Search in Google Scholar
Gaertner, S. D., & Dovidio, J. F. (2005). Understanding and addressing contemporary racism: From aversive racism to the common Ingroup Identity Model. Journal of Social Issues, (3), 615–639.10.1111/j.1540-4560.2005.00424.xSearch in Google Scholar
González, K. V., Verkuyten, M., Weesie, J., & Poppe, E. (2008). Prejudice towards Muslims in the Netherlands: Testing integrated threat theory. British Journal of Social Psychology, 47(4), 667–685. doi:10.1348/014466608X28444310.1348/014466608X284443Search in Google Scholar
Gorodzeisky, A., & Semyonov, M. (2016). Not only competitive threat but also racial prejudice: Sources of anti-immigrant attitudes in European societies. International Journal of Public Opinion Research, 28(3), 331–354. doi:10.1093/ijpor/edv02410.1093/ijpor/edv024Search in Google Scholar
Greenwald, A. G. (1995). Implicit social cognition: Attitudes, self-esteem, and stereotypes. Psychological Review, 102(1), 4–27.10.1037/0033-295X.102.1.4Search in Google Scholar
Inglehart, R., & Klingemann, H. D. (2000). Genes, culture, democracy, and happiness. In E. Diener & E. M. Suh (Eds.), Culture and subjective well-being (pp. 165–183). Cambridge, MA: The MIT Press.Search in Google Scholar
Jacobs, L., Damstra, A., Boukes, M., Swert, K. De, & Boukes, M. (2018). Back to reality: The complex relationship between patterns in immigration news coverage and real-world developments in Dutch and Flemish newspapers (1999 – 2015). Mass Communication and Society, 21(4), 473–497. doi:10.1080/15205436.2018.144247910.1080/15205436.2018.1442479Search in Google Scholar
Kroon, A. C., Kluknavská, A., Vliegenthart, R., & Boomgaarden, H. G. (2016). Victims or perpetrators? Explaining media framing of Roma across Europe. European Journal of Communication, 31(4). doi:10.1177/026732311664723510.1177/0267323116647235Search in Google Scholar
Mahfud, Y., Badea, C., Verkuyten, M., & Reynolds, K. (2018). Multiculturalism and attitudes toward immigrants: The impact of perceived cultural distance. Journal of Cross-Cultural Psychology, 49(6), 945–958. doi:10.1177/002202211773082810.1177/0022022117730828Search in Google Scholar
Mastro, D. (2009). Effects of racial and ethnic stereotyping. In J. Bryant & M. B. Oliver (Eds.), Media effects: Advances in theory and research (pp. 325–341). New York, NY: Routledge.Search in Google Scholar
Mastro, D. E., Behm-Morawitz, E., & Kopacz, M. A. (2008). Exposure to television portrayals of Latinos: The implications of aversive racism and social identity theory. Human Communication Research, 34(1), 1–27. doi:10.1111/j.1468-2958.2007.00311.x10.1111/j.1468-2958.2007.00311.xSearch in Google Scholar
Matamoros-Fernández, A. (2017). Platformed racism: the mediation and circulation of an Australian race-based controversy on Twitter, Facebook and YouTube. Information Communication and Society, 20(6), 930–946. doi:10.1080/1369118X.2017.129313010.1080/1369118X.2017.1293130Search in Google Scholar
Mikolov, T., Corrado, G., Chen, K., & Dean, J. (2013). Efficient estimation of word representations in vector space. ArXiv, 1–12.Search in Google Scholar
Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic regularities in continuous space word representations. Proceedings of NAACL-HLT, 746–751.Search in Google Scholar
Mutz, D. C., & Goldman, S. K. (2016). Mass media. In J. F. Dovidio, M. Hewstone, P. Glick, & V. M. Esses (Eds.), The SAGE handbook of prejudice, stereotyping and discrimination (pp. 241–258). London, England: Sage Publications.10.4135/9781446200919.n15Search in Google Scholar
Polek, E., Wöhrle, J., & van Oudenhoven, J. P. (2010). The role of attachment styles, perceived discrimination, and cultural distance in adjustment of German and Eastern European immigrants in the Netherlands. Cross-Cultural Research, 44(1), 60–88. doi:10.1177/106939710935277910.1177/1069397109352779Search in Google Scholar
Ramasubramanian, S. (2007). Media-based strategies to reduce racial stereotypes activated by news stories. Journalism & Mass Communication Quarterly, 84(2), 249–264. doi:10.1177/10776990070840020410.1177/107769900708400204Search in Google Scholar
Richardson, J. E. (2004). (Mis) representing Islam: The racism and rhetoric of British broadsheet newspapers. Amsterdam: John Benjamins Publishing.10.1075/dapsac.9Search in Google Scholar
Rudkowsky, E., Haselmayer, M., Wastian, M., Jenny, M., Emrich, Š., Sedlmair, M., …, Sedlmair, M. (2018). More than bags of words: Sentiment analysis with word embeddings. Communication Methods and Measures, 12(2–3), 140–157. doi:10.1080/19312458.2018.145581710.1080/19312458.2018.1455817Search in Google Scholar
Ruigrok, N., & Atteveldt, W. Van. (2007). Global angling with a local angle: How U.S., British, and Dutch newspapers frame global and local terrorist attacks. Press/Politics, 12(1), 68–90. doi:10.1177/1081180X0629743610.1177/1081180X06297436Search in Google Scholar
Savelkoul, M., Scheepers, P., Tolsma, J., & Hagendoorn, L. (2011). Anti-Muslim attitudes in the Netherlands: Tests of contradictory hypotheses derived from ethnic competition theory and intergroup contact theory. European Sociological Review, 27(6), 741–758. doi:10.1093/esr/jcq03510.1093/esr/jcq035Search in Google Scholar
Schnabel, T., Labutov, I., & Mimno, D. (2015). Evaluation methods for unsupervised word embeddings. Proceedings of the 2015 conference on empirical methods in natural language processing, 298–307.10.18653/v1/D15-1036Search in Google Scholar
Strabac, Z., & Listhaug, O. (2008). Anti-Muslim prejudice in Europe: A multilevel analysis of survey data from 30 countries. Social Science Research, 37(1), 268–286. doi:10.1016/j.ssresearch.2007.02.00410.1016/j.ssresearch.2007.02.004Search in Google Scholar
Taijfel, H., & Turner, J. (1979). An integrative theory of intergroup conflict. In W. G. Austin & S. Worchel (Eds.), The social psychology of intergroup relations (pp. 33–47). Monterey, CA: Brooks-Cole.Search in Google Scholar
Tulkens, S., Hilte, L., Lodewyckx, E., Verhoeven, B., & Daelemans, W. (2016). A dictionary-based approach to racism detection in Dutch social media.Search in Google Scholar
Van Atteveldt, W. (2008). Semantic network analysis: Techniques for extracting, representing and querying media content. Charleston SC: BookSurge Publisher.Search in Google Scholar
Van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9(2210), 2579–2605. doi:10.1007/s10479-011-0841-310.1007/s10479-011-0841-3Search in Google Scholar
Van Dijk, T. A. (2000). New(s) racism: A discourse analytical approach. In S. Cottle (Ed.), Ethnic minorities and the media (pp. 33–49). Keynes, UK: Open University Press.Search in Google Scholar
Van Osh, Y. M. J., & Breukelmans, S. M. (2012). Perceived intergroup differences as an organizing priniciple of intercultural attitudes and acculturation attitudes. Journal of Cross-Cultural Psychology, 43(43), 801–821. doi:10.1177/002202211140768810.1177/0022022111407688Search in Google Scholar
Vasta, E. (2007). From ethnic minorities to ethnic majority policy: Multiculturalism and the shift to assimilationism in the Netherlands. Ethnic and Racial Studies, 30(5), 713–740. doi:10.1080/0141987070149177010.1080/01419870701491770Search in Google Scholar
Verhaeghen, P., Aikman, S. N., & van Gulick, A. E. (2011). Prime and prejudice: Co-occurrence in the culture as a source of automatic stereotype priming. British Journal of Social Psychology, 50, 501–518. doi:10.1348/014466610X52425410.1348/014466610X524254Search in Google Scholar
Weisbuch, M., Pauker, K., & Ambaday, N. (2009). The subtle transmission of race bias via televised nonverbal behavior. Science, 5960(326), 1711–1714. doi:10.1126/science.117835810.1126/science.1178358Search in Google Scholar
Ye, B. H., Zhang, H. Q., Huawen Shen, J., & Goh, C. (2014). Does social identity affect residents’ attitude toward tourism development?: An evidence from the relaxation of the individual visit scheme. International Journal of Contemporary Hospitality Management, 26(6), 907–929. doi:10.1108/IJCHM-01-2013-004110.1108/IJCHM-01-2013-0041Search in Google Scholar
Appendix A
Top 100 most proximate words to Belgian(s) and Moroccan(s) in Dutch newspapers
Top 100 most proximate words to Belgian(s) and Moroccan(s).
Belgian + Belgians | Moroccan + Moroccans | ||||
Dutch | English translation | Cosine similarity | Dutch | English translation | Cosine similarity |
Fransen | French | 0.746 | Turk | Turk | 0.810 |
Duitser | German | 0.740 | Antillianen | Antilleans | 0.770 |
Italianen | Italians | 0.738 | Turken | Turkish | 0.725 |
Spanjaarden | Spaniards | 0.738 | Surinamers | Surinamese | 0.722 |
Noren | Norwegians | 0.731 | Surinamer | Surinamer | 0.717 |
Zweed | Swede | 0.710 | Moslim | Muslim | 0.695 |
Fransman | Frenchman | 0.709 | Rotjochies | Brats | 0.678 |
Tsjechen | Czechs | 0.700 | Jihadist | Jihadist | 0.669 |
Oostenrijkers | Austrian | 0.696 | Crimineeltjes | Criminals | 0.663 |
Duitsers | Germans | 0.692 | Algerijn | Algerian | 0.662 |
Vlamingen | Flames | 0.685 | Marokkaanse | Moroccan | 0.654 |
Denen | Danes | 0.684 | Hangjongeren | Loitering | 0.654 |
Zwitser | Swiss | 0.681 | Buitenlanders | Foreigners | 0.650 |
Routinier | Veteran | 0.678 | Allochtoon | Immigrant | 0.650 |
Luxemburger | Luxembourger | 0.677 | Moslims | Muslims | 0.647 |
Griek | Greek | 0.675 | Antilliaan | Antillean | 0.646 |
Nederlander | Dutchman | 0.674 | Jongens | Boys | 0.645 |
Tsjech | Czech | 0.673 | Boefjes | Crooks | 0.644 |
Portugezen | Portuguese | 0.669 | Molukkers | Moluccas | 0.644 |
Aussies | Aussies | 0.668 | Reljongeren | Rioters | 0.641 |
Vlaming | Fleming | 0.666 | Buurtvaders | Neighborhood fathers | 0.641 |
Oostenrijker | Austrian | 0.662 | Tasjesdief | Bag thief | 0.640 |
Slowaak | Slovak | 0.661 | Arabier | Arab | 0.639 |
Schlecks | Schlecks | 0.658 | Hindoestanen | Hindus | 0.638 |
Raborenner | Rabo rider | 0.656 | Imam | Imam | 0.637 |
Engelsen | English | 0.653 | Roemenen | Romanians | 0.632 |
Limburgers | Residents of Limburg | 0.649 | Marokkaans | Moroccan | 0.631 |
Haantjes | Machos | 0.649 | Jood | Jew | 0.631 |
Achterhoekers | Residents of the Achterhoek | 0.648 | Straatterroristen | Street terrorists | 0.630 |
Deen | Dane | 0.644 | Allochtonen | Allochtonen | 0.625 |
Finnen | Fins | 0.643 | Egyptenaar | Egyptian | 0.621 |
Catalanen | Catalans | 0.642 | Zigeuners | Gypsies | 0.620 |
Roemenen | Romanians | 0.642 | Jihadgangers | Jihad visitors | 0.618 |
Noor | Norwegian | 0.639 | Afghaan | Afghan | 0.618 |
Renners | Cyclists | 0.637 | Pedofiel | Pedophile | 0.617 |
Ajacieden | Ajacids | 0.633 | Marokkaantjes | Moroccans | 0.617 |
Sprinters | Sprinters | 0.633 | Koerd | Kurd | 0.615 |
Italiaan | Italian | 0.632 | Arabieren | Arabs | 0.614 |
Friezen | Friezes | 0.632 | Homohaters | Gay haters | 0.613 |
Hoste | Hoste | 0.631 | Joegoslaven | Yugoslavs | 0.612 |
Roemeen | Romanian | 0.631 | Ghanezen | Ghanaians | 0.612 |
Noorderlingen | Northerners | 0.630 | Moslimjongeren | Muslim youth | 0.609 |
Brabander | Resident of Brabant | 0.626 | Lastpakken | Troublemakers | 0.608 |
Sloveen | Slovene | 0.626 | Berbers | Berbers | 0.608 |
Hongaar | Hungarian | 0.625 | Islamieten | Islamists | 0.607 |
Routiniers | Veterans | 0.624 | Jongen | Youth | 0.607 |
Stybar | Stybar | 0.624 | Autochtonen | Natives | 0.605 |
Luxemburgers | Luxembourgers | 0.623 | Voetbalsupporters | Football supporters | 0.604 |
Brazilianen | Brazilians | 0.620 | Imams | Imams | 0.604 |
Raborenners | Cyclist with team Rabobank | 0.619 | Hooligans | Hooligans | 0.603 |
Topsprinters | Top sprinters | 0.617 | Skinheads | Skinheads | 0.600 |
Spanjaard | Spaniard | 0.617 | Asielzoeker | Asylum seeker | 0.600 |
Scandinaviers | Scandinavians | 0.616 | Raddraaiers | Hoodlums | 0.598 |
Cancellara | Cancellara | 0.615 | Meisjes | Girls | 0.595 |
Debutant | Debutant | 0.615 | Extremist | Extremist | 0.594 |
Slovenen | Slovenes | 0.614 | Medelanders | Fellow citizens | 0.594 |
Kopmannen | Leaders | 0.614 | Buitenlander | Outlander | 0.593 |
Argentijnen | Argentinians | 0.613 | Immigranten | Immigrants | 0.593 |
Kazak | Kazak | 0.612 | Probleemjongeren | Problem youth | 0.593 |
Brabanders | Residents of Brabant | 0.611 | Autochtone | Autochthonous | 0.592 |
Zabel | Zabel | 0.610 | Joden | Jews | 0.590 |
Profs | Pros | 0.610 | Rijksgenoten | Nationals | 0.590 |
Limburger | Limburger | 0.609 | Iranier | Iranian | 0.589 |
IJslander | Icelander | 0.607 | Rotjongens | Creeps | 0.589 |
Renner | Cyclist | 0.607 | Hangjeugd | Loiterers | 0.588 |
Veldrijder | Cyclocrosser | 0.607 | Afkomst | Descent | 0.588 |
Topfavorieten | Top favorites | 0.607 | Relschoppers | Rioters | 0.588 |
Ploegmaat | Team mate | 0.605 | Irakees | Iraqi | 0.587 |
Kopman | Leader | 0.604 | Straatterreur | Street terror | 0.585 |
Kazach | Kazakh | 0.604 | Jongeren | Youth | 0.584 |
Zwitsers | Swiss | 0.603 | Jongetjes | Little boys | 0.582 |
Bask | Basque | 0.602 | Djihadist | Jihadi | 0.581 |
Tijdritspecialist | Time trial specialist | 0.602 | Crimineel | Criminal | 0.581 |
Feyenoorders | Feyenoord | 0.601 | Veelpleger | Frequent offender | 0.580 |
Invallers | Substitutes | 0.601 | Bulgaren | Bulgaria | 0.578 |
Ieren | Irish | 0.600 | Algerijnen | Algerians | 0.578 |
Klassementsrenners | GC riders | 0.600 | Pooier | Pimp | 0.576 |
Jeugdinternationals | Youth internationals | 0.600 | Molukker | Moluccan | 0.575 |
Walen | Walloons | 0.600 | Bekeerling | Convert | 0.575 |
Impe | Impe | 0.599 | Somaliers | Somalians | 0.574 |
Toprenners | Top riders | 0.598 | Somalier | Somalian | 0.573 |
Veldrijders | Cross cyclists | 0.598 | Nederlanders | Dutch | 0.572 |
Amsterdammers | People from Amsterdam | 0.597 | Voetbalsupporter | Football supporter | 0.572 |
Slowaken | Slovaks | 0.596 | Terreurverdachte | Terror suspect | 0.572 |
Ritwinnaar | Stage winner | 0.596 | Jihadisten | Jihadists | 0.571 |
Pool | Pool | 0.596 | Hooligan | Hooligan | 0.569 |
Vedetten | Vedets | 0.595 | Vrouwenhandelaar | Trafficker | 0.569 |
Tukkers | Tukkers | 0.595 | Migrant | Migrants | 0.567 |
Verheyen | Verheyen | 0.595 | Criminelen | Criminals | 0.567 |
Jonkies | Young ones | 0.595 | Immigrant | Immigrant | 0.566 |
Tilburgers | People from Tilburg | 0.594 | Gastarbeider | Guest worker | 0.566 |
Topschutter | Top scorer | 0.594 | Straatschoffies | Urchin | 0.566 |
Kittel | Kittel | 0.593 | Jochies | Boys | 0.566 |
Australier | Australian | 0.593 | Griek | Greek | 0.566 |
Spurter | Sprinter | 0.592 | Kickbokser | Kickboxer | 0.565 |
Hongaren | Hungary | 0.592 | Djihadisten | Djihadists | 0.565 |
Boonen | Boonen | 0.591 | Metin | Metin | 0.565 |
Devolder | Devolder | 0.591 | Randgroepjongeren | Marginal youth | 0.563 |
Voigt | Voigt | 0.591 | Onruststokers | Nuisance | 0.562 |
Jalabert | Jalabert | 0.591 | Jeugdbendes | Youth gangs | 0.562 |
Appendix B
List of negative words derived from the word embeddings
terreurverdachte, jihadgangers, racisten, oorlogsmisdadiger, crimineeltjes, homohaters, extremist, prostituee, hangjongere, randgroepjongeren, criminelen, delinquenten, politieman, misdadiger, misdadigers, bedelaars, pedofielen, reljongeren, moordenaars, straatterroristen, terrorist, relschoppers, probleemjongeren, hangjongeren, extremisten, rotjochies, djihadisten, boefjes, jeugdbendes, geweldplegers, jihadstrijders, jihadisten, drugsdealer, tasjesdief, politieagente, terreurcel, crimineel, oplichter, pooier, hooligan, jihadist, onruststokers, pedofiel, hangjeugd, gedetineerde, zwervers, djihadist, strijder, straatschoffies, drugshandelaren, oorlogsmisdadigers, rotjongens, prostituees, straatterreur, politieagent, raddraaiers, maffiosi, bende, hooligans, lastpakken, vrouwenhandelaar, gangsters, veelpleger
© 2019 Kroon et al, published by De Gruyter
This work is licensed under the Creative Commons Attribution 4.0 International License.
