Accessible Requires Authentication Published by De Gruyter August 24, 2020

Who are Our Experts? Predictors of Participation in Expert Surveys

Christoph Valentin Steinert and Andrea Ruggeri

Abstract

Who are the colleagues participating when asked to complete expert surveys? This research note investigates which individuals’ characteristics associate with positive responses. Drawing on an expert survey dedicated to post-conflict trials, we collect data on various attributes of both respondents and non-respondents such as their age, sex, academic positions, disciplines, and research outputs. We expect that decisions to participate result from an interplay of (1) individuals’ levels of context-specific expertise, (2) the value attached to their expert role, (3) their confidence in making authoritative statements, and (4) resource constraints. Employing logistic regression models and statistical simulations (n = 414), we find that context-specific expertise is the primary, but not the only determinant of participation. On the one hand and luckily, individuals whose research corresponds closely to the object of study are most likely to participate. On the other hand and unfortunately, individuals with high citation outputs, female experts, and Area Studies-scholars are less likely to respond. Consequently, certain groups are under-represented in expert evaluations frequently considered as authoritative source of knowledge.


Corresponding author: Christoph Valentin Steinert, Department of Political Science IV, University of Mannheim, Parkring 47, 1. OG, 68159Mannheim, Germany, E-mail:

Appendices A. Overview of Predictors

Personal characteristics:

Female: Binary indicator of the sex of the contacted expert.

Year of birth: Continuous indicator of the year of birth of the contacted expert. If available, we used information in CVs and on homepages of scholars. Otherwise, we contacted them asking for this information.

Stages of academic careers:

Year of PhD: Continuous indicator capturing the year when an expert completed her/his PhD.

Professor: Binary measure capturing whether the contacted expert holds a full professorship.

Post-PhD: Binary variable indicating whether an expert holds a position as post-Doc, Assistant/ Associate/ Junior Professor. Coded as 0, if an expert holds a full professorship.

Emeritus: Binary indicator recording whether an expert is a retired professor.

Academic output:

Number of publications: Ordinal measure of the number of publications. If an expert has 1–5 publications, coded as 0. If s/he has 6−15 publications, coded as 1. If s/he has more than 15 publications, coded as 2.

Number of citations: Continuous variable indicating the number of citations in academic journals. We used Google Scholar profiles to collect these data. If no Google Scholar profiles were available, we added up the number of citations of authors’ identifiable publications ourselves using references in Google Scholar.

Location of research:

Academic: Binary measure indicating whether an expert works currently at a university. If s/he conducts research at a non-academic institution or s/he has a non-research job, coded as 0.

Western institution: Binary measure indicating whether an expert works at a US or European university or institution.

US institution: Binary measure capturing whether an expert works at a US institution.

Ivy League class university: Binary variable capturing whether an expert is employed at a ’self-declared’ top university. The following universities are deemed as Ivy League class universities: US Ivy League, Oxford, Cambridge, Science Po.

Academic discipline/ Research approach:

Anthropology: Binary indicator capturing whether an expert is primarily trained as anthropologist.

Area Studies: Binary measure signifying whether an expert is primarily trained as a specialist for area studies.

Economics: Binary variable capturing whether an expert is primarily trained as economist.

History: Binary measure indicating whether an expert is primarily trained as historian.

Law: Binary indicator denoting whether an expert is primarily trained as a lawyer.

Political Science: Binary variable indicating whether an expert is primarily trained in the field of political science.

Quantitative scholar: Binary measure capturing whether an expert works primarily with quantitative methods.

Specific expertise for object of research:

Match of publication: Ordinal variable capturing whether the selection publication pertains directly to the post-conflict trial. We hand-coded this variable reading Abstracts and screening full texts of scholars’ publications. Coded as 2, if the selection publication contains the respective post-conflict trial already in its title or abstract. Coded as 1, if the does not address the trial in its title or abstract but pertains directly to the political situation in the country during trial implementation. If the selection publication is only loosely connected to the post-conflict trial, coded as 0.

B. Tables

Table 3:

The post-conflict trials expert survey.

# of itemItemExtreme poles of scale (continuous in integers)
1Were all perpetrators of violence treated in an equal way or were some groups systematically discriminated?0 = unequal treatment 10 = equal treatment
2Were there indications that the government justified repression with reference to the justice process?0 = occurred frequently 10 = never occurred
3Were there incidences of violence related to the justice process such as targeting of judges and witnesses or retribution violence directed at perpetrators?0 = widespread violence 10 = absence of violence
4Did the scope of the process mandate concern only human rights violations perpetrated by certain groups or was violence from all sides (including the current government) considered?0 = extremely narrow focus 10 = complete inclusiveness
5Was the justice process restricted to a singular event or period of time or did it also concern potential backlash violence after the conflict/ genocide under investigation?0 = timewise restricted 10 = timewise unrestricted
6Did the narrative created by the justice process serve the purpose to consolidate the government?0 = distorted narrative 10 = objective narrative
7On a continuum from 0 to 10, whereby 10 indicates post-conflict fairness and 0 indicates post-conflict injustice: How would you evaluate the respective justice process overall?0 = post-conflict injustice 10 = post-conflict justice
Table 4:

Descriptive statistics.

VariablesnMeansdMinMaxCompletedExcused
Female4140.2730.4460115%8%
Professor4140.4860.5000119%12%
Post-PhD4140.3960.4900124%5%
Western4140.8450.3620119%11%
US4140.5390.4990122%10%
Ivy league class4140.0820.2750112%18%
Academic4140.8770.3290120%9%
Quantitative4140.1230.3290114%16%
Emeritus4140.1140.3180123%15%
Political science4140.4570.4990126%10%
Anthropology4140.0850.2790131%3%
Area studies4140.0700.256013%24%
Economics4140.0290.1680117%25%
History4140.2130.4100117%10%
Law4140.0480.2150110%5%
Sociology4140.0510.2200114%0%
# Of publications4141.7390.5300220% (at 2)10% (at 2)
Match of publication4140.5290.6980225% (at 2)0% (at 2)
Year of publication41420068.37619722017
Log # of citations4045.8921.51509.691
Year of birth222195612.9119261991
Year of PhD220199214.2319522017
Table 5:

Summary results statistical simulations.

Low bound (95% CI)Pr. meanUp bound (95% CI)
Model 1: Completed
 High match (2)0.12140.49070.8720
 Med. Match (1)0.04820.27880.6788
 Low match (0)0.01640.13010.4194
 Log citations (p25)0.01900.14840.4675
 Log citations (p75)0.01370.11130.3690
 Female0.00670.07430.2753
 Male0.01640.13010.4194
Model 2: Excused
 High match (2)0.00140.02280.1082
 Med. Match (1)0.00300.03860.1712
 Low match (0)0.00540.06760.2836
 Area studies0.12860.38880.7113
 Other discipline0.00540.06760.2836
Table 6:

Alternative model specifications (regressed on “Completed”).

(1)(2)(3)(4)(5)(6)
Female−0.724* (−2.02)−0.759* (−2.15)−0.712* (−1.96)−0.712* (−1.97)−0.717* (−1.98)−0.724* (−2.00)
Western−0.480 (−1.18)−0.662 (−1.71)−0.179 (−0.49)−0.179 (−0.49)−0.192 (−0.53)−0.190 (−0.52)
Quantitative scholar−0.824 (−1.62)−0.934 (−1.87)−0.716 (−1.44)−0.716 (−1.43)−0.713 (−1.44)−0.700 (−1.41)
Match of publication1.080*** (5.05)1.151*** (5.51)1.079*** (5.04)1.079*** (5.04)1.083*** (5.03)1.095*** (5.09)
Year of publication0.0291 (1.48)0.0367 (1.81)0.0286 (1.47)0.0286 (1.47)0.0286 (1.46)0.0255 (1.35)
Political science0.664 (1.18)0.795 (1.40)0.586 (1.04)0.586 (1.04)0.626 (1.11)0.602 (1.08)
Anthropology1.263 (1.92)1.485* (2.29)1.172 (1.80)1.172 (1.81)1.213 (1.86)1.241 (1.90)
Area studies−1.229 (−0.99)−1.089 (−0.88)−1.505 (−1.23)−1.506 (−1.24)−1.489 (−1.25)−1.456 (−1.22)
Economics1.245 (1.29)1.292 (1.33)1.113 (1.14)1.112 (1.15)1.160 (1.20)1.100 (1.14)
History0.454 (0.77)0.515 (0.87)0.412 (0.70)0.412 (0.70)0.455 (0.77)0.495 (0.84)
Law−1.085 (−1.10)−1.095 (−1.12)−1.094 (−1.12)−1.094 (−1.12)−1.047 (−1.07)−1.125 (−1.17)
Emeritus0.488 (1.22)0.391 (0.95)0.480 (1.18)0.479 (1.19)0.464 (1.14)
Ivy league class−0.495 (−0.80)−0.570 (−0.87)−0.404 (−0.65)−0.402 (−0.65)
Academic−0.0246 (−0.05)0.00570 (0.01)0.0102 (0.02)
US0.490 (1.51)0.566 (1.77)
Post-PhD0.119 (0.40)
Log citations−0.174 (−1.90)−0.212* (−2.21)−0.212* (−2.22)−0.217* (−2.27)−0.209* (−2.18)
Professor−0.0867 (−0.27)0.184 (0.55)0.186 (0.57)0.205 (0.63)0.250 (0.78)
Num. of publications−0.116 (−0.39)
Observations404414404404404404

  1. t statistics in parentheses.

Table 7:

Rare events logistic regression models.

(1) Completed(2) Excused
Female−0.676* (−1.97)0.178 (0.46)
Professor0.124 (0.39)0.697 (1.46)
Academic−0.0638 (−0.13)−1.032 (−1.90)
Log citations−0.188* (−2.06)0.112 (0.79)
Western−0.448 (−1.14)0.700 (1.08)
US0.429 (1.39)0.0274 (0.07)
Quantitative scholar−0.690 (−1.45)0.327 (0.57)
Match of publication1.022*** (4.95)−0.553* (−2.04)
Emeritus0.441 (1.13)0.279 (0.55)
Ivy league class−0.344 (−0.58)0.646 (1.20)
Year of publication0.0279 (1.50)−0.0004 (−0.02)
Political science0.377 (0.49)1.210 (1.22)
Anthropology0.955 (1.16)0.200 (0.14)
Area studies−1.049 (−0.82)2.066 (1.92)
Economics1.005 (0.94)1.710 (1.33)
History0.153 (0.20)1.008 (0.97)
Law−1.117 (−1.03)1.158 (0.84)
Sociology−0.219 (−0.23)
Constant−56.70 (−1.51)–3.086 (−0.07)
Observations404404

  1. t statistics in parentheses.

  2. *p < 0.05, **p < 0.01, ***p < 0.001.

Table 8:

Multinomial logistic regression.

(1) Excused(2) Completed
Female0.0525 (0.13)−0.729* (−2.01)
Professor0.763 (1.52)0.193 (0.57)
Academic−1.102 (−1.90)−0.151 (−0.28)
Citations_log0.0992 (0.66)−0.189 (−1.94)
Western0.789 (1.16)−0.407 (−0.99)
US0.127 (0.32)0.479 (1.46)
Quantitative scholar0.143 (0.24)−0.781 (−1.55)
Match of publication−0.385 (−1.26)1.039*** (4.74)
Emeritus0.341 (0.63)0.499 (1.22)
Elite university0.595 (1.06)−0.386 (−0.61)
Publication year0.00635 (0.28)0.0319 (1.61)
Political science1.837 (1.78)0.800 (1.44)
Anthropology0.425 (0.29)1.279* (1.98)
Area studies2.504* (2.23)−1.022 (−0.82)
Economics2.446 (1.82)1.490 (1.55)
History1.551 (1.42)0.525 (0.90)
Law1.037 (0.71)−1.013 (−1.04)
Constant−17.20 (−0.38)−65.07 (−1.63)
Observations404404

  1. t statistics in parentheses.

  2. *p < 0.05, **p < 0.01, ***p < 0.001.

C. Figure

Figure 3: Separation plot (based on Model 1).

Figure 3:

Separation plot (based on Model 1).

References

Arvanitidis, P. A., G. Petrakos, and S. Pavleas. 2010. “On the Dynamics of Growth Performance: An Expert Survey.” Contributions to Political Economy 29(1): 59–86, https://doi.org/10.1093/cpe/bzq001. Search in Google Scholar

Azzi, S. and N. Hillmer. 2013. “Evaluating Prime Ministerial Leadership in Canada: The Results of an Expert Survey.” Canadian Political Science Review 7(1): 13–23. Search in Google Scholar

Bakker, R., C. de Vries, E. Edwards, L. Hooghe, S. Jolly, G. Marks, J. Polk, J. Rovny, M. Steenbergen, and M. A. Vachudova. 2015. “Measuring Party Positions in Europe: The Chapel Hill Expert Survey Trend File, 1999-2010.” Party Politics 21(1): 143–52, https://doi.org/10.1177/1354068812462931. Search in Google Scholar

Binningsbø, H. and C. Loyle. 2012. Armed Conict and Post-Conict Justice Dataset: Background Narratives. Ed. by PRIO: Centre for the Study of Civil War. Also available at http://www.justice-data.com/pcj-dataset/. Search in Google Scholar

Binningsbø, H., C. Loyle, G. Scott, and J. Elster. 2012. “Armed Conict and Post-conict Justice, 1946-2006: A Dataset.” Journal of Peace Research 49(5): 731–40. Search in Google Scholar

Blom, A. G. and F. Kreuter. 2011. “Special Issue on Survey Nonresponse.” Journal of Official Statistics 27: 2. Search in Google Scholar

Bowler, S., D. M. Farrell, and R. T. Pettitt. 2005. “Expert Opinion on Electoral Systems: So Which Electoral System Is “Best”?” Journal of Elections, Public Opinion & Parties 15(1): 3–19, https://doi.org/10.1080/13689880500064544. Search in Google Scholar

Castles, F. G. and P. Mair. 1984. “Left-right Political Scales: Some ‘expert’ Judgments.” European Journal of Political Research 12(1): 73–88, https://doi.org/10.1111/j.1475-6765.1984.tb00080.x. Search in Google Scholar

Chernykh, S., D. Doyle, and T. J. Power. 2017. “Measuring Legislative Power: An Expert Reweighting of the Fish-Kroenig Parliamentary Powers Index.” Legislative Studies Quarterly 42(2): 295–320, https://doi.org/10.1111/lsq.12154. Search in Google Scholar

Cheung, K. L., M. Peter Cees Smit, H. de Vries, and M. E. Pieterse. 2017. “The Impact of Non-response Bias Due to Sampling in Public Health Studies: A Comparison of Voluntary versus Mandatory Recruitment in a Dutch National Survey on Adolescent Health.” BMC Public Health 17(1): 276–86, https://doi.org/10.1186/s12889-017-4189-8. Search in Google Scholar

Coma, F. M. I. and C. van Ham. 2015. “Can Experts Judge Elections? Testing the Validity of Expert Judgments for Measuring Election Integrity.” European Journal of Political Research 54(2): 305–25. Search in Google Scholar

Dahlberg, S., C. Dahlström, P. Sundin, and J. Teorell. 2013. “The Quality of Government Expert Survey 2008-2011: A Report.” QoG Working Paper Series 15. Search in Google Scholar

David, R. and I. Holliday. 2012. “International Sanctions or International Justice? Shaping Political Development in Myanmar.” Australian Journal of International Affairs 66(2): 121–38, https://doi.org/10.1080/10357718.2012.658615. Search in Google Scholar

Dion, M. L., J. Lawrence Sumner, and S. M. Mitchell. 2018. “Gendered Citation Patterns across Political Science and Social Science Methodology Fields.” Political Analysis 26(3): 312–27, https://doi.org/10.1017/pan.2018.12. Search in Google Scholar

Felisberti, F. M. and R. Sear. 2014. “Postdoctoral Researchers in the UK: A Snapshot at Factors Affecting Their Research Output.” PLoS One 9: 4, https://doi.org/10.1371/journal.pone.0093890. Search in Google Scholar

Finnemore, M. 1996. “Norms, Culture, and World Politics: Insights from Sociology’s Institutionalism.” International Organization 50(2): 325–47, https://doi.org/10.1017/s0020818300028587. Search in Google Scholar

Gervasoni, C. 2010. “Measuring Variance in Subnational Regimes: Results from an Expertbased Operationalization of Democracy in the Argentine Provinces.” Journal of Politics in Latin America 2(2): 13–52, https://doi.org/10.1177/1866802x1000200202. Search in Google Scholar

Gleditsch, N. P., P. Wallensteen, M. Eriksson, M. Sollenberg, and H. Strand. 2002. “Armed Conict 1946-2001: A New Dataset.” Journal of Peace Research 39(5): 615–37, https://doi.org/10.1177/0022343302039005007. Search in Google Scholar

Groves, R. M. and E. Peytcheva. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” Public Opinion Quarterly 72(2): 167–89, https://doi.org/10.1093/poq/nfn011. Search in Google Scholar

Guarino, C. M. and V. M. H. Borden. 2017. “Faculty Service Loads and Gender: Are Women Taking Care of the Academic Family?” Research in Higher Education 58(6): 672–94, https://doi.org/10.1007/s11162-017-9454-2. Search in Google Scholar

Huber, J. and R. Inglehart. 1995. “Expert Interpretations of Party Space and Party Locations in 42 Societies.” Party Politics 1(1): 73–111, https://doi.org/10.1177/1354068895001001004. Search in Google Scholar

Hunter, L. A. and E. Leahey. 2010. “Parenting and Research Productivity: New Evidence and Methods.” Social Studies of Science 40(3): 433–51, https://doi.org/10.1177/0306312709358472. Search in Google Scholar

Kato, J. and M. Laver. 1998. “Party Policy and Cabinet Portfolios in Japan, 1996.” Party Politics 4(2): 253–60, https://doi.org/10.1177/1354068898004002006. Search in Google Scholar

Kato, J. and M. Laver. 2003. “Policy and Party Competition in Japan after the Election of 2000.” Japanese Journal of Political Science 4 (1): 121–33, https://doi.org/10.1017/s146810990300104x. Search in Google Scholar

Kerby, M. and K. Blidook. 2014. “Party Policy Positions in Newfoundland and Labrador: Expert Survey Results in the Buildup to the 2011 Provincial Election.” American Review of Canadian Studies 44(4): 400–14, https://doi.org/10.1080/02722011.2014.976234. Search in Google Scholar

King, G., M. Tomz, and J. Wittenberg. 2000. “Making the Most of Statistical Analyses: Improving Interpretation and Presentation.” American Journal of Political Science 44(2): 341–55, https://doi.org/10.2307/2669316. Search in Google Scholar

King, G. and L. Zeng. 2001. “Logistic Regression in Rare Events Data.” Political Analysis 9(2): 137–63, https://doi.org/10.1093/oxfordjournals.pan.a004868. Search in Google Scholar

Kukulu, K., O. Korukcu, Y. Ozdemir, A. Bezci, and C. Calik. 2013. “Self-confidence, Gender and Academic Achievement of Undergraduate Nursing Students.” Journal of Psychiatric and Mental Health Nursing 20(4): 330–5, https://doi.org/10.1111/j.1365-2850.2012.01924.x. Search in Google Scholar

Laver, M. 1998. “Party Policy in Ireland 1997 Results from an Expert Survey.” Irish Political Studies 13(1): 159–71, https://doi.org/10.1080/07907189808406592. Search in Google Scholar

Lupu, N. and K. Michelitch. 2018. “Advances in Survey Methods for the Developing World.” Annual Review of Political Science 21: 195–214, https://doi.org/10.1146/annurev-polisci-052115-021432. Search in Google Scholar

Maestas, C. (2016). “Expert Surveys as a Measurement Tool – Challenges and New Frontiers”. In The Oxford Handbook of Polling and Survey Methods, edited by A. Lonna Rae and M. Alvarez. Oxford: Oxford Handbooks Online. https://www.oxfordhandbooks. com/view/10.1093/oxfordhb/9780190213299.001.0001/. Search in Google Scholar

Maliniak, D., P. Ryan, and B. F. Walter. 2013. “The Gender Citation Gap in International Relations.” International Organization 67(4): 889–922, https://doi.org/10.1017/s0020818313000209. Search in Google Scholar

Marquardt, K. L., D. Pemstein, B. Seim, and Y. Wang (2018). “What Makes Experts Reliable?” V-Dem Working Paper 68. Also available at: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3190946. Search in Google Scholar

McElroy, G. and K. Benoit. 2007. “Party Groups and Policy Positions in the European Parliament.” Party Politics 13(1): 5–28, https://doi.org/10.1177/1354068806068593. Search in Google Scholar

McLean, I., A. Blais, J. C. Garand, and M. Giles. 2009. “Comparative Journal Ratings: A Survey Report.” Political Studies Review 7(1): 18–38, https://doi.org/10.1111/j.1478-9299.2008.00168.x. Search in Google Scholar

O’Malley, E. 2007. “The Power of Prime Ministers: Results of an Expert Survey.” International Political Science Review 28(1): 7–27, https://doi.org/10.1177/0192512107070398. Search in Google Scholar

Orenstein, P. 2013. Schoolgirls: Young Women, Self-esteem, and the Confidence Gap. New York City: Anchor. Search in Google Scholar

Pétry, F., B. Collette, and H.-D. Klingemann (eds.). 2012. Left-Right in Canada: Comparing Data from Party Manifesto Content and Expert Surveys. Search in Google Scholar

Polk, J., J. Rovny, R. Bakker, E. Edwards, L. Hooghe, S. Jolly, J. Koedam, F. Kostelka, G. Marks, G. Schumacher, M. Steenbergen, M. Vachudova, and M. Zilovic. 2017. “Explaining the Salience of Anti-elitism and Reducing Political Corruption for Political Parties in Europe with the 2014 Chapel Hill Expert Survey Data.” Research & Politics 4: 1–9, https://doi.org/10.1177/2053168016686915. Search in Google Scholar

Ray, L. 1999. “Measuring Party Orientations towards European Integration: Results from an Expert Survey.” European Journal of Political Research 36(2): 283–306, https://doi.org/10.1111/1475-6765.00471. Search in Google Scholar

Ray, L. and H. M. Narud. 2000. “Mapping the Norwegian Political Space: Some Findings from an Expert Survey.” Party Politics 6(2): 225–39, https://doi.org/10.1177/1354068800006002007. Search in Google Scholar

Rohrschneider, R. and S. Whitefield. 2007. “Representation in New Democracies: Party Stances on European Integration in Post-communist Eastern Europe.” The Journal of Politics 69(4): 1133–46, https://doi.org/10.1111/j.1468-2508.2007.00613.x. Search in Google Scholar

Royston, P., J. B. Carlin, and I. R. White. 2009. “Multiple Imputation of Missing Values: New Features for MIM.” STATA Journal 9(2): 252–64, https://doi.org/10.1177/1536867x0900900205. Search in Google Scholar

Ruggeri, A., T.-I. Gizelis, and D. Han. 2011. “Events Data as Bismarck’s Sausages? Intercoder Reliability, Coders’ Selection, and Data Quality.” International Interactions 37(3): 340–61, https://doi.org/10.1080/03050629.2011.596028. Search in Google Scholar

Sax, L. J. and C. E. Harper. 2007. “Origins of the Gender Gap: Pre-college and College Influences on Differences between Men and Women”. Research in Higher Education 48(6): 669–94. Search in Google Scholar

Schmitt, H. and T. Loughran (eds.) 2017. Understanding Ideological Change in Britain: Corbyn, BREXIT, and the BES Expert Surveys. Search in Google Scholar

Steinert, C. 2019. “Trial Fairness before Impact: Tracing the Link between Post-conict Trials and Peace Stability.” International Interactions 45(6): 1003–31, https://doi.org/10.1080/03050629.2019.1657114. Search in Google Scholar

Szöcsik, E. and C. I. Zuber. 2015. “EPAC-a New Dataset on Ethnonationalism in Party Competition in 22 European Democracies.” Party Politics 21(1): 153–60, https://doi.org/10.1177/1354068812462927. Search in Google Scholar

Volken, T. 2013. “Second-stage Non-response in the Swiss Health Survey: Determinants and Bias in Outcomes.” BMC Public Health 13(1): 167–77, https://doi.org/10.1186/1471-2458-13-167. Search in Google Scholar

Warwick, P. 2005. “Do Policy Horizons Structure the Formation of Parliamentary Governments?: The Evidence from an Expert Survey.” American Journal of Political Science 49(2): 373–87, https://doi.org/10.1111/j.0092-5853.2005.00129.x. Search in Google Scholar

Received: 2020-02-17
Accepted: 2020-07-28
Published Online: 2020-08-24

© 2020 Walter de Gruyter GmbH, Berlin/Boston