Opening the methodological black box of research synthesis in language education: where are we now and where are we heading?

... full discovery will be made, if the inquirer be competent, conduct his researches with knowledge of the discoveries alreadymade, andmake them his starting-point; but anyone who, casting aside and rejecting all these means, attempts to conduct research in any other way or after another fashion, and asserts that he has found out anything, is and has been, the victim of deception. – Hippocrates (460 BC-377 BC)

by and for applied linguists and language educators.Firstly, the term 'research synthesis' has been used for a long time in this field, starting with Norris and Ortega's (2006) book, Synthesising Research on language Learning and Teaching, widely considered as seminal in research synthesis in applied linguistics (dos Santos Lima et al. 2016;Goo et al. 2015;Shin 2010;Sok et al. 2019).In Norris and Ortega's (2010) timeline published in Language Teaching, 'research synthesis' is used consistently to refer to reviews on substantive and methodological topics that employ systematic methodology and transparent reporting to bring together the findings of research addressing the same or similar questions.In 2022, the first Special Interest Group on research synthesis in our field was set up by the British Association for Applied Linguistics, called Research Synthesis in Applied Linguistics.Therefore, for the sake of consistency, and in-keeping with conventions in our field, we will use the term 'research synthesis' for this special issue.
The field of applied linguistics has had a varied relationship with research synthesis, but we have noticed thriving interest in it since the turn of the millennium, setting out to address the not inconsiderable challenge of "review [ing] the accumulated research on second language (L2) instructional effectiveness" (Norris and Ortega 2010, p. 461).In fact, in the nearly two decades since the publication of Norris and Ortega's (2006) seminal work, there has been a surge of research syntheses published in applied linguistics on a wide range of topics (see Chalmers, Brown & Koryakina, this issue).In particular, meta-analysis, referring to the statistical synthesis of quantitative data, has gained traction and remains one of the most widely published types of research synthesis in our field (Oswald and Plonsky 2010).Quantitative syntheses make valuable contributions to addressing questions of effectiveness as they relate to practice.It is well understood that a single study is rarely a sufficient basis on which to formulate confident conclusions about the relative effects teaching approaches or strategies.By considering the totality of the available evidence through careful curation of relevant research, assessing the quality of that research, and statistically synthesising its findings, we gain a better understanding of the likelihood of success should those approaches be adopted by practitioners.In addition, considering the totality of the evidence allows synthesists to identify contexts in which such approaches may be more or less effective, or where important contextual characteristics might modify an approach's effects.For example, by conducting subgroup analyses within a larger meta-analysisby age of participants or geographical location, for examplea more nuanced understanding of the likely effects of a given approach can be estimated.This allows us to move with more confidence beyond the somewhat reductive question of 'what works?' to the more nuanced 'what works, for whom, and under what circumstances?'.

Qualitative research synthesis and qualitative synthesis of research
In addition to meta-analysis, and its focus on the synthesis of quantitative data, the discipline of research synthesis has developed to incorporate alternative approaches to examining the totality of evidence in a given field.It is, therefore, important to note that research synthesis "transcends the narrower domain of meta-analysis and includes other quantitative as well as qualitative methods" (Norris and Ortega 2010, p. 461).In particular, as the field has developed we have begun to see types of research syntheses associated with the qualitative paradigm.We term these 'qualitative research syntheses' (research synthesis that employs qualitative methods to synthesise qualitative research evidence) and 'qualitative synthesis of research' (research synthesis that employs qualitative methods to synthesise research evidence derived from both qualitative and quantitative paradigms).
As with most types of syntheses, syntheses of qualitative evidence usually follow standard methodological stages: design research questions, identify keywords for literature search, conduct literature search, assess literature for eligibility using pre-specified inclusion and exclusion criteria, assess the quality of included research, extract data, synthesise data, and report the findings of that synthesis (Chong and Plonsky 2021).Qualitative syntheses differ from metaanalyses most substantially in the way that data are extracted and synthesised (Chong and Reinders 2021).Whereas meta-analyses are principally concerned with extracting summary data in the form of numbers, such as effect sizes, qualitative data extraction refers to extracting information in the form of words that are relevant to the aim of the synthesis.This process is similar to line-by-line coding or initial coding in primary studies that employ qualitative methods.Syntheses of the extracted data then involves the sense-making process using that information.The outcome of such a process is the development of higher-order themes based on the codes developed during data extraction.Data synthesis is reminiscent of focused and axial coding in qualitative studies that employ grounded theory (Charmaz 2006).In what follows, we outline three types of less common research synthesis in applied linguistics: scoping reviews, systematic literature reviews, and qualitative research syntheses.
Scoping reviews are a type of exploratory review that aims to survey the landscape of an emerging or existing field of research; sometimes, a scoping review is conducted to gauge the feasibility or necessity for conducting a larger-scale systematic review (Pham et al. 2014).Adhering to its exploratory nature, research questions of scoping reviews are generally broader; for example: 'What is the extent and nature of research on X?' (see Chong and Reinders (2022) for examples relating to conceptualisation, operationalisation, and evaluation in computerassisted language learning (CALL) research).The search strategy of scoping reviews prioritises inclusivity because the primary objective of a scoping review is to map out the evidence base of a field of enquiry, assess the extent of the available evidence, organise it into groups, and highlight gaps.In this case, standards used to appraise studies to be included in scoping reviews are usually less stringent than those employed in a systematic review or a qualitative research synthesis.
Systematic literature reviews can have a substantive or methodological focus.Compared to those in scoping reviews, research questions of systematic literature reviews are usually more specific and more focussed.A systematic literature review is expansive in scope, aiming to provide a thorough state-of-the-art review on a topic.Reflecting the greater scope of this type of research, Language Teaching journal, for example, expects reviews that it publishes to have approximately 100 references and increases the word allowance to 20,000, considerably more than the typical word limit for reports of primary research (Language Teaching 2022).Despite having a broad focus, systematic literature reviews require a more rigorous evaluation of the quality of included studies.Typically, constituent studies will be evaluated for methodological rigour and susceptibility to bias, using tools like Cochrane Risk of Bias tool for experiments (Sterne et al. 2019), Quality in Qualitative Evaluation for qualitative designs (Spencer et al. 2003), the Newcastle Ottawa Scale for observational designs (Wells et al. 2021), the Mixed Methods Appraisal Tool for reviews that include a variety of designs (Hong et al. 2018), and AXIS for cross-sectional designs (Downes et al. 2016).This means that, in addition to reporting the overall findings of a body of research, systematic literature reviews provide an estimate of trustworthiness, both in terms of individual studies and of the literature as a whole.
Qualitative research synthesis differs from scoping and systematic literature reviews because it summarises only qualitative research evidence.Published qualitative research syntheses in applied linguistics focus on research that is wholly qualitative in nature or by extracting only qualitative elements of mixed methods research (e.g., Chen 2016; Chong and Reinders 2020; Çiftçi and Savaş 2018).Following the qualitative research paradigm, qualitative research synthesis is underpinned by an interpretivist epistemological view, aiming to capture the richness and diversity of beliefs and experiences of social phenomena such as language learning and teaching.To represent the complex relationship among various themes, synthesised qualitative evidence needs to be presented in a way that highlights connections.This can be done through a conceptual mind map that accompanies a written report of themes.The qualitative research synthesis by Çiftçi and Savaş (2018) on telecommunication provides an excellent example of such conceptual mind map (p.291).
For a detailed analysis of these three (and other types of synthesis) particular to the field of applied linguistics, see Chong and Plonsky (this issue).

The next phase of development of research synthesis in applied linguistics
As the outline of developments in research synthesis in applied linguistics we have provided above suggests, we feel it is no exaggeration to say that the field is in an exciting phase of development.In this special issue, we aim to demonstrate the value of research synthesis in applied linguistics by reviewing and summarising the stateof-the-art as it stands in 2023, providing methodological commentary and accounts of innovation in the field, and showing how syntheses have contributedand can continue to contributeto better research to help to ensure that no applied linguistics researcher will fall foul of the deception that Hippocrates warned about more than two millennia ago.This special issue, consisting of five articles, showcases the diversity in research synthesis in our field.In addition to addressing the need for quality and rigour in research synthesis, and demonstrating good practice through related commentaries and field specific exemplars, we introduce less familiar approaches to synthesis such as scientometric reviews and qualitative research synthesis.We also include an article that demonstrates how research synthesis methodology can be applied to various sub-fields of applied linguistics, even in areas that are often perceived as less relevant to research synthesis such as corpus linguistics.
In response to the need to improve understanding of robust research synthesis in applied linguistics, the first article by Chong and Plonsky presents a muchneeded typology of secondary research.They identify no fewer than 13 unique types of synthesis, which they then exemplify through example reviews.They argue that, rather than rely on typologies that have been produced in other fields (e.g., healthcare, Grant and Booth 2009), a typology specific to the field of applied linguistics is needed.Despite the move towards more systematic approaches in recent years (Sutton et al. 2019), Chong and Plonsky also argue that traditional literature reviews and systematic research syntheses are both valuable, as they serve different purposes.
Reflecting a growing interest in bibliometric analyses (Chen et al. 2019), the second article by Zakaria and Aryadoust is a scientometric analysis of over 40,000 applied linguistics studies published in Q1 journals between 1970 and 2022.Through a co-citation analysis, the authors explore overall trends in the field, finding a high degree of theoretical interconnectedness, with a move towards specialised aspects of second language acquisition from 2000 onwards.They go on to suggest that further work is needed to expand scientometric approaches in the field.
An example of a meta review (a review of reviews), the third article by Li and Vuogan presents a systematic review of 120 meta-analyses in second language research, in order to gain a deeper understanding of the meta-analytic methods used in the field.Linking to an ongoing conversation in the field of education (e.g., Bond et al. 2021), the authors found that important study design details tend to be missing in the collected meta-analyses, and urge that greater transparency is needed in future syntheses, including providing the complete details of data extraction and analysis methods.This is particularly important for secondary research, as it is meant to help limit bias and provide a reliable overview or insight into a body of work.Li and Vuogan's recommendations are, therefore, a necessary contribution to the further development of the field.
Also finding room for improvement in the reporting quality of language education syntheses by conducting a meta review, Chalmers, Brown and Koryakina present an analysis of the 307 language education systematic reviews already included in the recently established International Database of Education Systematic Reviews (IDESR.org); a much needed repository for the field.The authors found an enormous upswing in the number of reviews being conducted in this field after 2016, with the Language Learning journal publishing the most reviews, and the field of educational technology by far the most prolific.Chalmers and colleagues advise researchers to conduct a thorough investigation of previous reviews that have been undertaken on the same or a similar topic before embarking on their own syntheses, to ensure that their work avoids unnecessary duplication of effort and wasting precious resources.The authors then provide several further recommendations, particularly around the careful and thorough reporting needed in evidence syntheses, which they find wanting in the assembled literature.
The final article by Liu and Chong is a "qualitative synthesis of research", synthesising 16 studies focused on bilingual education in China.Most of the studies focus on evaluating the effectiveness of bilingual education in relation to stakeholders' perceptions and experiences, although teacher perceptions were largely ignored.The findings of most of the included studies suggest that bilingual education in China is largely effective, although the research tools used to gauge its effectiveness focused on specific language skills (e.g., reading) rather than on learners' holistic linguistic competence.Similar to Li and Vuogan's review, Liu and Chong found a lack of reporting transparency in the primary studies, and also question the prevalence of shorter-term studies over those undertaken longitudinally.

Further development of the field
Compiling this special issue has not only allowed us to take stock of where we are with research synthesis in applied linguistic, but it has given us the opportunity to consider related aspects of the evolution of the field, allowing us to consider where we want to go next.To conclude this editorial, we will highlight some of the areas where we see positive movement related to research synthesis and which we expect to see reflected in related scholarship: open science in applied linguistics, closing the research-practice gap, and the growing recognition of the methodological requirements of evidence synthesis.

Applied linguistics and open science
Open science, or open scholarship, refers to practices that aim to "make multilingual scientific knowledge openly available, accessible, and reusable for everyone" (UNESCO 2021, p. 6).In addition to the prospective and open publication of systematic review protocols on IDESR.org(see Chalmers, Brown and Koryakina, this issue), a variety of other open science practices are gaining traction in our field.These include publishing materials and data on IRIS-database.org, creating textual and visual summaries of publications for audiences outside of academia (OASIS-database.organd TESOLgraphics.com), and translating abstracts into languages other than English (Multilingual Repository for Abstracts in Applied Linguisticsmultilingualrepository.org).Recently, the "Open Applied Linguistics" Research Network, affiliated with the International Association of Applied Linguistics (AILA), was established.The conveners held an online symposium on open scholarship in applied linguistics (Meng et al. 2022), following which a statement on open scholarship in applied linguistics, endorsed by AILA, was released and (Meng and Chong 2022).This statement identified areas where resources for facilitating open science in applied linguistics are growing, and urged take up of these resources by researchers and practitioners.Open science, and in particular open data, is highly relevant to research synthesis because making raw data and research materials publicly available is crucial for synthesists to conduct highquality reviews.It allows synthesists to examine the quality of the data collected and its analysis, thus helping to inform decisions of whether to include certain studies in reviews and facilitating more complete statistical syntheses.Additionally, open data enables the re-analysis of data by other synthesists.At present, however, the sharing of data does not appear to be common practice in applied linguistics research.For example, a meta-analysis conducted by Plonsky et al. (2015) reported that among 255 authors who were contacted to request the datasets generated by their research, the majority did not respond (n = 159).Of those that did, only 36 datasets were provided, of which only 25 were usable.If we strive to improve the quality of research synthesis in our field, in addition to rigorous methodology, synthesists need to be able to access usable raw data.Encouraging primary researchers to see the value in making their data publicly available via repositories like the IRIS database would be a positive first step.

Closing the research-practice gap
Researchers in our field have started to advocate the need for dialogues between researchers and practitioners.Sato and Loewen (2019) wisely urged researchers to "widen the door and increase the amount of dialogue" (p.9).This call to bridge the research-practice chasm points to the ongoing problem that educational research evidence is rarely used by teachers to inform their practice (see e.g., Bond et al. 2019).The reasons are manyfold.Medgyes (2017), for instance, contended that there is distrust between researchers and practitioners.Isaacs and Chalmers (in press) note the lack of involvement by teachers in setting research agendas.That is, teachers are not routinely asked what kind of research they need to inform their practices, which risks researchers addressing questions that are not of practical importance to teachers.Another reason relates to teachers' lack of access to language education journals.Although many teachers are interested in improving their practice on the basis of research evidence, they do not necessarily know where to locate credible research, and if they do, they are usually barred from doing so by journal paywalls (Sato and Loewen 2019).Even if they do have access, many report that it is a luxury for them to read academic texts given their heavy teaching and administrative workload (ibid).
In response to the above, Chong (2020Chong ( , 2022) ) argued that research synthesis has a vital role to play in bridging the research-practice gap.As research syntheses summarise the whole body of evidence from primary studies on topics of interest to teachers, it is less time-consuming for teachers to read and provides the overview necessary to interpret the extent of our knowledge on a given topic, rather than being misled by selective citation of single studies in, for example, the popular press.Moreover, research syntheses are more substantial pieces that (usually) involve the work of a team of researchers, and they are often supported by research funding.As such, research syntheses are more likely to be open access, allowing teachers to access research syntheses without paying a subscription fee.Lastly, unlike primary studies, synthesised findings are often presented visually, in tables and diagrams, collating information in a more reader-friendly manner.One example is Hung et al.'s (2018) scoping review of research on digital game-based language learning where the authors summarised the various types of digital games documented in research in a table, including their definitions and relevant primary studies.From the perspective of language teachers who may be unfamiliar with the literature base, accessible summaries like this are extremely helpful because they provide a bird's-eye view of the resources available to teachers and a summary of the research that has evaluated them.
With researchers' growing interest in establishing connections with practitioners, we urge synthesists to include concrete and evidence-informed recommendations for teachers and policymakers.Preferably, such information can be presented in a way that encourages views from colleagues in the profession, such as using various visual representation of the synthesised data.To give an example, TESOLgraphics (Chong and Sato 2022) is an initiative that creates one-page, infographic summaries of research syntheses for English teachers.Research syntheses were summarised by teacher members of the project team, with the intention to incorporate teachers' perspectives (e.g., what teachers view as important when reading a research synthesis; what practical insights teachers can get from reading the research synthesis) into the infographic.

Growing recognition of methodological requirements
Research synthesis has been increasingly recognised by journals in applied linguisticsnot only for its inherent value, but for its differences with primary studies.One such difference is length.Research syntheses, which summarise multiple primary studies in a systematic and rigorous manner, usually result in pieces much longer than the 7,000 to 8,000 words typical of published research.Journals such as Language Learning and Language Teaching have long been at the forefront of publishing research syntheses in our field, with dedicated sections that allow pieces that exceed 10,000 words.What is exciting is that more journals have started to include dedicated sections for research syntheses, including recently, for example, Studies in Second Language Acquisition and Language Testing.A new journal, Research Methods in Applied Linguistics published by Elsevier, also includes research synthesis ("reviews") as one of the article types, providing a more generous word limit of 10,000 words.

Concluding thoughts
In this editorial, we have shown that research synthesis has a long history of development outside of applied linguistics.Research synthesis gained recognition in our field by the influential work of, among others, Norris and Ortega (2006) and since then has enjoyed increasing attention and respect for what it has to offer our field.We underscored that there are various types of research synthesis, serving different purposes and employing various methodological approaches.It is one of the aims of this special issue to introduce to the field the types of synthesis that have received relatively less attention, such as qualitative research synthesis.We reviewed a number of advances in research synthesis in applied linguistics and how research synthesis can benefit from these.It is our aspiration that this special issue will provide a more comprehensive picture of what research synthesis is as a genre of research and methodological approach, and its potential to make substantive and important contributions to the field.