A typology of secondary research in Applied Linguistics

: Secondary research is burgeoning in the ﬁ eld of Applied Linguistics, taking the form of both narrative literature review and especially more systematic research synthesis. Clearly purposed and methodologically sound secondary research contributes to the ﬁ eld because it provides useful and reliable summaries in a given domain, facilitates research dialogues between sub-ﬁ elds, and reduces re-dundancies in the published literature. It is important to understand that secondary research is an umbrella term that includes numerous types of literature review. In this commentary, we present a typology of 13 types of well-established and emergent types of secondary research in Applied Linguistics. Employing a four-dimensional analytical framework, focus, review process, structure, and representation of text of the 13 types of secondary research are discussed, supported by examples. This article ends with recommendations for conducting secondary research and calls for further inquiry into ﬁ eld-speci ﬁ c methodology of secondary research.

f secondary research.

Secondary research, or literature review, refer to a scholarly review of a body of literature on a selected topic (Ellis 2015).In the field of Applied Linguistics, and elsewhere in the social and natural sciences, there are two main types of secondary research: traditional and systematic (Norris and Ortega 2007).Traditional (i.e., narrative-based) reviews bring together findings on the topic under investigation while providing a connection between the reported research and the macroscopic research terrain.While traditional reviews are often composed in a manner which aims to "tell a story" (Norris and Ortega 2006, p. 5), they also tend to include a critique of the literature at hand (an alternative name of some traditional reviews is "critical reviews").This is especially true in traditional reviews which are published as stand-alone journal articles in which researchers identify strengths and weaknesses of existing studies on a topic.Unlike systematic reviews, traditional reviews do not generally extract data in any formalized way from primary studies.Finally, such reviews usually also make suggestions for future research directions based on the reviewers' expert knowledge and/or based on gaps identified in the literature.

The other main family of secondary research is best characterized by a formal set of methods that are applied to the review process.Such 'systematic reviews', or research syntheses, have gained widespread popularity in Applied Linguistics research in recent years, especially those that aggregate quantitative findings (Chong et al. 2023).In addition to systematic reviews, a growing body of papers concerning their methodology can also be found (e.g., Chong and Plonsky 2021;Chong and Reinders 2021;Chong and Reinders 2022;Li and Wang 2018;Macaro 2020;Norris and Ortega 2006;Plonsky and Brown 2015;Plonsky and Oswald 2015) and several new contributions are underway (e.g., Norris and Plonsky In Preparation; Sterling and Plonsky In Preparation).Indeed, Norris and Ortega's (2007) prediction that research syntheses "will continue to thrive in our field" (p.812) has been borne out.Unlike traditional or narrative reviews, systematic research syntheses refer to a "protocoldriven and quality-focused approach" (Bearman et al. 2012, p. 625) to aggregate research evidence to enlighten theory, research, policy, and practice.

In parallel to primary research, system tic research syntheses can be broadly divided into two major types: qualitative and quantitative.Qualitative research synthesis, also called "qualitative evidence synthesis" (e.g., Chong and Reinders 2020), assembles qualitative research evidence to "reveal deep insights into disparate literature for future research" (Chen 2016, p. 387).Another purpose of qualitative research synthesis is to strengthen and deepen qualitative evidence by unraveling "multidimensions, varieties, and complexities" (Çiftçi and Savas 2017, p. 4) amongst studies.

Quantitative research syntheses, as the n me indicates, rely on quantitative data as a means to understand a given domain.The most well-known synthesis of this type is likely meta-analysis, which involves the statistical aggregation of effect sizes across studies (e.g., In'nami and Kozumi 2009).However, a number of other types of quantitative syntheses exist such as bibliometric reviews (see below). Figure 1 presents a visual of the basic breakdown of secondary research types introduced thus far.

Whilst traditional literature reviews conti ue to appear in Applied Linguistics journals and other outlets such as book chapters and monographs, systematic research syntheses are gaining prominence.Such growth can be attributed to several factors.First, the status of research synthesis has changed in recent years from a kind of 'state-of-the-art' think piece or commentary to a valuable form of empirical research necessary to consolidate and advance understanding of a given domain.To be sure, the systematicity, rigor, and transparency associated with research syntheses has greatly diminished the presence of bias and allowed for greater trust to be placed in secondary results.Secondly, reviews are no longer only produced by senior scholars who offer their authoritative opinions but also by more junior scholars as well as by synthesists working alongside researchers who specialize in the topic of the review (e.g., Visonà and Plonsky 2020).Such collaborations between experts in the substantive domain, on one hand, and in research syntheses on the other, bring a powerful synergy to the task of systematically reviewing a given body of work.A third reason for growth in the prominence of research synthesis is its place as "a cornerstone of the evidence-based practice and policy movement" (Dixon-Woods et al. 2007, p. 375) in education, medicine, and many other applied fields.Fourth, systematic research syntheses facilitate "epistemological diversity" (Norris and Ortega 2007, p. 813).Through more systematic and comprehensive literature searches (see Plonsky and Brown 2015), systematic research syntheses give voice to research findings published not only in prominent journals (mostly published in English) but also local and regional journals as well as other less visible yet valuable sources, such as masters and doctoral theses.

Despite the growth and prominence of secondary rese rch, we have noticed a lack of consistency in naming conventions in the synthetic literature in applied linguistics as well as in other disciplines (e.g., Sutton et al. 2019 underlying this inconsistency appears to be a misunderstanding of the different approaches to synthesis and their corresponding techniques and goals.One outcome of such inconsistency is confusion among authors, reviewers, editors, and readers (Chong et al. 2022).Perhaps more serious is the implicit message sent to readers regarding this type of work that almost anything goes and by any name.Consequently, we see an urgent need to clarify not only the labels but the characteristics and accepted norms for conducting systematic research syntheses in Applied Linguistics.


A typology of secondary research in Applied Linguistic

In the previous section, we established that secondary
esearch in Applied Linguistics has experienced a shift from the narrative tradition to a more systematic, empirical approach.While the development of systematic research synthesis is gaining momentum, it is unlikely that traditional literature reviews will be replaced entirely.Rather, in our view, it benefits the whole research field that the two families of secondary research co-exist, serving different purposes to advance our understanding.With this in mind, and to provide structure and guidance to future synthetic efforts, we address the following question in the remainder of this paper: "What are the different types of secondary research in Applied Linguistics?"A number of taxonomies or typologies have been proposed to standardise conventions and practices of secondary research, for example, in healthcare (Grant and Booth 2009), health sciences (Littell 2018), medical sciences (Munn et al. 2018), and social sciences (Cooper 1988).With the proliferation of secondary research in Applied Linguistics in recent years, we believe the development of a field-specific typology is necessary.At the outset, however, a few disclaimers are warranted.Firstly, this typology is not a result of a comprehensive and fully systematic literature search, although considerations were given as to what reviews to include as examples in the typology (see below).Secondly, given the nature of this article (which is a commentary), analysis of the included reviews will indubitably represent, to a certain extent, our own views, biases, and experiences as primary and secondary/ synthetic researchers.Thirdly, this typology is not to be seen as conclusive or definitive.Owing to the dynamic development of secondary research in Applied Linguistics, we regard this typology as a work in progress and invite contributions and revisions which build upon it.Finally, the purpose of presenting this preliminary typology is not to prescribe best practices in secondary research.Rather, we hope but to offer an overview of this versatile and emergent set of methodological techniques as a means to standardize some of the naming and methodological conventions being employed.We also seek to expose the field to some of the potential approaches to and outcomes of secondary research that might otherwise be unfamiliar or overlooked.

In the remainder of this article, we discuss features of 13 types of secondary research in Applied Linguistics.Discussions on relevant features are based on the frameworks by Gough et al. (2012) and Grant and Booth (2009).The features analyzed include purpose of review, process of review (search strategy, appraisal mechanism, synthesis techniques, analysis techniques), and product of review (i.e., structure and representation of text).Features of each type of secondary research will first be discussed (see Table 1).Then, the 13 types of secondary research will be compared to identify issues and shared attributes (see breakdown in Table 2) and suggest a way forward.It must be emphasized that the types of literature review/secondary research above are by no means mutually exclusive.It is common, in fact, for secondary studies to overlap.For example, Tullock and Ortega (2017) is a scoping review and a meta-analysis.


Overview of the 13 types of research synthesis in Applied Linguistics

(1)

ritical review: The purpose of a critical review is to explore prevale
t views in a research topic and offer alternative perspectives.Muñoz and Singleton (2011), for instance, reviews and challenges the prevalent view of maturational constraints toward second language acquisition.Moreover, this type of review often discusses key questions within the target domain.Lai and Li (2011) provide another example in their critical review on the intersection of technologies and task-based language teaching.Regarding the review process, there is generally no explicit description of how literature included in the review was searched and appraised, nor does it mention how the findings from existing research are synthesized and analyzed.The text of a critical review of literature takes a narrative approach, discussing prevalent issues thematically, with arguments or views illustrated by individual studies.In terms of product, a critical review often has a customized structure with headings related to the specific prevalent issues.Unlike the majority of secondary research types, information in a critical review is usually represented using only text and without the use of figures or tables.(2) Meta-analysis: Meta-analysis is one of the most prominent types of systematic re earch synthesis in Applied Linguistics (Plonsky 2014).It aims to consolidate similarities and clarify conflicting findings.Given its quantitative nature, synthesists who conduct meta-analyses are often interested in effects or efficacy of a particular intervention.For example, Kang and Han's (2015) metaanalysis quantified the overall effects of written corrective feedback on improving linguistic accuracy of second language writers; as is typical in metaanalysis, the study also identified a number of factors which moderate the effect of written corrective feedback.The review process of meta-analysis is documented in detail, usually in a separate 'method' section, with sub-sections dedicated to the literature search, coding process, and analysis.The two metaanalyses included in Table 1 reported how data were synthesized through coding of the features of primary studies and how the reliability of the coding was upheld (e.g., Teimouri et al. 2019).The synthesized data were then analyzed using descriptive and inferential statistics to estimate and identify effect sizes, outliers, publication bias, and moderators.The procedures involved in 'reliability generalization meta-analysis', a particular type of metaanalysis, are nearly identical to meta-analysis.However, the focus is on aggregating estimates of measurement error (see, e.g., Plonsky 2017Plonsky , 2019) ) rather than effect sizes.As a type of systematic research synthesis, metaanalysis follows a rather standard structure like a primary study, usually with additional materials included as supplementary online documents (e.g., coding scheme of Teimouri et al. 2019, is made available on IRIS).In addition to text-based explanations, meta-analyses frequently utilise figures, tables, and bullet points to document the review procedure and make the otherwise information-dense piece more reader-friendly.(3) Methodological synthesis: This type of research synthesis can take two different approaches.Some focus on assessing the methodological approaches, designs, tools, and so forth within a given substantive domain, as in Plonsky and Gass' (2011) review of interactionist research in second language acquisition (SLA).Other methodological syntheses take as their defining principle a particular research technique or tool.This approach can be seen, for example, in Marsden et al.'s (2018) review of self-paced reading tasks.Concerning the review process, the two reviews included in Table 1 represent a similarly systematic approach (Crowther et al. 2021;Plonsky and Kim 2016).Similar to meta-analysis, Plonsky and Kim (2016) documented the coding process and how reliability of coding was maintained.In relation to how data were analyzed, the calculated percentages and frequencies of different research and reporting practices.These data are used to describe and evaluate the methods in the domain in question as well as to provide empirically-grounded recommendations for future research.Like other types of research synthesis, the structure of a methodological synthesis conforms to that of the typical empirical study in the social sciences.(4) Mixed review: As the name suggests, this type of research synthesis is a combination of two types of review.For example, Jackson and Suethanapornkul ( 2013) is a combination of a qualitative research synthesis and metaanalysis on task complexity whilst Marsden et al.'s (2018) extensive work is a presentation of a narrative and systematic literature review on replication studies in second language research.The review process, textual structure, and representation vary and are contingent on the types of studies being included.For instance, Jackson and Suethanapornkul (2013) follow a systematic approach to research synthesis with explicit description of how the included literature was searched, selected, extracted, and synthesized.As for structure, two approaches are noted.In Jackson and Suethanapornkul (2013), a convergent approach was adopted by presenting the qualitative research synthesis and meta-analysis as one single review, with the findings of each complementing each other (like the convergent design of mixed-methods studies noted in Creswell and Clark 2017).Another structural approach is sequential, meaning that one review is presented at a time (e.g., a narrative review is first presented in Marsden et al. 2018 and then the systematic literature review).When there are two reviews in a given report (sequential), the first is usually broader with the second one being based on a subset of the first (e.g., Yan et al.'s 2015, review and meta-analysis of elicited imitation).

Because of the profound breadth and depth of the work (see, e.g., Marsden et al. 2018), mixed reviews often include additional online materials.

(5) Narrative r view: This research-focused review aims to map the 'state of the art' in a given domain.Often written by leaders in a particular domain, the objective of this type of review is "to authoritatively answer particular research questions, and to identify gaps in research methodologies" (Norris and Ortega 2006, p. 4).Referring to the two included narrative reviews in Table 1, there seems to be divergence in terms of how the review process is enacted.In DeKeyser and Botana (2015), a narrative review is presented in the form of a traditional research synthesis with a customised structure addressing various facets of the research topic of L2 grammar acquisition.On the contrary, Thomson and Derwing (2015) adopt a more systematic approach, which is exemplified from the brief section on how studies were appraised and how findings of the included studies were analysed by calculating percentages.(We would note, however, that Thomson and Derwing's choice to include this information is atypical of this type of review.)Both reviews include individual studies as examples of points of interest to substantiate their arguments.However, neither of these narrative reviews describe their search strategy or synthesis procedure.It appears that both reviews adopt a thematic approach (as reflected in the section headings) despite not stating how these themes are generated.In terms of structure, sections similar to a typical primary study are found in Thomson and Derwing (2015) whilst a thematically-driven organisation is found in DeKeyser and Botana (2015).Usually only textual representation of results is found in these narrative reviews.(6) Qualitative research synthesis: This type of systematic research synthesis is dedicated to the aggregation of qualitative research evidence, usually in classroom-based studies, to unravel complexities of ecological and naturalistic research studies.Chong and Reinders (2020) for example, synthesizes findings from students' and teachers' perception of technology-mediated task-based language teaching.Like other types of systematic research synthesis such as meta-analysis, qualitative research synthesis strictly follows a systematic procedure of searching and selecting relevant literature.Due to its qualitative nature, much attention is paid to ensure 'openness' in the data coding process to fully capture emerging themes.Both Chen (2016) and Chong and Reinders (2020) adopt the constant comparison method of grounded theory to perform initial, focused, and axial coding of data (Charmaz 2006).Unlike meta-analysis, however, reliability of coding of qualitative research synthesis is usually achieved through multiple rounds of discussions between reviewers.As for its structure, qualitative research synthesis follows the standard structure of a research paper, with important textual information underscored using diagrams and tables.

(7) Research agenda: The purpose of a research agenda is to "discuss research tasks" (Svalberg 2012, p. 376) and "review existing studies … to summarize current knowledge and to identify research gaps" (Lou and Noels 2019, p. 1).Similar to the aforementioned types of traditional literature reviews, authors of a research agenda do not specify the methods used to search and screen relevant studies.As far as analysis of literature is concerned, at least two forms can be employed.First, as illustrated in Svalberg (2012), the reviewed literature is presented narratively and thematically.In each theme, the reviewer offers an overview statement or argument which is supplemented with a few studies as examples.Each section concludes with research tasks which spring from the limitations of the reviewed studies.Another approach is the one adopted in Lou and Noels (2019) which focuses on an emerging research field, mindsets in language teaching and learning.In their research agenda, the authors first offer a review of the current research base in a thematically threaded fashion, analogous to Svalberg's (2012) approach.Nevertheless, instead of adding research tasks at the end of each thematic section, Lou and Noels (2019) examine the way forward in a separate conclusion section.Regardless of the approach, a research agenda is usually structured thematically with section headings.Sometimes, crucial information is presented in tabular form to enhance clarity and impact (e.g., the construct of engagement with language in Svalberg 2012).( 8) Research into practice: The objective of this type of research synthesis is selfexplanatory.Being a specific type of review of the journal Language Teaching, a 'research into practice' review resembles a narrative review but with a more practical focus, that is, the application of research findings in language classrooms.For instance, Lee (2013) and Graham (2017) examine aspects of research on written corrective feedback and listening strategies which are over-and under-applied in classrooms.As a type of traditional research synthesis, there is no mention of how the topical literature was collected and assessed.Taking a thematic approach to analyze literature, pedagogicallyrelated topics are examined by citing individual studies, and in some cases, with the addition of personal experience (e.g., Graham 2017).A "research into practice" synthesis adopts a customized structure presented in textual form.(9) Scoping review: Taking a systematic and inclusive approach, a scoping review aims to map the research landscape of a given domain (Pham et al. 2014).Adhering to its systematic nature, the review process is conducted in a structured manner and each step in the process is reported extensively to maintain transparency, objectivity, and credibility (e.g., the inclusion and exclusion criteria).The included research is coded, and measures are in place to ascertain inter-coder reliability (e.g., through coder training in Visonà and Plonsky 2020).As for the analysis of coded research, synthesists conducting a scoping review usually present the overview of a research base by calculating frequencies and percentages.Depth of analysis is maintained through the discussion of individual studies of interest (Tullock and Ortega 2017).Due to their systematic nature, scoping reviews are structured as a standardized research paper, with texts being accompanied by tables and figures.Finally, although scoping reviews tend to appear as 'stand-alone' pieces, they can also serve to identify specific subdomains ripe for further inquiry via metaanalysis or systematic reviews.( 10) State-of-the-art review: State-of-the-art reviews are usually a more exhaustive and comprehensive review of a specific research topic.For example, the journal Language Teaching requires authors of state-of-the-art reviews to include approximately 100 works in the bibliography (Language Teaching 2020).Ultimately, the mission of this kind of research synthesis is to "point out contradictions and omissionsas well asagreement" (Language Teaching 2020) in the field.In the two examples, methods pertaining to the literature search and selection are explicitly mentioned in one (Hanks 2019) but not the other (Bardovi-Harlig and Stringer 2010).As for data analysis, a more personal approach is adopted to offer an insider's perspective.For example, a dialogic approach between the author and other researchers is adopted in Hanks ( 2019) and a thematic approach is adopted in Bardovi-Harlig and Stringer (2010).In both cases, the analysis of the literature is done through the provision of overview statements, followed by substantiation with a number of studies as examples.

Generally being more extensive, a state-of-the-art review o ten relies on tables and figures in addition to narrative exposition (see Bardovi-Harlig and Comajoan-Colomé 2020 for an example, which is published as a dedicated article type called "State of the Scholarship" in Studies in Second Language Acquisition).(11) Systematic literature review: A systematic literature review (also called 'research synthesis') can be practice-focused or research-focused.Practiceoriented systematic literature reviews provide arguments in favor of a practice) and evaluate issues pertaining to enactment of such practice (Macaro et al. 2018).As for research-focused systematic literature reviews, their aim is to assess the quality and range of studies and provide an updated literature search to inform new research questions (Macaro et al. 2018).Literature search strategies are devised and the selection of literature is carefully planned and executed.The included literature is coded by multiple reviewers and the reliability of coding is emphasised.'Data' are analyzed using thematic analysis and, like other types of systematic reviews, by calculating frequencies and percentages to observe patterns and trends in the sample.In many ways, a systematic literature review can be seen as overlapping with meta-analysis in all respects except for the aggregation of effect sizes.A typical research paper structure is adopted, with texts supplemented with figures and tables.(12) Historical review: The review types discussed thus far generally treat their target domains as a whole or perhaps in different groupings based on features or variables shared across certain studies.By contrast, historical reviews use time as an organizing principle and as a means to understand the theoretical and/or methodological development that has taken place in a given domain.Boo et al. (2015), for example, traced the recent history of L2 motivation research, focusing on changes in theoretical models, research designs, and target languages, among other features.The authors describe their methods in detail and followed procedures much like those of other systematic review types, presenting results in tables and figures to enhance clarity.Historical reviews can also, however, take the form of a timeline.The journal Language Teaching has published many reviews of this type.Timelines generally begin with a broad introduction to situate the domain in question and to highlight major themes and/or developments that have taken place.Landmark publications and other pertinent events are then presented chronologically, usually accompanied by annotations as well as thematic indicators presented in the introduction.See Isaacs and Harding's (2017) timeline of pronunciation assessment in L2 research as a prime example of this type of synthesis.(13) Bibliometric review: As in other types of reviews, bibliometric analyses focus on the study/report as the unit of analysis.However, the focus here is on publication meta-data such as citation counts (co-)citation within and across publications, authorship attributes (e.g., number, gender, language and geographic background), article titles and keywords, and so forth.Bibliometric research is fairly new to Applied Linguistics, but it appears to be surging as evident in the number of recent studies in a bibliography of bibliometric research in applied linguistics (Plonsky, N.D).Hyland and Jiang (2019), for example, analyzed the frequency as well as linguistic and stylistic features of citation patterns both over time and across four disciplines including applied linguistics.(For another recent bibliometric analysis of citation patterns, see Lei and Liu 2018.)Aryadoust and Ang (2021) take on a narrower domain, that of eye-tracking research in the language sciences.Using a sample of 341 publications, the authors extract and analyze citations and co-citations as well as research affiliations and countries, among other types of publication meta-data.The authors' co-citation analysis found a number of prominent 'clusters' of scholarly activity and influence.As in other quantitative and systematic review types, bibliometric analyses often utilize tables and figures to present their findings.


Four continua of research synthesis in Applied Linguistics

Having laid out the 13 t

es of secondary research being employed in Applied Linguist
cs, we now turn to four dimensions, or continua, that characterize and distinguish between them.These include (a) research versus practice; (b) more versus less systematic; (c) more versus less structured presentation; and (d) multiversus mono-modal (see Figure 2).We note that new continua and ways of describing secondary research many be needed as new types of secondary research emerge.


The 'research-versus practice-focused' continuum

Unsurprisingly, the objective of the

jority of secondary research types is researchori
nted.Purely research-focused reviews can be used to summarize the current state of research or aggregate research evidence (e.g., state-of-the-art review), to explore emerging research areas (e.g., scoping review), to challenge mainstream perspectives (e.g., critical review), to address methodological issues in research (e.g., methodological synthesis), or to suggest new research directions (e.g., research agenda).On the other hand, two types of secondary research, systematic literature review and qualitative research synthesis, can address both research-and practicerelated issues.For instance, in their qualitative research synthesis, Chong and Reinders (2020) draw implications from the synthesized research evidence to inform technology-mediated task-based language teaching research and practice.As for systematic literature reviews address and express caution around the use of course grades as metrics to assess language proficiency levels in classrooms and in L2 research.Amongst the 13 types of secondary research identified previously, there is only one type which focuses exclusively on pedagogical practices: research into practice.'Research into practice' reviews concern the extent to which findings from classroom-based studies are applicable to authentic educational contexts.In particular, reviewers are interested in underscoring areas of research findings which are under-, over-, and possibly mis-applied.

There is an unequal focus of secondary work in Applied Linguistics on research and researchers as the sole interest and audience, respectively, as opposed to teaching and teachers, for example.Currently, there is a dearth of reviews which focus on practice, as exemplified from the fact that 'research into practice' reviews can only be found with regularity in one journal (i.e., Language Teaching).While it is the primary purpose of most secondary research to target researchers, it is meaningful to explore review types which focus primarily on informing the practice of language teaching and learning as well as other practical realms within applied linguistics such as assessment (Chong 2020).In addition to evaluating the success of knowledge transfer from academia to the professional context, practice-focused secondary research can focus on synthesizing data from naturalistic, classroombased studies to showcase evidence-based practices.Moreover, it is important for synthesists to specify the purpose of their reviews.The connection between the purported goal of the reviews and the review process should be stated explicitly.'Meta-reviews', or 'reviews of reviews' can be conducted to examine the alignment between the objectives in each type of secondary research.Moreover, investigations can be carried out on the (mis)alignment between the purpose of research syntheses and their review processes.


The 'more systematic versus less systematic' continuum

Regarding systematicity of the review proces

the reviewed examples show that there is a fairly even
distribution of the 13 types of research synthesis on the continuum.Five types of secondary research adopt a systematic approach and traditional (non-empirical) approach to reviewing, respectively (meta-analysis, systematic literature review, qualitative research synthesis, scoping review, methodological review).Moreover, three types (mixed review, narrative review, the-art review) utilize a mixture of systematic and narrative review strategies.Research syntheses which embrace the more systematic review process include a discrete 'methodology' section detailing the various stages of the review process namely literature search strategy, inclusion and exclusion criteria, a coding scheme for data extraction/collection, and data analysis.To reduce bias and increase credibility, these systematic research syntheses are transparent with their coding procedure, often sharing their coding scheme as supplementary materials or appendices (e.g., Chen 2016), calculating inter-coder reliability (e.g., Larsson et al. 2020), and describing strategies to resolve disagreement in the synthesis and analysis stage (e.g., resolving disagreement through discussions in Chong and Reinders 2020;Liu and Chong 2023, in this special issue).By contrast, the three types of traditional secondary research (critical review, research into practice, research agenda), tend to employ a less systematic process.Resembling the 'commentary' style of secondary research, these research syntheses do not describe how the review process is conducted nor do they provide justifications for the adoption of a particular review protocol.

Based on the typology in Table 1, another observation is the ambiguity of the review processes of some types of sec ndary research.With regards to the aforementioned examples, mixed reviews, narrative review, and state-of-the-art review do not demonstrate a consistent take on the systematicity of the process.While we can understand that the systematicity of mixed reviews is contingent on the mixture of the review types, it raises concerns regarding the stark contrast in the methods used in the paired examples of the other two review types.Without a consensual view towards review methodology, we are likely to encounter challenges for researchers who want to conduct such kinds of review.It is imperative to conduct comprehensive reviews on the methodologies used in each type of secondary research and the variation thereof as a means to bring consistency to the field.It would be fruitful to review methodologies of (a) earlier and more recent secondary research; (b) secondary research published in top-tiered and less prestigious journals/outlets; (c) written by researchers at various career stages, and (d) published in different sub-domains of Applied Linguistics.Despite being a strenuous task, such work would yield valuable insights into the differentiation between 'ideal', 'acceptable', and 'unacceptable' review protocols of each type of secondary research.(For an example of an evaluation of meta-analytic techniques in applied linguistics, see Plonsky and Talip 2015).

Secondly, three types of secondary research adopt a less systematic review process (critical review, research agenda, rese rch into practice), relying mainly on the authors' expertise and judgement to identify important themes and studies as examples.We recommend a dose of caution when interpreting such reviews because less systematicity can lead to inconsistency, idiosyncrasy, and bias (not that systematicity eliminates bias).The majority of these reviews do not include an explanation of how themes are generated or how studies are chosen.Although credibility of these reviews may not be an issue when they are written by leading scholars in the field who offer an 'authoritative stance', it is important to minimize bias by providing a brief statement on the reflective process of the reviewers.Hanks (2019) is a commendable example in this respect.This review, which adopts a "dialogic approach", makes clear that the review process is rigorous despite not having "a third-party stance" (p.145).The author acknowledges a personal stance and explains how reviewer perspectives are co-constructed through interactions with researchers, professionals, and doctoral students.


3.3

The 'more standardized structure versus less standardized structure' continuum

From a genre analysis perspective, some sec

dary
research types adhere to a more standardized structure while structures of others are more flexible.The kinds of secondary research which follow a more standardized reporting structure are also those which implement a more systematic review protocol, including meta-analysis, systematic literature review, qualitative research synthesis, methodological synthesis, bibliometric review, and scoping review.Their standardized organizational approach comprises six structural components: introduction, background literature, method, findings, discussion, implications/conclusion, in parallel to a typical primary research paper.These types of research synthesis often include additional online supplementary information such as coding schemes and bibliographical summaries of the synthesized studies.Secondary research types which adopt a less standardized structure are those which employ a less systematic and rigorous review procedure (i.e., critical review, research into practice, research agenda).These types of secondary research, which narrate the synthesized findings through a thematic approach, organize the review text based on prominent themes.An interesting observation emanating from this comparison is found in the two examples of narrative review.Traditionally, a narrative review is viewed as a type of secondary research that relies on a less systematic synthesis process; nevertheless, one narrative review mentioned previously follows a primary research paper structure (Thomson and Derwing 2015), whilst the other uses a customized thematic structure (DeKeyser and Botana 2015).This example, which demonstrates a lack of consensus of a preferred structure for narrative reviews, is made even more intriguing as these two reviews were published in the same issue of the same journal (i.e., Applied Linguistics).While the discussion of only two examples makes it impossible to draw any definitive conclusion, it suggests that the two sets of authors of these review articles may have different perceptions or conceptualizations towards 'narrative review', with Thomson and Derwing (2015) leaning more towards the "synthesis as research" side while DeKeyser and Botana (2015) regard narrative review as an authoritative commentary.

The major takeaway, pertaining to the structural organization of secondary research, is quite apparent: Reviews that adopt a more systema ic and scientific review protocol are represented in a more standardized structure while those that are written in a 'commentary' style are more flexible with their structures.From the examples drawn from the typology, the only type of secondary research that shows disagreement in terms of structural approach is narrative review.To further understand structural variations of each type of secondary research, genre analysis can be conducted to identify the "cognitive structuring" (Garzone 2015, p. 6) or "schematic structure of the discourse" (Swales 1990, p. 58) used to convey communicative purposes in each research synthesis type.


The 'multimodal text versus monomodal text' continuum

Referring to the analytical framework (see Figure 2), the representation of the rev

w text is another dimension of variation across review
types.Over half of the secondary research types adopt multimodal representation, using a combination of text, figures, and tables.Among these types of secondary research, six (meta-analysis, systematic literature review, qualitative research synthesis, scoping review, methodological synthesis, bibliometric review) adopt systematic review protocols.To present large sums of information in a clear and systematic manner, these reviews utilize tables and figures to document the steps of the review and collate bibliographical, substantive, and methodological information from the selected studies (e.g., Chen 2016).By contrast, the four types of text which generally rely on monomodal text (i.e., narrative review, critical review, research into practice, research agenda) represent the types of secondary research that follow a more flexible mechanism of review.

Similar to the previous continuum, the decision on whether to utilize different modes to present secondary results seems to be associated with he systematicity of the review.For more systematic types of reviews, figures and tables are usually used to present methodological flow and underscore synthesized findings.Bullet points are sometimes used to list factual information (e.g., the list of journals consulted, the list of search terms used, inclusion/exclusion criteria).Future research on secondary research practices might look into the "science of using science", which refers to the study of "the efficacy of interventions applied to increase decision-makers' use of research in various decision arenas" (Laurenz et al. 2016, p. 1).One form of intervention to enhance evidence-informed decision making that can be investigated is the mode of presentation of synthesized research evidence (e.g., the use of infographical representation of synthesized findings to facilitate use of research findings by practitioners).


Conclusion

This article provided an overview of the common types of secondary research in the field of Applied Linguistics (see Table 2 for a sum

ry).The mai
objective of this piece is to demonstrate the similarities and differences among types of secondary research through four perspectives: topic (research/practice focus), review process (systematic/traditional), structure (standardized/customized), and representation of text (textual/multimodal).The typology presented will be useful to not only novice reviewers as a primer for what is possible using secondary methods, but also seasoned synthesists to explore different secondary research orientations and options.Most importantly, we hope that this typology can serve as a catalyst to facilitate discussions on secondary and synthetic research methodologies and, ultimately, lead to the formulation of methodological guidelines for each type of secondary research for our own field.

Figure 2 :
2
Figure 2: Four areas for analyzing secondary research in applied linguistics.




).More importantly,
Literaturereview/SecondaryresearchT