The Classification of Q1 SJR-Ranked Library and Information Science Journals by an AI-driven “Suspected Predatory” Journal Classifier

1 Why do Academics Need to Screen Journals to Assess their Scholarly Nature?

It is generally understood that academics would want to select the best journals in which to represent their work. For many reasons (e.g., insufficiently robust analysis, small sample sizes or insufficient replicates, poor experimental design, insufficient novelty), it might not always be possible for an academic to publish in their desired target journal, so they might seek alternative journals in which to publish their work, particularly in response to one or more desk rejections (Dwivedi et al., 2022). Since competition for space in journals can be strong and given that high barriers to entry exist in some high-ranked journals, it is not uncommon for some authors to experience multiple desk rejections among such journals before their work eventually finds its way into a lower-ranked journal, which may exercise lower standards of screening, have a higher level of acceptance, or a unique editorial view on the work, thereby allowing publication. In some cases, academics might select journals that do not exercise sufficiently stringent academic standards (Frandsen, 2019; Mertkan, Aliusta, & Suphi, 2021). Even though the assessment of work based the perceived “quality” of research, as reflected in the journal’s impact factor or similar journal-based metrics (JBMs) carries flaws, and should thus not be used in official decision-making (Paulus, Cruz, & Krach, 2018), researchers nonetheless try to publish in highest ranked journals, typically first quartile (Q1) journals, because they might receive more benefits such as funding, than if they were to publish in lower-ranked journals (e.g., in Brazil; McManus & Baeta Neves, 2021), and because papers in such journals tend to gather more citations and/or attention by the peer community (Miranda & Garcia-Carpintero, 2019). Even though researchers should be free to select their target journal of choice, their decision is often determined by their employer’s (e.g., research institute or university) objectives and prerogatives, which tend to place pressure on publication in higher-ranked journals, thereby stimulating the publish-or-perish culture (Grimes, Bauch, & Ioannidis, 2018). That choice might also be biased toward open access (OA) journals, if the perception is that there is a citation advantage (Bautista-Puig, Lopez-Illescas, de Moya-Anegon, Guerrero-Bote, & Moed, 2020). This article does not aim to resolve these issues.

Jeffrey Beall, a US librarian, established a blog with blacklists of what he perceived to be “predatory” OA journals that took advantage of this publishing model to generate as much revenue as possible (Beall, 2017). However, those lists have become outdated, following a closure of his blog in 2017, and due to reliability issues, researchers should refrain from using or relying on them for advice or other academic purposes, including the selection of venues in which to publish (or not) their work (Teixeira da Silva & Kendall, 2023a). Despite those efforts, there is little clarity about what a “predatory journal” actually is, and since many facets may enter the realm of deception (Eriksson & Helgesson, 2017), journals in a “gray” zone of scholarly behavior display borderline activity that lies between predation and legitimacy (IAP, 2022; Siler, 2020).

2 Are SJR Q1 Journals Likely to Be “Predatory”? A Generalist’s View

There is ample discussion in the academic literature arguing that the use of JBMs to assess journal quality is fraught with limitations and that more factors need to be benchmarked when deciding the choice of publishing venue (Björk & Holmström, 2006). The topics of JBMs, ranking, indexing, and journal quality are often interlinked and are central to academic performance and librarianship, the former because academics are judged and rewarded based on the venues where they publish, the latter because a journal’s prestige and reputation depend on reliable bibliometric classification, coverage, and ranking in order to attract prospective authors. Coverage of journals among different journal ranking lists (JRLs) can differ, so some academics have attempted to find ways to appreciate whether journal ranking across such lists is consistent or overlapping (Anderson, Elliott, & Callahan, 2021; Jaafar, Pereira, Saab, & El-Kassar, 2021). Cognizant of a wealth of literature on such comparisons, this article focuses on only one such JRL, the SCImago Journal Rank (SJR), which draws information about journals that are indexed in Elsevier’s Scopus, which itself is a curated database (Baas, Schotten, Plume, Côté, & Karimi, 2020), and one of the largest and most prominent databases, although it is neither free of errors nor perfect (Pranckutė, 2021). Moreover, some of the lower-ranked journals may exercise poor scientific standards (Frandsen, 2017). There is a high (>99%) overlap between Scopus-indexed journals and those that are indexed in the Web of Science (Singh, Singh, Karmakar, Leta, & Mayr, 2021). Given its reliability, generally speaking, SJR is used for rewarding researchers’ academic achievement (Garner, Hirsch, Albuquerque, & Fargen, 2018).

SJR is a JRL that ranks journals based on citations (Brown & Gutman, 2019), then classifies them into four groups or quartiles, for example, Q1 represents the top 25% or quartile (Ahmad et al., 2019). If one considers an unscholarly journal, i.e., one that does not apply scholarly principles to its publishing model, then it can be argued that Q1-ranked journals in SJR are highly likely to contain less such journals than Q4 journals, if any at all, by virtue of the fact that a double-filter system of quality control is in place: the first by the ranked journal itself, the second by an independent team of experts^[1] who monitor, filter, and evaluate journals for inclusion in SJR (Sonntag, 2023). Seasoned academics who publish widely and who have experience with journal content and even a journal’s handling of papers, such as peer review, might broadly agree that Q1 journals tend be more academically stringent than Q4 journals because they might exercise more quality control, although this is a loose generalization because listing and ranking in Q1–Q4 is based on metrics such as citations, and not on quality control. However, as a result, robustly published science papers may result in more citations, thereby feeding this cycle of quality = rank analogy. Not always, as evidenced by the citation (sometimes highly) of retracted and invalidated papers (Frandsen, 2017), but for the purpose of this discussion, and for simplicity’s sake, the rank of a journal (e.g., in SJR) is considered to be proportional to its level of quality control.

The SJR list related to library and information science (LIS) journals (SJR, 2023) is a list of reasonably reliable – in the author’s opinion, and based on experience and indicators in the literature – scholarly LIS journals that academics can submit to, and that tend to exercise acceptable – even high – standards of rigorous screening and peer review. Stated in another way, seasoned academics would probably not consider Q1 LIS journals indexed in SJR to be “predatory.” Bibliometric performance of a set of journals blacklisted by Beall was lower than an SJR-selected set of journals that was not blacklisted (Moed, Lopez-Illescas, Guerrero-Bote, & de Moya-Anegon, 2022). Can LIS researchers reliably detect a predatory from a non-predatory LIS journal?

Academics’ experiences and motivations may differ considerably (Mills & Inouye, 2021), so there is a need to remove human-based subjectivity when relying on decision-making related to journal quality (Yamada & Teixeira da Silva, 2022), and one potentially effective way to achieve this would be to rely on artificial intelligence (AI). However, to the author’s knowledge, only four studies have proposed AI to resolve the issue of predatory publishing: Adnan et al. (2018), Ateeq and Al-Khalifa (2023), Bedmutha and Bedmutha (2022), Chen, Su, Liao, Wong, and Yuan (2023), and Gaffney and Townsend (2022). This article focuses on the latter study.

3 Is Academic Journal Predatory Checking (AJPC) system an Effective AI Tool for Differentiating “Predatory” Journals?

In February of 2023, a ground-breaking paper published in Springer Nature’s Scientific Reports (#11 Q1 journal in SJR’s Multidisciplinary field) announced the development of a simple AI-based tool, the AJPC system, which the authors claimed was able to effectively distinguish suspected predatory journals (SPJs) from normal journals (NJs) (Chen et al., 2023). AJPC system is indicated by Chen et al. as functioning by screening a journal’s website for language-based signals, verifying such listing against two blacklists or watchlists (Beall’s Lists and a derivative of Beall’s Lists) and/or one whitelist or safelist (BIH Quest), classifying it as either an NJ or an SPJ. Prospective authors can verify this classification at AJPC system, via a one-click step after pasting the URL of their target journal into an online prompt. This tool is free, and open, so it is an attractive solution. In recent years, academics have relied heavily on two watchlists, Beall’s Lists^[2] and Cabells’ Predatory Reports,^[3] the former free and the latter pay-walled, but both blacklists have sensitivity issues caused by some unclear criteria, leading some journals or publishers to be likely incorrectly classified as “predatory,” or vice versa (Teixeira da Silva et al., 2022; Teixeira da Silva, Moradzadeh, Yamada, Dunleavy, & Tsigaris, 2023a). For this reason, AJPC system would be a practical solution for academics around the globe to identify predatory journals, provided it is able to do so effectively.

As part of an ongoing exercise to assess the reliability of this AI-driven tool, AJPC system was put to the test, using a methodology that was used by Teixeira da Silva, Tsigaris, and Moussa (2023b). In the past few months, three important tests and findings were made and published. First, the classification of Scientific Reports, the journal in which the introduction of AJPC system by Chen et al. (2023) was published, was modified from an SPJ to an NJ within about 2 weeks (Teixeira da Silva & Daly, 2023). Second, a web-scraping approach was used to assess about 17,000 journals, including 2500 SJR-ranked journals, and also 15,000 prominent journals of leading (in terms of market size, journal volume, and/or academic status, standing or prominence) publishers (Elsevier, Frontiers, MDPI, Springer Nature, Taylor & Francis, Wiley), finding, among some of the most incredulous results, that AJPC system classified 100% of Elsevier journals (n = 4,756) as SPJs (Teixeira da Silva & Kendall, 2023b). In that study, using OMICS as an outsider, and given its classification as a predatory publisher in a US court of law (Manley, 2019), it was discovered that about 7% of 705 journals were classified as NJs. Third, using a similar approach followed in this article, but testing the famous Financial Times top 50 (FT50) journals in fields such as financing, marketing, and other related fields, an incredulous 88% were classified as SPJs (Teixeira da Silva et al., 2023b), despite knowledge that their quality control measures are robust (Fassin, 2021).

4 AJPC system Put to the Test: Q1 SJR LIS Journals

The Q1-ranked LIS journals in SJR (SJR, 2023) – 64 in total – were selected, in the belief that these represent the crème-de-la-crème of LIS journals. Eventually, it is reasonable to expect that one or two journals that might not exercise pristine scholarly principles might be identified, as was argued earlier, but overall, the expectation was that none of these top Q1-ranked journals would be considered as SPJs. Departing from this belief, URLs of the top pages of all 64 journals were manually entered into AJPC system, to appreciate the outcome, and then verified again on 2 June 2023 (Suppl. file). Surprisingly, 27 (or 42%) of these 64 Q1 LIS journals were classified as SPJs, including some journals that, to the author’s knowledge and/or experience, have very rigorous peer review and editorial policies in place, aspects that would not typically be associated with “predatory” behavior. AJPC system indicates that most (13/27, or 48%) of these SPJs are published by Taylor & Francis.

What then might explain this highly unlikely output? At least three possibilities might explain this observation: (1) an inaccurate tool (AJPC system) that is not able to clearly distinguish an NJ from an SPJ because it relies on a fundamentally flawed (inaccurate) watchlist, Beall’s Lists; (2) an outdated dataset (the AJPC system website indicates that trained data reflect a 11 November 2020 date; Figure 1a and b); (3) an over-simplistic binary output (NJ or SPJ) that does not accompany that classification with any list of indicators or properties that could inform users of the scholarly characteristics that led it to be classified as an NJ, or, more importantly, the unscholarly characteristics that led it to be classified as an SPJ; (4) the lack of criteria used to differentiate an SPJ from an NJ; (5) the relative lack of open and public accountability by the authors and Scientific Reports to resolve these discrepancies; and (6) poorly trained AI with erroneous input parameters or data.

Figure 1

AJPC system classified Wiley’s Journal of the Association for Information Science and Technology, a Q1 ranked 27th by the SJR (Supplementary file) as an SPJ on 18 March 2023 (a), but as an NJ on 2 June 2023 (b). What happened in the interim? The author submitted a report to Journal of the Association for Information Science and Technology. It is possible (conceivable) that the journal challenged its classification as an SPJ, sharing the data submitted by the author to that journal, and that Chen et al. modified the code and underlying data so that an NJ classification would be reflected sometime between March and June 2023. No public explanation for this change can be found.

Of potentially greater concern is the change in classification of journals that were once (March 2023) classified as SPJs to NJs (by June 2023) (see one example in Figure 1). Although in March 2023, the author initially analyzed the AJPC system classification of only 50 of the 64 Q1 SJR-ranked LIS journals, at that time, 54% of the journals were classified as SPJs, including the top six. Comparing the 50 classifications made on March 2023 (reflecting a 2021 SJR rank) with an overlap in the June 2023 classifications (reflecting a 2022 SJR rank), 18 journals’ classification as an SPJ changed to NJ, while in eight cases the classification changed from NJ to SPJ (see yellow and blue entries, respectively, in second spreadsheet in Suppl. file). Chen et al. provide no public explanation for these changes.

Did any journal or publisher put pressure on Chen et al. to modify AJPC system’s classification?

5 Is AJPC system a Viable Assistant for Differentiating Scholarly from Predatory Journals?

In the world of academic publishing, the risk of a journal’s misclassification can cause heavy – and in extreme cases, irreversible – reputational damage, both to the journal that is classified as such, as well as to academics that may select it for publishing their work and who then receive benefits based on their publishing performance (Hulsey et al., 2023). Consequently, if a journal is characterized as “predatory” (or in this case, as an SPJ) when it is not (in this case, an NJ), this could “cost” the journal and publisher if the tool (in this case, AJPC system) becomes widely used. Research institutes and university authorities need to thus rethink their reward policies for publishing, offering better guidance to their researchers, especially students and early career researchers, and ensure that those who publish do so in journals that truly follow best publishing practices (Khan, Vieira Armond, Ghannad, & Moher, 2022). LIS researchers, following such advice, and confident in their perception that it is safe to submit to a Q1 SJR-ranked LIS journal (SJR, 2023), but turning to AJPC system to verify their choice, might be confused and surprised to learn that 42% of them are classified as SPJs. In the absence of any additional explanations by AJPC system, Chen et al. (2023), or Scientific Reports, and not wishing to take any reputational risks, they might then turn to another Q1 SJR-ranked LIS journal that is, using the same two-step process, classified as an NJ by AJPC system. In this example, LIS researchers would likely have been misled by AJPC system. Moreover, these top Q1 SJR-ranked LIS journals might lose an opportunity of peer reviewing and publishing these researchers’ papers. For the publishers of these journals, the loss of each paper is not only the loss of indexed intellect and copyright, but also in the case of gold OA journals, the loss of an article processing cost (APC). Cumulatively, if AJPC system gains traction over time and becomes a popular AI tool among students and seasoned academics to screen journals to determine if they are safe to publish in, this might cause reputational damage to those journals that are classified as SPJs, when they are not. In addition, assuming that a substantial mass of papers might be “lost” due to failed submissions caused by hesitation, this might “cost” the publisher subscriptions and APCs, i.e., both intellectual and financial costs. As one concrete example, how many authors have the Taylor & Francis LIS journals, the majority stake-holder of AJPC system SPJ classification, lost due to authors who might have turned away and submitted to another journal that is classified as an NJ?

6 Conclusion, Limitations, and Suggestions for Improvement

The classification of the majority of Q1 SJR-ranked LIS journals as SPJs is likely a seriously erroneous classification error. The creators of AJPC system (Chen et al., 2023) and Scientific Reports need to address these odd classifications urgently and publicly, explaining what characteristics led them to classify all journals classified as SPJs as such. There also needs to be some clarification as to whether AJPC system was trained on Beall’s Lists, which were focused exclusively on OA (Krawczyk & Kulczycki, 2021). In the absence of journal-by-journal evidence (which currently does not exist and is not provided by Chen et al.), AJPC system cannot – in its current state of development – be considered to be a trustworthy AI-based tool and is destined to be relegated to the history books, similar to Beall’s Lists, unless the six aforementioned limitations are fully addressed. Any output by AJPC system should not be accompanied by a simplistic binary NJ/SJR output, but also supplemented by a list of scholarly and unscholarly characteristics in journals that led to their SJR classification, because predation is not binary, but rather a gradient of negative characteristics (IAP, 2022; Yamada & Teixeira da Silva, 2022). In the absence of accountability, were AJPC system to become a popular or widely used online tool for academics to screen journals before submission, allowing them to discern NJs from SPJs, and then make a selection for manuscript submission based on AJPC system’s classification, this may threaten the reputation of truly scholarly LIS journals, throwing them into disrepute. There is a silver lining, however: AI can be trained if accurate information is fed to it by its human managers, so AJPC system could still become a promising tool to assist academics in journal selection, provided that the underlying data that have led to the classification of a journal as an NJ or an SPJ are open and public. Although Chen et al. provide the code in OA at GitHub, what data have changed and the dates when changes have been made to reflect manipulated classifications (Figure 1a and b; Teixeira da Silva & Daly, 2023; Yamada & Teixeira da Silva, 2023) are neither clear nor open.

Of course there is one additional (less plausible, in the author’s opinion, and based on the lack of published evidence) possibility, namely that AJPC system’s assessment is accurate while that of SJR is not, i.e., that there are in fact predatory journals in the ranks of SJR-ranked journals. However, this possibility is slim. Even Chen et al. recognize that AJPC system has flaws and errors, and is limited to knowledge known until the end of 2020, i.e., almost 3 years outdated.^[4]

This article only presents findings of SJR’s Q1 journals for the LIS category. Interested researchers are encouraged to expand the analysis to encompass Q2–Q4, keeping in mind that as Q increases, scholarly quality or editorial scrutiny might decrease. Despite the relative simplicity of this analysis, journals in other fields of study also need to be tested and evidence (i.e., the AJPC system output) needs to be archived via screenshots (e.g., Figure 1) because the output cannot be archived in the Internet Archive. Those findings should be recorded and published in academic journals for posterior appreciation of how the AJPC system has fared over time.