The Stories We Live By and the stories we won’t stand by: Measuring the impact of a free online course in ecolinguistics

1 Introduction

The advent of the Internet has brought new ways in which individuals engage with learning and resources. It is common nowadays to find a plethora of online courses and blended learning opportunities across disciplines whereby participants enjoy the benefits of both synchronous and asynchronous teaching methods. Online courses have become very popular within higher education environments to reduce the carbon footprint that is naturally created with face-to-face modes only (Castle and McGuire 2010) and have become a necessity during the COVID-19 pandemic (e.g. Adnan and Awar 2020; Bao 2020). While these practices have expanded, measuring the impact of digital resources has proven to be a difficult task, particularly as these incorporate academic and non-academic impacts. Traditionally, impact has been measured by means of metrics relating to journal citations and books. However, bibliometrics that rely on the journal impact factor (IF) and the h-index (Hirsch 2005) provide only a misrepresentation of the data due to the lack of normalization (Bornmann and Marx 2013). Moreover, Penfield et al. (2014) point out five challenges when undertaking research impact assessment, namely, time lag, the developmental nature of impact, attribution, knowledge creep, and gathering evidence. The time frame to assess impact may be insufficient in some cases as some impacts may take longer to appear. This closely links to the following challenge: impact is not static and there may be temporal and dissipated impacts which should therefore be considered in the broader picture. Another crucial aspect is attribution: since research usually involves various actors and organizations it is hard to link those who were crucial to the creation of a specific impact. Closely related to the latter is knowledge creep, i.e. when new data informs existing research, a practice that makes attribution problematic. The last challenge finally relates to gathering evidence: the problem lies in collating evidence that may no longer be available or may be non-existent and having the tools that adequately measure and capture the data.

Traditional tools of assessment for impact in the humanities, such as citations, appear to be an inaccurate measure for digital resources (Hughes et al. 2015). Scholars have long reflected on the challenges that new ways of acquiring knowledge pose on impact assessment. The Oxford Internet Institute developed in 2011 the Toolkit for the Impact of Digital Scholarly Resources (TIDSR), a notable example of the efforts to contribute to new methods of impact assessment. The TIDSR describes a variety of broad-based quantitative and detailed qualitative methods that assist in providing a broader picture of the array of impacts (Meyer 2011). Quantitative measures include but are not limited to website analytics, surveys, and webometrics while qualitative measures include referrer analyses, interviews, feedback, and focus groups. Combining different methods will depend on the size of the corpus and the data that is available. By drawing on a balanced amount of qualitative and quantitative evidence it is possible to illustrate the impact of research on various levels. Since digital resources are usually specialized and catering for a smaller audience, the TIDSR highlights the need for using a wide range of methods, as relying solely on one will prove insufficient.

In the context of the United Kingdom, the Research Excellence Framework (REF) is responsible for setting out a current definition of impact and how this should be assessed within academic institutions. The appraisal of non-academic impacts is crucial for the allocation of funding in the universities, a topic which has not been exempt from criticism. Currently impact is defined in the REF (2021: 81) as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.” Nonetheless, institutions and researchers understand impact in different ways, which is reflected in the range of pathways to impact provided in the REF 2014 case studies (Terämä et al. 2016). In a REF submission, academics are required to submit a qualitative impact case study which can provide evidence of the types of impact and external sources that can substantiate these impacts. An impact case study is divided into five sections: (a) summary of the impact; (b) a description of the underpinning research; (c) references; (d) details of the impact; and (e) sources that corroborate the impact. The criteria for assessment is reach, that is, how spread the research effect has been on the intended beneficiaries, and significance, the overall influence. Ravenscroft et al. (2017) highlight a number of issues with this relatively straightforward system. Since the submissions are only three to five pages long, they usually feature only a small number of high impact projects, which is detrimental for academics contributing small amounts in various projects. Interdisciplinary submissions are considered within the REF but the depth of the assessment alongside the inherent qualitative nature of the procedures question the system’s robustness, particularly in terms of the assessment of digital components. In addition, a report from King’s College London looking at REF 2014 submissions found that quantitative data used to support impact were not consistent and presented in diverse ways, which reveals the need of standardizing numerical values if impact metrics are to be developed and put in place at some point (King’s College London and Digital Science 2015).

In an attempt to compensate this, digital humanities scholars at King’s College London developed a checklist to help scholars prepare their digital research outputs for submission to the REF 2021 (Ciula 2019). The questions range from work attribution, accessibility, user experience, look & feel and citability, amongst other items in the checklist. This list aims to provide the preliminary non-subject specific criteria to assess whether a digital output is suitable for a REF 2021 submission.

As for impact case studies in the field of linguistics, the search in the REF 2014 shows 106 case studies ranging from explorations in foreign language teaching to language preservation. Within ecolinguistics, impact assessment appears sparser with only a few instances of case studies highlighting the impact of Arran Stibbe’s pioneering research (REF 2014). The joint work between the University of Maribor and the University of Gloucestershire in conjunction with universities in Italy and Turkey for the “Me We Whole” project also account for the impact of the teaching pack Living in the weatherworld: Reconnection as a path to sustainability. The impacts of the materials showed to influence teaching practices as well as individuals working in tourism who incorporated the materials for their daily activities.

In this paper, we investigate how the free online course in ecolinguistics The Stories We Live By (hereafter TSWLB) has impacted the everyday habits of those who completed it in terms of both their thinking and their behavior. The emphasis on impacts that go beyond the mere academic scope is crucial for determining how the online course TSWLB can account as a contribution to the expansion of the field of ecolinguistics not only in terms of its development within academia but also across the general public. As an online, digital tool of self-learning, TSWLB fits within the realm of the digital humanities, an aspect which must be considered when engaging with impact assessment for the reasons previously set out. The data stems from an independent consultancy project for the University of Gloucestershire (see Roccia 2020 for the final report). A brief introduction of the course, its content and its aims are outlined in Section 2. This is then followed by a description of the methods used to gather data and the coding system utilized for analyses purposes (see Section 3); these are semi-structured interviews, anonymous comments left on TSWLB website, and feedback received by students when submitting their certificate request form. An overview of the themes emerged as results of the analysis is presented in Section 4 and a discussion of these is provided in Section 5 in which we argue for the benefits of using this mixed-method design to assess impacts in the digital humanities without forgetting about its limitations. The article then concludes with a brief proposition for a quantitative method to standardize the discussion across impact studies in the digital humanities (see Section 6).

2 The Stories We Live By: A free online course in ecolinguistics

The Stories We Live By: A free online course in ecolinguistics (hereafter “the course”) is, as the name would suggest, an online course open to all for free with accreditation from the University of Gloucestershire in the form of a certificate upon successful completion. Students are required to register on the website^[1] by creating a free account which will give them access to all materials and resources, as well as to discussion groups and to the support of voluntary tutors who can provide advice and mentoring in 12 different languages. The course can be completed in whatever order and time the student wishes to and, upon completion, the student can fill in a feedback form to request their certificate. Although the completion of this course does not provide university credits, the certificate is nonetheless an official document which is fully accredited by the University of Gloucestershire in the United Kingdom.

The course reflects both in content and structure Arran Stibbe’s Ecolinguistics: Language, ecology and the stories we live by (2015) in which the author delves into the one fundamental aspect of human existence that is shared across all human beings and therefore transcends social constructs such as race, gender, and religion. That is the need to make sense of our world which is accomplished by the creation of mental models – stories. When these stories are shared by a community of people, and therefore across an entire culture, Stibbe argues that they become stories-we-live-by.

With his book and this course, Stibbe aims to provide a solid linguistic methodological framework through which discourse can be analyzed. By doing this, Stibbe is one of the first scholars to combine ecology and linguistics into one discipline. Indeed, the central theme that runs through the course is one concerned with how language shapes the relationships between humans and the natural world and what linguistics can do to contribute to their reconnection. Throughout the course a wide range of real-world examples are provided in order to concretize abstract concepts. These examples are taken from Stibbe (2015) in which the author analyses data selected from textbooks of economics, as well as machinery handbooks, ecological reports, and nature writing texts. Stibbe’s analysis focuses on the linguistic features found throughout the texts in terms of, for example, lexical choices, passive and active grammatical structures, and intertextuality. When combined together in one text, these linguistic features constitute the main message, or indeed ideology, implied behind the semantic interface of sentences. Stibbe goes one step further and overtly exposes these stories by questioning them and by offering alternative ones, of positive nature. For the purpose of questioning a text in terms of whether it encourages people to care for the world or, instead, damage it, a set of ethical guidelines have to be established. These guidelines will constitute the basis of the framework within which the analysis can be conducted; they will be decided by the author of the analysis, based on their ethical values – their ecosophy. Discourse can then be evaluated as being beneficial, ambivalent, or destructive, depending on the degree with which they promote sustainable practices of interconnections between humans and the rest of life on Earth.

Overall, the course offers a model through which discourse can be analyzed with the purpose to reveal the eight types of story described by Stibbe (2015): ideologies, framings, metaphors, evaluations, identities, convictions, erasure, and salience. With “Introduction” forming part 1 of the course, the types of stories constitute the rest of course as namely, part 2 to 9. Ideologies can be represented in various ways and whenever a specific choice of vocabulary is used to trigger a particular concept, Stibbe refers to this as “framing”. Stating that sustainability should be reframed as a supply chain challenge is an example of this (Stibbe 2015: 26). Metaphors are a specific type of framing whereby an abstract area of life is very vividly conceptualized by applying properties of other more concrete areas of life: “climate change is a global-scale violence” (Stibbe 2015: 65). Furthermore, the common idea, in British culture, that rainy weather is bad is an example of how stories can be evaluated by groups of people (hence the term “evaluation”) either positively or negatively. Similarly, stories in people’s minds about the truthfulness of an area of life are defined as “convictions” (Stibbe 2015: 129). Finally, stories can be about “identities”, or, more explicitly, about being a specific kind of person, such as a consumer, a manager, and so on.

The connection between ideology, framing, identity, evaluation, and conviction and the way people’s behavior can be manipulated is evident. Whether these consequences are worthy of consideration in a discourse, constitutes two different kind of stories, namely, “erasure” and “salience”. While “erasure” is a story that renders an area of life completely unimportant (such as erasing the individuality of fish by referring to them as “fish catch”), “salience” has the opposite effect. Indeed, it is stories of salience that can provide beneficial alternatives of reconnection and appreciation of the natural world, such as in the works of new nature writing and Japanese poetry.

Through this online course, it is therefore possible to further increment the chances of making an impact in the lives of people who would not ordinarily read academic texts. The course presently counts more than 1,500 subscribers from six continents, and different backgrounds. Of particular interest is the incorporation of the course into an English Language and Translation undergraduate programme at the University of Cagliari, Italy (UNICA 2021). Due to the current pandemic and in an attempt to compensate for the lack of face-to-face interaction, the course was added as extra reading materials. However, the course proved very popular among the students and the programme leader has now decided to include the course as part of the curriculum. It must be noted that even though both geographical and socio-economical representation would seem to suggest that the course has been successful in reaching a wide range of people, the data also reveals some interesting points worth further investigating. For example, in terms of education, only 6% of students have no formal higher education qualification against 94% of participants holding at least one degree (Roccia 2020). A similar gap can also be seen in geographical representation with 68% of participants residing in the United Kingdom (in first position), 38% from the USA (in second position) and 11% from China (in third position) (Roccia 2020). While the latter can be justified by factors such as the language used for the course and its materials, and the pre-existing body of interest in ecolinguistics in these territories, the data on education is more problematic. Both location and education percentages seem to show limitations which are inherent to the structure and nature of the course. These limitations are especially true for the level of education reached so far and strategies to improve these numbers may be worth designing and implementing so that having a degree in higher education does not act as a sort of gatekeeper to materials created for the benefit of all.

3 Materials and methods

Data stems from three different sources: (a) semi-structured email interviews, (b) anonymous comments on the Stories We Live By website, and (c) course completion certificates. Due to the inquisitive nature of the research design, data was analyzed following a content-driven approach aimed to identify the types of impacts that were most predominant. A coding system was devised in line with the emerging themes to classify the impacts into three high-levels: personal, work-related and other impacts; these were further sub-classified into lower levels with further subcategories. Table 1 below illustrates the coding system.

As seen in Table 1, the three main types of impact that emerged from the coding are Personal (IMP), Work-related (IMW), and Other (IMO). As reported in Roccia (2020), nearly 45% of respondents are academics, 18% are students in Higher Education and nearly 27% selected “Other” as their occupation. With a total ratio of 63% of respondents working and/or studying in an academic context and a disproportionate majority of participants with at least a Master’s degree (78%), the impacts reported seem to reflect this orientation. Indeed, work-related impacts were further divided into two subcodes, namely, “Research” (IMWR) and “Teaching” (IMWT); while IMWR contains impacts of how researchers changed their research and/or academic work as a direct result of completing the course, IMWT contains a larger variety of impacts. Some teachers/educators stated that the course impacted their teaching by bringing ecological issues to their curriculum (IMWT1), others were inspired to add a focus on the role of language in society (IMWT2) and others still commented on the changes they brought about on their resources (IMWT3). As for personal impacts, responses exhibited high levels of introspective thought on how language operates in society. Such was the fine level of introspection in participants’ responses that a more detailed sub-coding of IMP was not only justifiable but needed. Personal impacts were first subdivided into three main categories: “Thinking” (IMPT), “Communicating” (IMPC), and “Doing”(IMPD). Data then led us to further divide the latter into practical changes in behavior in everyday life (IMPD1), such as cutting down on meat consumption or being more careful with recycling, and changes of practical nature but on a larger scale, such as taking up activism and campaigning (IMPD2). In terms of impacts on a participant’s thinking behavior (IMPT), three further sub-themes emerged. Some respondents commented on changes in their awareness (IMPTA) of how language shapes society (IMPTA1) and, more generally, towards nature (IMPTA2). These were responses that did not mention changes of a practical nature nor did they indicate a change in attitude, rather they simply disclosed a sense of discovery. Therefore, these impacts were labeled as separate from changes in thinking (IMPTC). Likewise, changes in feelings (IMPTF) pertain to a shift in a participant’s emotional disposition towards specific areas of life, such as hope and/or despair for the future of the planet. Once again, these comments did not allude to changes of a practical nature nor of an attitudinal stance. Some respondents stated that the completion of the course has equipped them with communicative tools to better communicate ecological issues with others; these are labeled as IMPC. Finally, only a very few numbers of responses included impacts of other types (IMO) which for the sake of transparency have all been presented and discussed in Section 4.3.

4 Results

4.1 Personal related impacts – when “thinking” is not enough

One of the emerging themes throughout the data was the changes in thinking (IMPT-Thinking) the course TSWLB prompted in the participants. Figure 1 below illustrates all six categories within Level 2 and shows a marked tendency within the data extracted from the course completion forms and the semi-structured email interviews to highlight the importance of this impact. This is followed by a reported impact on research (IMWR-Research) within the course completion forms, and an impact on doing (IMPD-Doing) within the semi-structured email interviews. The website comments reported a marked impact on teaching (IMWT-Teaching); this is presumably due to the interest the course generated amongst language teachers. The second highest reported impact found in the website comments relates to thinking (IMPT-Thinking).

Figure 1:

Overall Level 2 by source.

Within the personal dimension, the thinking process is divided into further subcategories in Level 3: IMPTA-awareness of how language shapes society (IMPTA1) and nature (IMPTA2); IMPTC-change in thinking; IMPTF-change in feelings, i.e. hope/despair. Figure 2 below displays the reported impacts at Level 3.

Figure 2:

Total number of personal-related impacts at Level 3.

Figure 2 confirms that IMPTA1 (how language shapes society) accounts for the highest number of responses followed by an “impact on doing” IMPD1 – practical everyday changes. Two exemplary responses from each category can be found below in Semi-structured Interview (SSI) 5, IMPTA1 and Certificate (CERT) 23, IMPD1: the first is an excerpt from a semi-structured email interview and the second is from a course completion form.^[2]

SSI 5, IMPTA1: “I hope that my being more aware of ecolinguistics will help me to write with more awareness of the stories I’m telling, so that I don’t accidentally spread harmful ones in both my fiction and non-fiction writing.”

CERT 23, IMPD1: “These decisions included things related to my: diet, home environment, and (mode of) travel choices. The course has been part of an important process for me of reorienting my own life to better reflect my values rather than simply going along with the destructive values and ‘stories’ that are pervasive in so many areas of life.”

Responses like the previous examples often reflect on the language/action dichotomy, a relationship which appears crucial for ecolinguistics. The data suggests that course participants translated the newly acquired contents into more practical and environmentally attuned daily choices. This positioning strongly resonates with the often-cited gap problem that occurs in developing environmental awareness, i.e. “knowing of the impact of human behavior on the environment” (Kollmuss and Agyeman 2002: 253) and responding to it. Several models have aimed at revealing the underlying factors that shape decision making for pro-environmental behaviors, however, not all of them encompass the multiple dimensions intervening the process. Kollmuss and Agyeman (2002) argue that there is a “pro-environmental consciousness” which comprises values, attitudes, knowledge and emotional involvement within the individual’s personality traits and value system (internal factors). External factors include infrastructure, political, social and cultural factors, and economic situation. The greater pro-environmental behaviors occur when the internal and external factors cooperate with each other.

Therefore, it can be argued that the course encouraged some of the participants to align the internal and external factors to achieve pro-environmental behaviors in their daily lives; thus, bridging the gap between action/knowledge.

4.2 Work-related impacts

The second category that emerged from the data relates to impacts around work, more precisely on teaching (IMWT) and research (IMWR). Figure 3 below shows the total number of impacts at Level 3 for this category.

Figure 3:

Total work-related impacts (IMW).

The highest number of responses was ranked at the IMWR level in the category “impact on own research and academic work”. This is presumably due to the ways the course has been disseminated, mainly through academic networks which take an interest in the topic, and amongst higher education students/academics (as seen in the participants’ description in Section 2). In a large proportion, participants expressed how the course provided them with a new angle to their own work, such as in the following indicative quote:

CERT 23, IMWR: “Seeing how language and discourse could be analysed with a view toward eco-ethics in broad range of social and linguistic contexts contributed to a change in how I imagined what I could do in my research.”

Environmental issues, such as climate change, have been a concern for the ever-interdisciplinary field of Critical Discourse Analysis for quite some time now (Wodak and Chilton 2005). The previous indicative quote (CERT 23, IMWR) highlights one of the benefits of the course in terms of crossover with other social sciences. This offers an interesting approach for a wide range of academics working at the interface of language and the environment, which is further emphasized by the digital nature of the resources. In relation to this, Tanner (2015) notes in his blog that “digital humanities scholarship offers potential for evidence of interactions with a wide range of beneficiaries from our research outcomes and impact on an international scale. Digital humanities is in many ways a foster parent for change and for outcomes that resonate beyond the Academy with the wider public”. However, this perceived wider reach is particularly difficult to measure when it comes to the REF. Aside from the metrics, the lack of specific Digital Humanities Panels also adds to the complex scenario of evaluating digital humanities outputs in a large scale.

The second highest impact within work-related impacts was “IMWT3-resources”. This high occurrence corresponds to the nature of the free online course and with a high number of teachers and tutors downloading the free resources for their classes. The majority of the impacts for this category were collected from the TSWLB website as seen in the three indicative quotes below website comment WC 9, WC 10 and WC 12:

WC 9-13-07-2017 [feedback@ecoling.net]: “I’m a teacher, and this course gave me wonderful material to work in class at secondary school. Thanks a lot, from Argentina!!”

WC 10-21-08-2017 [feedback@ecoling.net]: “[…] I cannot thank you enough for these amazing resources! I am preparing to teach Poetry by Ted Hughes, from an Ecocritical theoretical perspective, for year 13 Literature coursework and your Ecolinguistics course has enabled me to structure my work scheme and given me key terms and up to the moment ideas by which to make clear what is meant by an Ecocritical perspective of literature – and helped me explain what Ted Gifford meant when he called Hughes a ‘post-pastoral eco-poet’.”

WC 12-10-5-2017 [feedback@ecoling.net]: “The course is really helpful whether you are an expert in the field or not, easy to understand and comprehensible.”

These indicative quotes reflect an interest to integrate environmental issues into the language teaching curriculum, particularly in EFL (English as a Foreign Language), to increase students’ awareness and encourage them to take action. In this regard, Hauschild et al. (2012: 3) list a number of advantages for including content-based environmental education into the language classroom: “(1) heighten students’ interest in contemporary issues that might directly influence their futures; (2) teach students how to contribute to a healthier, more sustainable world; and (3) promote language learning and meaningful communication.” However, bringing attention to these issues in the classroom imposes a challenge on educators to intuitively transmit attitudes and values which may not be explicitly considered in the formal curriculum (Cotton 2006). By questioning dominant discourses that have detrimental impacts on the environment and challenging ideological constructs, ecolinguistics offers a framework through which educators can introduce and critically analyse language use around environmental issues within their local context as seen in the following indicative quotes from a website comment (WC) and one of the course completions forms:

WC 7, IMWT2: “I am from Indonesia, a big country granted with rich bio-and-cultural diversity. This is really a new domain for me and in the long run will surely attract more and more interested parties to join in and be concerned with. More and more students of mine need to be introduced to this domain to help them widen their horizons of how language constitutes an indispensable part of their life and is inseparable from the nature they live in.”

CERT 22, IMWT1: “In my classes I have started telling the children about the activity they should do in future with respect to the society and the ecology. I showed some of the slides that I viewed here. Also, I took them to the places near our school and showed them how the environment is badly affected and that they are also responsible for the same.”

The categories IMWT2-focus on language and IMWT1-ecological issues accounted for the lowest reported impacts. This could be due to participants already working at the interface of the two categories and reporting on newly acquired methods and contents they can use in their work. Notably, the responses report an interest in encouraging others – in this case the participants’ students – to engage in a reflective practice towards preserving their local environment and becoming global citizens (Jacobs and Cates 1999).

5 Discussion

The purpose of this paper was twofold: firstly, it sought to assess the impact of a free online course in ecolinguistics by revealing the personal and work-related impacts reported by the sampled participants. Secondly, at a meta-level, this paper aimed to reflect on the methodological challenges that arise when assessing the impact of a free online course and to propose an alternative that can assist in standardizing systems and methods.

In order to provide a broad representation of the impact the course has generated, a range of data collection techniques were employed: semi-structured email interviews, completion forms and website comments. This contributed to overcome some of the challenges of conducting impact assessment as outlined in Penfield et al. (2014). For example, the use of self-reporting in the certificate request forms helped capture impacts minimizing potential time delays as participants were required to reflect on any changes in their behavior as a direct consequence of completing the course. Self-reporting was also a successful method insofar that it fostered a broadening of answers through introspection and self-reflection that could be otherwise lost in structured interviews and surveys. Finally, all three data sources aptly provided data that revealed the various layers of impact attribution and facilitated the collating of evidence that would have otherwise not been available at the time or been non-existent.

The data reported a number of impacts at personal and work-related levels, predominantly with participants reporting impacts at the level of (a) thinking how language shapes society; (b) changing how research or academic work is being conceived; and (c) doing practical daily choices. These results indicate that there appears to be a positive correlation between the course’s aim – i.e. providing the linguistic tools to critically reveal harmful stories, challenging them from an ecological perspective and searching for new and encouraging stories – and the impact outcomes. Moreover, the results suggest that there is a gradual transition from rational scrutiny and self-reflective practice to a more active response with the potential of enabling social change. In this sense, the category of “Other” related impacts (IMOs) can offer further insights into the participants’ negotiation between personal and work-related values and how these can be integrated in various ways, not only by encouraging pro-environmental habits, but by triggering curiosity in other areas of life. Since environmental knowledge and environmental awareness account only for a small percentage of pro-environmental behavior (Geiger et al. 2019; Kollmuss and Agyeman 2002), developing resources that provide the critical tools to encourage action are essential. Digital humanities courses like TSWLB, which emphasize the interdisciplinary nature of environmental issues, can significantly assist in bridging the gap between internal and external factors to create and promote pro-environmental behaviors. This is an area in which research should be encouraged and that could be particularly appealing to scholars interested in articulating their subject-disciplines with environmental and digital humanities methodologies, not only in terms of content creation, but, crucially in terms of impact assessment and how assessment links to processes and outcomes (Morton 2015).

Despite its efforts, the study still has some limitations, such as understanding which new impacts have developed in the COVID-19 era and how previous impacts have been modified by the pandemic. Given the increasing reliance on digital resources as a result of the pandemic, and the recent decision by the University of Cagliari to include the TSWLB course as a compulsory item in their curriculum, it is crucial to collect further data that can identify new impacts and link old ones to new research. Furthermore, the study is concerned with assessing impacts in a very specific way determined by the REF, a framework which is culturally and geographically bound to the United Kingdom. This in itself poses a limitation insofar that only certain assumptions of what constitutes impacts are considered, thus neglecting and potentially ignoring other relevant criteria. Examples of other approaches to impacts can be found in League of European Research Universities (2013), Hicks et al. (2004) for the USA context, Jones et al. (2004) for an Australian perspective, Bao et al. (2004) for Environmental Impact Assessment in China, and Momtaz (2002) for a similar study in Bangladesh. Nonetheless, the framework provided by the REF does offer analytical aspects that can be transferable and addressed in transnational and transcultural discussions.

At the level of metanalysis, this study discusses the multiple difficulties that resources like the TSWLB course can encounter when being analyzed under the scope of the REF since citation indexes may not be an accurate representation of the impact this resource has had. Therefore, by using a combination of qualitative techniques it was possible to thematically analyse the impact of the course beyond academia. The REF (2021: 90) excludes the “advancement of academic knowledge within the HE sectors”; however, it includes “impacts on students, teaching or other activities both within and beyond submitting HEI” (REF 2021: 90). This is an important distinction since, at first sight, it would seem that the work-related impacts appear predominantly within this category. Introducing a quantitative model for impact assessment would further assist in revealing the nuances of impact beyond academia and the REF as well as procuring a more robust analysis. The following section suggests a complementary approach to the qualitative models already discussed in this study.

6 Quantifying impact in the digital humanities: A proposition

As set out in Section 1, in this paper we wish to propose an additional analytical tool to the qualitative nature of the REF assessment by putting forward a quantitative model for impact assessment. As encouraged in TIDSR (see Section 1), the model is intended as a complementary method to qualitative analyses with the only scope to facilitate the standardization and homogenization of impact assessments across the discipline (also King’s College London and Digital Science [2015]; and “normalization” in Bornmann and Marx [2013]) so that researchers may make reference to a type of result that is immediately recognizable and comprehensible to others. The model can be a useful tool to readily interpret impacts as well as to gauge in what areas the phenomenon investigated yielded higher impacts; an analytical outcome especially useful when working towards changes in policy-making decisions (e.g. O’Connell and Hurley 2009). In this section, the model will be presented from a theoretical perspective and then applied on the results yielded from the survey responses related to TSWLB (reported in Roccia 2020) as a case example for illustrative purposes.