Skip to content
BY 4.0 license Open Access Published by De Gruyter Mouton July 23, 2021

The Stories We Live By and the stories we won’t stand by: Measuring the impact of a free online course in ecolinguistics

  • Mariana Roccia EMAIL logo and Jessica Iubini-Hampton


Current dominant narratives of economic growth, consumerism, and anthropocentric views on human existence, to name a few, are behind the driving forces responsible for the increasing destruction of the very own ecological systems that all life depends on. By utilizing tools of Discourse Analysis while adopting an ecological perspective, the free online course The Stories We Live By (TSWLB) offers a practical and accessible framework in which stories can be critically evaluated, questioned, and resisted. Crucially, students are encouraged to apply their newly acquired theoretical insights to search for alternative stories to live by. While current approaches of impact assessment beyond academia can be measured more readily in the sciences, it is harder to assess whether, how, and to what extent humanities research produces change in society; arguably, the digital format of the course adds to the complexity of assessing its impact. In this article, the authors aim to draw attention on the inherent value of the dissemination of traditional academic tools beyond academia. By combining both qualitative and quantitative methodologies in relation to the free online course TSWLB as a case study, the article provides an innovative tool to effectively measure impact which renders itself suitable for a wider range of disciplines across both traditional and digital humanities.

1 Introduction

The advent of the Internet has brought new ways in which individuals engage with learning and resources. It is common nowadays to find a plethora of online courses and blended learning opportunities across disciplines whereby participants enjoy the benefits of both synchronous and asynchronous teaching methods. Online courses have become very popular within higher education environments to reduce the carbon footprint that is naturally created with face-to-face modes only (Castle and McGuire 2010) and have become a necessity during the COVID-19 pandemic (e.g. Adnan and Awar 2020; Bao 2020). While these practices have expanded, measuring the impact of digital resources has proven to be a difficult task, particularly as these incorporate academic and non-academic impacts. Traditionally, impact has been measured by means of metrics relating to journal citations and books. However, bibliometrics that rely on the journal impact factor (IF) and the h-index (Hirsch 2005) provide only a misrepresentation of the data due to the lack of normalization (Bornmann and Marx 2013). Moreover, Penfield et al. (2014) point out five challenges when undertaking research impact assessment, namely, time lag, the developmental nature of impact, attribution, knowledge creep, and gathering evidence. The time frame to assess impact may be insufficient in some cases as some impacts may take longer to appear. This closely links to the following challenge: impact is not static and there may be temporal and dissipated impacts which should therefore be considered in the broader picture. Another crucial aspect is attribution: since research usually involves various actors and organizations it is hard to link those who were crucial to the creation of a specific impact. Closely related to the latter is knowledge creep, i.e. when new data informs existing research, a practice that makes attribution problematic. The last challenge finally relates to gathering evidence: the problem lies in collating evidence that may no longer be available or may be non-existent and having the tools that adequately measure and capture the data.

Traditional tools of assessment for impact in the humanities, such as citations, appear to be an inaccurate measure for digital resources (Hughes et al. 2015). Scholars have long reflected on the challenges that new ways of acquiring knowledge pose on impact assessment. The Oxford Internet Institute developed in 2011 the Toolkit for the Impact of Digital Scholarly Resources (TIDSR), a notable example of the efforts to contribute to new methods of impact assessment. The TIDSR describes a variety of broad-based quantitative and detailed qualitative methods that assist in providing a broader picture of the array of impacts (Meyer 2011). Quantitative measures include but are not limited to website analytics, surveys, and webometrics while qualitative measures include referrer analyses, interviews, feedback, and focus groups. Combining different methods will depend on the size of the corpus and the data that is available. By drawing on a balanced amount of qualitative and quantitative evidence it is possible to illustrate the impact of research on various levels. Since digital resources are usually specialized and catering for a smaller audience, the TIDSR highlights the need for using a wide range of methods, as relying solely on one will prove insufficient.

In the context of the United Kingdom, the Research Excellence Framework (REF) is responsible for setting out a current definition of impact and how this should be assessed within academic institutions. The appraisal of non-academic impacts is crucial for the allocation of funding in the universities, a topic which has not been exempt from criticism. Currently impact is defined in the REF (2021: 81) as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.” Nonetheless, institutions and researchers understand impact in different ways, which is reflected in the range of pathways to impact provided in the REF 2014 case studies (Terämä et al. 2016). In a REF submission, academics are required to submit a qualitative impact case study which can provide evidence of the types of impact and external sources that can substantiate these impacts. An impact case study is divided into five sections: (a) summary of the impact; (b) a description of the underpinning research; (c) references; (d) details of the impact; and (e) sources that corroborate the impact. The criteria for assessment is reach, that is, how spread the research effect has been on the intended beneficiaries, and significance, the overall influence. Ravenscroft et al. (2017) highlight a number of issues with this relatively straightforward system. Since the submissions are only three to five pages long, they usually feature only a small number of high impact projects, which is detrimental for academics contributing small amounts in various projects. Interdisciplinary submissions are considered within the REF but the depth of the assessment alongside the inherent qualitative nature of the procedures question the system’s robustness, particularly in terms of the assessment of digital components. In addition, a report from King’s College London looking at REF 2014 submissions found that quantitative data used to support impact were not consistent and presented in diverse ways, which reveals the need of standardizing numerical values if impact metrics are to be developed and put in place at some point (King’s College London and Digital Science 2015).

In an attempt to compensate this, digital humanities scholars at King’s College London developed a checklist to help scholars prepare their digital research outputs for submission to the REF 2021 (Ciula 2019). The questions range from work attribution, accessibility, user experience, look & feel and citability, amongst other items in the checklist. This list aims to provide the preliminary non-subject specific criteria to assess whether a digital output is suitable for a REF 2021 submission.

As for impact case studies in the field of linguistics, the search in the REF 2014 shows 106 case studies ranging from explorations in foreign language teaching to language preservation. Within ecolinguistics, impact assessment appears sparser with only a few instances of case studies highlighting the impact of Arran Stibbe’s pioneering research (REF 2014). The joint work between the University of Maribor and the University of Gloucestershire in conjunction with universities in Italy and Turkey for the “Me We Whole” project also account for the impact of the teaching pack Living in the weatherworld: Reconnection as a path to sustainability. The impacts of the materials showed to influence teaching practices as well as individuals working in tourism who incorporated the materials for their daily activities.

In this paper, we investigate how the free online course in ecolinguistics The Stories We Live By (hereafter TSWLB) has impacted the everyday habits of those who completed it in terms of both their thinking and their behavior. The emphasis on impacts that go beyond the mere academic scope is crucial for determining how the online course TSWLB can account as a contribution to the expansion of the field of ecolinguistics not only in terms of its development within academia but also across the general public. As an online, digital tool of self-learning, TSWLB fits within the realm of the digital humanities, an aspect which must be considered when engaging with impact assessment for the reasons previously set out. The data stems from an independent consultancy project for the University of Gloucestershire (see Roccia 2020 for the final report). A brief introduction of the course, its content and its aims are outlined in Section 2. This is then followed by a description of the methods used to gather data and the coding system utilized for analyses purposes (see Section 3); these are semi-structured interviews, anonymous comments left on TSWLB website, and feedback received by students when submitting their certificate request form. An overview of the themes emerged as results of the analysis is presented in Section 4 and a discussion of these is provided in Section 5 in which we argue for the benefits of using this mixed-method design to assess impacts in the digital humanities without forgetting about its limitations. The article then concludes with a brief proposition for a quantitative method to standardize the discussion across impact studies in the digital humanities (see Section 6).

2 The Stories We Live By: A free online course in ecolinguistics

The Stories We Live By: A free online course in ecolinguistics (hereafter “the course”) is, as the name would suggest, an online course open to all for free with accreditation from the University of Gloucestershire in the form of a certificate upon successful completion. Students are required to register on the website[1] by creating a free account which will give them access to all materials and resources, as well as to discussion groups and to the support of voluntary tutors who can provide advice and mentoring in 12 different languages. The course can be completed in whatever order and time the student wishes to and, upon completion, the student can fill in a feedback form to request their certificate. Although the completion of this course does not provide university credits, the certificate is nonetheless an official document which is fully accredited by the University of Gloucestershire in the United Kingdom.

The course reflects both in content and structure Arran Stibbe’s Ecolinguistics: Language, ecology and the stories we live by (2015) in which the author delves into the one fundamental aspect of human existence that is shared across all human beings and therefore transcends social constructs such as race, gender, and religion. That is the need to make sense of our world which is accomplished by the creation of mental models – stories. When these stories are shared by a community of people, and therefore across an entire culture, Stibbe argues that they become stories-we-live-by.

With his book and this course, Stibbe aims to provide a solid linguistic methodological framework through which discourse can be analyzed. By doing this, Stibbe is one of the first scholars to combine ecology and linguistics into one discipline. Indeed, the central theme that runs through the course is one concerned with how language shapes the relationships between humans and the natural world and what linguistics can do to contribute to their reconnection. Throughout the course a wide range of real-world examples are provided in order to concretize abstract concepts. These examples are taken from Stibbe (2015) in which the author analyses data selected from textbooks of economics, as well as machinery handbooks, ecological reports, and nature writing texts. Stibbe’s analysis focuses on the linguistic features found throughout the texts in terms of, for example, lexical choices, passive and active grammatical structures, and intertextuality. When combined together in one text, these linguistic features constitute the main message, or indeed ideology, implied behind the semantic interface of sentences. Stibbe goes one step further and overtly exposes these stories by questioning them and by offering alternative ones, of positive nature. For the purpose of questioning a text in terms of whether it encourages people to care for the world or, instead, damage it, a set of ethical guidelines have to be established. These guidelines will constitute the basis of the framework within which the analysis can be conducted; they will be decided by the author of the analysis, based on their ethical values – their ecosophy. Discourse can then be evaluated as being beneficial, ambivalent, or destructive, depending on the degree with which they promote sustainable practices of interconnections between humans and the rest of life on Earth.

Overall, the course offers a model through which discourse can be analyzed with the purpose to reveal the eight types of story described by Stibbe (2015): ideologies, framings, metaphors, evaluations, identities, convictions, erasure, and salience. With “Introduction” forming part 1 of the course, the types of stories constitute the rest of course as namely, part 2 to 9. Ideologies can be represented in various ways and whenever a specific choice of vocabulary is used to trigger a particular concept, Stibbe refers to this as “framing”. Stating that sustainability should be reframed as a supply chain challenge is an example of this (Stibbe 2015: 26). Metaphors are a specific type of framing whereby an abstract area of life is very vividly conceptualized by applying properties of other more concrete areas of life: “climate change is a global-scale violence” (Stibbe 2015: 65). Furthermore, the common idea, in British culture, that rainy weather is bad is an example of how stories can be evaluated by groups of people (hence the term “evaluation”) either positively or negatively. Similarly, stories in people’s minds about the truthfulness of an area of life are defined as “convictions” (Stibbe 2015: 129). Finally, stories can be about “identities”, or, more explicitly, about being a specific kind of person, such as a consumer, a manager, and so on.

The connection between ideology, framing, identity, evaluation, and conviction and the way people’s behavior can be manipulated is evident. Whether these consequences are worthy of consideration in a discourse, constitutes two different kind of stories, namely, “erasure” and “salience”. While “erasure” is a story that renders an area of life completely unimportant (such as erasing the individuality of fish by referring to them as “fish catch”), “salience” has the opposite effect. Indeed, it is stories of salience that can provide beneficial alternatives of reconnection and appreciation of the natural world, such as in the works of new nature writing and Japanese poetry.

Through this online course, it is therefore possible to further increment the chances of making an impact in the lives of people who would not ordinarily read academic texts. The course presently counts more than 1,500 subscribers from six continents, and different backgrounds. Of particular interest is the incorporation of the course into an English Language and Translation undergraduate programme at the University of Cagliari, Italy (UNICA 2021). Due to the current pandemic and in an attempt to compensate for the lack of face-to-face interaction, the course was added as extra reading materials. However, the course proved very popular among the students and the programme leader has now decided to include the course as part of the curriculum. It must be noted that even though both geographical and socio-economical representation would seem to suggest that the course has been successful in reaching a wide range of people, the data also reveals some interesting points worth further investigating. For example, in terms of education, only 6% of students have no formal higher education qualification against 94% of participants holding at least one degree (Roccia 2020). A similar gap can also be seen in geographical representation with 68% of participants residing in the United Kingdom (in first position), 38% from the USA (in second position) and 11% from China (in third position) (Roccia 2020). While the latter can be justified by factors such as the language used for the course and its materials, and the pre-existing body of interest in ecolinguistics in these territories, the data on education is more problematic. Both location and education percentages seem to show limitations which are inherent to the structure and nature of the course. These limitations are especially true for the level of education reached so far and strategies to improve these numbers may be worth designing and implementing so that having a degree in higher education does not act as a sort of gatekeeper to materials created for the benefit of all.

3 Materials and methods

Data stems from three different sources: (a) semi-structured email interviews, (b) anonymous comments on the Stories We Live By website, and (c) course completion certificates. Due to the inquisitive nature of the research design, data was analyzed following a content-driven approach aimed to identify the types of impacts that were most predominant. A coding system was devised in line with the emerging themes to classify the impacts into three high-levels: personal, work-related and other impacts; these were further sub-classified into lower levels with further subcategories. Table 1 below illustrates the coding system.

Table 1:

Coding system.

Personal (IMP) [Level 1] Work-related (IMW) [Level 1] Other (IMO) [Level 1]
IMPT – Thinking [Level 2]

[Level 3]
  1. IMPTA – awareness

    • IMPTA1 of how language shapes society

    • IMPTA2 of nature

  2. IMPTC – change in thinking

  3. IMPTF – change in feelings, i.e. hope/despair

IMWT – Teaching [Level 2]

[Level 3]
  1. IMWT1 ecological issues

  2. IMWT2 focus on language

  3. IMWT3 resources

IMO – other impact [Level 2]

[Level 3]
  1. IMO any other relevant impact

IMPC – Communicating [Level 2]

[Level 3]
  1. IMPC – Being able to communicate ecological issues

IMWR – Research [Level 2]

[Level 3]
  1. IMWR – impact on own research and academic work

IMPD – Doing [Level 2]

[Level 3]
  1. IMPD1 – practical everyday choices

  2. IMPD2 – larger scale changes related to activism and campaigning

  1. Note: The table is the original work of the authors and was produced for the scope of this manuscript. This table also appears in Roccia (2020).

As seen in Table 1, the three main types of impact that emerged from the coding are Personal (IMP), Work-related (IMW), and Other (IMO). As reported in Roccia (2020), nearly 45% of respondents are academics, 18% are students in Higher Education and nearly 27% selected “Other” as their occupation. With a total ratio of 63% of respondents working and/or studying in an academic context and a disproportionate majority of participants with at least a Master’s degree (78%), the impacts reported seem to reflect this orientation. Indeed, work-related impacts were further divided into two subcodes, namely, “Research” (IMWR) and “Teaching” (IMWT); while IMWR contains impacts of how researchers changed their research and/or academic work as a direct result of completing the course, IMWT contains a larger variety of impacts. Some teachers/educators stated that the course impacted their teaching by bringing ecological issues to their curriculum (IMWT1), others were inspired to add a focus on the role of language in society (IMWT2) and others still commented on the changes they brought about on their resources (IMWT3). As for personal impacts, responses exhibited high levels of introspective thought on how language operates in society. Such was the fine level of introspection in participants’ responses that a more detailed sub-coding of IMP was not only justifiable but needed. Personal impacts were first subdivided into three main categories: “Thinking” (IMPT), “Communicating” (IMPC), and “Doing”(IMPD). Data then led us to further divide the latter into practical changes in behavior in everyday life (IMPD1), such as cutting down on meat consumption or being more careful with recycling, and changes of practical nature but on a larger scale, such as taking up activism and campaigning (IMPD2). In terms of impacts on a participant’s thinking behavior (IMPT), three further sub-themes emerged. Some respondents commented on changes in their awareness (IMPTA) of how language shapes society (IMPTA1) and, more generally, towards nature (IMPTA2). These were responses that did not mention changes of a practical nature nor did they indicate a change in attitude, rather they simply disclosed a sense of discovery. Therefore, these impacts were labeled as separate from changes in thinking (IMPTC). Likewise, changes in feelings (IMPTF) pertain to a shift in a participant’s emotional disposition towards specific areas of life, such as hope and/or despair for the future of the planet. Once again, these comments did not allude to changes of a practical nature nor of an attitudinal stance. Some respondents stated that the completion of the course has equipped them with communicative tools to better communicate ecological issues with others; these are labeled as IMPC. Finally, only a very few numbers of responses included impacts of other types (IMO) which for the sake of transparency have all been presented and discussed in Section 4.3.

3.1 Semi-structured email interviews

One way to overcome the geographical distance between the researchers and the participants was to conduct email interviews (Bampton et al. 2013). An introductory email was sent to the online course’s mailing list to ask participants to voluntarily take part in the interviews. A total of fourteen semi-structured email interviews were completed between July and August 2017. The process followed an interactive conversation between the researcher(s) and the participants-with introductory questions, some comments on the previous response and further questions (Bampton and Cowton 2002). Overall, the interviews sought to elicit the following data: (1) profession and life situation; (2) exposure to ecolinguistics; and (3) the impact of ecolinguistics on participants’ lives and work.

3.2 Anonymous comments on TSWLB website

The data also included a series of 18 indicative comments extracted from the comments section on the TSWLB website.

3.3 Course completion forms

Course participants are encouraged to submit a reflective essay upon completion of the nine parts. The essay prompts participants to elaborate on their learning experience as well as reflecting on how they can apply the contents of the course to their daily life/professional practice. Twenty-three completed forms were considered for the analysis.

All three methods of data collection are inherently biased insofar that they naturally attract participants whose feedback tends to be of a positive inclination. This, however, is not unique to these sources of data and research design. Even in methods such as questionnaires, respondents can be seen to be more prone to provide positive answers in order to be seen as more socially attractive (Fisher 1993). The aim and focus of this study are to gauge the ways in which the completion of the course may have impacted a participant’s way of thinking or doing. In other words, it is the accounts of those whose everyday habits have changed as a result of the course that was of interest rather than showing who was impacted and who was not. These sources of data therefore proved to be apt and fitting to the scope of the study.

4 Results

4.1 Personal related impacts – when “thinking” is not enough

One of the emerging themes throughout the data was the changes in thinking (IMPT-Thinking) the course TSWLB prompted in the participants. Figure 1 below illustrates all six categories within Level 2 and shows a marked tendency within the data extracted from the course completion forms and the semi-structured email interviews to highlight the importance of this impact. This is followed by a reported impact on research (IMWR-Research) within the course completion forms, and an impact on doing (IMPD-Doing) within the semi-structured email interviews. The website comments reported a marked impact on teaching (IMWT-Teaching); this is presumably due to the interest the course generated amongst language teachers. The second highest reported impact found in the website comments relates to thinking (IMPT-Thinking).

Figure 1: 
Overall Level 2 by source.
Figure 1:

Overall Level 2 by source.

Within the personal dimension, the thinking process is divided into further subcategories in Level 3: IMPTA-awareness of how language shapes society (IMPTA1) and nature (IMPTA2); IMPTC-change in thinking; IMPTF-change in feelings, i.e. hope/despair. Figure 2 below displays the reported impacts at Level 3.

Figure 2: 
Total number of personal-related impacts at Level 3.
Figure 2:

Total number of personal-related impacts at Level 3.

Figure 2 confirms that IMPTA1 (how language shapes society) accounts for the highest number of responses followed by an “impact on doing” IMPD1 – practical everyday changes. Two exemplary responses from each category can be found below in Semi-structured Interview (SSI) 5, IMPTA1 and Certificate (CERT) 23, IMPD1: the first is an excerpt from a semi-structured email interview and the second is from a course completion form.[2]

SSI 5, IMPTA1: “I hope that my being more aware of ecolinguistics will help me to write with more awareness of the stories I’m telling, so that I don’t accidentally spread harmful ones in both my fiction and non-fiction writing.”

CERT 23, IMPD1: “These decisions included things related to my: diet, home environment, and (mode of) travel choices. The course has been part of an important process for me of reorienting my own life to better reflect my values rather than simply going along with the destructive values and ‘stories’ that are pervasive in so many areas of life.”

Responses like the previous examples often reflect on the language/action dichotomy, a relationship which appears crucial for ecolinguistics. The data suggests that course participants translated the newly acquired contents into more practical and environmentally attuned daily choices. This positioning strongly resonates with the often-cited gap problem that occurs in developing environmental awareness, i.e. “knowing of the impact of human behavior on the environment” (Kollmuss and Agyeman 2002: 253) and responding to it. Several models have aimed at revealing the underlying factors that shape decision making for pro-environmental behaviors, however, not all of them encompass the multiple dimensions intervening the process. Kollmuss and Agyeman (2002) argue that there is a “pro-environmental consciousness” which comprises values, attitudes, knowledge and emotional involvement within the individual’s personality traits and value system (internal factors). External factors include infrastructure, political, social and cultural factors, and economic situation. The greater pro-environmental behaviors occur when the internal and external factors cooperate with each other.

Therefore, it can be argued that the course encouraged some of the participants to align the internal and external factors to achieve pro-environmental behaviors in their daily lives; thus, bridging the gap between action/knowledge.

4.2 Work-related impacts

The second category that emerged from the data relates to impacts around work, more precisely on teaching (IMWT) and research (IMWR). Figure 3 below shows the total number of impacts at Level 3 for this category.

Figure 3: 
Total work-related impacts (IMW).
Figure 3:

Total work-related impacts (IMW).

The highest number of responses was ranked at the IMWR level in the category “impact on own research and academic work”. This is presumably due to the ways the course has been disseminated, mainly through academic networks which take an interest in the topic, and amongst higher education students/academics (as seen in the participants’ description in Section 2). In a large proportion, participants expressed how the course provided them with a new angle to their own work, such as in the following indicative quote:

CERT 23, IMWR: “Seeing how language and discourse could be analysed with a view toward eco-ethics in broad range of social and linguistic contexts contributed to a change in how I imagined what I could do in my research.”

Environmental issues, such as climate change, have been a concern for the ever-interdisciplinary field of Critical Discourse Analysis for quite some time now (Wodak and Chilton 2005). The previous indicative quote (CERT 23, IMWR) highlights one of the benefits of the course in terms of crossover with other social sciences. This offers an interesting approach for a wide range of academics working at the interface of language and the environment, which is further emphasized by the digital nature of the resources. In relation to this, Tanner (2015) notes in his blog that “digital humanities scholarship offers potential for evidence of interactions with a wide range of beneficiaries from our research outcomes and impact on an international scale. Digital humanities is in many ways a foster parent for change and for outcomes that resonate beyond the Academy with the wider public”. However, this perceived wider reach is particularly difficult to measure when it comes to the REF. Aside from the metrics, the lack of specific Digital Humanities Panels also adds to the complex scenario of evaluating digital humanities outputs in a large scale.

The second highest impact within work-related impacts was “IMWT3-resources”. This high occurrence corresponds to the nature of the free online course and with a high number of teachers and tutors downloading the free resources for their classes. The majority of the impacts for this category were collected from the TSWLB website as seen in the three indicative quotes below website comment WC 9, WC 10 and WC 12:

WC 9-13-07-2017 []: “I’m a teacher, and this course gave me wonderful material to work in class at secondary school. Thanks a lot, from Argentina!!”

WC 10-21-08-2017 []: “[…] I cannot thank you enough for these amazing resources! I am preparing to teach Poetry by Ted Hughes, from an Ecocritical theoretical perspective, for year 13 Literature coursework and your Ecolinguistics course has enabled me to structure my work scheme and given me key terms and up to the moment ideas by which to make clear what is meant by an Ecocritical perspective of literature – and helped me explain what Ted Gifford meant when he called Hughes a ‘post-pastoral eco-poet’.”

WC 12-10-5-2017 []: “The course is really helpful whether you are an expert in the field or not, easy to understand and comprehensible.”

These indicative quotes reflect an interest to integrate environmental issues into the language teaching curriculum, particularly in EFL (English as a Foreign Language), to increase students’ awareness and encourage them to take action. In this regard, Hauschild et al. (2012: 3) list a number of advantages for including content-based environmental education into the language classroom: “(1) heighten students’ interest in contemporary issues that might directly influence their futures; (2) teach students how to contribute to a healthier, more sustainable world; and (3) promote language learning and meaningful communication.” However, bringing attention to these issues in the classroom imposes a challenge on educators to intuitively transmit attitudes and values which may not be explicitly considered in the formal curriculum (Cotton 2006). By questioning dominant discourses that have detrimental impacts on the environment and challenging ideological constructs, ecolinguistics offers a framework through which educators can introduce and critically analyse language use around environmental issues within their local context as seen in the following indicative quotes from a website comment (WC) and one of the course completions forms:

WC 7, IMWT2: “I am from Indonesia, a big country granted with rich bio-and-cultural diversity. This is really a new domain for me and in the long run will surely attract more and more interested parties to join in and be concerned with. More and more students of mine need to be introduced to this domain to help them widen their horizons of how language constitutes an indispensable part of their life and is inseparable from the nature they live in.”

CERT 22, IMWT1: “In my classes I have started telling the children about the activity they should do in future with respect to the society and the ecology. I showed some of the slides that I viewed here. Also, I took them to the places near our school and showed them how the environment is badly affected and that they are also responsible for the same.”

The categories IMWT2-focus on language and IMWT1-ecological issues accounted for the lowest reported impacts. This could be due to participants already working at the interface of the two categories and reporting on newly acquired methods and contents they can use in their work. Notably, the responses report an interest in encouraging others – in this case the participants’ students – to engage in a reflective practice towards preserving their local environment and becoming global citizens (Jacobs and Cates 1999).

4.3 Other impacts-IMOs

The last category that emerged from the data was “Other related impacts-IMO”. Although only four items could be classified within this category (one website comment and three extracts from the semi-structured interviews – see below), they are nonetheless a clear reflection of the influential range of impacts TSWLB has had and could shed further light on other important areas of impact which have not been considered so far:

WC 18-11-06-2017[]: “Really, I do appreciate the great efforts exerted to present such a course. The material is useful, apt and comprehensive. The course encourages me to read more and more about the topic. Ecolinguistics is very appealing.”

SSI 5: “The excellent references have led me to explore further; for example, I’m now studying ecofeminism which I might not have come across otherwise.”

SSI 11: “There is a custom in our religion to sacrifice sheep or other animals to thank God, to prevent a bad event happening to us, our children or anyone important for us. After sacrificing the sheep, we offer the meat to poor people. After completing this course, my family was faced with a few bad events. Two of them were about my daughters and all my family recommended us to sacrifice a sheep but my wife and I decided to plant a tree and we are sure that God was happier in that way.”

SSI 4: “I am, however, discussing with colleagues in Environmental Sciences, Political Science, Digital Humanities, Art, and Engineering the possibility of creating a multi-disciplinary seminar, maybe a freshman seminar, with Ecolinguistics as the focus.”

These indicative quotes suggest an interest in the participants to integrate the course with complex aspects within their lifestyles, such as religious worldviews (SSI 11), as well as sparking curiosity and encouraging participants to engage in other areas of enquiry. It remains unclear whether this enthusiasm (WC 18 and SSI 5) is geared towards personal development or professional development – such as in SSI 4. What is interesting about this category, is the overlapping between what appears to be both personal and work-related motivations that could enable a granular change at a more holistic level.

5 Discussion

The purpose of this paper was twofold: firstly, it sought to assess the impact of a free online course in ecolinguistics by revealing the personal and work-related impacts reported by the sampled participants. Secondly, at a meta-level, this paper aimed to reflect on the methodological challenges that arise when assessing the impact of a free online course and to propose an alternative that can assist in standardizing systems and methods.

In order to provide a broad representation of the impact the course has generated, a range of data collection techniques were employed: semi-structured email interviews, completion forms and website comments. This contributed to overcome some of the challenges of conducting impact assessment as outlined in Penfield et al. (2014). For example, the use of self-reporting in the certificate request forms helped capture impacts minimizing potential time delays as participants were required to reflect on any changes in their behavior as a direct consequence of completing the course. Self-reporting was also a successful method insofar that it fostered a broadening of answers through introspection and self-reflection that could be otherwise lost in structured interviews and surveys. Finally, all three data sources aptly provided data that revealed the various layers of impact attribution and facilitated the collating of evidence that would have otherwise not been available at the time or been non-existent.

The data reported a number of impacts at personal and work-related levels, predominantly with participants reporting impacts at the level of (a) thinking how language shapes society; (b) changing how research or academic work is being conceived; and (c) doing practical daily choices. These results indicate that there appears to be a positive correlation between the course’s aim – i.e. providing the linguistic tools to critically reveal harmful stories, challenging them from an ecological perspective and searching for new and encouraging stories – and the impact outcomes. Moreover, the results suggest that there is a gradual transition from rational scrutiny and self-reflective practice to a more active response with the potential of enabling social change. In this sense, the category of “Other” related impacts (IMOs) can offer further insights into the participants’ negotiation between personal and work-related values and how these can be integrated in various ways, not only by encouraging pro-environmental habits, but by triggering curiosity in other areas of life. Since environmental knowledge and environmental awareness account only for a small percentage of pro-environmental behavior (Geiger et al. 2019; Kollmuss and Agyeman 2002), developing resources that provide the critical tools to encourage action are essential. Digital humanities courses like TSWLB, which emphasize the interdisciplinary nature of environmental issues, can significantly assist in bridging the gap between internal and external factors to create and promote pro-environmental behaviors. This is an area in which research should be encouraged and that could be particularly appealing to scholars interested in articulating their subject-disciplines with environmental and digital humanities methodologies, not only in terms of content creation, but, crucially in terms of impact assessment and how assessment links to processes and outcomes (Morton 2015).

Despite its efforts, the study still has some limitations, such as understanding which new impacts have developed in the COVID-19 era and how previous impacts have been modified by the pandemic. Given the increasing reliance on digital resources as a result of the pandemic, and the recent decision by the University of Cagliari to include the TSWLB course as a compulsory item in their curriculum, it is crucial to collect further data that can identify new impacts and link old ones to new research. Furthermore, the study is concerned with assessing impacts in a very specific way determined by the REF, a framework which is culturally and geographically bound to the United Kingdom. This in itself poses a limitation insofar that only certain assumptions of what constitutes impacts are considered, thus neglecting and potentially ignoring other relevant criteria. Examples of other approaches to impacts can be found in League of European Research Universities (2013), Hicks et al. (2004) for the USA context, Jones et al. (2004) for an Australian perspective, Bao et al. (2004) for Environmental Impact Assessment in China, and Momtaz (2002) for a similar study in Bangladesh. Nonetheless, the framework provided by the REF does offer analytical aspects that can be transferable and addressed in transnational and transcultural discussions.

At the level of metanalysis, this study discusses the multiple difficulties that resources like the TSWLB course can encounter when being analyzed under the scope of the REF since citation indexes may not be an accurate representation of the impact this resource has had. Therefore, by using a combination of qualitative techniques it was possible to thematically analyse the impact of the course beyond academia. The REF (2021: 90) excludes the “advancement of academic knowledge within the HE sectors”; however, it includes “impacts on students, teaching or other activities both within and beyond submitting HEI” (REF 2021: 90). This is an important distinction since, at first sight, it would seem that the work-related impacts appear predominantly within this category. Introducing a quantitative model for impact assessment would further assist in revealing the nuances of impact beyond academia and the REF as well as procuring a more robust analysis. The following section suggests a complementary approach to the qualitative models already discussed in this study.

6 Quantifying impact in the digital humanities: A proposition

As set out in Section 1, in this paper we wish to propose an additional analytical tool to the qualitative nature of the REF assessment by putting forward a quantitative model for impact assessment. As encouraged in TIDSR (see Section 1), the model is intended as a complementary method to qualitative analyses with the only scope to facilitate the standardization and homogenization of impact assessments across the discipline (also King’s College London and Digital Science [2015]; and “normalization” in Bornmann and Marx [2013]) so that researchers may make reference to a type of result that is immediately recognizable and comprehensible to others. The model can be a useful tool to readily interpret impacts as well as to gauge in what areas the phenomenon investigated yielded higher impacts; an analytical outcome especially useful when working towards changes in policy-making decisions (e.g. O’Connell and Hurley 2009). In this section, the model will be presented from a theoretical perspective and then applied on the results yielded from the survey responses related to TSWLB (reported in Roccia 2020) as a case example for illustrative purposes.

6.1 The model

What follows is a theoretical explanation of the model and its implementation. The first criterion for the implementation of the model is for the researcher to include a method of data collection in which participants must provide their answers by grading it out of a limited set of integer numbers (such as a Likert Scale). The questions should aim at capturing a change in behavior in the participants as a direct consequence of whatever phenomenon the researcher is investigating the impacts of (e.g. “I am now more attentive to buying products in environmentally sustainable packaging” – Participants to respond by grading their answer on a 1–5 scale, whereby one stands for “I disagree completely” and five for “I completely agree”). At this stage, the researcher can choose to select whatever scale may fit their research purposes. We would however advise a scale which allows for coding towards positive outcomes (i.e. the higher the score, the more positive the answer) and whose total is an odd number (such as 1–5 or 1–7, etc.) so that a middle, neutral point in the scale may be identified. Once datasets are collected and analyzed for the scope of research, the researcher can then make use of the same datasets to calculate the percentage in decimals of all positive responses that are at least one value above the neutral score on the scale used for data collection. The model can be represented with the following algebraic expression whereby “Ne” stands for “Neutral score” on whatever scale used and “Nr” stands for the total number of responses: Σ ( N e + 1 ) N r .

To illustrate this, let us take a hypothetical example in which the data collection yielded 50 responses on a Likert Scale of 1–5 and out of these responses 35 are of scores of 4 or 5 (that is at least one above the neutral, middle value of 3). 35 is therefore the numerator in the model above and 50 is the denominator. By dividing 35 by 50 we obtain 0.7; that is 70% in decimal form. A minimum of 0.5 would, in our opinion, show an indicator of “good impact” as it would mean that at least 50% of respondents declared to have changed their behavior as a direct consequence of phenomenon studied.

6.2 Case example

Let us now take the results yielded by a specific question in the survey conducted by Roccia (2020) in relation to the impacts reported by participants upon completion of TSWLB. Data for this survey was collected in 2019 through the JISCMAIL[3] list compiled for members of the International Ecolinguistics Association. The survey comprised eight questions which, specifically, sought information on a respondent’s occupation, location, education, current stage in the completion of the course, estimated amount of reading on Stibbe’s work, and finally the impacts experienced as a direct result of taking TSWLB. The survey made use of both open-ended and multiple-choice questions with a total number of 217 responses. Of these, the model will be run on the results yielded by the question on impacts for which respondents were asked to mark the impacts generated by Stibbe’s research on their world and life out of three options: “Very little or none”, “A significant amount”, and “A very significant amount”. Out of 217 respondents, 138 answered and 78 skipped this question. 18% of respondents declared to have experienced very little or no amount of impacts from Stibbe’s work, 52.5% reported a significant amount of impacts and 29.5% a very significant amount. Reconverting these percentages in numerical values, we obtain 72 responses for “A significant amount”, 41 for “A very significant amount”, and 25 for “Very little or none”. This now allows us to apply the model for which all responses that are at least one value above the neutral score on the scale should be identified. The scale used in the survey employed a three-value system in which the middle answer “Very little or none” acts as neutral value. This answer was identified as the neutral value based on semantics (i.e. a significant number of impacts cannot be taken to indicate neutrality) rather than choosing the second option out of three. This shows the importance of considering the use of the module on the outset of a study so that answers can be elicited to better fit this quantification process. All responses submitted as “A very significant amount” and “A significant amount” are therefore to be taken as the numerator in the algebraic expression of the model. This number is 113 which divided by the total number of responses, 138, equals 0.82. The overall score of 0.82 would therefore indicate that students on the course who participated in the survey found the experience as highly impactful.

The model here proposed was not implemented on the data collected for this study as outside the immediate scope. This case example is provided to the reader with the sole scope of acting as an illustrative explanation. Though it can be argued that the model may be applied to the entirety of our data retrospectively, this was not pursued and is in fact, in our opinion, inadvisable due to the level of researcher’s influence and arbitrariness that would be required to transform the participants’ comments into a sort of rating scale. Instead, the model should be seen as one of the tools available at the outset to researchers engaged with impact studies in the digital humanities.

As with all other solely quantitative methods, in order to avoid the reductionism and essentialization of complex and intersectional dynamics at play in our social lives (e.g. Fuchs 2009), this model should not be used on its own but should instead be used as one of an array of mixed methods. Researchers interested in using the model are therefore advised to design their methodology to include the use of the model from the outset and plan accordingly to the needs of this strategy (e.g. Morton 2015).

This model is meant as a proposition and a starting point in the discussion. We therefore welcome others to join us in this debate and work together in optimizing the model and its intended application. While we do not claim that this is an infallible and fit-for-all approach, we do believe that this value can provide a standard point of reference to discuss the level of impact generated by the phenomenon under investigation across the digital humanities.

Corresponding author: Mariana Roccia, The International Ecolinguistics Association, Cheltenham, UK, E-mail:


Adnan, Muhammad & Kainat Anwar. 2020. Online learning amid the COVID-19 pandemic: Students’ perspectives. Journal of Pedagogical Sociology and Psychology 2(1). 45–51. in Google Scholar

Bampton, Roberta & Christopher J. Cowton. 2002. The e-interview. Forum: Qualitative Social Research 3(2). (accessed 3 April 2021).Search in Google Scholar

Bampton, Roberta, Christopher Cowton & Yvonne Downs. 2013. The e-interview in qualitative research. In Natalie Sappleton (ed.), Advancing social and business research methods with new media technology, 329–343. Hershey: IGI Global.10.4018/978-1-4666-3918-8.ch019Search in Google Scholar

Bao, Wei. 2020. COVID‐19 and online teaching in higher education: A case study of Peking University. Human Behavior and Emerging Technologies 2(2). 113–115. in Google Scholar

Bao, Cunkuan, Yongsen Lu & Jincheng Shang. 2004. Framework and operational procedure for implementing strategic environmental assessment in China. Environmental Impact Assessment Review 24(1). 27–46. in Google Scholar

Bornmann, Lutz & Werner Marx. 2013. How good is research really?: Measuring the citation impact of publications with percentiles increases correct assessments and fair comparisons. EMBO Reports 14(3). 226–230. in Google Scholar

Castle, Sidney R. & Chad J. McGuire. 2010. An analysis of student self-assessment of online, blended, and face-to-face learning environments: Implications for sustainable education delivery. International Education Studies 3(3). 36–40. in Google Scholar

Ciula, Arianna. 2019. KDL checklist for digital outputs assessment. (accessed 3 April 2021).Search in Google Scholar

Cotton, Deborah R. E. 2006. Teaching controversial environmental issues: Neutrality and balance in the reality of the classroom. Educational Research 48(2). 223–241. in Google Scholar

Fisher, Robert. 1993. Social desirability bias and the validity of indirect questioning. Journal of Consumer Research 20(2). 303–315. in Google Scholar

Fuchs, Stephan. 2009. Against essentialism: A theory of culture and society. USA: Harvard University Press.Search in Google Scholar

Geiger, Sonja Maria, Mattis Geiger & Oliver Wilhelm. 2019. Environment-specific vs. general knowledge and their role in pro-environmental behavior. Frontiers in Psychology 10. 1–12. in Google Scholar

Hauschild, Staci, Poltavtchenko Elena & Fredricka L. Stoller. 2012. Going green: Merging environmental education and language instruction. English Teaching Forum 50(2). 2–13.Search in Google Scholar

Hicks, Diana, Hiroyuki Tomizawa, Yoshiko Saitoh & Shinichi Kobayashi. 2004. Bibliometric techniques in the evaluation of federally funded research in the United States. Research Evaluation 13(2). 78–86. in Google Scholar

Hirsch, Jorge E. 2005. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences 102(46). 16569–16572. in Google Scholar

Hughes, Lorna M., Paul S. Ell, Gareth A. G. Knight & Milena Dobreva. 2015. Assessing and measuring impact of a digital collection in the humanities: An analysis of the SPHERE (Stormont Parlamentary Hansards: Embedded in Research and Education) project. Digital Scholarship in the Humanities 30(2). 183–198. in Google Scholar

Jacobs, George M. & Kip Cates. 1999. Global education in second language teaching. K@ta: A Biannual Publication on the Study of Language and Literature 1(1). 44–56.10.9744/kata.1.1.44-56Search in Google Scholar

Jones, Lawrence R., Kuno Schedler & Riccardo Mussari (eds.). 2004. Strategies for public management reform. Amsterdam & London: Elsevier.10.1016/S0732-1317(2004)13Search in Google Scholar

King’s College London and Digital Science. 2015. The nature, scale and beneficiaries of research impact: An initial analysis of Research Excellence Framework (REF) 2014 impact case studies. Bristol, United Kingdom: HEFCE.Search in Google Scholar

Kollmuss, Anja & Julian Agyeman. 2002. Mind the Gap: Why do people act environmentally and what are the barriers to pro-environmental behavior? Environmental Education Research 8(3). 239–260. in Google Scholar

League of European Research Universities. 2013. Research universities and research assessment. Position papers. Leuven: LERU. (accessed 3 April 2021).Search in Google Scholar

Meyer, Eric T. 2011. Splashes and ripples: Synthesizing the evidence on the impact of digital resources. London: JISC. (accessed 3 April 2021).Search in Google Scholar

Momtaz, Salim. 2002. Environmental impact assessment in Bangladesh: A critical review. Environmental Impact Assessment Review 22(2). 163–179. in Google Scholar

Morton, Sarah. 2015. Progressing research impact assessment: A “contributions” approach. Research Evaluation 24(4). 405–419. in Google Scholar

O’Connell, Emer & Fintan Hurley. 2009. A review of the strengths and weaknesses of quantitative methods used in health impact assessment. Public Health 123(4). 306–310.10.1016/j.puhe.2009.02.008Search in Google Scholar

Penfield, Teresa, Matthew J. Baker, Rosa Scoble & Michael C. Wykes. 2014. Assessment evaluations, and definitions of research impact: A review. Research Evaluation 23. 21–32. in Google Scholar

Ravenscroft, James, Maria Liakata, Amanda Clare & Daniel Duma. 2017. Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements. PloS One 12(3). (accessed 3 April 2021).10.1371/journal.pone.0173152Search in Google Scholar

Research Excellence Framework. 2014. (accessed 3 April 2021).Search in Google Scholar

Research Excellence Framework. 2021. (accessed 3 April 2021).Search in Google Scholar

Roccia, Mariana. 2020. Changing lives and professional practice: A report on the impact of ecolinguistics. Language & Ecology. (accessed 3 April 2021).Search in Google Scholar

Stibbe, Arran. 2015. Ecolinguistics: Language, ecology and the stories we live by. London & New York: Routledge.10.4324/9781315718071Search in Google Scholar

Tanner, Simon. 2015. 3 reasons why REF2014 was good for digital humanities scholars. When the data hits the fan! (accessed 3 April 2021).Search in Google Scholar

Terämä Emma, Melanie Smallman, Simon J. Lock, Charlotte Johnson & Martin Zaltz Austwick. 2016. Beyond Academia – Interrogating Research Impact in the Research Excellence Framework. PLoS One 11(12). (accessed 3 April 2021).10.1371/journal.pone.0168533Search in Google Scholar

UNICA. 2021. corso di ecolinguistica online. (accessed 3 April 2021).Search in Google Scholar

Wodak, Ruth & Paul Chilton (eds.). 2005. A new agenda in (critical) discourse analysis: Theory, methodology and interdisciplinarity. Amsterdam: John Benjamins.10.1075/dapsac.13Search in Google Scholar

Received: 2020-10-26
Accepted: 2021-03-18
Published Online: 2021-07-23
Published in Print: 2021-04-27

© 2021 Mariana Roccia and Jessica Iubini-Hampton, published by De Gruyter, Berlin/Boston

This work is licensed under the Creative Commons Attribution 4.0 International License.

Downloaded on 2.12.2023 from
Scroll to top button