The EMI^QM certification procedure consists of three unique characteristics:

1. it takes place in an authentic setting and not in a simulated testing environment as is the case in (semi-)commercial language tests such as IELTS or TOEPAS (Kling & Stæhr 2012);

2. (b) it integrates pluri-perspective feedback from experts on teaching in English as well as from the students and the lecturer him/herself;

3. it assesses specific linguistic and communicative competencies for teaching in the usually non-native language to an international student body, instead of focusing primarily on general language competencies.

This quality assurance procedure certifies the linguistic and communicative quality of English-medium instruction competencies. If at least 80 % of permanent and long-term teaching staff in an English-taught degree programme is certified, the programme is awarded a quality seal attesting to the certified competencies of teaching staff in an English-medium instruction classroom context in higher education (Figure 1).

Figure 1:

The EMI^QM quality seal, attesting to certified competencies for English-medium instruction.

In light of unforeseeable sick leaves and staff turnover, an 80 % threshold was agreed upon to make the seal administratively attainable for programmes. The quality seal is valid for five years and teachers who have already been certified need not undergo further assessment should the programme decide to have the seal renewed after it expires, the rationale behind this rule being that language and communicative competencies rarely deteriorate while a teacher has a continuous teaching load (rather, the opposite is the case). Once the interested programme director of an English-taught programme has proposed the certification procedure to the teaching staff and there is staff consensus on pursuing the quality seal, the EMI team organizes the actual procedure directly with teaching staff.

The EMI^QM certification procedure entails three steps. The first step is a classroom visit whereby the EMI team agrees on a date with the teacher and video-records one English-taught class for between 45 and 120 minutes (depending on the schedule). The teacher is free to suggest a specific class in which s/he wants to be assessed. All students are notified about the visit, its purpose and the recording in advance. In addition, the teacher provides the EMI team with the teaching material envisaged for the class so as to help the assessors prepare for the visit and take better notes during the visit. Upon conclusion of the class (10 minutes earlier than its regular end) the EMI team hands out feedback questionnaires to the students and a self-assessment questionnaire to the teacher. The feedback questionnaire collects students’ perceptions on the linguistic and communicative quality of the lesson with regard to their lesson comprehension. The teacher’s self-assessment serves as a tool to reflect on the communicative intentions of and actions taken in the lesson.

The second step of the procedure entails a detailed analysis and triangulation of the collected data. The EMI team assessors individually analyse their notes taken during the classroom visit based on the specific EMI^QM expert assessment criteria (see Section 3), compare them with the video-recording and transcribe parts of the lecture so as to document relevant instances (positive or negative) to substantiate their rating. Once the individual expert assessment is finished, the EMI team discusses and merges their individual results into a joint EMI team expert feedback. In the meantime, the university’s central evaluation service processes and summarizes the student feedback questionnaires and sends the results to the EMI team, who then juxtapose students’ views with those of the teacher from his/her self-assessment. The analysis step culminates in the triangulation of results: the combined scores from expert feedback and student feedback account for the quantitative feedback (ratio 2:1). The self-assessment, the student feedback and the expert comments constitute the qualitative feedback for the teacher. More details on the scoring scheme can be found in Section 4.

The third step in the assessment procedure is an individual feedback meeting with the teacher approximately 7 to 14 days after the classroom visit. During this 30-minute meeting, the teacher receives the video recording of the lesson, the summarized results of the student feedback and his/her self-assessment, and the detailed EMI analysis of the lesson based on specific linguistic and communicative criteria. The holistic feedback provided by the EMI team highlights linguistic and communicative strengths, addresses weaknesses, and proposes minor adjustments and add-ons moving forward.

In developing the assessment criteria, various sources were considered. The general language criteria are by and large based on the Common European Framework of Reference for Languages (CEFR) developed by the Council of Europe. But they also incorporate elements from band descriptors of the International English Language Testing Service (IELTS) developed by the British Council, from the Examination for the Certificate of Proficiency in English (ECPE) developed by Cambridge Michigan Language Assessments, and elements of Celce-Murcia et al.’s (1995) communicative competence model. The criteria that focus more specifically on language use for teaching draw on insights from a range of research studies on English medium instruction in higher education (Gundermann 2014; Kling & Stæhr 2012; Pilkinton-Pihko 2013; Smit 2010; among many others). The criteria were discussed with fellow English-medium instruction experts Patrick Studer and Paul Kelly from the Zurich University of Applied Sciences and were piloted in the English Seminar at the University of Freiburg. Upon applying the criteria in select classes at the English Seminar, the EMI team was able to test the criteria, gain valuable insights from colleagues, and fine-tune the wording of the criteria to set benchmarks.

The assessment criteria are divided into two categories: linguistic competencies and communicative competencies. Each category is made up of five criteria. The five linguistic criteria are based to a large extent on C1 standards in the CEFR as well as the IELTS band descriptors for a 7.0 speaking level. In an observed lesson, the EMI team assesses the teacher’s overall linguistic competencies via the following criteria: fluency, pronunciation and articulation, grammar, lexical accuracy and range, and code consistency. Table 1 provides the threshold descriptor for each linguistic criterion. The linguistic criterion L.5 was specifically formulated to address the use of languages other than English (in speech or writing, e.g., slides), which potentially excludes some students from part of a lesson (Gundermann 2014).

The five communicative criteria assess the use of English for a specific purpose, i.e., teaching specific subject content to a multilingual, multicultural student body at the tertiary level. During an observation, the EMI team assesses the teacher’s communicative competencies via the following criteria: cohesion, prosody, initiation and integration of student input, responses to student input, and intercultural transparency. Table 2 provides the threshold descriptor for each communicative criterion.

This section outlines how an observed lesson is evaluated and how the scores from the EMI team feedback and student feedback are merged into a final result. For the purposes of demonstrating the evaluation and scoring process, we will focus on one criterion, namely lexical accuracy and range (see Table 1), and provide excerpts of the EMI expert scoring sheet as well as the student feedback questionnaire. Each of the ten criteria in the assessment scheme includes two or three sub-criteria.

Table 3 shows the two sub-criteria the EMI team evaluates for the linguistic competency criterion of lexical accuracy and range. The scoring ranges from “to the fullest extent” – which equates to a score of 1 in our scheme – to “to a small extent” – which equates to a score of 4. Thus, the score of 1.0 is the best achievable score whilst 4.0 is the lowest (analogous to the German university grading scheme), the threshold score being 2.0.

After having observed a lesson, the EMI team reviews parts of the recording to time-reference and document evidence supporting the given score for each sub-criterion. Table 3 contains two time-referenced comments and the corresponding scores from the two EMI experts, which are indicated by an X. When there is a difference of one column, the average is taken. If there is a difference of two columns, the EMI team reassesses the criterion and revises the score. In this particular lesson, the EMI team gave a score of 2.5 for lexical accuracy and 1.5 for lexical range. These two sub-criteria scores average out to a score of 2.0 for lexical accuracy and range.

Student perceptions on the teacher’s linguistic and communicative competencies are collected via a questionnaire at the end of the observed lesson. Table 4 shows the questionnaire item pertaining to lexical accuracy and range.

In order to generate scores from student feedback, the university’s central evaluation service electronically processes all student responses and summarizes the results in a report, which is sent to the teacher and the EMI team. Figure 2 depicts an excerpt from the report pertaining to the question on lexical accuracy and range. It shows the number of student responses (n), the average score (av.), and the standard deviation (dev.). For this particular item, student feedback equated to a score of 1.1.

Figure 2:

EMI^QM Student feedback result for the item pertaining to the linguistic competency of lexical accuracy and range.

Once the EMI team has scored a lesson and received the summarized student feedback, it merges the two scores using a 2:1 weighting. In our example for lexical accuracy and range, the EMI team expert feedback resulted in a score of 2.0 and student feedback resulted in a score of 1.1. Thus, the teacher received a score of 1.7 for lexical accuracy and range ([2 x 2.0] + 1.1) /3=1.7.

The evaluation process and scoring scheme is the same for all ten criteria. Once a merged score has been calculated for each criterion, it is averaged to produce a score for the category. Table 5 illustrates the scoring of linguistic competencies for this one particular lesson.

The threshold score for certification is 2.0. If the final score for both categories – linguistic and communicative competencies – is between 1.0 and 2.0, the quality threshold is met. If the final score of one or both categories is above 2.0, the quality threshold is not met and the teacher is not certified. In such cases, the teacher can opt for relevant training measures as outlined by the EMI team and undergo re-assessment, i.e. a second classroom visit, within the following 3–12 months.

Over the course of two semesters in 2015–16, the EMI team certified the English-taught Masters of Science programme Microsystems Engineering. ¹ All lessons observed lasted approximately 90 minutes and were lecture-based with varying degrees of interaction between students and teacher. Thirty-six teachers took part in the certification procedure and 34 teachers were certified, 30 of whom are native speakers of German. In total, 424 student feedback questionnaires were filled in with an average class attendance of 11.7 students. Of course, many students filled in the questionnaire multiple times. Figure 3 depicts the aggregate results for the linguistic competencies of participating teaching staff.

Figure 3:

Aggregate results for linguistic competencies in the Master’s Programme Microsystems Engineering.

Each bar represents the score range for the respective competency and the red line illustrates the mean score for that competency. As Figure 3 indicates, linguistic competencies in this programme are generally strong. All teachers the EMI team observed demonstrated high degrees of fluency, strong command of grammar, and extensive academic and field-specific vocabulary. There were, however, recurring pronunciation issues (phonetics and syllable stress) typical for German native speakers of English, most of which did not seem to lead to confusion in the classroom based on student responses. Vocabulary use was quite accurate minus occasional inaccuracies, e.g. *unsymmetrical instead of asymmetrical, *self-explaining instead of self-explanatory. Instances of false friends were observed, e.g. isolation instead of insulation, eventually instead of possibly, and immediate feedback was provided, as such errors can inhibit student comprehension or even lead to acquisition of faulty terminology. Where code consistency was an issue in an observed class, it was almost exclusively tied to figures and terminology on slides which were displayed in German without the teacher explicitly translating/explaining the terms in English, a practice which can exclude and frustrate students with no knowledge of German.

Figure 4 indicates the aggregate results for the communicative competencies of participating teaching staff. While there is a wider score distribution for communicative competencies compared to linguistic competencies, teachers in this programme generally demonstrated solid communicative competencies.

Figure 4:

Aggregate results for communicative competencies in the Master’s Programme Microsystems Engineering.

Language use on a micro-level to give lessons cohesion – i.e., signal transitions, highlight important information, communicate connections between previous and/or future lessons – was a recurring strength. Moreover, many lessons included closed and open-ended questions as well as constructive responses to student input. Nevertheless, there were recurring weaknesses. Explicit learning objectives and lesson outlines for macro-level cohesion were not the norm, an important point which the EMI team continuously raised in feedback meetings as a helpful guide for students in an international programme. In addition, some lessons lacked a real attempt at facilitating student input even though the teacher clearly possessed the linguistic competencies to do so. Although the EMI team observed only a handful of opaque references to local culture, i.e., references that require knowledge of the local/German context, students tended to indicate that there were some and this led to a wider score distribution. The EMI team believes this is most likely due to students’ interpretation of the question and this is a criterion that will require more attention in future.

As of April 2017, the EMI team had certified in total four English-taught programmes at the University of Freiburg. In addition to Microsystems Engineering, three more programmes, one large programme (>35 lecturers) and two smaller programmes (<10 lecturers each) had been certified. It is planned to assess further English-taught programmes in coming semesters.

Despite mostly positive feedback from teachers on the certification procedure, there are two issues we intend to address moving forward. The first one pertains to the difficulty in assessing intercultural transparency, especially student interpretations of this. As indicated earlier, some lessons scored low on this criterion although the EMI team observed no instances of intercultural murkiness. Could students be answering the question with other aspects of the course in mind, i.e., not exclusively focusing on the one observed class? Researching student opinions and experiences via semi-structured personal or focus group interviews could provide an answer to that question.

The second question addresses whether fluency, grammar, and code consistency should or should not be score-contributing criteria. Our experience so far has shown that participating teachers are fluent speakers with good command of grammar thanks to their extensive experience abroad as students, academics, and/or professionals in their respective fields. Scores for these criteria may be positively skewing overall results. In other words, we may have to simply assume that the linguistic competencies are present and focus exclusively on how well the language is used to teach with a linguistically and culturally heterogeneous student body at the tertiary level. A re-assessment of the existing criteria in theory (through further literature reviews and expert discussions) as well as in practice (testing and piloting a revised version of the criteria in an authentic setting) could bring about meaningful results not only for the EMI^QM certification procedure but also on a more general level for discussion of language competencies for teaching with a multilingual, multicultural student body.