In line with the ongoing internationalization of higher education, more and more universities offer English-medium instruction programmes. The University of Freiburg currently offers 16 degree programmes which are taught entirely in English and cater to both domestic and international students. The challenges of learning and teaching in a (usually) non-native language with a multilingual, multicultural student body are manifold. In order to cater for these challenges and to continue offering quality education, the University of Freiburg established the English Medium Instruction unit at the university’s Language Teaching Centre in December 2011. This unit is one of seven measures within the so-called Quality Pact for Teaching, a large scale and long-term project funded by the German Federal Ministry of Education and Research (funding code 01PL110007). The two-person English Medium Instruction team (hereafter EMI team) has been tasked with providing a variety of formats for improving the language skills of teaching staff (e.g., courses, workshops, online modules, coaching) and to develop a diagnostic assessment procedure which ensures the linguistic and communicative quality of teaching in English. The latter is the focus of this activity report.
2 Certification of quality teaching in English: Rationale and procedure
The EMIQM certification procedure consists of three unique characteristics:
- it takes place in an authentic setting and not in a simulated testing environment as is the case in (semi-)commercial language tests such as IELTS or TOEPAS (Kling & Stæhr 2012);
- (b) it integrates pluri-perspective feedback from experts on teaching in English as well as from the students and the lecturer him/herself;
- it assesses specific linguistic and communicative competencies for teaching in the usually non-native language to an international student body, instead of focusing primarily on general language competencies.
This quality assurance procedure certifies the linguistic and communicative quality of English-medium instruction competencies. If at least 80 % of permanent and long-term teaching staff in an English-taught degree programme is certified, the programme is awarded a quality seal attesting to the certified competencies of teaching staff in an English-medium instruction classroom context in higher education (Figure 1).
In light of unforeseeable sick leaves and staff turnover, an 80 % threshold was agreed upon to make the seal administratively attainable for programmes. The quality seal is valid for five years and teachers who have already been certified need not undergo further assessment should the programme decide to have the seal renewed after it expires, the rationale behind this rule being that language and communicative competencies rarely deteriorate while a teacher has a continuous teaching load (rather, the opposite is the case). Once the interested programme director of an English-taught programme has proposed the certification procedure to the teaching staff and there is staff consensus on pursuing the quality seal, the EMI team organizes the actual procedure directly with teaching staff.
The EMIQM certification procedure entails three steps. The first step is a classroom visit whereby the EMI team agrees on a date with the teacher and video-records one English-taught class for between 45 and 120 minutes (depending on the schedule). The teacher is free to suggest a specific class in which s/he wants to be assessed. All students are notified about the visit, its purpose and the recording in advance. In addition, the teacher provides the EMI team with the teaching material envisaged for the class so as to help the assessors prepare for the visit and take better notes during the visit. Upon conclusion of the class (10 minutes earlier than its regular end) the EMI team hands out feedback questionnaires to the students and a self-assessment questionnaire to the teacher. The feedback questionnaire collects students’ perceptions on the linguistic and communicative quality of the lesson with regard to their lesson comprehension. The teacher’s self-assessment serves as a tool to reflect on the communicative intentions of and actions taken in the lesson.
The second step of the procedure entails a detailed analysis and triangulation of the collected data. The EMI team assessors individually analyse their notes taken during the classroom visit based on the specific EMIQM expert assessment criteria (see Section 3), compare them with the video-recording and transcribe parts of the lecture so as to document relevant instances (positive or negative) to substantiate their rating. Once the individual expert assessment is finished, the EMI team discusses and merges their individual results into a joint EMI team expert feedback. In the meantime, the university’s central evaluation service processes and summarizes the student feedback questionnaires and sends the results to the EMI team, who then juxtapose students’ views with those of the teacher from his/her self-assessment. The analysis step culminates in the triangulation of results: the combined scores from expert feedback and student feedback account for the quantitative feedback (ratio 2:1). The self-assessment, the student feedback and the expert comments constitute the qualitative feedback for the teacher. More details on the scoring scheme can be found in Section 4.
The third step in the assessment procedure is an individual feedback meeting with the teacher approximately 7 to 14 days after the classroom visit. During this 30-minute meeting, the teacher receives the video recording of the lesson, the summarized results of the student feedback and his/her self-assessment, and the detailed EMI analysis of the lesson based on specific linguistic and communicative criteria. The holistic feedback provided by the EMI team highlights linguistic and communicative strengths, addresses weaknesses, and proposes minor adjustments and add-ons moving forward.
3 Certification of quality teaching in English: Assessment criteria
In developing the assessment criteria, various sources were considered. The general language criteria are by and large based on the Common European Framework of Reference for Languages (CEFR) developed by the Council of Europe. But they also incorporate elements from band descriptors of the International English Language Testing Service (IELTS) developed by the British Council, from the Examination for the Certificate of Proficiency in English (ECPE) developed by Cambridge Michigan Language Assessments, and elements of Celce-Murcia et al.’s (1995) communicative competence model. The criteria that focus more specifically on language use for teaching draw on insights from a range of research studies on English medium instruction in higher education (Gundermann 2014; Kling & Stæhr 2012; Pilkinton-Pihko 2013; Smit 2010; among many others). The criteria were discussed with fellow English-medium instruction experts Patrick Studer and Paul Kelly from the Zurich University of Applied Sciences and were piloted in the English Seminar at the University of Freiburg. Upon applying the criteria in select classes at the English Seminar, the EMI team was able to test the criteria, gain valuable insights from colleagues, and fine-tune the wording of the criteria to set benchmarks.
The assessment criteria are divided into two categories: linguistic competencies and communicative competencies. Each category is made up of five criteria. The five linguistic criteria are based to a large extent on C1 standards in the CEFR as well as the IELTS band descriptors for a 7.0 speaking level. In an observed lesson, the EMI team assesses the teacher’s overall linguistic competencies via the following criteria: fluency, pronunciation and articulation, grammar, lexical accuracy and range, and code consistency. Table 1 provides the threshold descriptor for each linguistic criterion. The linguistic criterion L.5 was specifically formulated to address the use of languages other than English (in speech or writing, e.g., slides), which potentially excludes some students from part of a lesson (Gundermann 2014).
EMIQM Linguistic Competencies for quality teaching in English.
|Linguistic competencies for English-medium instruction|
|L.1 Fluency||Speech is fluent with rare instances of language-related hesitations which do not disrupt comprehension.|
|L.2 Articulation and Pronunciation||Pronunciation (phonemic sound contrasts) is clear to understand, word stress is accurate according to target-language standards, and articulation does not require extra listener effort.|
|L.3 Grammatical accuracy||Grammar is accurate according to target-language standards with only minor or rare inaccuracies which do not disrupt comprehension.|
|L.4 Lexical accuracy and range||Lexical choice is accurate according to target-language standards and semantically transparent (avoidance of opaque idiomaticity), lexical range is broad enough to elaborate on subject-specific content and to compensate any lexical gaps.|
|L.5 Code consistency||Code is consistently English, both in speech and writing. If a language other than English is used, a follow-up explanation or translation in English is provided.|
|L total||A certified lecturer speaks fluently with no or few instances of language-related hesitations, articulates and pronounces clearly with no or few instances where confusion might occur, and uses grammar accurately with minor inaccuracies. The lecturer’s lexical choice is accurate and the lexical range is broad enough to explain subject-specific content and to compensate occasional lexical gaps, while avoiding opaque idiomaticity. He/she consistently uses English in speech and writing and any use of a language other than English is followed by an explanation or translation in English. The overall linguistic performance might occasionally require extra listener effort but does not impede comprehension.|
The five communicative criteria assess the use of English for a specific purpose, i.e., teaching specific subject content to a multilingual, multicultural student body at the tertiary level. During an observation, the EMI team assesses the teacher’s communicative competencies via the following criteria: cohesion, prosody, initiation and integration of student input, responses to student input, and intercultural transparency. Table 2 provides the threshold descriptor for each communicative criterion.
EMIQM Communicative Competencies for quality teaching in English.
|Communicative competencies for English-medium instruction|
|C.1 Cohesion||Cohesion in the session is achieved through a range of cohesive devices, the structure and objectives of the session are clearly expressed, and the lesson pace is appropriate.|
|C.2 Prosody||Speech rate is appropriate and does not require extra listener effort and prosodic variation (intonation, stress, pauses) to enhance student comprehension can be observed.|
|C.3 Initiation and integration of student input||Student input and comprehension are facilitated through teacher questions and student contributions are anchored and integrated into ongoing classroom discourse.|
|C.4 Responses to student input||Responses to student questions or contributions are sociolinguistically appropriate, if necessary comprehension is negotiated through adaptation of (non- or para-)verbal communication (variation in prosody, use of additional media or body language).|
|C.5 Intercultural transparency||Locally specific concepts or matters are contextualized and explained in advance for the multicultural classroom.|
|C total||A certified lecturer produces coherent speech through a range of cohesive devices to structure the session, speaks at an appropriate rate and uses prosodic variation (intonation, stress, pauses) to support communicative intention. During a session, he/she facilitates student input through questions, integrates student contributions into ongoing discourse, responds appropriately to student input and negotiates comprehension through adaptation of his/her (non- and para-)verbal communication if necessary. Locally specific concepts and matters are contextualized and explained in advance for the multicultural classroom. The communicative performance stimulates student participation and facilitates comprehension.|
4 Certification of quality teaching in English: Evaluation process and scoring
This section outlines how an observed lesson is evaluated and how the scores from the EMI team feedback and student feedback are merged into a final result. For the purposes of demonstrating the evaluation and scoring process, we will focus on one criterion, namely lexical accuracy and range (see Table 1), and provide excerpts of the EMI expert scoring sheet as well as the student feedback questionnaire. Each of the ten criteria in the assessment scheme includes two or three sub-criteria.
Table 3 shows the two sub-criteria the EMI team evaluates for the linguistic competency criterion of lexical accuracy and range. The scoring ranges from “to the fullest extent” – which equates to a score of 1 in our scheme – to “to a small extent” – which equates to a score of 4. Thus, the score of 1.0 is the best achievable score whilst 4.0 is the lowest (analogous to the German university grading scheme), the threshold score being 2.0.
EMIQM expert scoring sheet excerpt for the linguistic competency of lexical accuracy and range.
|Linguistic competencies||To the fullest extent||To a large extent||To a moderate extent||To a small extent||Evidence /Comments|
|L4. Lexical (word) choice was accurate according to target-language standards.||X||X||Lexical inaccuracies: (00:41) “We have to a bit share and divide characteristics and functions.” (categorize, differentiate between)|
|L4. Lexical (word) range was broad enough to elaborate on subject-specific content and manage classroom activities.||X||X||Language for classroom management: (00:32) “Now this is the first point where I want to guide you.” ((teacher starts clarifying the difference between function and characteristic))|
After having observed a lesson, the EMI team reviews parts of the recording to time-reference and document evidence supporting the given score for each sub-criterion. Table 3 contains two time-referenced comments and the corresponding scores from the two EMI experts, which are indicated by an X. When there is a difference of one column, the average is taken. If there is a difference of two columns, the EMI team reassesses the criterion and revises the score. In this particular lesson, the EMI team gave a score of 2.5 for lexical accuracy and 1.5 for lexical range. These two sub-criteria scores average out to a score of 2.0 for lexical accuracy and range.
Student perceptions on the teacher’s linguistic and communicative competencies are collected via a questionnaire at the end of the observed lesson. Table 4 shows the questionnaire item pertaining to lexical accuracy and range.
EMIQM Student feedback questionnaire item pertaining to the linguistic competency of lexical accuracy and range.
|I agree||I tend to agree||I tend to disagree||I disagree|
|I was familiar with the words and expressions used by the lecturer||□||□||□||□|
In order to generate scores from student feedback, the university’s central evaluation service electronically processes all student responses and summarizes the results in a report, which is sent to the teacher and the EMI team. Figure 2 depicts an excerpt from the report pertaining to the question on lexical accuracy and range. It shows the number of student responses (n), the average score (av.), and the standard deviation (dev.). For this particular item, student feedback equated to a score of 1.1.
Once the EMI team has scored a lesson and received the summarized student feedback, it merges the two scores using a 2:1 weighting. In our example for lexical accuracy and range, the EMI team expert feedback resulted in a score of 2.0 and student feedback resulted in a score of 1.1. Thus, the teacher received a score of 1.7 for lexical accuracy and range ([2 x 2.0] + 1.1) /3=1.7.
The evaluation process and scoring scheme is the same for all ten criteria. Once a merged score has been calculated for each criterion, it is averaged to produce a score for the category. Table 5 illustrates the scoring of linguistic competencies for this one particular lesson.
Merged results from the EMI expert team and student feedback on a teacher’s linguistic competencies
|Linguistic competencies: Merged results from EMI team and student feedback|
|L.1. – Fluency||1.0|
|L.2. – Articulation and pronunciation||1.4|
|L.3. – Grammar||1.3|
|L.4. – Lexical range and accuracy||1.7|
|L.5. – Code consistency||1.0|
The threshold score for certification is 2.0. If the final score for both categories – linguistic and communicative competencies – is between 1.0 and 2.0, the quality threshold is met. If the final score of one or both categories is above 2.0, the quality threshold is not met and the teacher is not certified. In such cases, the teacher can opt for relevant training measures as outlined by the EMI team and undergo re-assessment, i.e. a second classroom visit, within the following 3–12 months.
5 Certification of quality teaching in English: Summarized results from a certified English-taught programme
Over the course of two semesters in 2015–16, the EMI team certified the English-taught Masters of Science programme Microsystems Engineering. 1 All lessons observed lasted approximately 90 minutes and were lecture-based with varying degrees of interaction between students and teacher. Thirty-six teachers took part in the certification procedure and 34 teachers were certified, 30 of whom are native speakers of German. In total, 424 student feedback questionnaires were filled in with an average class attendance of 11.7 students. Of course, many students filled in the questionnaire multiple times. Figure 3 depicts the aggregate results for the linguistic competencies of participating teaching staff.
Each bar represents the score range for the respective competency and the red line illustrates the mean score for that competency. As Figure 3 indicates, linguistic competencies in this programme are generally strong. All teachers the EMI team observed demonstrated high degrees of fluency, strong command of grammar, and extensive academic and field-specific vocabulary. There were, however, recurring pronunciation issues (phonetics and syllable stress) typical for German native speakers of English, most of which did not seem to lead to confusion in the classroom based on student responses. Vocabulary use was quite accurate minus occasional inaccuracies, e.g. *unsymmetrical instead of asymmetrical, *self-explaining instead of self-explanatory. Instances of false friends were observed, e.g. isolation instead of insulation, eventually instead of possibly, and immediate feedback was provided, as such errors can inhibit student comprehension or even lead to acquisition of faulty terminology. Where code consistency was an issue in an observed class, it was almost exclusively tied to figures and terminology on slides which were displayed in German without the teacher explicitly translating/explaining the terms in English, a practice which can exclude and frustrate students with no knowledge of German.
Figure 4 indicates the aggregate results for the communicative competencies of participating teaching staff. While there is a wider score distribution for communicative competencies compared to linguistic competencies, teachers in this programme generally demonstrated solid communicative competencies.
Language use on a micro-level to give lessons cohesion – i.e., signal transitions, highlight important information, communicate connections between previous and/or future lessons – was a recurring strength. Moreover, many lessons included closed and open-ended questions as well as constructive responses to student input. Nevertheless, there were recurring weaknesses. Explicit learning objectives and lesson outlines for macro-level cohesion were not the norm, an important point which the EMI team continuously raised in feedback meetings as a helpful guide for students in an international programme. In addition, some lessons lacked a real attempt at facilitating student input even though the teacher clearly possessed the linguistic competencies to do so. Although the EMI team observed only a handful of opaque references to local culture, i.e., references that require knowledge of the local/German context, students tended to indicate that there were some and this led to a wider score distribution. The EMI team believes this is most likely due to students’ interpretation of the question and this is a criterion that will require more attention in future.
6 Certification of quality teaching in English: Future prospects
As of April 2017, the EMI team had certified in total four English-taught programmes at the University of Freiburg. In addition to Microsystems Engineering, three more programmes, one large programme (>35 lecturers) and two smaller programmes (<10 lecturers each) had been certified. It is planned to assess further English-taught programmes in coming semesters.
Despite mostly positive feedback from teachers on the certification procedure, there are two issues we intend to address moving forward. The first one pertains to the difficulty in assessing intercultural transparency, especially student interpretations of this. As indicated earlier, some lessons scored low on this criterion although the EMI team observed no instances of intercultural murkiness. Could students be answering the question with other aspects of the course in mind, i.e., not exclusively focusing on the one observed class? Researching student opinions and experiences via semi-structured personal or focus group interviews could provide an answer to that question.
The second question addresses whether fluency, grammar, and code consistency should or should not be score-contributing criteria. Our experience so far has shown that participating teachers are fluent speakers with good command of grammar thanks to their extensive experience abroad as students, academics, and/or professionals in their respective fields. Scores for these criteria may be positively skewing overall results. In other words, we may have to simply assume that the linguistic competencies are present and focus exclusively on how well the language is used to teach with a linguistically and culturally heterogeneous student body at the tertiary level. A re-assessment of the existing criteria in theory (through further literature reviews and expert discussions) as well as in practice (testing and piloting a revised version of the criteria in an authentic setting) could bring about meaningful results not only for the EMIQM certification procedure but also on a more general level for discussion of language competencies for teaching with a multilingual, multicultural student body.
Celce-Murcia, Marianne, Zoltan Dörnyei & Sarah Thurrell. 1995. Communicative competence: A pedagogically motivated model with content specifications. Issues in Applied Linguistics 6(2). 5–35.
Gundermann, Susanne. 2014. English-medium instruction: Modelling the role of the native speaker in a lingua franca context. Freiburg: University of Freiburg doctoral dissertation. https://freidok.uni-freiburg.de/data/9795 (accessed 27 July 2017).
Kling, Joyce & Lars Stenius Stæhr. 2012. The development of the test of oral English proficiency for academic staff (TOEPAS). Centre for Internationalisation and Parallel Language Use (CIP), University of Copenhagen. http://cip.ku.dk/forskning/cip_publikationer/CIP_TOPEPAS_Technical_Report.pdf (accessed 27 July 2017).
Pilkinton-Pihko, Diane. 2013. English-medium instruction: Seeking assessment criteria for spoken professional English. Helsinki: University of Helsinki doctoral dissertation. http://urn.fi/URN:ISBN:978-952-10-9520-7 (accessed 24 July 2017)
Smit, Ute. 2010. English as a lingua franca in higher education: A longitudinal study of classroom discourse. Berlin /New York: De Gruyter.
The programme director has given permission for information pertaining to the Masters in Microsystems Engineering to be published.