Development and implementation of an online platform for curriculum mapping in medical education

Introduction

Medical education incorporates a mixture of theoretical and clinical issues that should be well combined and applied in various learning and teaching strategies to ensure the medical students will be competent and thus best prepared for everyday clinical practice. The design of learning and teaching activities is usually reflected in the curriculum and its structure integrated in a particular school or even in a whole education system. Medical schools, similarly to other higher education institutions, move towards electronic support of curriculum management utilising either self-developed or proprietary external tools or technical infrastructure.

In general, a curriculum should be developed in a way that is clear and understandable not only to the school management and teachers, but to the learners too. Furthermore, it should be easily revisable. However, while inexperienced observers often consider the curriculum is merely a list of courses offered by the school, the truth is it refers more in depth to the knowledge and skills management the learners are expected to master upon graduation. This includes various learning standards, objectives, outcomes, learning units, study materials, different assessment methods etc. used in the education process.

In response to ever increasing demands on quality of undergraduate education, universities require effective management strategies to maintain high quality and well-balanced curricula with optimised outcomes. A well-designed quantitative and qualitative overview of the curriculum helps in identification of strengths, weaknesses and gaps. This helps in making suggestions for changes in discussions with curriculum leaders or planners. The process of curriculum evaluation is not easy and requires systematic approaches. However, once completed, it is likely to deliver significant benefits. For example, Wright and Mynett assessed their medical school curriculum context and concluded that the evaluation was useful in enabling them to visualise the strengths and opportunities for improvement in terms of teaching resilience and resourcefulness they focused on [1].

Even if the curriculum is well designed, approved by statutory authorities and successfully implemented into educational practice, it has to be regularly re-evaluated. The typical reasons for such re-evaluation include, but are not limited to, the need to stay up-to-date while teaching and assessing the most relevant topics in the given area applying evidence-based educational methods; to comply with the various changing regulatory policies, directives and standards as well as to be ready to meet the requirements related to the accreditation processes.

Curriculum mapping exposes educational content, applied pedagogy and assessment methods by which it facilitates transparency and communicability of the curriculum. These attributes are essential for explaining and understanding of what is when and how taught [2]. The curriculum mapping activities were shown to be beneficial in various medical and health care related areas and study programs including medicine [3, 4], dentistry [5], nursing [6], gerontology [7], veterinary medicine [2] or pharmacy [8].

Medical terminologies and classification systems play a significant role in curriculum mapping as it was proved in the study of Komenda et al. [9]. As medical education is moving toward outcome-based education, the role of a unified language for categorising and linking all components of a study program is considered essential. This fact was emphasised also in the report of Stoddard and Brownfield [10], who used a controlled vocabulary of 291 identifiers to categorise educational activities. In general, controlled vocabularies of international medical classifications, such as International Statistical Classification of Diseases and Related Health Problems (ICD) or Medical Subject Headings (MeSH) help curriculum designers in semantic indexation of learning objectives and subsequent analysis of a curriculum.

Specification and grouping of learning objectives into a comprehensive catalogue may lead to easier ways to design and analyse curriculum. Furthermore, such catalogue(s) can be used to compare education institutions and their study programs at national or even at international level. For instance in Germany there is the National Competency-based Learning Objective Catalogue for Undergraduate Medical Education (NKLM) and Dental Medicine (NKLZ) [11, 12]. Similar efforts to establish nationwide competency catalogues or frameworks exist in many countries worldwide. For example, Sohrmann et al. [13] presented the national outcomes reference framework for the undergraduate medical curriculum in Switzerland.

Efforts to comprehensively manage a curriculum and design curriculum maps brought various online tools and systems. For instance, Fritze et al. [14] used MERlin, a standardised, common tool for curriculum mapping. As they reported 14 of 38 German faculties used this platform at the time of publication. Another example is the system LOOOP (Learning Opportunities, Objectives and Outcome Platform) in use by several German medical schools [15]. Some other systems (such as “medtrics” [16], OPTIMED [17], prudentia [18], pharmacy curriculum management system [7], CurrMIT [19]) were described in a recent systematic review by Kononowicz et al. [20].

The number and diversity of publications presenting software solutions in curriculum mapping shows high interest in electronic support for the complex task of curriculum management and mapping. Yet, hardly any of those support international design and harmonization of curricula in higher education. For the purposes of improving the long-term process of medical and healthcare curricula harmonization, we initiated a cooperation of five medical faculties from five European countries. These institutions include Jagiellonian University in Krakow, Poland (JU); Masaryk University in Brno, Czech Republic (MU); University of Augsburg, Germany (UAU); University of Medicine and Pharmacy of Iasi, Romania (UMF) and Pavol Jozef Safarik University in Kosice, Slovakia (UPJS). Our research activities, supported by the Erasmus+ strategic partnership entitled Building Curriculum Infrastructure in Medical Education (BCIME), aimed to develop an innovative and well-structured electronic system for curriculum mapping and management, easily applicable in everyday academic practice in several European countries. This paper shares our experiences with building and implementing EDUportfolio, a platform developed within our consortium to facilitate optimisation of a medical curriculum and effective education in the context of health professions education.

Methods

Our research activities were initiated by a survey conducted among medical faculty staff members involved in curriculum development or management at BCIME project institutions at the turn of 2017 and 2018. Above all, our primary goal was to understand the principal ways of curriculum management at partner universities. Additionally, we aimed to identify general requirements for a potential modern Information and communication technology (ICT) – based solution to facilitate efficient curriculum mapping and management. The online structured survey was constructed in Google Forms based on previous experience with curriculum development, management, mapping and harmonisation. In addition, the personal face-to-face discussions with these persons were conducted in order to clarify the purpose of the survey as well as to minimize the risks of misunderstandings or misinterpretations of the survey and/or its questions. The survey was divided into two sections: (i) general information with questions relating to the identification of institutions and relevant staff members and (ii) curriculum management with questions relating to the curriculum development and management. The structure of the questionnaire 1 is presented in Supplementary material of this paper.

In the light of the preliminary general findings, we deepened the analysis of individual expectations and requirements on the electronic curriculum management system by a combination of a second online survey and unstructured interviews and group discussions. In this activity, conducted during the last quarter of 2018, we collected responses from persons involved in curriculum design or administration at our faculties. The second online survey was also divided into two sections: (i) general information with questions relating to the identification of institutions and (ii) curriculum development, management and harmonisation with questions relating to the details the curriculum management platform should offer to their users. The structure of the questionnaire 2 is presented in Supplementary material of this paper. Personal discussions were conducted as part of bilateral meeting either directly at the institution from which we collected the requirements or using remote video conferencing technologies. No predefined scenario was held during these interviews, the primary attention was paid on a process how institution stakeholders are used to work and how the particular curricula are designed and organised in practice. All technical and methodological details were mentioned for easier implementation into the final platform, which was planned to be unified for all partner institutions. We acknowledge that the fact that the interviews were unstructured represent a minor limitation of the study, however we hoped by an unconstrained flow of the discussion to acquire more in-depth and open-ended feedback necessary for our further research and development activities. All the responses were analysed and as a result we prepared a set of requirements that informed the development of guidelines and implementation of the software platform across partners’ institutions.

A compilation of the needs identified at our institutions and the most recent recommendations from the curriculum management literature guided us in the development and implementation of a new curriculum management software platform. During the development phase of the EDUportfolio platform, the emphasis was placed on testing and verification of system functionality, usability, practical applicability and visual appearance. The testing process was performed in two main steps. Firstly, the platform was populated with simulated content to determine the accessibility and functionality of all elements. This approach has worked well because it helped us to detect bugs and errors that have been fixed before releasing the platform and filling it with the real curriculum content. Moreover, ideas and the need for some new features appeared. The platform was tested on multiple devices and browsers by the users and by automated or semi-automated exploratory tests. Secondly, the EDUportfolio platform and its functionalities were tested and verified continuously during pilot real world operation while describing the actual content and metadata of selected study disciplines. Errors and inconsistencies across devices were reported to the development team in order to be fixed. As a result the platform improved according to the identified requirements.

The EDUportfolio platform is based on previously tested methodology, which guides curriculum designers during a complicated process of definition of learning outcomes and learning units devoted to particular courses. EDUportfolio uses a detailed schema of descriptive attributes to specify curriculum in a structured form suitable for semi-automatic data processing. Individual attributes include category, assessment form, duration of teaching, type of teaching, importance, description, keywords, significant terms and study materials. A complete curriculum mapped in EDUportfolio consists of individual building blocks that represent basic units for curriculum development. The main components of curriculum building blocks are study program, discipline, sequence block (course, module, unit, block, clerkship), event (instructional or assessment session) and competence (learning outcomes – measurable description of what students are able to demonstrate in terms of knowledge, skills and values).

To evaluate the curricula described using EDUportfolio platform from the outcome-based education perspective, all the learning outcomes (LO) were coded to cover the following categories: (i) Learning category – cognitive, psychomotor and affective; (ii) Bloom’s knowledge dimension – factual, conceptual, procedural, and metacognitive; (iii) Bloom’s cognitive process dimension – remember, understand, apply, analyse, evaluate and create; (iv) MeSH chapter “Anatomy” and its following main categories – body regions, musculoskeletal system, digestive system, respiratory system, urogenital system, endocrine system, cardiovascular system, nervous system, sense organs, tissues, cells, fluids and secretions, stomatognathic system, hemic and immune system and embryonic system. The text descriptions of learning units (LU) were analysed based on presence of MeSH terms only, as they did not include specific verbs indicating Bloom’s levels. After reaching consensus, the results were presented to and discussed with all partners.

Results

Needs and requirements

As the result of our initial survey all institutions confirmed that the process of curriculum management is challenging but at same time very important for them. No specialized tools, applications or systems of curriculum building were used at partners’ universities. Most of the work related to curriculum design was done manually by staff members in parallel to many other professional activities, without dedicated software-based support. Only UAU had a dedicated curriculum designer role and MU used their own web-based system called OPTIMED for curriculum development [17, 21]. However, the respondents identified their local learning management systems as places where the curriculum information is available to teachers and students, typically in the form of syllabi and course related electronic documents. The representatives of BCIME partner institutions expected the relations of learning objectives/topics in the curriculum should be verified using computerized tools and most importantly, the respondents would like to use electronic curriculum management tools or systems.

A set of requirements was prepared after the subsequent survey and unstructured interviews and group discussions were analysed. This set of requirements informed the development of guidelines and implementation of the software platform across partners’ institutions. The key characteristics and features of the curriculum management system derived from this needs analysis were: (1) Available online; (2) visual overview of curriculum; (3) integration of different user roles; (4) export of curricula by course, study field, department, faculty; (5) visual relations between various components of curriculum; (6) possibilities to search by keywords; (7) integration of international recommendations; (8) possibility to modify reports and outputs according to the institutional requirements; (9) evaluation of learning objectives; (10) identification of redundancies in learning objectives; (11) outcome-based education compatibility and (12) complex reporting based on available curriculum building blocks [22]. Those 12 key characteristics were selected as a coding frame to structure the compilation of recommendations and best practices to improve the platform for medical and healthcare curriculum management based on a literature review [20]. In addition, we extended those points by presenting (13) step-by-step recommendations for a process of implementing the requested features and (14) reflection on generalisability of the results to other disciplines [23].

The graphical overview of particular identified features that should be incorporated into the curriculum management tool is presented in Figure 1.

Figure 1:

A mind-map of features to consider while developing and implementing a curriculum mapping system.

Curriculum management platform

The new curriculum management platform with curriculum mapping functions developed as part of the BCIME project is called EDUportfolio [24]. Our platform was tailored to the needs of all five medical faculties involved in our project that formulated the main input parameters for its development [25, 26]. Based on the needs analysis and review of recent methodologies, the EDUportfolio platform was designed as a web-based system. The technical part of the development was implemented using the PHP Symfony 4.4 framework together with the Twig template engine and the Doctrine ORM library for object mapping. For data storage the PostgreSQL open-source object-relational database system was selected. Yarn was used to manage dependencies on the frontend, and Composer was used to manage backend dependencies for third-party libraries. The Zurb Foundation framework, using the jQuery library, was applied to develop a responsive frontend. The asset administration is dealt by the webpack’s derivate so-called webpack-encore, which comes with the Symfony. Various JavaScript libraries were used for the purposes of individual modules, such as d3.js, NVD3 and Datatables (interactive visualisations of data), or select2.js, sweetalert2.js, jstree.js and featherlight.js (improvements in user experience). Eduportfolio supports a hierarchy of different user roles using access control list mechanism. Implemented visualisations give a comprehensive and interactive overview of the curriculum with various forms of presentations. It allows access to current competency frameworks relevant for the given institution, which includes support of national learning objectives catalogues, but may also contain local institutional target outcomes or specialised competencies of selected medical fields. Controlled vocabularies like the MeSH thesaurus facilitate an intelligent free-text search functionality. The user interface is intuitive and easy to use, but powerful enough to support the stakeholders in coping with the effort of describing and analysing the curriculum. The home screen of the EDUportfolio platform used to map the curricula of all project partners in English is shown in Figure 2.

Figure 2:

The home screen of EDUportfolio.

The system is based on a compounded curriculum model, which makes it possible to define a hierarchy of teaching blocks and thus characterise it in a structured, parametric, way at different levels of detail (e.g. for study program, medical discipline, course, learning unit and learning outcome) in accordance with international standards provided by the MedBiquitous association.

All BCIME project partners use a central EDUportfolio instance, available in English, to manage, share and compare their curricula. The EDUportfolio platform consists of modules as it is summarised in Table 1, through which authorised users manage individual parts of the curriculum.

Table 1:

Modules of EDUportfolio.

Module	Description
User administration module	Administration of individual user accounts and their roles in the system. There are four different user roles including administrator, student, teacher and curriculum designer.
Curriculum development modules	Modules that allow curriculum designers or administrators to view, to add and to modify the content of curriculum. Modules are organised in the top-down principle and correspond to curriculum building blocks: Study programmes, medical disciplines, courses, learning units, learning outcomes and study materials.
Module for curriculum browsing	Module for searching and browsing of existing curriculum, available to all users.
Module for study materials browsing	A module that allows users to browse content not only within the curriculum, but also within the recommended study materials.
Module for curriculum reporting	A standalone module that allows users to examine aggregated statistics about teaching and learning involved in the curriculum maps.
Module for publications	A freely accessible module for overview of all our relevant publications, which relate to the domain of curriculum development and mapping.

The most important modules regarding the curriculum design and curriculum mapping processes are the modules developed to meet requirements of a wide range of educational institutions. They are grouped in the content management option of EDUportfolio main menu. These modules allow users with the role of curriculum designer (typically teachers and guarantors of individual courses) or administrators to view and edit the content of the curriculum. Figure 3 shows the screens of the curriculum development modules.

Figure 3:

Web-based forms of curriculum development modules.

In addition to the core EDUportfolio platform, we installed another five instances in partner’s local languages. These platforms are available for the needs of individual partner institutions to map and describe curricula of their study programs using the provided methodology independently and in their national languages.

Visualisation and data analysis

To guide the visualisation of data we developed a set of curriculum-related research questions, based on our literature review, needs analysis survey and responses of stakeholders while using EDUportfolio. Our aim was that these questions could be answered using EduPortfolio. Project partners rated the collected questions in a nominal group consensus building process [29]. The top 10 prioritised questions include five descriptive and five analytical questions. Descriptive are (1) What should students learn by the end of the unit for which I am responsible?; (2) What is expected of me in a particular course?; (3) What have the students learned before they start my unit?; (4) Who is responsible for this part of the course? and (5) What learning outcomes are covered in year X? Analytical questions are (1) How to identify overlaps in curricula?; (2) Do we have overlaps within and between content domains in our curriculum?; (3) Are the ILOs assessed with appropriate assessment methods? (knowledge, skills and competencies); (4) How balanced is the curriculum in terms of type of taught competency (knowledge, skills and attitudes)? and (5) What are the core elements of well-built curricula?

Teachers and learners can find the answers to the descriptive questions by browsing the curriculum map of EDUportfolio. Additionally, the metadata from EDUportfolio was exported and visualised using selected third-party services, Google Data Studio, which enabled us to turn the data into additional illustrative, readable and shareable forms. Figure 4 shows the overview of this visualisation with all details related to learning outcomes, assessment and responsible guarantors.

Figure 4:

Visualisation of curriculum metadata in Google Data Studio.

In case of analytical questions, the methods for proper text cleaning and pre-processing, work with stop word sheets and metrics for calculating similarity were applied. The best and most credible results in the past, which were also verified by experts from practice, were achieved by using cosine similarity between documents. Therefore, for computation of similarities and finding overlaps as well, the cosine similarity was used. All textual data from the EDUportfolio were used for analyses: description of courses, importance of courses and learning outcomes. Analyses of overlaps were performed on two levels: (i) institutions and (ii) courses. For institutions, the document term matrix (DTM) consisted of five rows (each university was represented by one big document, all textual data were merged together). For courses, the DTM matrix consisted of nine rows (each course was represented by one document). For both levels, description statistics over pre-processed data were performed.

The highest similarity in anatomy curriculum was identified between MU and JU (65.45%). More than 50% similarity was discovered also between UPJS and JU. Conversely, the lowest degree of similarity was shown between UPJS and UAU (only 2.14%). However, little similarity was identified between UAU and all other partners as well, which might be explained by differences in the education system. Similar education systems, as a historical fact, proved higher similarities between UPJS, MU and JU. At the level of individual courses, the highest degree of similarity was identified between anatomy I and anatomy II at MU (90.83%). This also indicates the high level of overlaps in the curriculum of anatomy at MU. High levels of similarity was detected between anatomy with embryology (JU) and anatomy I (MU) with 62.07%; anatomy with embryology (JU) and anatomy I (MU) with 65.88% that confirms similarity of MU and JU curriculum. Similarity levels of 36.58%, 44.96% and 47.80% were detected between JU and UPJS courses, respectively. The similarity values between 25% and 50% were discovered also between MU and UPJS courses. That is consistent with analysis at institution level. The smallest degrees of similarity were identified between UAU and all the other courses reaching maximum similarity level less than 10%. The full report on this study can be found in [30].

Discussion

Well-balanced and organised curriculum mapping tools facilitate the curriculum management in higher education including the linking of learning outcomes, the documentation of compliance with accreditation standards and performing many other administrative and analytical tasks. Hierarchical metadata structures describe the organisation of teaching and assessment ranging from a few study blocks (disciplines) that represent the part of the whole curriculum to the level of individual items in multiple-choice tests. Fine-grained descriptions leverage a powerful search functionality and high quality reporting, but at the same time this generates a lot of effort in data entry and management that might easily go beyond the available time and personnel resources.

A major aspect of our project is its internationalisation and proves general usage in given institutions without any major restrictions. While most of the available systems are national solutions, EDUportfolio serves the needs of five European countries each with individual requirements. The multilingual platform also supports different competency frameworks and, with some minor modifications, can be applied in other healthcare related disciplines or even non-healthcare study programs. The results of our study showed that the number of LOs to describe partners’ anatomy curricula varies significantly due to different levels of granularity. This finding is consistent with recent literature, which still does not give a clear answer on the optimal level of granularity for curriculum mapping. This suggests that there is a need to deliver not only a curriculum system, but also the methodology and recommendations for curriculum mapping and management. Based on our experiences, we recommend describing curricula at a high level of granularity, as such fine-granular LOs are more specific and helpful for learners and educators and also better applicable in analysing a curriculum.

In our study, we mapped only a small part of the complete medical curricula, but we have nevertheless identified elements of interdisciplinarity, i.e. relations between LOs from different medical disciplines, such as anatomy and surgery or radiology. Following our approach and extending it to a whole medical curriculum, we anticipate that the level of interdisciplinarity can be identified.

Finally, we see a need for the development of national competency-based learning objectives catalogues as a national or even European initiative. Such catalogues are currently available in Germany (NKLM) and Poland. LOs catalogues can significantly improve and unify education of particular study programs across educational institutions at national and European level.

Conclusions

This study reported on the experiences of the BCIME project gained while implementing a new online curriculum mapping platform. The project team collected institutional requirements across five European universities, as well as recommendations and good practices from the literature. We integrated this know-how into technical curriculum mapping guidelines and the newly developed web-based platform called EDUportfolio that was implemented and tested across partners’ institutions. The platform consists of modules that offer curriculum designers possibilities to map and visualise their curriculum easily. Thanks to the modules for searching curricula and study materials, teachers as well as students can easily browse and use individual building blocks of the curriculum. An open architecture of the platform allows its further development and extensions.

Curriculum mapping platform has become an indispensable tool in the hands of curriculum designers. Additionally, it facilitates comparisons of study programs and informs formal evaluations. However, development and institutional implementation of such a system is just one stepping stone in the complex process of curriculum management and harmonisation. To achieve the best possible outcomes of student learning, all curricula should be regularly reviewed and updated following a carefully planned curriculum mapping process to ensure that what is declared and delivered is in tandem with what is learned or “tested”. No matter which tool is used, one has to bear in mind the curriculum mapping process still requires significant personal resources. We hope that the presented outcomes of the BCIME project contributed to streamlining this process.