Development and implementation of an online platform for curriculum mapping in medical education

Objectives: Nowadays universities face ever-increasing demands on quality of education, which is crucial from perspective of future graduates. In face of the need of constant quality improvements of medical curricula, it is important to seek strategies for their efficient management. The general trend is to develop electronic support tools to streamlinethecurriculardesign,analysisandharmonization. Methods: Based on the requirements we have identified by the needs analysis among curriculum designers, teachers and managers at five universities involved in the Building Curriculum Infrastructure in Medical Education (BCIME) project, and evidence published in literature on curriculum development, we have developed methodological guidelines on curriculum innovations and a software-based tools that help manage, map and analyse curricula in the medical and healthcare study fields. Results: In this paper, we share our experiences with building and implementation of EDUportfolio, an online platform developed within our consortium and intended to facilitate harmonisation and optimisation of medical outcome-based curricula. Its functionalities and outputs were veri ﬁ ed by pilot mapping of Anatomy curricula as taught at partner universities in ﬁ ve European countries. Conclusions: The visualisation and the analysis of described curriculum data using natural language processing techniques revealed both the hidden relations between curriculum building blocks and a set of overlaps and gaps in curricula. In addition, we demonstrate both the usability of the platform in the context of the involved academic environments and the capability to map and compare curricula across different institutions and different countries.


Introduction
Medical education incorporates a mixture of theoretical and clinical issues that should be well combined and applied in various learning and teaching strategies to ensure the medical students will be competent and thus best prepared for everyday clinical practice. The design of learning and teaching activities is usually reflected in the curriculum and its structure integrated in a particular school or even in a whole education system. Medical schools, similarly to other higher education institutions, move towards electronic support of curriculum management utilising either self-developed or proprietary external tools or technical infrastructure.
In general, a curriculum should be developed in a way that is clear and understandable not only to the school management and teachers, but to the learners too. Furthermore, it should be easily revisable. However, while inexperienced observers often consider the curriculum is merely a list of courses offered by the school, the truth is it refers more in depth to the knowledge and skills management the learners are expected to master upon graduation. This includes various learning standards, objectives, outcomes, learning units, study materials, different assessment methods etc. used in the education process.
In response to ever increasing demands on quality of undergraduate education, universities require effective management strategies to maintain high quality and wellbalanced curricula with optimised outcomes. A welldesigned quantitative and qualitative overview of the curriculum helps in identification of strengths, weaknesses and gaps. This helps in making suggestions for changes in discussions with curriculum leaders or planners. The process of curriculum evaluation is not easy and requires systematic approaches. However, once completed, it is likely to deliver significant benefits. For example, Wright and Mynett assessed their medical school curriculum context and concluded that the evaluation was useful in enabling them to visualise the strengths and opportunities for improvement in terms of teaching resilience and resourcefulness they focused on [1].
Even if the curriculum is well designed, approved by statutory authorities and successfully implemented into educational practice, it has to be regularly re-evaluated. The typical reasons for such re-evaluation include, but are not limited to, the need to stay up-to-date while teaching and assessing the most relevant topics in the given area applying evidence-based educational methods; to comply with the various changing regulatory policies, directives and standards as well as to be ready to meet the requirements related to the accreditation processes.
Curriculum mapping exposes educational content, applied pedagogy and assessment methods by which it facilitates transparency and communicability of the curriculum. These attributes are essential for explaining and understanding of what is when and how taught [2]. The curriculum mapping activities were shown to be beneficial in various medical and health care related areas and study programs including medicine [3,4], dentistry [5], nursing [6], gerontology [7], veterinary medicine [2] or pharmacy [8].
Medical terminologies and classification systems play a significant role in curriculum mapping as it was proved in the study of Komenda et al. [9]. As medical education is moving toward outcome-based education, the role of a unified language for categorising and linking all components of a study program is considered essential. This fact was emphasised also in the report of Stoddard and Brownfield [10], who used a controlled vocabulary of 291 identifiers to categorise educational activities. In general, controlled vocabularies of international medical classifications, such as International Statistical Classification of Diseases and Related Health Problems (ICD) or Medical Subject Headings (MeSH) help curriculum designers in semantic indexation of learning objectives and subsequent analysis of a curriculum.
Specification and grouping of learning objectives into a comprehensive catalogue may lead to easier ways to design and analyse curriculum. Furthermore, such catalogue(s) can be used to compare education institutions and their study programs at national or even at international level. For instance in Germany there is the National Competency-based Learning Objective Catalogue for Undergraduate Medical Education (NKLM) and Dental Medicine (NKLZ) [11,12]. Similar efforts to establish nationwide competency catalogues or frameworks exist in many countries worldwide. For example, Sohrmann et al. [13] presented the national outcomes reference framework for the undergraduate medical curriculum in Switzerland.
Efforts to comprehensively manage a curriculum and design curriculum maps brought various online tools and systems. For instance, Fritze et al. [14] used MERlin, a standardised, common tool for curriculum mapping. As they reported 14 of 38 German faculties used this platform at the time of publication. Another example is the system LOOOP (Learning Opportunities, Objectives and Outcome Platform) in use by several German medical schools [15]. Some other systems (such as "medtrics" [16], OPTIMED [17], prudentia [18], pharmacy curriculum management system [7], CurrMIT [19]) were described in a recent systematic review by Kononowicz et al. [20].
The number and diversity of publications presenting software solutions in curriculum mapping shows high interest in electronic support for the complex task of curriculum management and mapping. Yet, hardly any of those support international design and harmonization of curricula in higher education. For the purposes of improving the long-term process of medical and healthcare curricula harmonization, we initiated a cooperation of five medical faculties from five European countries. These institutions include Jagiellonian University in Krakow, Poland (JU); Masaryk University in Brno, Czech Republic (MU); University of Augsburg, Germany (UAU); University of Medicine and Pharmacy of Iasi, Romania (UMF) and Pavol Jozef Safarik University in Kosice, Slovakia (UPJS). Our research activities, supported by the Erasmus+ strategic partnership entitled Building Curriculum Infrastructure in Medical Education (BCIME), aimed to develop an innovative and well-structured electronic system for curriculum mapping and management, easily applicable in everyday academic practice in several European countries. This paper shares our experiences with building and implementing EDUportfolio, a platform developed within our consortium to facilitate optimisation of a medical curriculum and effective education in the context of health professions education.

Methods
Our research activities were initiated by a survey conducted among medical faculty staff members involved in curriculum development or management at BCIME project institutions at the turn of 2017 and 2018. Above all, our primary goal was to understand the principal ways of curriculum management at partner universities. Additionally, we aimed to identify general requirements for a potential modern Information and communication technology (ICT)based solution to facilitate efficient curriculum mapping and management. The online structured survey was constructed in Google Forms based on previous experience with curriculum development, management, mapping and harmonisation. In addition, the personal face-to-face discussions with these persons were conducted in order to clarify the purpose of the survey as well as to minimize the risks of misunderstandings or misinterpretations of the survey and/or its questions. The survey was divided into two sections: (i) general information with questions relating to the identification of institutions and relevant staff members and (ii) curriculum management with questions relating to the curriculum development and management. The structure of the questionnaire 1 is presented in Supplementary material of this paper.
In the light of the preliminary general findings, we deepened the analysis of individual expectations and requirements on the electronic curriculum management system by a combination of a second online survey and unstructured interviews and group discussions. In this activity, conducted during the last quarter of 2018, we collected responses from persons involved in curriculum design or administration at our faculties. The second online survey was also divided into two sections: (i) general information with questions relating to the identification of institutions and (ii) curriculum development, management and harmonisation with questions relating to the details the curriculum management platform should offer to their users. The structure of the questionnaire 2 is presented in Supplementary material of this paper. Personal discussions were conducted as part of bilateral meeting either directly at the institution from which we collected the requirements or using remote video conferencing technologies. No predefined scenario was held during these interviews, the primary attention was paid on a process how institution stakeholders are used to work and how the particular curricula are designed and organised in practice. All technical and methodological details were mentioned for easier implementation into the final platform, which was planned to be unified for all partner institutions. We acknowledge that the fact that the interviews were unstructured represent a minor limitation of the study, however we hoped by an unconstrained flow of the discussion to acquire more in-depth and open-ended feedback necessary for our further research and development activities. All the responses were analysed and as a result we prepared a set of requirements that informed the development of guidelines and implementation of the software platform across partners' institutions.
A compilation of the needs identified at our institutions and the most recent recommendations from the curriculum management literature guided us in the development and implementation of a new curriculum management software platform. During the development phase of the EDUportfolio platform, the emphasis was placed on testing and verification of system functionality, usability, practical applicability and visual appearance. The testing process was performed in two main steps. Firstly, the platform was populated with simulated content to determine the accessibility and functionality of all elements. This approach has worked well because it helped us to detect bugs and errors that have been fixed before releasing the platform and filling it with the real curriculum content. Moreover, ideas and the need for some new features appeared. The platform was tested on multiple devices and browsers by the users and by automated or semi-automated exploratory tests. Secondly, the EDUportfolio platform and its functionalities were tested and verified continuously during pilot real world operation while describing the actual content and metadata of selected study disciplines. Errors and inconsistencies across devices were reported to the development team in order to be fixed. As a result the platform improved according to the identified requirements.
The EDUportfolio platform is based on previously tested methodology, which guides curriculum designers during a complicated process of definition of learning outcomes and learning units devoted to particular courses. EDUportfolio uses a detailed schema of descriptive attributes to specify curriculum in a structured form suitable for semi-automatic data processing. Individual attributes include category, assessment form, duration of teaching, type of teaching, importance, description, keywords, significant terms and study materials. A complete curriculum mapped in EDUportfolio consists of individual building blocks that represent basic units for curriculum development. The main components of curriculum building blocks are study program, discipline, sequence block (course, module, unit, block, clerkship), event (instructional or assessment session) and competence (learning outcomesmeasurable description of what students are able to demonstrate in terms of knowledge, skills and values).
To evaluate the curricula described using EDUportfolio platform from the outcome-based education perspective, all the learning outcomes (LO) were coded to cover the following categories: (i) Learning categorycognitive, psychomotor and affective; (ii) Bloom's knowledge dimensionfactual, conceptual, procedural, and metacognitive; (iii) Bloom's cognitive process dimensionremember, understand, apply, analyse, evaluate and create; (iv) MeSH chapter "Anatomy" and its following main categoriesbody regions, musculoskeletal system, digestive system, respiratory system, urogenital system, endocrine system, cardiovascular system, nervous system, sense organs, tissues, cells, fluids and secretions, stomatognathic system, hemic and immune system and embryonic system. The text descriptions of learning units (LU) were analysed based on presence of MeSH terms only, as they did not include specific verbs indicating Bloom's levels. After reaching consensus, the results were presented to and discussed with all partners.

Needs and requirements
As the result of our initial survey all institutions confirmed that the process of curriculum management is challenging but at same time very important for them. No specialized tools, applications or systems of curriculum building were used at partners' universities. Most of the work related to curriculum design was done manually by staff members in parallel to many other professional activities, without dedicated software-based support. Only UAU had a dedicated curriculum designer role and MU used their own web-based system called OPTIMED for curriculum development [17,21]. However, the respondents identified their local learning management systems as places where the curriculum information is available to teachers and students, typically in the form of syllabi and course related electronic documents. The representatives of BCIME partner institutions expected the relations of learning objectives/topics in the curriculum should be verified using computerized tools and most importantly, the respondents would like to use electronic curriculum management tools or systems.
A set of requirements was prepared after the subsequent survey and unstructured interviews and group discussions were analysed. This set of requirements informed the development of guidelines and implementation of the software platform across partners' institutions. The key characteristics and features of the curriculum management system derived from this needs analysis were: (1) Available online; (2) visual overview of curriculum; (3) integration of different user roles; (4) export of curricula by course, study field, department, faculty; (5) visual relations between various components of curriculum; (6) possibilities to search by keywords; (7) integration of international recommendations; (8) possibility to modify reports and outputs according to the institutional requirements; (9) evaluation of learning objectives; (10) identification of redundancies in learning objectives; (11) outcome-based education compatibility and (12) complex reporting based on available curriculum building blocks [22]. Those 12 key characteristics were selected as a coding frame to structure the compilation of recommendations and best practices to improve the platform for medical and healthcare curriculum management based on a literature review [20]. In addition, we extended those points by presenting (13) stepby-step recommendations for a process of implementing the requested features and (14) reflection on generalisability of the results to other disciplines [23].
The graphical overview of particular identified features that should be incorporated into the curriculum management tool is presented in Figure 1.

Curriculum management platform
The new curriculum management platform with curriculum mapping functions developed as part of the BCIME project is called EDUportfolio [24]. Our platform was tailored to the needs of all five medical faculties involved in our project that formulated the main input parameters for its development [25,26]. Based on the needs analysis and review of recent methodologies, the EDUportfolio platform was designed as a web-based system. The technical part of the development was implemented using the PHP Symfony 4.4 framework together with the Twig template engine and the Doctrine ORM library for object mapping. For data storage the PostgreSQL open-source object-relational database system was selected. Yarn was used to manage dependencies on the frontend, and Composer was used to manage backend dependencies for third-party libraries. The Zurb Foundation framework, using the jQuery library, was applied to develop a responsive frontend. The asset administration is dealt by the webpack's derivate so-called webpack-encore, which comes with the Symfony. Various JavaScript libraries were used for the purposes of individual modules, such as d3.js, NVD3 and Datatables (interactive visualisations of data), or select2.js, sweet-alert2.js, jstree.js and featherlight.js (improvements in user experience). Eduportfolio supports a hierarchy of different user roles using access control list mechanism. Implemented visualisations give a comprehensive and interactive overview of the curriculum with various forms of presentations. It allows access to current competency frameworks relevant for the given institution, which includes support of national learning objectives catalogues, but may also contain local institutional target outcomes or specialised competencies of selected medical fields. Controlled vocabularies like the MeSH thesaurus facilitate an intelligent free-text search functionality. The user interface is intuitive and easy to use, but powerful enough to support the stakeholders in coping with the effort of describing and analysing the curriculum. The home screen of the EDUportfolio platform used to map the curricula of all project partners in English is shown in Figure 2.
The system is based on a compounded curriculum model, which makes it possible to define a hierarchy of teaching blocks and thus characterise it in a structured, parametric, way at different levels of detail (e.g. for study program, medical discipline, course, learning unit and All BCIME project partners use a central EDUportfolio instance, available in English, to manage, share and compare their curricula. The EDUportfolio platform consists of modules as it is summarised in Table 1, through which authorised users manage individual parts of the curriculum. The most important modules regarding the curriculum design and curriculum mapping processes are the modules developed to meet requirements of a wide range of educational institutions. They are grouped in the content management option of EDUportfolio main menu. These modules allow users with the role of curriculum designer (typically teachers and guarantors of individual courses) or administrators to view and edit the content of the curriculum. Figure 3 shows the screens of the curriculum development modules.
In addition to the core EDUportfolio platform, we installed another five instances in partner's local languages. These platforms are available for the needs of individual partner institutions to map and describe curricula of their study programs using the provided methodology independently and in their national languages.

Metadata description
To process the data related to a curriculum and to allow particular groups of users advanced mapping and overview features, we extracted and entered the content of the Anatomy curricula from five BCIME project partner institutions into the newly developed platform. This step enabled text-based analyses. All partners used their real curriculum content as it is taught at their universities and processed it in English for subsequent comparative studies. Latin terms were specified in keywords too, as it is commonly used in this medical discipline. The entered descriptions were heterogeneous across the consortium, as the curricula have followed different models, have been at various stages of development, and there have been differences in education systems in partner countries. For example, the curriculum metadata at UAU was translated and entered into the EDUportfolio in the form of learning outcomes. They used the learning outcomes that relate to anatomy from the NKLM catalogue and mapped them to the four modules they plan to develop for their curriculum. Since UAU does not have a discipline-based curriculum, this was a feasible way to do this metadata description and without generating any "artificial" learning units in the curriculum description. On the other hand, MU and UPJS curricula showed a high similarity as their educational systems share common roots and tradition for historical reasons, even though there are no national learning outcomes or learning objectives catalogues in these partner countries.
A summary of the Anatomy metadata description, entered in the EDUportfolio platform, is presented in Table 2.
Furthermore, one complementary discipline was described by each partner university in their local instance of EDUportfolio. In this way, we tested the local instances with data from complementary disciplines curricula including conservative and prosthetic dentistry, clinical and communication skills, analysis, data management and informatics for healthcare specialisation and neurosurgery [27].

Module Description
User administration module Administration of individual user accounts and their roles in the system. There are four different user roles including administrator, student, teacher and curriculum designer.

Curriculum development modules
Modules that allow curriculum designers or administrators to view, to add and to modify the content of curriculum. Modules are organised in the top-down principle and correspond to curriculum building blocks: Study programmes, medical disciplines, courses, learning units, learning outcomes and study materials. Module for curriculum browsing Module for searching and browsing of existing curriculum, available to all users.

Module for study materials browsing
A module that allows users to browse content not only within the curriculum, but also within the recommended study materials. Module for curriculum reporting A standalone module that allows users to examine aggregated statistics about teaching and learning involved in the curriculum maps. Module for publications A freely accessible module for overview of all our relevant publications, which relate to the domain of curriculum development and mapping.

Outcome-based curriculum evaluation
An in-depth analysis of the learning outcomes (LOs) and learning unit (LU) descriptions used to describe the curricula of anatomy was realised in a quantitative as well as in a qualitative way. These LOs and descriptions have been reviewed and validated by content matter experts. First, we quantitatively analysed the LOs and LU descriptions provided by each partner, while the irrelevant  words were removed. The number of learning outcomes to describe the curriculum of Anatomy at partner medical schools ranges from 13 (JU) up to 225 (UPJS), indicating a heterogeneous level of granularity and different mapping approach, well-known challenges in curriculum mapping. The LOs have been defined either on course level (JU and UAU) or LU level (MU, UMF and UPJS). No LOs have been specified on a module level. The most frequently used terms in LOs include "explains", "structures", "systems", "describes," and "functions". In LOs, the most frequent verbs were "explain", "describe", "identify", "name" and "characterise". Second, for qualitative analysis, all LOs were coded deductively and independently by two experienced BCIME team members (from JU and UAU) with a health profession education background. Divergent codings were solved by discussion and consensus has been reached in all cases. For the purpose of the coding a coding guideline has been developed [28]. From the total of 805 LOs, 794 have been categorized as cognitive, four as affective (all from JU), and seven as psychomotor (five from UMF, one from MU and one from UPJS). The psychomotor LOs covered physical examination skills for which anatomical knowledge is essential, such as palpation. The affective LOs covered aspects such as professionalism and teamwork. The 794 cognitive LOs were further coded based on Bloom's taxonomy ( Table 3). LOs of most partners were categorized as factual, except for UAU, where about 67% of LOs were categorized as conceptual and only about 30% as factual. Procedural LOs were less common for all partners.

Visualisation and data analysis
To guide the visualisation of data we developed a set of curriculum-related research questions, based on our literature review, needs analysis survey and responses of stakeholders while using EDUportfolio. Our aim was that these questions could be answered using EduPortfolio. Project partners rated the collected questions in a nominal group consensus building process [29]. The top 10 prioritised questions include five descriptive and five analytical questions. Descriptive are (1) What should students learn by the end of the unit for which I am responsible?; (2) What is expected of me in a particular course?; (3) What have the students learned before they start my unit?; (4) Who is responsible for this part of the course? and (5) What learning outcomes are covered in year X? Analytical questions are (1) How to identify overlaps in curricula?; (2) Do we have overlaps within and between content domains in our curriculum?; (3) Are the ILOs assessed with appropriate assessment methods? (knowledge, skills and competencies); (4) How balanced is the curriculum in terms of type of taught competency (knowledge, skills and attitudes)? and (5) What are the core elements of well-built curricula?
Teachers and learners can find the answers to the descriptive questions by browsing the curriculum map of EDUportfolio. Additionally, the metadata from EDUportfolio was exported and visualised using selected thirdparty services, Google Data Studio, which enabled us to turn the data into additional illustrative, readable and shareable forms. Figure 4 shows the overview of this visualisation with all details related to learning outcomes, assessment and responsible guarantors.
In case of analytical questions, the methods for proper text cleaning and pre-processing, work with stop word sheets and metrics for calculating similarity were applied. The best and most credible results in the past, which were also verified by experts from practice, were achieved by using cosine similarity between documents. Therefore, for computation of similarities and finding overlaps as well, the cosine similarity was used. All textual data from the EDUportfolio were used for analyses: description of courses, importance of courses and learning outcomes. Analyses of overlaps were performed on two levels: (i) institutions and (ii) courses. For institutions, the document term matrix (DTM) consisted of five rows (each university was represented by one big document, all textual data were merged together). For courses, the DTM matrix consisted of nine rows (each course was represented by one document). For both levels, description statistics over pre-processed data were performed. The highest similarity in anatomy curriculum was identified between MU and JU (65.45%). More than 50% similarity was discovered also between UPJS and JU. Conversely, the lowest degree of similarity was shown between UPJS and UAU (only 2.14%). However, little similarity was identified between UAU and all other partners as well, which might be explained by differences in the education system. Similar education systems, as a historical fact, proved higher similarities between UPJS, MU and JU. At the level of individual courses, the highest degree of similarity was identified between anatomy I and anatomy II at MU (90.83%). This also indicates the high level of overlaps in the curriculum of anatomy at MU. High levels of similarity was detected between anatomy with embryology (JU) and anatomy I (MU) with 62.07%; anatomy with embryology (JU) and anatomy I (MU) with 65.88% that confirms similarity of MU and JU curriculum. Similarity levels of 36.58%, 44.96% and 47.80% were detected between JU and UPJS courses, respectively. The similarity values between 25% and 50% were discovered also between MU and UPJS courses. That is consistent with analysis at institution level. The smallest degrees of similarity were identified between UAU and all the other courses reaching maximum similarity level less than 10%. The full report on this study can be found in [30].

Discussion
Well-balanced and organised curriculum mapping tools facilitate the curriculum management in higher education including the linking of learning outcomes, the documentation of compliance with accreditation standards and performing many other administrative and analytical tasks. Hierarchical metadata structures describe the organisation of teaching and assessment ranging from a few study blocks (disciplines) that represent the part of the whole curriculum to the level of individual items in multiple-choice tests. Fine-grained descriptions leverage a powerful search functionality and high quality reporting, Majerník et al.: Online platform for curriculum mapping in medical education but at the same time this generates a lot of effort in data entry and management that might easily go beyond the available time and personnel resources.
A major aspect of our project is its internationalisation and proves general usage in given institutions without any major restrictions. While most of the available systems are national solutions, EDUportfolio serves the needs of five European countries each with individual requirements. The multilingual platform also supports different competency frameworks and, with some minor modifications, can be applied in other healthcare related disciplines or even non-healthcare study programs. The results of our study showed that the number of LOs to describe partners' anatomy curricula varies significantly due to different levels of granularity. This finding is consistent with recent literature, which still does not give a clear answer on the optimal level of granularity for curriculum mapping. This suggests that there is a need to deliver not only a curriculum system, but also the methodology and recommendations for curriculum mapping and management. Based on our experiences, we recommend describing curricula at a high level of granularity, as such fine-granular LOs are more specific and helpful for learners and educators and also better applicable in analysing a curriculum.
In our study, we mapped only a small part of the complete medical curricula, but we have nevertheless identified elements of interdisciplinarity, i.e. relations between LOs from different medical disciplines, such as anatomy and surgery or radiology. Following our approach and extending it to a whole medical curriculum, we anticipate that the level of interdisciplinarity can be identified.
Finally, we see a need for the development of national competency-based learning objectives catalogues as a national or even European initiative. Such catalogues are currently available in Germany (NKLM) and Poland. LOs catalogues can significantly improve and unify education of particular study programs across educational institutions at national and European level.

Conclusions
This study reported on the experiences of the BCIME project gained while implementing a new online curriculum mapping platform. The project team collected institutional requirements across five European universities, as well as recommendations and good practices from the literature. We integrated this know-how into technical curriculum mapping guidelines and the newly developed web-based platform called EDUportfolio that was implemented and tested across partners' institutions. The platform consists of modules that offer curriculum designers possibilities to map and visualise their curriculum easily. Thanks to the modules for searching curricula and study materials, teachers as well as students can easily browse and use individual building blocks of the curriculum. An open architecture of the platform allows its further development and extensions.
Curriculum mapping platform has become an indispensable tool in the hands of curriculum designers. Additionally, it facilitates comparisons of study programs and informs formal evaluations. However, development and institutional implementation of such a system is just one stepping stone in the complex process of curriculum management and harmonisation. To achieve the best possible outcomes of student learning, all curricula should be regularly reviewed and updated following a carefully planned curriculum mapping process to ensure that what is declared and delivered is in tandem with what is learned or "tested". No matter which tool is used, one has to bear in mind the curriculum mapping process still requires significant personal resources. We hope that the presented outcomes of the BCIME project contributed to streamlining this process.
Acknowledgements: Results presented in this paper were obtained with the support of the Erasmus+ project 2018-1-SK01-KA203-046318 "Building Curriculum Infrastructure in Medical Education" and the national agency grant KEGA 011UPJS-4/2019 "Increasing of competences and critical thinking level in students of medical study programs using simulation tools of Problem-Based Learning and Evidence-Based Medicine". We would like to acknowledge the help of those collaborators who helped in the BCIME project at some stage but were not mentioned in the authors' list.
Research funding: Parts of this research were funded by the European Commission, ERASMUS+ BCIME -Building Curriculum Infrastructure in Medical Education program (reg. no.: 2018-1-SK01-KA203-046318). European Commision, ERASMUS+ programme played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication. Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission. Competing interests: Authors state no conflict of interest. Informed consent: Not applicable. Ethical approval: Not applicable.