Flipped learning in faculty development programs: opportunities for greater faculty engagement, self-learning, collaboration and discussion

Objectives: This study aimed to evaluate the impact of flipped learning in faculty development program (FDP) dedicated for developing key-feature questions (KFQs), and to compare examinees ’ success and their opinions for the KFQs in a board exam. Methods: The study group consisted of two cohorts – Pediatric Endocrinology and Diabetes Society Board members (n=11) and Pediatric Endocrinology Subspecialty 2019 Board Examination examinees (n=22). We used ﬂ ipped learning approach to develop KFQs. This entailed writing KFQs while learning took place online. The discussion and re ﬁ nement of the questions for the exam were done during the face-to-face meeting. At the end, feedback was taken from both cohorts. Results: The faculty members developed 14 KFQs during the flipped learning process. They found the training useful (Md=9, IQR=1), and expressed high satisfaction (Md=9, IQR=1)for ﬂ ippedlearningapproach.According tothefaculty members, peer review was an important step in the ﬂ ipped learning instructions for revising and improving the KFQs. Conclusions: A successful intervention of flipped learning approach has been tested in a FDP dedicated for KFQ writing. During flipped learning, the board (faculty) members were more engaged, motivated for self-learning and collaboration, and encouraged for robust face-to-face discussions.


Introduction
Faculty development programs (FDPs) help faculty members throughout their careers [1], however, attendance and engaging participation in the program activities are challenging issues [2]. Clinician educators especially, have a lower engagement in the activities than the program demands [3][4][5]. Many approaches to faculty development (e.g., seminars, longitudinal programs) showed successful results albeit self-reported [6,7]. Studies involving students, the true end-user of the outcomes of faculty development, are largely overlooked to show the effectiveness of the programs [8,9]. Faculty development is required to alternate approaches to promote and cultivate interactive experiences as suggested in the future trends of faculty development [10,11]. During the coronavirus disease 2019 (COVID-19) pandemic, many faculty have experienced online teaching and learning [12]. We assume that when the pandemic is over, blending online and face-to-face programs will be expected. However, the literature in faculty development suffers from the lack of empirical evidence on how learner centered approaches can contribute to faculty learning and effect end-users on the outcomes of faculty development.
Flipped learning, a learner-centered approach, utilizes active learning, engagement, motivation, and attainment [13,14] to achieve faculty development goals. Lately, flipped learning is influencing the conventional learning methods in medical education [15]. Active learning supported by cutting-edge technology enables an improved learning process and significantly contributes to both learning and teaching in medical education [16][17][18]. The benefits of flipped learning are apparent not only for trainees, but also provide trainers the opportunity to experience flipped learning, while participating in faculty development activities. Faculty members learn effectively and contribute more under flexible learning atmosphere [19]. However, sparing time from the workload for the same, although not obligatory may become a low priority. The present study evaluates the flipped learning approach in designing a FDP to improve assessment skills of clinician educators in an engaging manner.
The assessment of higher cognitive skills such as clinical decision making is an important aspect of health science education. A vast number of methods are used to assess this aspect of the knowledge domain. Page et al. [20] focused on the key-features (KFs) approach in assessing the key elements of clinical decision making (i.e., certain steps of information gathering, diagnosis, diagnostic tests or patient management) which is essential for the resolution of any clinical problem [21,22]. Examinees are most likely to make errors regarding KF, which is difficult/challenging aspect of a clinical problem, however a critical step towards the resolution of a problem [23]. Key-features questions (KFQs), based on KFs, starts with a brief clinical scenario followed by two to three short answer questions with write-in or short/long menu formats (Box 1) [21][22][23]. These questions target to assess the unique aspects in each case. An example case with questions is given in Box 1 (see Supplementary Appendix 1 for answers with scoring key).
Until now, the KFs assessment is not utilized widely, and faculty members are not familiar with it. Hence, faculty development and training focused on KFQs formulation is central to the assessment of cognitive skills such as clinical Box 1: Example of a key-feature question.

Adapted from the Turkish Pediatric Endocrinology and Diabetes Society Board Examination (2019)
Case-1 A four-week-old baby boy was referred to you for his TSH level 7 mIU/mL measured in the newborn screening program. It was stated that the baby was born at 40 weeks of gestation, 3,200 g, 48 cm with normal spontaneous delivery. In the physical examination, height was at 25th percentile, weight at 25th percentile; penis length was 2.5 cm, testicles 1 mL with high scrotal location; sclera and trunk were icteric and liver 1 cm palpable. Other system examinations revealed no pathological findings.
Diagnostic tests TSH 1.6 mIU/mL (0.8-10) T. bilirubin 10.8 mg/dL FT4 0.6 ng/dL (0.7-1.9) D. bilirubin 3 mg/dL AST 60 U/L ALP 310 U/L ALT 80 U/mL GGT 90 U/L Blood glucose was reported as 30 mg/dL CBC and urine sediment were within normal limits Answer the three (3) questions below using the information given in Case-1 How will you manage this patient at this point? List only one (1). 1. ______________ decision making. While KFQs provide high stake assessment tool, the development of a KFQ is relatively hard and complex and requires extensive training on faculty. While practicing on developing KFQs will improve quality, active learning and peer feedback may increase the quality of the questions. Flipped learning approach allows active participation and peer discussion while theoretical learning is self-regulated and takes place online. That is being said, flipped learning has high potential to examine in complex learning skills in FDPs.
Despite growing literature on online learning for faculty development, little is known on how flipped learning can allow active learning to improve the learning of faculty members. Moreover, little attention has been paid to the learners who are actively affected by the outcomes of faculty development. Keeping aforementioned scenario of learning approach of FDPs in view, the present study was designed to evaluate the intervention of flipped learning approach in FDP dedicated for developing KFQs, and to gather examinees' opinions for the KFQs in the board exam.

Study group
The study sample consisted of two cohorts -The Turkish Pediatric Endocrinology and Diabetes Society (TPEDS) Board Members (n=11) and Pediatric Endocrinology Subspecialty 2019 Board Examination participants (n=22). We excluded two board members who neither completed the online training nor participated in the board meetings for the exam preparation thus only nine board members (81.8%) participated in the study.

Study setting
The TPEDS has been conducting board examinations since 2015. The paper-pencil exam consists of 100 multiple-choice questions (MCQ) aiming to assess examinees' basic knowledge as well as clinical reasoning skills. The TPEDS Board examination is open to Pediatric Endocrinology specialists and to the candidates in their last year of training at the time of the exam. In year 2018, the TPEDS Board members deliberated upon the ways to improve the quality of the examinations and eventually decided to implement KFQs after reviewing various question formats. Subsequently, we developed a FDP using flipped learning to teach KFQs.

Procedures
Flipped learning procedure: An asynchronous online learning environment affording faculty members to learn at their own pace and convenience was created. OpenLabyrinth software, used to host our instructional design process, is an open-source platform for creating conditional contents based on the answer provided by the user [24]. A total of four germane interactive videos on KFQs were recorded using a Green box video studio. The recorded videos were reviewed by three external experts to ascertain and fulfill the specific needs of flipped learning and KFQs. Our instructional design primarily focused on developing KFQs while learning through the system. The overall learning environment was revised and updated after the feedback of the review by three experts from pediatric endocrinology. The development of a KFQs framework was based on the literature [20,22,23,25,26] in 10 steps guided design of the training (Supplementary Appendix II).
There were short explanations including an example for completing each step along with a short video pertaining to the respective section. The online learning environment also supported detailed learning experience wherein each step was explained conceptually by the instructor.
While progressing in the flipped learning, we created a What-sApp group for the support and discussion. The recruited faculty members used synchronous online support during the training phase via this platform.

Study procedure
The faculty members followed flipped learning for a week (March 2019) prior to the TPEDS board meeting. During this time, they attended the training and developed KFQs. The evaluation took place online before the face-to-face meeting. At the end of the process, the owner of the questions (question setter) received all the review feedback and revised them accordingly. Finally, faculty members discussed and revised the KFQs during the board meeting and selected the best five for inclusion in the exam set. A consensus was achieved on the selection of KFQs according to the relevancy with the daily practice and incidence of complaints. Each KFQ consisted of two to three short answer write-in questions related to the case scenarios and three points allocated to each KFQs (Supplementary Appendix I). The exam was held in April 2019 and consisted of 85 MCQs and 5 KFQs.

Data collection and analysis
The feedback of the recruited faculty members was collected using two online questionnaires after completion of the training. The first survey questionnaire was seeking feedback on flipped learning process using 10 structured and eight semi-structured free-text items. The faculty members were asked to rate their flipped learning experience using a Likert type scale (1: Strongly disagree, 10: Strongly agree) based on the structured items and express their opinions in a short write-up against the semi-structured items. The faculty members reviewed their peers' KFQs using the second form, which consisted of three structured and one open ended question dealing with the quality and the appropriateness of each KFQ in the exam. Therefore, we included free-text questions from feedback survey and communication logs held over WhatsApp to qualitative data analysis.
In addition, the results of the exam were included in the overall analysis. The examinees' feedback about KFQs was gathered via inclusion of four structured items to be rated with a Likert type scale (1: Strongly disagree, 9: Strongly agree) and three free-text items in the post-exam survey. The free-text questions of the feedback form were included in qualitative data. Faculty members signed a consent form and the examinees were verbally informed about the study before data collection. Those who volunteered to participate filled out the questionnaires.
The quantitative data have been presented with descriptive tables. Percentage distributions were used for categorical variables, whereas mean (M), standard deviation (SD), median (Md) and interquartile range (IQR) were used for the numerical variables. One-way variance analysis was used for the comparisons and the confidence interval as p<0.05 was considered significant. Cronbach's alpha was calculated for internal consistency. Microsoft Excel and IBM SPSS 21.0 were used for the analysis.
We used six-steps approach for qualitative data analysis as described by Braun and Clarke [27]. Qualitative data had three sources: faculty feedback form, faculty WhatsApp communication logs, and examinees feedback form. We gathered all the responses and the logs in a Word document and familiarized with the data. Next, two researchers with a qualitative data analysis background created initial codes and searched for themes. On another meeting, we discussed the themes and refined them by creating their final theme names. While reporting our results, we quoted participants where it is applicable.

Ethics
The data collection in the present study was conducted after the approval of Ege University Scientific Research, and Publication Ethics Boards dated 14 February 2019 Ref. 147.

Online training results
During the flipped learning training, out of 11 board members, nine attended the online learning part and they developed 14 KFQs. Each faculty member developed at least one KFQ, while five developed two, followed by review and feedback of the other participants (Table 1).

Board members' feedback on learning process
All the nine board members gave feedback for flipped learning process (Table 2). Based on the analysis, the faculty members found the training useful (Md=9, IQR=1). They expressed high satisfaction (Md=9, IQR=1) for receiving the desired information on KFQs development. Training not only encouraged the faculty members to develop KFQs (Md=9, IQR=1) but it also imparted technique of framing new question type using an appropriate tool for the evaluation of clinical decisions (Md=9, IQR=2). Flipped learning guided the faculty members to organize their learning time effectively and develop KFQs (Md=7, IQR=1).
Four main themes emerged from the free-text question and WhatsApp communication logs analysis which are interactive learning experience, time management, critical appraisal through peer review, and drawbacks of flipped learning. We explained the themes below along with quotes from faculty participants.
All the participants found the given videos instructive and helped them to understand KFQs and its concept quickly. One participant commented "The videos taught me how to develop a KFQ", whereas another participant said: "I was able to watch and develop a question on my own time." One participant stressed "The discussion during the face-to-face meeting was deeper, and I expected Is the current form of KFQ ready to use for the exam?
Please write your revision suggestions. Faculty#: Question  can be formulated as "Which hormonal imbalances do you consider at this time?'". Faculty#: In question , question stem should be re-written in a clear way and "thyrotropin-releasing hormone (TRH)" should be included as a correct answer. Faculty#: Regarding to the answers of Q -I think-"starting thyroxine + hydrocortisone treatment" should be according to low-dose ACTH-stimulation test result. Also "starting thyroxine + hydrocortisone treatment until the ACTH-stimulation test result arrives" should be considered as a correct answer in this case". Y: Yes, N: No, C: Yes, but with corrections. more comprehensive information on videos." Although, a detailed information was provided in the additional documents of the training material, more comprehensive information (detailed) based videos were desired by the faculty members.
The faculty members found the overall flipped learning process impressive and effective, since the learning is aligned with real-time working (i.e., developing KFQs, while learning it). One faculty member abridged the process as "I think it is very effective. I was able to learn KFQs completely during online learning. I understand that discussions following the feedback is a crucial process for the learning." Another comment received was "It is not suitable for my age and my workload … " however they also expressed their view as "I think it (i.e., flipped learning) is very good. First you learn on your own by introducing the new information, then you discuss more (with your peers)." The faculty members described the learning process as timesaver, comprehensive, efficient, and providing a good preparation in advance for face-to-face discussion.
The faculty members were quite content about their online learning experience and a new type of assessment strategy for clinical decision making by developing KFQs. "I have understood that it (KFQs) is an important method to emphasize the topic of what I want to teach." and they compared it over MCQs, "KFQs can provide more objective assessment and better than MCQs by avoiding memorization." Moreover, they were satisfied with the instructional design (i.e., organization, amount, and presentation of the content). The faculty members mentioned that they would prefer the same approach for learning new content in the future.
We sought opinions of the faculty members about timing and participation in online training. The faculty members informed that they participated in online training as much as possible. Whereas very few participants mentioned that office workload and hectic clinical duties restricted them to follow online training comprehensively, but those respondents at least completed the requisite steps.
Peer review was a crucial step in the flipped learning instruction to revise and improve the KFQs. The participants were highly satisfied and delighted about this intervention since the peer feedback helped them to fix the points often overlooked. Most of the participants stressed how the feedback enabled them to improve or correct their questions. One of the participants highlighted the peer review process as "I have realized that some questions cannot be KFQs for PE residents. I noticed that some of my questions are for residents in pediatrics." In addition to the positive aspects of the flipped learning approach, the faculty members did comment on its drawbacks. Although faculty did not have any issues during the training, few faculty foresee that potential issues may arouse in the future for those who have busy workload. One faculty member pointed out KFQ as "it's hard to prepare and time-consuming." They also requested more video-based learning content in the future.

Examinees' feedback
The examinees liked the KFQs and found that the questions were aligned with their daily clinical practice (Md=7, IQR=2). One important finding based on examinees' feedback was that KFQs assessed clinical problem-solving skills of the examinees (Md=7, IQR=2). Examinees found that the difficulty of KFQs was moderate (Md=5, IQR=2), and they were in favor of the integration of KFQs in future board exams (Md=8, IQR=2) ( Table 4). Analysis on the free text responses of the examinees revealed two themes: satisfaction with the overall quality of the exam and alignment with daily practice. One examinee commented on KFQs as "I have seen that the exam is more clinically oriented compared to the previous years, so I think it is very useful for aligning my daily practice with the exam questions."

Discussion
This study examined flipped learning approach in a FDP aimed for KFQs teaching and evaluated its results from the end-user achievement. Our findings showed that faculty were able to develop high-stake assessment questions using flipped learning. Faculty reported satisfaction with the flipped learning approach with specific focus on self-paced learning in which they were able to complete the tasks. They also reported that peer discussion and feedback enabled them to increase quality of the KFQS. The examinees found KFQs are useful in their assessment.
Flipped learning is a new window into a learning approach for the FDPs that offers a greater faculty engagement, self-and collaborative learning along with rich faceto-face discussions. Earlier studies have proven that busy schedule especially due to clinical responsibilities and associated tasks are the main shortcomings for a successful faculty development strategy [6,7,28]. However, the prioritization may differ on what to do [29], and is a key element in the decision making to eliminate or include in the format of the program. Flipped learning has the features of timely delivery, focus oriented, outcome-based training [30], which may provide prioritization for faculty development. Instant messaging platforms for discussion and help purposes enable just-in-time communication and provide resolution rapidly [31]. Our findings showed that faculty members easily solicited help over WhatsApp group and received instant replies to their questions which enabled quick resolution for improvement. This approach is quite useful when faculty members allocate their time for research, teaching, and other duties in their role. Instant resolutions save faculty time and get the task done within the allocated time.
In this study, the faculty members were required to attend the face-to-face meeting with the KFQs they had prepared. Therefore, they were able to prioritize the development of KFQs and took the online learning in a serious and a timely manner. More than half of the faculty members developed two KFQs. Developing more than one question was desired because of repeating of the learning process. This suggests that the faculty members' prioritization align with completing the minimum requirements, and leave out extras, when they do not have time. Flipped learning enables self-regulated learning where every faculty can learn and complete their task on their schedule. In our study group, every faculty developed their KFQs in their own time while learning how to develop it. The design of flipped learning environment allow many opportunities for online learning and in-person discussions in medical education [32], as our study provides similar insights toward faculty development.
The experiences of the faculty members were very positive for flipped learning. They mentioned that the learning process was enjoyable during the development of KFQs. Flipped learning enabled the faculty members to participate actively and impact their skills related to education, similar to other flipped learning [33][34][35]. Flipped learning also facilitates self-paced learning wherein the faculty members can learn during the training process and practice the skills (in the present case, KFQs development) related to the educational outcomes. Flipped learning is an important approach for harnessing result oriented outcomes from workplace learning [30]. Based on faculty members' feedback on the learning process, we found that flipped learning is a rigorous training approach to support rich (means, result oriented, meaningful, and extensive) discussion while learning. Flipped learning-based FDP of KFQs development provided online as well as in-person social interaction with the participants and offered possibilities of greater faculty engagement, self-learning, collaboration and extensive discussion contrary to what critics say about online learning [19]. Developing and tailoring custom content for KFQs can be extremely helpful to accomplish the results of the study. A step-by-step supported with time-line methodology guide template as utilized in this study for implementing flipped learning in FDP dedicated for KFQs development has been provided as Supplementary Appendix III. In this study, the participants were able to develop KFQs content by employing flipped learning design elements. This design approach may be helpful particularly for flipped learning wherein it reduces split-attention in the content thereby cognitive load [36].
Interestingly, all the faculty members were able to develop KFQs by only watching and following the online instructions, which suggests that the online learning part of the flipped learning process is adequate to make learn and being able to develop KFQs as instructed. Online synchronous discussion helped the faculty members in resolving problems if they encountered or seek support from the peers.
The participants' average exam performance scores of both MCQs and KFQs were quite close to each other and were around mid-level. But, when compared to their failed counterparts, the participants who passed the exam performed better on MCQs. Hence, it can be inferred that the performance for MCQs part of the examination gives an edge by securing an increasing factor of chance of passing. Thus, in this study, we found that well-designed KFQs reduce the chance and increase to assess clinical decisions in an exam as aligned with the published literature [25,37].
The reliability coefficient of MCQs part of the exam was found high which was also reported satisfactory according to the latest literature review [38]. In this study, the reliability of KFQs calculated as the cases were considered as units. Although, there is a wide variety of reliability coefficients reported (0.49-0.95) in diverse use of KFQs in different levels of education, we calculated a lower reliability coefficient than reported in the literature [21]. The literature suggests conducting 40 cases KFQs to reach a reliability of 0.80 level that requires 4.1 h of testing time [37]. In our study, low reliability most probably originated from both a small number of participants and KFQs cases.
At the end, the examinees provided positive feedback for the exam in which KFQs are an important part of the exam. However, KFQs and MCQ scores are nowhere linked with each other. It can be assumed that the present results support the discrimination of KFQ type over MCQ. The results of the exam suggest that KFQs indicate important findings for the development process, which contribute to the overall quality of the exam and evaluation of the study. Both were taken into consideration, developing KFQ with flipped learning is of value for the future of exam set preparations.

Limitations
The number of faculty members recruited for flipped learning training as well as exam participants were very low which was inevitable in our situation. Also, the implementation of KFQs to the TPED Board Exam was the very first time, so the Board members wanted to have a limited number of cases in the exam. Although the examinees are in the same field of expertise, they are trained in different institutions which could be considered as a confounding factor. In addition to this, the examinees might have felt pressure to succeed in the board exam as the pediatric endocrinology community is relatively small which might be the origin of this type of stress.

Conclusion
Flipped learning carries importance for the future of faculty development to harness active and smart learning, and to elaborate on a given topic among faculty members as a community. Faculty members engaged, learned, and enjoyed by following online instructions to learn KFQs development along with robust face-to-face discussions. Flipped learning is a well-accepted instructional approach, and here in this study, it was used in the FDP dedicated to developing KFQs for board examinations. Our current findings demand for future validation studies involving more participants. Overall, we conclude flipped learning is a modern approach that can add great value to our classical FDPs if exploited judiciously. In addition, future research is encouraged to focus on other topics in addition to KFQs teaching using flipped learning as the focus on retention of the training in a long period. This study also paves the way for exploiting the potential of flipped learning approach in similar FDPs or other faculty educational/training programs with desired modifications in the near future after rigorous validation studies.