The Assessment of Reasoning Tool (ART): structuring the conversation between teachers and learners

Introduction

Developing and ensuring competence in clinical reasoning is one of the most important goals of medical education programs. The 2015 National Academies of Science, Engineering and Medicine report Improving Diagnosis in Health Care [1] highlighted the prevalence of diagnostic errors and recommended that training programs improve learners’ performance in the diagnostic process to decrease diagnostic errors.

A productive feedback conversation requires that clinical teachers and learners share a common understanding of the processes used to make diagnostic decisions. This shared mental model is an essential but often elusive starting point because clinical reasoning is idiosyncratic [2]. Clinicians rely upon their own unique clinical experiences and knowledge to address clinical problems; hence, there may be as many paths to a diagnosis as there are diagnosticians. Further, expert diagnosticians often use non-analytic, experience-based approaches to make decisions and may be unaware of the underlying cognitive processes. The context-specific nature of clinical reasoning necessitates that teachers converse with learners about situational factors (e.g. student’s clinical experience with the chief complaint) that can impact clinical reasoning performance [3], [4]. Lastly, teachers’ own knowledge, experience, ability and even personal cognitive biases may influence his or her assessment of learners [5], [6].

Despite these limitations and challenges, assessment of learners’ clinical reasoning in medical education programs is essential to identify opportunities for improvement. Such identification is useful when teachers can follow with questions that help learners calibrate their reasoning processes and provide learners with strategies for improvement in future encounters. Questions asked during a clinical reasoning assessment allows learners to reactivate existing knowledge, incorporate newly acquired knowledge and then assimilate the two in increasingly organized and rich illness scripts [7]. In summary, a tool that facilitates assessment for learning will be most useful in guiding learners in the development toward mastery of clinical reasoning skills. Though numerous approaches can be used to assess components of clinical reasoning ability, very few assessment tools to guide feedback are comprised entirely of elements specific to the reasoning process [4], [6], [8], [9], [10], [11], [12], [13], [14].

To address this gap, we developed the Assessment of Reasoning Tool (ART). The goal of the tool is to facilitate teachers’ formative assessment of learners’ oral presentations for competence in clinical reasoning and provide a structure for conversations between teachers and learners during a feedback session. Recognizing that an informed faculty member is essential to successful ART use, we developed instructional videos for asynchronous, just-in-time faculty development for the five ART domains. This paper describes the tool development process and provides preliminary validity evidence.

Methods

A subcommittee of the Society to Improve Diagnosis in Medicine Education Committee, composed of nine medical educators with expertise in teaching clinical reasoning and an interest in decreasing diagnostic errors, was formed. The development process for the ART followed the scale development guidelines recommended by DeVellis [15]. The study protocol was approved by the Institutional Review Board at the Baylor College of Medicine.

Determining the assessment target

We determined the ART should be used to assess learners’ performance in clinical reasoning during an oral presentation of a patient encounter. The ART may be used in the context of a direct observation of a clinical encounter although that was not our primary intent, as the opportunities for direct observation in many medical education programs are limited. Through an iterative process, we formulated a conceptual framework for a diagnostic reasoning process based on prevalent reasoning concepts (e.g. structural semantics, script theory) [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], and proposed error-reduction strategies (e.g. cognitive debiasing) [28], [29]. We used this framework to define domains to be assessed (Figure 1).

Figure 1:

A proposed conceptual framework for the clinical reasoning process derived from literature and expert consensus.

Defining features are clinical elements that are characteristic of a disorder; discriminating features are clinical elements that are useful for distinguishing one disorder from another. A problem representation is a concise synthesis of the clinical problem that needs to be solved. Semantic qualifiers are specific terms (e.g. epidemiological features, severity description, temporal association) used to frame the patient presentations into clear and recognizable logical paths or patterns. Embedded in these terms are some opposing qualities (e.g. acute vs. chronic) used to facilitate differentiation between potential diagnoses. Pattern recognition is a form of reasoning that is driven by recognition of familiar combination of data. Metacognition is an awareness of one’s own thinking process that enables self-assessment, monitoring and reflection during the reasoning process. Cognitive debiasing is a bias-reduction strategy within judgment or decision-making. Cognitive biases are systematic patterns of deviation from normative judgment. Metacognition, not debiasing strategy, was used as a construct of the final Assessment of Reasoning Tool.

Results

The ART is an assessment tool that delineates five domains of the clinical reasoning process and allows for both specific and global assessment of learners’ diagnostic reasoning performance (Figure 2). The tool provides teachers with benchmarks and language that can be used during formative feedback discussions with learners.

Figure 2:

The Assessment of Reasoning Tool includes five domains of the clinical reasoning process and an overall assessment of the learner’s performance.

Table 1 demonstrates the results of the pilot testing by 10 clinical faculty who evaluated the initial six characteristics of the ART. The faculty favorably rated the ART’s relevance to their own assessment practices and provision of feedback to learners. The potential for learning about clinical reasoning through the tool was rated lower than the other metrics (median rating of 4 on a five-point scale). Faculty commented that some terms used in the ART such as problem representation and high-priority diagnoses were either unfamiliar or ambiguous and that they would benefit from training to properly guide their learners. A common suggestion was to establish a process for faculty training using practice cases so that faculty could be better prepared to teach the components of clinical reasoning to their learners and accurately assess their presentations. Surveyed faculty also mentioned that clarity regarding which level of performance reflected competence would be helpful for summative assessments.

The faculty training module comprises five short (3–4 min) whiteboard animation videos (available at www.improvediagnosis.org/art). Each video highlights one domain of the ART, describes the rationale for the domain, defines clinical reasoning terms and gives examples of high- and low-performing learners. Ten clinical faculty rated the faculty training module favorably. They rated the all five videos as useful, educational and enjoyable (medians between 4 and 5 of five-point scale) and anticipated that the videos would provide effective guidance to faculty using the ART (median 4).

Discussion

Clinical reasoning has been called the “Holy Grail” of assessment in the health professions due to its great importance juxtaposed against the lack of a gold-standard method of measurement [2]. Through a collaborative effort of multi-specialty experts, we created a theory-informed tool for the assessment of clinical reasoning along with faculty development videos. The ART provides an explicit structure for assessment of clinical reasoning and shared terminology to facilitate formative feedback.

Multiple tools have been developed to assess clinical reasoning. The methods often capture particular components or sub-tasks of the clinical reasoning process [4]. Few of the published general assessment tools that include clinical reasoning specify multiple domains of competence within the clinical reasoning process [12]. For example, the mini-Clinical Evaluation Exercise (mini-CEX) asks clinicians to rate “clinical judgment” on a nine-point scale [13]. The Integrated Direct Observation Clinical Encounter Examination (IDOCEE) tool includes three domains (data gathering, reasoning and analysis, decision-making) which address clinical reasoning but does not deconstruct it. The Problem Representation, Background Evidence, Analysis, Recommendation (PBEAR) tool [14] includes a comprehensive list of concrete tasks or actions within history taking and physical examination, which provides useful guidance for communication during an oral case presentation. Though the PBEAR construct implies several theoretical principles in clinical reasoning, few terms or concepts from the clinical reasoning literature are used explicitly.

The Interpretive summary, Differential diagnosis, Explanation of reasoning, Alternatives (IDEA) tool [8] developed for assessing medical students’ clinical documentation includes important elements of clinical reasoning along with concise descriptors of those elements, but omits several elements of the clinical reasoning process featured in contemporary literature (e.g. hypothesis-directed data gathering). The Reading Hospital mini-CEX rating instrument assesses some domains of clinical reasoning (e.g. data collection, interpretation and synthesis of data, diagnostics and therapeutic reasoning). However, the tool also covers other competencies (e.g. interpersonal/communication skills, professional conduct and system-based practice) [10]. Furthermore, each domain combines multiple behaviors encompassing knowledge, skills and attitudes, so teachers must synthesize several aspects into one assessment rating. Most aligned with our proposed conceptual framework is a diagnostic skill evaluation form developed by Haight and DePriest [11] that uses milestone language to create clinical reasoning benchmarks within domains of competence.

The ART is distinguished from these tools by providing detailed behavioral anchors using contemporary clinical reasoning terminology to guide assessor observations or judgments. The ART is grounded in a theoretical framework integrated with a contemporary practice goal (i.e. HVC) and proposed strategies for reduction of errors (e.g. metacognition).

The ART is also distinguished from other assessment tools by the accompanying faculty development modules that facilitate implementation. Teachers can use these videos to familiarize themselves with core concepts of clinical reasoning and prepare for an upcoming ART assessment with a learner. Clinical reasoning terms represent a shared language for teachers and learners to dissect how they think about and learn from clinical problems [36]. Providing specific feedback to learners can be difficult if the teacher is not versed in vocabulary that can be used to describe specific behaviors within the clinical reasoning process. For example, some learners may be unable to give a clear synopsis of the clinical problem (e.g. merely restating facts and findings). Asking the learners to synthesize a case without providing an explicit guidance may leave some learners wondering how they can improve. A teacher can use the ART to assess the oral case presentation and then guide learners in applying the concepts and terms pertaining to a problem representation to structure the case synthesis. Teachers can also refer learners to the accompanied training module to expand their conceptual understanding about the concepts. Teachers can use these videos for either asynchronous or synchronous learning in a faculty development program to enhance teaching and learning about the diagnostic reasoning process. Teachers can also share these videos with learners in advance of or following an ART-guided conversation.

The ART has limitations. The conceptual framework derived from our interpretation and synthesis of literature to guide the construct of the ART reflects one model of clinical reasoning. By its nature, a conceptual framework emphasizes certain aspects and inherently disregards or dismisses other aspects of the subject being studied [37]. During the development process, our expert consultants advised us to capture important elements but not all aspects of the clinical reasoning process. Thus the ART captures only parts of the complex and broad construct of diagnostic reasoning process. Summative assessment of diagnostic reasoning skills requires a combination of non-workplace-based and workplace-based longitudinal assessments across multiple chief complaints and diseases in diverse clinical settings with numerous raters. The ART was not developed to displace well-established summative diagnostic reasoning assessment methods (e.g. multiple choice question, objective structured clinical examination). Rather, it complements them in the formative domain because it promotes explicit teaching of the diagnostic reasoning process. Another invaluable suggestion from experts was to develop a simple tool for a broad range of users and then spend an equal amount of time developing the faculty training module. The training videos introduce clinical reasoning concepts but are not a comprehensive overview of the field. Among the specific behavioral descriptors in the ART, some are difficult to assess objectively. Hypothesis-directed data gathering in the first domain, for instance, is a partially subconscious process that can only be inferred but not directly assessed, even with real-time observation at the bedside. This can be improved by direct questioning of the learner. The fifth domain addresses metacognition with a simplified approach. Metacognitive abilities cannot be easily assessed or taught by simply asking “What else could this be?” or “Have you reflected about this case?”. Teachers can use the ART to prompt a conversation with learners around cognitive tendencies and emotional/affective factors and to raise awareness of how these factors may have influenced their thinking.

The collective evidence of validity of the ART was embedded in the development process. We ascertained evidence of validity pertaining to construct and content through a systematic approach and integration of theories, expert opinion, consensus and comments from the end-users. By piloting the tool with clinician-educators we demonstrated some response process validity. A validation study is underway to gather further evidence about the response process, internal structure and relationships to other variables.

Conclusions

Through a collaborative effort of multi-specialty experts, we created a theory-informed assessment tool of clinical reasoning along with faculty training videos. The ART provides behavioral descriptors which serve as standards for assessment and benchmark practices that can be used by evaluators and learners for effective performance feedback.