Next, I will introduce my co-moderator, Dr. Brijan Shah, who is a full professor of medicine at the Icahn School of Medicine at Mount Sinai. We'll be discussing competency in endoscopy, a topic that's very near and dear to all our hearts. And thank you to the course organizers for inviting me. I'm going to spend the next 15 or 20 minutes briefly talking about updates and assessing competency in endoscopy, trying to build a little bit more off of what Drs. Williams and Whitson spoke about earlier this morning. So I have no disclosures. Basically, I have two learning objectives over the course of this session. The first is to list some competency-based tools to aid endoscopic assessment, mostly to jog your memory of what's out there. And then second, I'm going to spend the bulk of the time in this space is to discuss the current state of assessment, first in trainees, and then close with some comments about how we might do this in practicing physicians. So as we first think about this, you know, this is really aimed at anybody who's involved in any aspect of training, anybody who does endoscopy. So whether you're a program director, you work occasionally with fellows, or you're in charge of running some aspect of assessment in your institution, or you're involved in credentialing, I hope you'll be able to think about this after we depart today. Just to level set, I thought it'd be helpful to go through what is competency-based education or training, which is the current model in which we've been living in in medical education for nearly 20 years. So first of all, competency is an observable ability of a health professional integrating multiple components of knowledge, skill, values, and attitudes. So it's those four things that we're trying to bring together and become the basis not only for the curriculum we develop, but the assessments that we then engage in, whether it's endoscopy, patient care, or some other element of physician development. So this is the current list of tools that I could come up with from the literature. I'm sure there are others that I have not found. Lots of alphabet soup here. But if you pick any of them, you should be able to find the relevant articles which derive these tools. I think what I would say about this is, in terms of the journey, we have many tools. I'm going to share with you a few more. I think now where we need to begin to go is understanding how to deploy them and how to use the information in them in order to get people to where we would like them to be at the end of their training, and then to continue to develop their skill base as they go towards mastery. Competency-based education really, at least in the U.S. graduate medical education space, came to being in about 2012, 2013. At that time in GI, there was the development of entrustable professional activities. So these were documents or resources to help take very common tasks within the field and bring together those knowledge, skills, and attitudes in a way that people could understand. These documents outlined these areas, and they also suggested assessment tools. And I'm really glad to see that since about seven years or eight years since the development of these, there are even more assessment tools out there for us. And then these were mapped to the milestones. At that time, the milestones were generic milestones for internal medicine fellowship trainees and were not specific to GI. In developing these EPAs, though, there were two EPAs which were developed specific to endoscopy. The first was the ability to perform upper and lower endoscopic evaluation of the luminal GI tract for screening, diagnosis, and intervention. And the second was the performance of endoscopic procedures for the evaluation and management of GI bleeding. So the authors at that time, the writing group felt that it was important to separate that out. Kind of moving forward, this is a figure from Dr. Walsh's paper, which you saw earlier in the prior talk related to education, is really thinking about the broader view of the skills and knowledge and attitudes needed when trying to develop an endoscopist. Often we focus on the technical competencies, which are on the one side of the slide, and we do need to, and we have increasing recognition of the need for the cognitive competencies. But what you see in the middle are those integrative competencies, such as teamwork, leadership, professionalism, and safety, which we also want to make sure are part of the way in which we think about how we assess our fellows. In 2020, 2021, we were able to update the milestones. And for the U.S. training programs, we now have specific milestones related to endoscopy. And I think that was a really nice move in the right direction to be able to call out the importance of endoscopic procedures in our field. These were split into two sections. One was a milestone specifically related to the cognitive components. And there are three themes within this milestone. One is procedure selection and indication. The second is interpretation. And the third element is plan of care. So this resource today can help anybody that's involved in assessment get the kind of the words, the language, the idea of what you need to be looking at when you're engaging in assessment. And, in fact, for those of you that are at places with the fellowship program, this is what your clinical competency committee is making an assessment of every six months on each of your trainees. And this is the milestone for technical components, again, being able to select the right, select and perform the correct components along with the respective therapeutic interventions. And for those of you that are not familiar with this, the goal is to get everybody into that level four column or close to it by the time they graduate. And really level five is meant for development once you're in practice towards mastery learning. And then lastly, all of this is built on this idea of the Dreyfus model of development. And the Dreyfus model talks about taking somebody from novice, where they don't know what they don't know, all the way through taking somebody to expert in mastery, which is at the bottom. And really our focus in fellowship training is sort of in that middle. We're looking at competency and proficiency and making sure our assessment data is giving us information in order to get there. So I thought that would be helpful for you all as we talk about what's new in assessment, just to kind of know the framework that shapes our current model of education and assessment. So as we think about what are the current updates for assessment for fellows, I've organized my comments into what to assess, how to assess it, and then very lastly, how does assessment support development? Because ultimately, that's the purpose of assessment. How are we using this information to further the growth of the learner? So in terms of what to assess, I gave you a little bit of a teaser of that on a few slides back, but there are a few topics that we might want to think about in what ways are our programs currently assessing. The first is the non-technical skills for endoscopy. I think this is increasingly important because we know that it's not just the technical ability to perform endoscopy that's going to make somebody successful, but it is going to be things like how can they lead, what is their situational awareness of what's going on in the room and with the patient, what is their ability to communicate with other healthcare professionals, and their overall professionalism. It's often these things that tend to get in the way when people go into practice. We often have stories of people who are technically very proficient, but it's one of these items that gets in their way of either the patient care aspect or being able to function in a greater team. There are many different tools that are available to help you with the assessment of these components. The GIECAT is one that has the ability to look at this in addition to the technical components. You can use any type of 360-degree evaluation where you're asking non-physician team members to give you input on a trainee or a physician's communication skills, professionalism, and ability to team, and in fact, the ACGME in their current model of assessing learning environments has really focused on teaming as a really critical component to know if the training environment is appropriate for fellows. And then lastly, I shared with you some milestone data. There are milestones specific to communication skills and professionalism, so you could look towards those in order to get the language to be able to know what to look for as you develop assessment tools for your own institution. Second, I wanted to talk briefly about sedation. There are these three really nice papers on both sedation curricula and assessment, and I think that depending on where you practice, it's going to depend on whether or not you're using anesthesia for some or all your cases. Where I practice, we use anesthesia for all of our cases, but others of you may be practicing in places where you're still doing a lot of moderate sedation. In either case, we have to think a little bit about the fact that we're training people to ultimately practice in a setting that may not have the same resources where they're training. So I thought that it would be helpful to share with you this schematic that comes from one of these papers about how they developed a really nice sedation assessment program. So there was a formative period where they were engaged in cases with a supervisor for about 30 cases, and in each of those cases, there was a debrief, there was an analytic reflection, and then there was a written exam at the end related to the principles of sedation. And then for their summative assessment, they had three different assessors over a series of cases who were looking at how they performed sedation, and then again, there was a final written summary exam to make sure they understood the cognitive components of sedation. And there's some nice high-fidelity simulation resources that exist out there that can support this so that you don't have to just do this at the bedside. Simulation can be really effective, especially for the communication skills and managing emergencies that can come up during sedation. Next, I think it's important that we call out the knowledge aspects of this. We don't routinely spend a lot of time on the knowledge components when we think about our high-stakes board exams. There are plenty of cognitive questions on the board exams, but we don't have a lot of cognitive questions as it relates to some aspect of endoscopy. And these are some papers, although there are others that have really used a lot of some form of online testing as part of their curriculum to move people through a skill development. The first is this endoscopy diploma, which is a European program. They were taking surgeons and non-surgeons, and what they showed through a 300-hour, year-long curriculum, they combined both online testing, hands-on sessions, and clinical rotations, and they found that for the majority of the folks in this program who had no endoscopy experience, they were able to increase their gauges assessment score, which was significant. And then on the other side of the slide is a snapshot from a simulation course, and what they did over this two-day course is they really utilized quite a bit of knowledge assessment in phase two and, again, later at the very end in order to solidify and make sure that they had assessed the cognitive knowledge in addition to the technical skills. So there might be opportunity, I think, working collaboratively across programs to think about how we might share knowledge-based assessments as it relates to different aspects of endoscopy. You heard Renee and Matt talk about pre-, intra-, and post-procedure, and each of those phases has its own respective knowledge components which could be assessed. Next, just a brief comment on, you know, can simulation help us with competency assessment? There's been certainly a lot of papers about this, and I think the jury is a bit out. On one hand, it's a very attractive item to use as we think about assessment because it can be done independently, you know, people don't need to – you're not doing this with the patient. It's certainly very, very safe. On the con side, it is quite costly, and so far some of the literature that's out there shows that it's not able to discriminate between those with a high degree of expertise and experience versus those with a lower degree of expertise and experience. And lastly, these simulation-based assessments might not be predictive of performance when you get to the bedside, so I think thinking a little bit about simulation with caution is important. I'm curious with augmented and virtual reality if there'll be any changes or opportunity there, so I think that's something to look out for in the coming years. Just related to this, I found this one paper which came out of a group in Saudi Arabia. Actually, they were looking to develop a really good simulation-based assessment tool for endoscopy, and they did this while using the DOPS endoscopy assessment as kind of their gold standard. They went through a Delphi approach with experts, and they came up with a five-question assessment which could be used if you're teaching endoscopy in a simulated setting. And the thing I like about this checklist is it's nicely behavior-based, and it basically says is something done or not done. So it's very easy to use and train people on. I think when we think about assessment, and we have a very critical need for faculty development, and many of our programs lack this, but we cannot get to good quality assessment if we don't spend time helping to develop our faculty. Faculty development is critical to help develop a shared mental model to make sure that we're all looking at and assessing the same thing. It will also help us mitigate bias as we're moving from learner to learner with different backgrounds. It will also help us to tell people how do you assess the non-technical skills in endoscopy. And then finally, this is an opportunity for us to teach people how to give effective feedback, which we know is one of the things that's lacking in being able to give people timely feedback as they move from procedure to procedure. And again, this table, which comes from the paper of the Saudi Arabia group, one thing that I liked is sometimes when I do faculty development for our group, it's helpful for me to, of course, to tell them what to look for, but sometimes it's easier for them to see the things that we don't want to see. And this paper actually highlighted about 15 behaviors that when you see them in an endoscopic setting might be things that we want to look out for, as opposed to always like looking for the perfect. So I wanted to include this in my slides in case that was helpful to any of you. Of course, coming on the heels of the last couple talks, I thought I would make a comment about AI. This comes from a figure of a paper that Dr. Keshwani and colleagues wrote. And yes, I think AI is going to have a role to help us with assessment. I think its role is really in the middle box here. The data in some of these AI platforms can help us with giving some feedback on grading accuracy. I think it can help us with giving learners some information about assessing their withdrawal techniques and helping to ultimately standardize benchmarks across training programs, as well as it can be a tool as we're teaching people about polyp type, polyp size, and how to go about sampling and removing things. So I think AI will become part or an adjunct to our assessment, but it's not going to replace direct observation anytime soon. And so now I want to turn our attention on the training realm to thinking about development. We're really fortunate that lots of great people have spent time over the last five to seven years developing learning curves across a variety of procedures. So we have a better sense of how people develop a variety of skills, whether it's an ERCP, colonoscopy, or EGD. We now know this based on cases and time. And I think that that's really, really important. I think where we have to go next is how are we using these curves that have been developed to help our learners as they go through their fellowship training to understand where they're at and how can that help to feed the development of an individualized learning plan for the next four to five, six months of their training. So I think, you know, I hope people will think a little bit about that both at their own institution and more broadly as these types of papers continue to come up in the literature. Related to this, I wanted to share with you a really interesting development from the folks at the ACGME. So Eric Holmboe, who runs the milestone initiative for the ACGME, which has now been around for almost 10 years, has some really interesting data with all this milestone information they've been collecting. What they have is they now have thousands of learners' worth of data on their milestone development. And they've been able to develop these positive predictive value charts for each milestone within each discipline. So this is an example of plastic surgery for their patient care milestone of aesthetic surgery. And what this helps you to do is it tells you that what level, where were they in their training, what was their milestone rating at that point in time, and what is the likelihood that they will or will not get to level four, which is where you want to be at graduation, by the time they finish training. So in this example, if you take a year two plastic surgery resident on this milestone and they received a mark of two and a half or lower, there's a 66% chance that they will not make it to level four by the end of their plastic surgery residency. That's pretty powerful stuff for those of us that are program directors or very involved in training, because this gives us the sign that we need to help figure out what are we going to do to give them what they need to get them to that point by the end of their training. So GI, in terms of our milestones, we only started using the current milestones in 2021, so it's going to be a few years before we have this kind of data, but I think this kind of data is going to really nicely augment the learning curve research that is already out there. So in the last couple minutes, I just want to turn my attention to talking about attending physicians or those of us in practice and how we might think about assessment. So classically, assessment was usually a one-time thing that happened when you got privileges at whatever institution you're working at. It was mostly based on volume with no data submitted about your quality. I thought this was an interesting paper because it kind of looked across the world at what the threshold is for the number of cases needed in a variety of endoscopic procedures, and you can see there's a high degree of variability across the world as to the number of procedures needed in order to get privilege for being able to do that. In my view, I think there's three time points or three items as we think about endoscopic privileging. One is initial privileging, and we do have to move beyond case volume, and I think our research is going to help us with that. The question is, can we get information about the quality of the cases that were performed, even if in training? Can we get an assessment of people's non-technical skills to help us make that judgment? And then lastly, what other limits might we have in terms of being able to get this information? This is not going to be easy, but if we ever get to a point where there's a centralized assessment platform for some of our procedures, I think we might be able to help make an impact there. Next, it's really getting comfortable at the fellowship to practice transition and having those people who lead fellowship programs be honest with the next group that's going to get that doctor. If somebody hasn't gotten enough cases or isn't competent yet, to be honest and then have hospitals have really good train-up programs that allows them to continue getting the training and then ultimately get evaluated so they can have an unconditional privilege, I think will really help us in order to move forward. Obviously, this is operationally very difficult because many places don't have excess numbers of physicians that can do the training, and sometimes you hire people because you have a real need for patient care. And then lastly, thinking about what should be the role of the types of data we need in ongoing assessment. It should be more than volume data, although in many places we don't even have that. And we should really consider using quality metrics as part of things like ongoing professional practice evaluation for meaningful assessment. So what do we know? We know that knowledge and non-technical skills assessment should be part of our way that we assess going forward. The role of simulation assessment is still yet to be determined and needs further research. And the competency-based model of skill development endoscopy is becoming more and more clear, which I think is an amazing thing if we look back to where we were about 10 or 15 years ago. So what can you do going forward or thinking about how to use this information? One, hopefully you can reflect a little bit on how do you assess the development of endoscopic skills in your work and in your programs. Two, consider using multiple modes of assessment in the context of endoscopy so you can get at these things. And then lastly, ask yourself in your program, are you engaging in assessment really for learning or is it just to make a high-stakes decision about graduation? Really should be doing the former and doing a little bit less of the latter. Thank you for your attention.

competency

endoscopy

assessing competency

competency-based education

tools and assessments

faculty development