Brenda Stutsky (University of Manitoba); Marilyn Singer (University of Manitoba); Jose Francois (University of Manitoba); Robert Renaud (University of Manitoba)

Synopsis

The Manitoba Practice Assessment Program has been designing and testing a performance assessment program for practicing physicians on a provisional register. Hear about the results of the pilot study used to test selected processes and examine the preliminary reliability and validity of the practice assessment tools.

Background

The Manitoba Practice Assessment Program (MPAP) was developed based on an identified need from the College of Physicians and Surgeons of Manitoba (CPSM) and involves the assessment of practicing physicians on the CPSM provisional register. Assessment tools developed, using the CanMEDS as the underlying framework, include tools completed by physician candidates, physician and interprofessional colleagues, patients, and assessors.

Purpose

The purpose of the pilot study, conducted from June to December 2011, was to test selected MPAP processes and examine the preliminary reliability and validity of the practice assessment tools.

Method

Using a mixed methods design, the pilot involved six main stages: (a) training of assessors (n=12) using a blended learning approach; (b) selection of practicing physicians registered with the Canadian College of Family Physicians of Canada (n=6) and first year family medicine residents (n=6); (c) completion of the self-assessment process and 360 degree surveys; (d) onsite assessment including direct observation, chart auditing, and a chart stimulated recall session; (e) collation of data and feedback to the physician candidates; and (f) a final debriefing session involving both assessors and physician candidates.

Results and Conclusions

Results and conclusions include both qualitative and quantitative measures. Expected results supporting the validity of the assessments include experienced physicians obtaining higher performance ratings compared to first year residents, and correlations between different measures of the same trait (e.g., Medical Expert self-rating and Medical Expert colleague-rating) being stronger than the correlation between the same measure of different traits (e.g., Medical Expert self-rating and Communicator self-rating).