Mary J. Bell (Department of Medicine, University of Toronto); Edmund Lorens (Department of Medicine, University of Toronto); Christian Base (Department of Medicine, University of Toronto)

Synopsis

Building upon previous work, this study aims to empirically validate reliability estimates and discern presentation characteristics most associated with positive presenter evaluation scores. Criteria most predictive of favourable overall presentation assessments include enthusiasm, stimulation, appropriate level of presentation, effective style, and good rapport. Theoretical reliability estimates were empirically validated and the questionnaire instrument appears to be a valid tool for obtaining feedback.

Purpose

Building upon previous work (Rothman & Sibbald, JCEHP 2002), this study aims to empirically validate reliability estimates and discern presentation characteristics most associated with positive presenter evaluation scores (PES). Over 51,000 evaluation forms have been amassed 2002-2009.

Methods

Intra-Round Dispersion of mean PES was assessed against number of forms. Intra-class correlations were calculated as a function of the number of forms. Intra item correlations were measured using Spearman’s rho.

Results

Consistent with earlier findings, review of inter-item correlations suggests appropriate inter-item coherence and user discrimination between items. Intra-round standard deviations were normally distributed with 0.48 mean central tendency; the mean number of forms associated with this was 29 forms (median=22). Relating these findings to observed intra-class correlations and number of forms relationships, we estimate achievement of a reliability rating of ~0.82-0.85. Consistent with theoretical estimates, diminishing returns relative to the number of forms was empirically demonstrable. Criteria most predictive of favourable overall presentation assessments include enthusiasm stimulation, appropriate level of presentation, effective style, and good rapport. Overall longitudinal improvement was also observed from 2002 to 2009, subject to further statistical testing.

Conclusions

Theoretical reliability estimates were empirically validated. The questionnaire instrument appears to be a valid tool for obtaining feedback.

Funding Sources

This project was funded through internal Departmental resources.

Declaration

The authors have no disclosures to make with respect to this abstract.