“Rater bias” is present when a significant amount of variation in an assessment’s scores can be attributed to factors related to the person completing the assessment, rather than factors related to the skills being assessed. For example, consider a teacher-completed assessment of students’ SEL skills. If all male teachers tend to rate students as exhibiting more SEL skills than female teachers, or if all experienced teachers tend to rate students as exhibiting fewer SEL skills than first-year teachers, the assessment could be showing evidence of systematic rater bias. If present, rater bias affects our ability to accurately measure students’ skills.
Should you be concerned about rater bias when implementing a teacher-completed behavior rating scale to assess students’ social and emotional competence? In general, no. There is no evidence to suggest that teacher-completed measures are inherently more biased than any other measures of social and emotional competence. Here are some things to keep in mind when considering a teacher-completed assessment.
1. Teacher-completed behavior rating scales are widely used in schools. This method of assessing students’ strengths is not new, but rather has been used in schools for decades. Standards for Educational Testing have even been developed by experts in the field to advise educators on selecting high-quality behavior rating scales for their purposes.
2. When selecting an assessment to implement, there are often are ways to evaluate each individual tool’s overall error (and specifically error related to rater bias). An assessment’s internal consistency is an indicator of how well the tool’s items work together to measure its underlying construct. Internal consistency is inversely related to total error, so when a tool’s internal consistency is high, we can be confident that all sources of error (including rater bias) will have a small combined impact on students’ scores.
The most common indicator of internal consistency, Cronbach’s Alpha, ranges from 0 to 1, so values close to 1 indicate the lowest overall error. Information about a tool’s internal consistency can typically be found in a technical manual or by contacting the assessment developers.
Many teacher-completed assessments will also provide information on the tool’s interrater reliability. This is a measure of how similarly multiple teachers tend to rate the same students on an assessment. High interrater reliability means that scores tend to be very similar between teachers and that users can be confident in a lack of bias.
3. The potential effect of rater bias can often be reduced through preservice training. This recent study examined one teacher-completed measure of Social and Emotional Competence, the DESSA-mini, for potential rater bias and found that little variance in students’ scores could be attributed to the teachers who rated them. Moreover, they found that schools can reduce this variance by over 50% by providing teachers with adequate preservice training.
4. What looks like “rater bias” in practice may simply be a reflection of differences in students’ behavior across different settings. For example, consider a school where math teachers’ ratings of students’ social and emotional skills tend to be lower than those of other teachers. Is this an indicator of a biased tool? Perhaps. But first, consider whether this difference in ratings reflects a true difference in student behavior. Do the math classes at this school offer fewer opportunities to practice and display social and emotional skills than other classes? Rather than an indication of rater bias, this observation may provide an opportunity to reflect on ways to further incorporate SEL into a school community.
Ultimately, teacher-completed behavior rating scales can be excellent options for assessing students’ social and emotional skills. Informed assessment selection, adequate preservice training, and careful intervention planning can help to mitigate concerns of rater bias and support the collection of high-quality social and emotional data in schools. Interested in learning more about the DESSA Comprehensive SEL System? Contact our team today through the form below.