Table of Contents

Misconception: It is harder to make growth with students from certain demographic or socioeconomic backgrounds.

It is widely known that students with certain socioeconomic or demographic (SES/DEM) characteristics tend to score lower, on average, than students with other SES/DEM characteristics, and there is concern that educators serving those students could be systematically disadvantaged in the modeling.

However, this adjustment is not statistically necessary for the most sophisticated value-added models, such as those used for EVAAS in Michigan. This is because EVAAS uses all available testing history for each individual student and does not exclude students who have missing test data. Each student serves as their own control, and to the extent that SES/DEM influences persist over time, these influences are already represented in the student's data.

EVAAS in Theory

As a 2004 Ed Trust study stated, specifically with regards to the EVAAS modeling

[I]f a student's family background, aptitude, motivation, or any other possible factor has resulted in low achievement and minimal learning growth in the past, all that is taken into account when the system calculates the teacher's contribution to student growth in the present.*

This approach has been confirmed through a variety of robust statistical analyses. In 2004, a SAS and Vanderbilt team published a study that closely examined SES/DEM adjustments and concluded:

SES and demographic covariates add little information beyond that contained in the covariance of test scores.

This finding has been confirmed independently by prominent value-added experts who have replicated a variety of value-added models, including SAS EVAAS models. More specifically, a 2007 paper by RAND researchers J.R. Lockwood and Dan McCaffrey explicitly verified the models used for EVAAS district, school, and teacher reporting when they wrote:

William Sanders, the developer of the TVAAS model, has claimed that jointly modeling 25 scores for individual students, along with other features of the approach is extremely effective at purging student heterogeneity bias from estimated teacher effects...The analytic and simulation results presented here largely support that claim.

An economist-based perspective by UCLA researchers Kilchan Choi, Pete Goldschmidt, and Kyo Yamashiro provided a similar finding in their study comparing value-added models:

First, adding in an adjustment for student SES (as measured by eligibility for free- or reduced-price lunch) adds very little once a student's initial status is controlled... This indicates that student initial status captures many of the effects that SES is attempting to measure. In other words, by controlling for initial status, the model already captures the preceding effects that SES might have on students.§

EVAAS in Practice

Although the statistical literature presents evidence that educators are not advantaged or disadvantaged by the type of students that they serve in sophisticated value-added reporting, actual data might be the most readily apparent evidence to support this belief.

The first figure below plots the percentage of tested students who are considered economically disadvantaged at each school in Michigan against the school's growth index (the value-added estimate divided by its standard error) for M-STEP Mathematics in grades 4-8 in 2019. Regardless of the school's student characteristics, there is little to no correlation to the growth index. In other words, the dots representing each school do not trend up or down as the percentage increases; the cluster of dots is fairly even across the spectrum.

MICHIGAN GROWTH INDEX VERSUS PERCENT TESTED ECONOMICALLY DISADVANTAGED BY SCHOOL