SGPs: Bringing Accountability to School Systems

by John Butcher

For much too long, the principal measures of educational quality were inputs: budgets, teacher salaries, class sizes, pupil/teacher ratios, and the like.  Grades did not compare from school to school or even class to class. Specialized tests such as the SAT reached only limited numbers of students and, in the SAT case, measured intelligence rather than achievement.

This situation was disrupted in 2002 by the No Child Left Behind Act, which conditioned Title I federal support on statewide testing.  Whatever their other strengths and weaknesses — see, e.g., the Wikipedia discussion — the resulting testing schemes gave us measures of outputs.

The scores, however, were strongly dependent on the economic status of the students being tested.  For example, on Virginia’s 2014 English Reading SOL tests, a 10% increase in the division count of economically disadvantaged students was associated with a 3.4% decrease in the division average pass rate.

butcher1

(Click for more legible image.)

The Federales figured this out and, beginning in 2011, required states to develop a growth measure with data for reading/language arts and mathematics in the tested grades. Virginia settled on the Student Growth Percentile, the “SGP.”

Stated briefly, the SGP compares the progress (year-to-year score change) of each student with the other students in the state who had similar prior scores.  That progress then is expressed as a percentile ranking.  Thus, a student with a 60 SGP had a score increase better than 60% of the students with the same “prior achievement.”

The Virginia Department of Education (VDOE) has a page of discussion with further links here.  A 2011 PowerPoint from VDOE’s Director of Research and Strategic Planning has a very clear discussion.

butcher2

(Click for more legible image.)

The advantage of the SGP is that it compares students with peers as to achievement.  Thus, students with low SOL’s (and, likely, low family wealth) are compared with students in similar circumstances.  The Virginia reading data demonstrate how this produces data that are largely independent of economic disadvantage.

Note here the weak correlation (11% R2, vs. 57% for the SOL chart above) and the weaker effect of economic disadvantage (14% SGP decrease compared to a 38% decrease in the SOL).

Data elsewhere show an even weaker relationship between SGP and economic disadvantage:

butcher3

To the bad, the SOL score is imprecise to some degree.  VDOE has a mind-numbing discussion here.  For the present, it is enough to notice that taking the difference between two SOL scores to calculate an SGP produces a datum that is even less precise than either of the two SOL data.  Thus, any single SGP, or any small collection, is weak evidence as to the teacher, school, or division.

All that said, Virginia has been calculating SGPs since 2011 for most students (about 70%) who take the SOL tests for reading and math in grades 4-8 and Algebra I through grade 9. VDOE has data that can be used, albeit with caution, to evaluate the progress of individual students and the effectiveness of their teachers, their schools, and their school divisions.

Tennessee puts these data on the Web.  For instance, the Volunteer State provides data on the progress of the Alamo district and the schools in that district, e.g., the Alamo Elementary School.

Yet Brian Davison, parent of two Loudoun County school children, had to sue VDOE to get the Loudoun County data.  Fortunately for us, Brian also got the statewide data by school and by division (posted here; the 16709 set is more current).  He still is litigating the data by teacher (the teachers’ union and, to their shame, the Loudoun School Board and VDOE are trying to suppress these data).

butcher8The 2013-14 dataset (the latest of three) occupies 833 MB when imported into Microsoft Access.  There’s a lot of information there.  For a start, here is a graph of the 2013-14 division reading SGP v. SOL scores.  Caution: These are not the division average SOLs, but the division averages of the SOLs of students for whom VDOE calculated an SGP.

To the point here, Franklin City and Charles City County, the green points, are taking students with low and nearly average SOLs and achieving large gains in reading.  Poquoson, the yellow point, is doing the same thing with high-SOL students.  At the red points, Henry and Highland counties, in contrast, are the leading underachievers in this group. (To view the data for each school division, click here.)

This graph is a snapshot that shows only one subject for one year, but it makes two important points:

  • We now have data that can measure year-to-year improvement in the SOL scores by division and by school. If Brian can win his suit, we’ll have the data parents will want, by teacher.
  • Some divisions are teaching a lot to students with low achievement. If Franklin City can do that, so can the schools in your division and mine!  No wonder the teachers’ union and the Loudoun School Board want these data suppressed.

For a sample of the further kinds of analyses available, see the data here and here and here.

If you would like to dig into your own school division, pull down the raw data from Brian’s post. You’ll need to strip out the SOL scores for students for whom VDOE did not calculate the SGP.  Unless you’re a database whiz, you’ll also need to restrict the dataset (probably by year) to get under Excel’s 1.024 million row limit.

Or, even better, you can demand that your School Board analyze and post its results.



The “Diff” column shows how many percentiles the calculated number (“Calc’d”) is above or below the fitted line indicating the norm. The higher the positive number, the more the school division out-performs its peers; the greater the negative number, the more the school division under-performs.

butcher6

There are currently no comments highlighted.

18 responses to “SGPs: Bringing Accountability to School Systems

  1. Really, really powerful blog post and much credit to Jim B for his efforts.

    just the last chart showing the county’s aggregate SOL scores is especially informative.

    and Jim says this:

    ” Or, even better, you can demand that your School Board analyze and post its results.”

    good luck on that. Most school systems do not even post their division scores much less individual school scores .. and this is an insult to taxpayers and citizens.

    also – with all the hate talk these days about NCLB and the Federal govt imposing standards, testing and accountability – if we listen to the anti-govt, anti-common core, critics – we’d get the Feds out of this all together and tell me what the schools would do if the Feds were taken out of this.

    in other words, we ONLY have this “big data” – uniform standards – because of the big bad, oppressive, top-down, incompetent, corrupt – govt.

    Next – why don’t the critics and especially those who are opposed to public schools – set up competing non-public school alternatives that really do provide a similar level of accountability of non-public school performance so that parents would actually have legitimate and assured alternatives so taxpayer voucher money can be provided with a level of assurance?

    Again – these posts by Jim provide significant information to readers – not generally known.

    I’ve checked with some current teachers and they themselves are only slightly aware of it because their schools have not yet stood up programs like the ones that Jim has highlighted.

    Finally – this is going to get to the heart of why some schools in some neighborhoods inside the same school district have such terrible school level scores.

    it’s going to call into question – the division policies.. with respect to addressing the core academic needs of economically disadvantaged kids – which was the original namesake of
    NCLB – No Child Left Behind – one of the most important legacies of a much maligned POTUS.

    • Thanks for the kind words, Larry, but the real credit goes to Brian Davison, who had to sue VDOE to get the data, and to John Butcher, who wrote this story. The only credit I claim is recognizing the importance of their work.

      • Jim – would you consider going over what Brian had to do to get the data?

        why would VDOE refuse to provide this data?

        and John Buchers work – I did not see links to either Brian or John’s work or perhaps I missed them or they were in prior posts.

  2. Oops – I just assumed the blog was by Jim .. my mistake!

    good POST – excellent !

    doing a little more reading on the Butcher lawsuit… I support the FOIA – VDOE has/had no legitimate reason for not releasing the data but this is
    not the only thing that VDOE and Va schools systems will not release ..and they are not alone – State Agencies and localities are playing this game with increasing frequency and it’s just plain wrong so I support folks like Mr. Butcher , in fact cheer them on – and I am a financial supporter of Virginia Open Govt.

    however, I think his motive is misguided because this is not going to end up being about bad teachers – per se – because many teachers, good teachers, are put in untenable teaching situations without sufficient resources and help for the kids that are behind.

    It won’t be long – if it has not happened already before some teacher who feels like he/she is being scapegoated is going to subpoena this data to
    use in their defense to basically demonstrate that its’ the school and school administration that is not providing adequate resources and attempting to
    shift blame on the teacher.

    I also would not be surprised that parents of kids in bad neighborhood schools are also going to get that data to prove their school is not being equitably staffed also.

    and I’m totally in favor of the data to be used to hold schools and school systems totally accountable – whatever their motives are – just keep in mind that different folks have different reasons for supporting accountability and from my point of view the “bad teacher” idea is simplistic and wrongheaded especially in a right-to-work state like Va.

    don’t confuse due-process rights with “union protection” and this will be found out pretty quickly once a discharged teacher hires a lawyer and get the SGP data to show how the school and school system deals with at-risk and economically-disadvantaged kids. the only “bad teachers” that will actually “stick” will be the ones that have typical classes and the whole class loses progress.

  3. Brian and John are my heroes!

    • did folks notice the SOL scores for Fairfax and nearby Arlington?

      that’s pretty dang impressive! you’d think with a super-huge school system
      that managing resources to produce such good outcomes would be a huge task.

  4. “Or, even better, you can demand that your School Board analyze and post its results.”

    good luck on that. Most school systems do not even post their division scores much less individual school scores .. and this is an insult to taxpayers and citizens.

    You got that right!

    I sent this data to the Lynchburg School Board asking that they release this data to the public. I just got a response pooh-poohing the SGPs in their entirety from the Superintendent and School Board chair. Kinda like the loser at the poker table, doesn’t want to talk about it, “just shut up and deal!”
    Every year there is less accountability and more and more excuses for failing schools…but it is NEVER the teachers or administrators fault.

    • HCJ – you should do a FOIA. I think you already know how.. it’s fairly simple because you KNOW the data you seek.

      you know there is a HUGE irony to all of this going back to NCLB and how the Feds enforce it – holding the Title 1 funding …

      so Virginia and many school districts would not even fund TITLE 1 resources if it were not for the much-maligned Feds who offer it – with strings attached.

      And we know what happened before Title 1 – there was no equivalent local or state funding for the kids that Title 1 provided resources for. It took Title 1 to address the needs of at-risk kids and then Title 1 allowed the Feds to institute NCLB which then enabled the use of SGP data then both local schools and the VDOE attempted to deny that data to the public.

      so there you have it… the dirty little story of how Virginia and local school districts would deal with at risk kids if the Feds were not involved.

      All this talk about “local control” of schools, and not having top-down, govt-required standards and accountability – would – and will today, if you listen to the anti-common-core idiots – lead to local control, no uniform standards, and zero accountability.

      this all came about because of top-down govt and I hope the “public schools suck” chattering class here in BR will admit it.

  5. I have the data!

    I asked the School Board to release it to the public and use it as a part of the evaluations.

    They refused!

    I sent the data to the press, asked them to publish it…nothing as of yet.

    • well perhaps you might give Jim or John Butcher a crack at it and/or they could tell you how to process it into useful info.

      I still think trying to use it to target a given teacher is wrong-headed and will
      lead to unintended outcomes.. for those that think it’s teachers who are the reason for SOL shortfalls.

      Most teachers – teach what they are told to teach – and they do it to an entire class – and the only teachers who are going to be successful targeted are those whose whole class fails to progress – but by the time you get to that data you’re going to see a lot of other data – for the school itself that is going to cause other questions.

      the most at-risk teachers are going to be those at schools where the whole school has a rank SGP and his/her class is below it – class wide.

      I applaud the SGP – because it offers hope that :

      1 – problems WILL be identified
      2. – there will NOT be an automatic presumption that a teacher caused it
      3. – if there is -the school better have virtue in the numbers for the school and can clearly show it’s a teacher-specific problem.
      4. – I’d be willing to bet that most principles and administrators will find that kind of a process to be entirely uncomfortable… but we’ll see.

    • Jim… maybe we need to tag team on this. Lynchburg’s performance against other school systems is atrocious, in my opinion. I tried sorting the data and no matter how it was sorted, Lynchburg lagged behind.

  6. re: ” For much too long, the principal measures of educational quality were inputs: budgets, teacher salaries, class sizes, pupil/teacher ratios, and the like. ”

    totally true.

    but before now – and even now – there are voices that say that unaccountable public schools should be abandoned and/or challenged by voucher/charter/non-public schools with one huge significant difference.

    In that advocacy, there are NO PROPOSALS for the non-public schools to be held accountable … as far as I know. it’s like it’s not even a part of what they propose!

    seems like we’re throwing out the baby with the bath water or advocating a de-facto different standard (any standard?) for the supposed better education alternatives.

    that’s why I find the school voucher/choice movement to be so bogus. Why not actually compete head-to-head on superior performance rather than claim you deserve the money – without any accountability because the public schools don’t perform?

    like many other things these days – it appears that all the opponents have to do – to justify alternatives is to prove the current approach fails.

    Many of the alternatives proposed – have no track record. they have no demonstrated record of performance. they actually have no proof that their alternative concept actually works but worse than all of that – they don’t seem to offer to compete on transparency and accountability.

    they just want the job – with no strings attached.

    why should taxpayers buy such a pig in a poke to start with?

    why do the folks who say the public schools have failed and choice/charter should have a shot – never say they want choice/charter to be transparent and accountable also?

    I’m all for the challengers. Let’s have competition but this stuff about no standards, no transparency and no accountability for the alternatives is a bunch of hooey.

    Let Heritage Action, CATO and ALEC and whoever else is exercising their anti-public school vocal cords to get their fat rumps into this game – fully.

    lets see some real alternative schools that actually do compete on academic performance… and the proof is in the data… that they are indeed superior.

    until then – I chalk this up the same way I chalk up critics of other things like health care – put your money where your mouth is on the alternatives or admit that what you’re really about is not performance but just ideology.

    “ideas” are not solutions.. any tom, dick and harry can put together a half-butted “idea” – hells bells, we got a Congress of half wits that already do that … but like the choice school issue – do more than blather about cockamamie ideas – put those ideas to the test in real school environments… and show that your ideas are actually superior… and most of all that you do know how to ferret out and get rid of “bad” teachers that then leads to better results.

    how about it?

  7. I am lost on the logic of opposing the common core. If not at the national level then why not at the state level? The more curricula is common the more incentive there is for technology-based entrepreneurs to offer computer assisted teaching and tutoring. When curricula are managed at the local level there is much less interest.

    I’ve been told that elementary school textbooks cost more than $100. I must assume that middle school and high school texts are more expensive still. It must cost less to publish these books electronically. So, how much would a low end PC or tablet really cost when the reduced cost of textbooks is taken into account?

    Beyond that, there is a growing belief that sophisticated software can record a child’s early efforts to learn (via ongoing quizzes in the text) and try different teaching styles. Some kids are visual learners, some are audio learners, some prefer quantitative explanations, some do better with words. The software would start to try different teaching styles in an effort to determine the style which best teaches that student. Sure, the teacher still has to teach the class the same way but the electronic teacher’s assistant does not.

    • re: ” the logic of opposing the common core”

      that’s a subject I’d like to hear Jim Bacon blog on… common core, SOLs and standards in general… and whether they should be nationwide or statewide or not and just local.

      In a day and time when people change jobs and move – around – one wonders why a standardized curriculum is such a bad, even unacceptable thing and most folks with a decent education themselves KNOW the importance and WANT some level of standardized benchmarking so they’ll know where their own kids are academically – a responsibility that parents do have – beyond politics.

      The anti-common-core folks don’t have a real alternative plan. Their “plan” is to oppose government and in their mind – standardization means govt and that’s why you hear “local control” and it presumably includes the SOLs in Virginia.

      NCLB, SOLs and Common-core would be gone in a heartbeat if they decided the issue.

      what is their plan? what do they want to do instead to assure their kids are growing up with enough education to successfully get a job in the 21st century ?

      not the first or hardly the last thing on their agenda – it’s much more important to go after teacher unions – real or imagined and the “bad” teachers who have not educated their kids properly.

      This is a pattern with the ignorati these days. They’re almost monolithically opposed to the “govt” being involved in anything from the environment, to health care to education. To them – government is a brand/category of liberalism and true conservatism means as less govt as they can get by destroying as much as they can to get back to only what was specifically
      articulated in the Constitution which means about half or more of the Cabinets would go away all together.

      The “plan” for education is the exact same plan for health care and other things – get the Federal and State government – out of it. Turn over education to no-public schools and let those who are sick and old go seek charity at the churches.

      when will this foolishness end because right now – as a county moving forward – we’re not – we’re in a death spiral from our own American brand of suicide bombers.

      It will end when parents stand up and swat down the ignorati… at the local school – encouraged them to leave to actually seek the real alternatives they
      never spoke of as opponents.

      Much as been said on these pages about the plight of economically disadvantaged kids especially minorities and how the parents are “lazy”, passing on “bad genes” and, in general – illiterate because they were more interested in gansta rap than education …

      but Now, there’s another group out there these days – just as bad -worse. They actually have better educations and have access to world-class (lite) education for their kids but they’ve rejected it.

      It’s one thing for mom to have a stunted 11th grade education and dad is off in prison somewhere because he couldn’t make a legal living with a crippled education – it’s quite another for people who ARE actually fairly-well educated to essentially advocate for weakening standards or have no standards at all for their own kids education. They actually SEEM to WANT to go back to a world where kids graduated high school and were “job ready”.

      this is a new breed of folks who basically see a standards-based public education more as a “liberal plot” than something the country, their kids – must have to stay competitive in the 21st century.

      They seem to be wanting a return to a world that no longer exists – one that is black and white simplistic where only the most minimal govt is needed.

      they’re certainly welcome to their opinion – and even their vote on the issues but the rest of us including the remaining part of the GOP that is still dealing with the real world – must move on and do the things that are necessary for us to remain successful and competitive as a country.

      Some day – the part of the GOP that is still connected to the real world – needs to take a firm and unambiguous stand on education – and the other important issues like health care and AGW, immigration and a few other things or else
      gridlock will become the new normal.

  8. I’m still not sold that the SOL is worth anymore than student grades in predicting and analyzing things like whether a student will graduate, is prepared for college or is actually learning anything.

    The said, the SGPs are great and VDOE was wrong to keep them under wraps. I doubt the utility of – generally – getting anything useful out of looking at these on an individual teacher level. The trend that emerged last time we looked at aggregated data about school system performance seems to be emerging here, that smaller school districts tend to educate all their students better – Franklin and Charles city both have only one elementary, one middle and one high school – and I’d bet money that applied to individual teachers you’d see a mostly random distribution but that teachers in more successful districts would perform better than teachers in less successful districts.

    That said, I’m heartened to see we can finally – at least in the Richmond region – but to rest the bullshit shibboleth that “the suburbs are where the good schools are” since it looks like according to these SGPs Richmond is outperforming Chesterfield, Hanover, Henrico and Goochland.

    • The SGPs would not be possible without a uniform standardized measure used by all schools in a state.

      The other states do not have SOLs but they do have their own standardized way to measure and that’s the value and usefulness of the data for SGP. Without the standardize test SGPs would not be possible. Other states that use SGP -do base it on their version of standardized testing but it’s not SOLs which are unique to Va.

      there is much hew and cry about “high stakes” testing of SOLs in Va and their counterpart in other states.

      Part of the problem is they have more SOLs subjects than we should have or need and partly because the GA wanted them – but it’s gotten out of hand and it really (these are my views of course) -dissipates the importance and value of the core – core academic subjects – the ones that other countries focus on and that we are compared to.

      so we’re a mile wide and an inch deep and we stress the teachers and students with the higher frequency of high stakes testing.

      but we do need standards or else you could not compare between schools, districts or the state.

  9. I do not understand why those of a conservative bent are so opposed to SOLs (or SGBs), when it was conservatives who originally led the charge for greater accountability in our secondary schools. Yes, mandatory testing smacks of big brother education (and may even expose more than a few creationist curricula), but it provides the basis for parent knowledge, and ultimately parent choice. Perhaps that choice ought to be exercised politically in the first instance, to reform the public schools, but where that has proved impossible (or where one’s children need a good education NOW), why not also support the competitive choice of tuition vouchers to private schools? Again, it was conservatives who led the way here; why are they so busy trying to repeal SOL/common core standards now?

Leave a Reply