About Those SOL Numbers….

Matt Hurt, director of the Comprehensive Instructional Program coalition of rural school districts, provided detailed feedback on a recent post, “School Districts’ COVID Recovery Varied Widely

.” His comments contribute significantly to the discussion of the impact of  COVID school closures in 2020-21 on Standards of Learning test scores in 2021-22, so, I republish them with minor modifications here. — JAB

by Matt Hurt

According to my calculations, 47% of the variance among overall division pass rate differences in 2021 was due to the amount of in-person instruction offered combined with the poverty rate of the division.

When I discussed this with our teachers and administrators, it seemed to explain a lot. The way that they put it, kids of affluent parents tended to have the supports in the home to make sure the kids attended to their work when they were participating in virtual instruction. Kids who live in poverty were much less likely to have anyone standing over them making them do their work. Given this situation, these kids found much more engaging activities in which to invest their time than school work. I suspect that the amount of in-person instruction in 2021 did affect the 2022 outcomes, but I haven’t had the time to calculate that yet.

Second, the designations posted on the VDOE website were a very poor representation of how much in-class time was offered to students. For example, I am not aware of any division that offered a full school day, five days a week of in-person instruction during 2021, even some of those that were designated as having done so. The much more reliable measure can be found in the reports that were provided to VDOE during the summer of 2021. These reported the actual number of hours of in-person instruction that were offered to students that year.

We attempted to look at student-level data indicating the number of hours each student attended in-person versus virtual, but we found that data was too unreliable, as everything was in chaos and some kids changed between the two more than they changed their underwear. Therefore, the summer reports (which were used to create the table below) I consider to be the most reliable, accurate data we have of in-person instruction.

Third, the cut scores used to determine Reading SOL scores from 2013 through 2019 are listed below as the “Old Cut.” In 2021, we began to administer the newer SOL Reading test, and those cut scores are the “New Cut” scores in this table. As you can see there is a significant difference between the two, with the newer cut scores being lower, thus it is now easier for a kid to pass his/her reading test than prior to the pandemic. The math cut scores were lowered in 2019 and have remained constant since, so the math comparisons from 2019 to 2022 are apples-to-apples, while the reading comparisons are apples to Volkswagens.

I’m still digesting the data, and once I get it all right in my head, and my colleagues confirm by finding our study reasonable, I’ll fill you in on what we find.

BTW, when the state accreditation numbers are published next month, the scores will magically and significantly improve, at least for 3-8 reading and math. The 2021 General Assembly through [its] “growth” bill will make things look much better, but a bunch of that seems to be “growth” in name only.

Share this article


(comments below)


(comments below)


6 responses to “About Those SOL Numbers….”

  1. Kathleen Smith Avatar
    Kathleen Smith

    Matt, you cleared up another point. Does the build a table for 2018-19 include growth? This was right on time.

    1. Matt Hurt Avatar
      Matt Hurt

      No ma’am, the build-a-table page displays the federal accountability calculations for student achievement- no growth, no recoveries, and no transfer adjustments.

  2. James Wyatt Whitehead Avatar
    James Wyatt Whitehead

    Can’t Glenn Y. do something about those ridiculous cut scores? Testing at that low of a bar is a waste of time and bread. Passing score should be 70% or above. Back in 1998, when SOLS began, the cut scores elementary to high school varied between 65% to 70%. The retreat from high standards must stop.

  3. madness Avatar

    Not sure how Matt is measuring 2021 and full in-person. I know that Hanover County Public Schools had in-person school for the entirety of 2020-2021 school year, of which about two-thirds of the regular school population participated. They didn’t switch back and forth, except for individual students involved in COVID protocols.

    HCPS did stand up an online school for that year, which families had to opt into. And those students had to stay in the online school at least for the semester, if not the full-year. No switching back and forth, except at the semester break (space permitting).

    HCPS still has its online schooling, even this year (third of operation), but the numbers are shrinking each year. But it’s a separate, standalone school for the purposes of education delivery, with its own principals and dedicated teachers. But the students still belong to their home schools for things like graduation.

    Maybe that online school makes him categorize it differently, but they ran an in-person school for the entire year and didn’t flip the schools back and forth between in-person and virtual.

    1. Kathleen Smith Avatar
      Kathleen Smith

      Each division reported their percentages to the state superintendent. Keep in mind, some guessed, and some actually calculated.

  4. Kathleen Smith Avatar
    Kathleen Smith

    The change in cut score will decrease the achievement gap. Will we look better in the Secretary’s honesty gap data? Kudos for her, but it was put in place by democrats and I don’t see the new board republican board changing it. The new growth measure will inadvertently set even lower expectations by allowing a pass when the student isn’t near grade level, he/she/they just moved up a slight increment from the beginning of the year and it is counted as a pass. If I were an administrator, I would make sure the Fall growth test was on the hottest day of the season and the AC was out. More kids would show growth. Or maybe, I would give the kids a pre-Halloween party just before the test so they were all on a sugar high and would concentrate less. Or, maybe host and SOL sleepover and keep them up late with movies the night before the test.

Leave a Reply