Virginia Public Schools and Learning Losses – Part 1 – Winners and Losers

by James C. Sherlock

This article is the first in a series about COVID-associated learning losses in Virginia public schools.

The contribution I hope to make is to measure learning losses and correlating factors in each of 132 school divisions horizontally against its own pre-COVID learning assessment results.

That is different than comparing Richmond to Falls Church to Wise County vertically. We will do that too, but only in knowledge differentials — gains and losses — across all grade levels, not in specific levels of knowledge attained by the students before and after COVID interruptions.

Then we will seek correlation of learning losses with other factors. At this level of aggregation of statistics, correlation is what can be done. Causation assessment requires far more information than is available to the public.

I have left out race as a factor on purpose, at least at this time. I have found that when race is included all of the rest of the data tend to be ignored. A mistake in my view. I have included a factor of percentage of students in each school division economically disadvantaged for this data run. I may check it against racial correlations later.

The measures of student achievement used here for measuring learning losses are:

  • the last SOLs taken before COVID in 2018-2019 and
  • the SOLs in the post-COVID-shutdown year of 2021-22.  

Resources over that period were teachers, kids and their parents. Some turnover in teachers and kids, but not significant at this level of aggregation. The kids were three years older, replaced by younger ones in each grade. Since SOL testing does not begin until 3rd grade, virtually all that took SOLs in 2021-22 were in the system in 2018-19.

You will see that some divisions — teachers, students and their parents together — navigated the three years between the spring of 2019 and the Spring of 22 well. Some very well. Others failed in what they tried to achieve. Some badly.

As I roll out the data in a series of articles I think readers will find the learning loss data and its horizontal and vertical correlations informative.

And in some cases surprising.

Since 2018-19, much education data have been either missing or unusable because of COVID interruptions. I decided to fill in that gap in a way that I hope will provide meaningful lessons and fair assessments of correlation of learning losses.

Horizontal data fields — Fredericksburg example. To show what types of correlation are coming, I offer one example to show the scope of the project. Here is a look at the entire horizontal spreadsheet element so far for Fredericksburg Public Schools. I have the same for each of Virginia’s districts.

You will note that Fredericksburg’s 2020-21 chronic absentee statistics correlate with massive declines in 2021-22 SOLs from already poor 2018-19 SOL scores in Fredericksburg. VDOE is still attempting to confirm the 71% number.

But we can see the two data elements correlate: poor attendance and bad learning losses.

The learning losses also correlate with late resumption of in-person learning in 2020-21. We see that the division had a relatively standard number of economically disadvantaged kids and students with disabilities compared to state averages, and a high percentage of English Learners against the state average.

We also see that Fredericksburg has adopted the Virginia Tiered Systems of Supports (VTSS) framework designed to improve academics, behavior, and social-emotional wellness. Most school divisions have not. This approach did not seem to help Fredericksburg schools mitigate COVID learning losses.

Vertical comparisons. But it is not only internal division correlation that I am seeking. We will look at vertical correlations (and lack of correlation) across all school divisions in later articles in this series.

You will see that some of the data fields are pending. VDOE has agreed to provide me data for 2018-19 chronic absenteeism so that we can see trends in that domain. I may add data fields to the master later.

I will roll this out a little at a time.

This first article shows vertical results — best and worst. Correlations will come later.

  1. the divisions with best learning gains and smallest learning losses– those SOLs revealing less than 3% declines in student pass rates of SOLs between Spring of 2019 and Spring of 2022  are here.
  2. the worst of the learning losses — those SOLs revealing more than 20% declines in student pass rates of SOLs between Spring of 2019 and Spring of 2022 — are here.

Reading gains and losses. A case can be made that reading skills can be maintained by individual students above grade 3 without school support. Reading gains and losses across the 132 divisions are not shown but are listed below.

  • Gains in reading skills over the three years were seen in Lexington, Buena Vista, Clarke County, Falls Church, Alleghany County, Colonial Beach and Botetourt County.
  • The worst losses in reading skills were (between 15 and 19% lower pass rates in 2021-22) in Hopewell, Charles City County, Bath County and Franklin.

Best school divisions in mitigating learning losses. So what do we see among the best performing school divisions in preventing learning losses? Color coding goes from lime green (outstanding) to red (bad) and every shade in between so that viewers can see visually what went on.

  • The size of the student population does not correlate to learning loss.
  • Students in these schools (and across Virginia) suffered their smallest learning loss in the subject that they were most likely to do on their own — reading.
  • You will also see that only one school division in the state was able to maintain student SOL scores in math — Lexington City.
  • Lexington City Public Schools, a small division, started with good scores and ran the table raising the performance of their students. Students actually improved their scores over the COVID years. Norton and Staunton raised the bar or prevented significant learning loss on three SOLs.
  • Some of the rest of the schools started with high scores and maintained them. Some with poor or bad scores maintained them.
  • Staunton’s 16 point increase in English Writing score was amazing. Similarly Norton’s 13 point increase and Highland County’s 11 point boost in the same SOL. From before COVID.

But we also see in a different view that Lexington above all, Highland County, Middlesex, Norton and Staunton students maintained skills in one or more subjects in addition to reading and writing:

  • math (Lexington only among 132 school divisions),
  • history and social science; and
  • science.

If we take out reading, for which some case can be made could have been maintained by the students and parents themselves at home more than the other subjects, we see that writing skill loss mitigation dominated.

Worst school divisions in mitigating learning losses. How about the worst performing divisions. The attached spreadsheet shows only the SOLs in which more than 20% fewer students passed in the Spring of 2022 than passed the same test in the same division in 2019.

What are some of the things we see?

  • Franklin, Harrisonburg and Hopewell were complete train wrecks. Petersburg went from very bad to inexplicable.
  • Some of the divisions that finished 2019 with excellent results suffered major learning losses. Galax, Amelia County and Caroline County are examples. Some went from bad to worse.
  • Major learning losses in mathematics were found in some surprising divisions. Surprising to me anyway. Alexandria, Chesterfield County, Galax, Henrico County, Madison County, Orange County, Stafford County, Suffolk. There are more, but I did not expect those.
  • There were no reading losses of that magnitude, though writing skills suffered major losses in many.
  • Major SOL losses in science were nearly constant, followed by History and Social Science, the only subjects that students learn almost exclusively at school.

The school division with major learning learning losses in three or more SOLs were: Franklin and Harrisonburg with four and with three were Amelia County, Amherst County, Brunswick County, Caroline County, Galax, Hopewell, King and Queen County, Madison County, Newport News, Northampton County, Nottaway County, Portsmouth, Radford and Suffolk.

I choose my words carefully here. Whatever they did over those three years failed.

That does not assign blame, and we will look at other correlations, but they failed to do what they tried to do.

Much more follows.


Share this article



ADVERTISEMENT

(comments below)



ADVERTISEMENT

(comments below)


Comments

13 responses to “Virginia Public Schools and Learning Losses – Part 1 – Winners and Losers”

  1. LarrytheG Avatar

    This is quite excellent. Kudos! data centric. non-partisan. objective.

  2. LarrytheG Avatar

    The problem with correlations is so many variables that cannot be held “constant” so that valid correlations between two variables can be accomplished.

    So what Sherlock is doing is breaking the data down into classes of data where some variables are essentially the same/held constant to allow some correlations. Even then, there are other variables involved as he points out that some schools were “good” before COVID and after and other schools were “bad” before Covid and remained bad.

    As pointed out before, all of this is possible because of the rules that public schools collect data and make it available, something non-public schools are not required to do.

    During COVID, non-public schools also did various versions of in-person, virtual and hybrid… but we have no real way to assess them nor compare them to public schools.

    1. Dick Hall-Sizemore Avatar
      Dick Hall-Sizemore

      It is possible to identify which one of several variables has the most effect on an outcome. The technique is multivariate analysis. To apply multivariate analysis to a situation like this requires specialized software and training in the use of the various techniques.

      1. James C. Sherlock Avatar
        James C. Sherlock

        It is not that challenging for someone with my systems engineering background to identify variables that roughly correlate to results, in this case learning losses as I have defined and calculated them, with the methodology I am using.

        I expect one of those correlating variables will be chronic absenteeism for example. Another economic disadvantage. Another percentages of ELL kids. We’ll see. And we’ll see if any one stands out.

        You are right about the final determination, but the visualizations I have added and the power of the spreadsheet itself to sort by different variables will give us a pretty good initial picture.

      2. LarrytheG Avatar

        and someone who is schooled in data analysis and how to use the software and is inclined to have it peer reviewed.

        just simple things like how “virtual” was done , how much was done, many variables just for that.

        Even a “before and after” is tough if the actual curriculum content was different. Can’t assume the exact same material and curriculum was repeated.

        Paying attention to what was successful could be useful (but still speculative) , but looking at failure and trying to understand why involves so many different things that cannot be easily held constant.

        1. James C. Sherlock Avatar
          James C. Sherlock

          Perfection is the enemy of good enough.

          1. LarrytheG Avatar

            and faulty/wrong correlations may be the enemy of “good enough”.

            So I complimented you on the data collection and recognizing the differences in the data and hope you continue that approach when you start to get to correlations and conclusions.

            Some data that you don’t have and won’t be able to “correlate” is the data from ”
            73% of private schools said they had to move some or all their classes online during the pandemic”

            https://uploads.disquscdn.com/images/dc1c1716069b6fdd753b1c93a4b8dd21ab00d9c95114ea4798459792d8d10936.jpg

            https://www.usnews.com/news/education-news/articles/2022-03-15/how-traditional-public-private-and-charter-schools-responded-to-the-pandemic#:~:text=Overall%2C%20remote%20learning%20was%20a,classes%20online%20during%20the%20pandemic.

  3. “You will note that its 2020-21 chronic absentee statistics correlate with massive declines in 2021-22 SOLs from already poor 2018-19 SOL scores in Fredericksburg. VDOE is still attempting to confirm the 71%
    number.

    But we can see the two data elements correlate – poor attendance
    – bad learning losses….

    And that Fredericksburg has adopted the Virginia Tiered
    Systems of Supports (VTSS) framework designed to improve
    academics, behavior, and social-emotional wellness. Most school
    division have not. It did not seem to help Fredericksburg
    schools mitigate COVID learning losses.”

    Seems you’re on to something. We can’t teach kids who are not
    there. The chronic absentee numbers are an indicator of bad SOL results to follow. Fredericksburg is right down there at the bottom with
    Richmond, Petersburg and Danville.

    Here’s my followup in your “Challenge Accepted” post:

    Here’s an article that addresses F’burg City school problems and
    accepts the chronic absenteeism report as correct.

    https://fredericksburg.com/opinion/columnists/its-time-for-fredericksburg-to-face-the-real-problem/article_e8a89687-7a94-56ba-a7fe-fb6170952c30.html

    Look at JABs report on aggregate SOL numbers. It shows F’burg City
    schools as 6th from the bottom. They’re right down there with
    Richmond, Petersburg and Danville.

    It may just be that F’burg City schools really suck.

    The Superintendent is a F’burg native and has been part of the
    city school system for 40+ years, starting as a 6th grade teacher.
    Her education is thoroughly Virginia (to add fuel to that fire),
    Virginia State, UVa, VPI. She has post graduate certs from Oxford,
    Harvard and Howard.

    The Deputy Super has a more varied career where it is reported
    that he actually accomplished some things for school systems. It
    seems possible he was brought in as a fixer.

    It looks like the real story is that the VDOE graph is right and
    that it is just the tip of the iceberg of a F’burg City schools
    debacle.

    I’ll eat my crow for assuming that the absentee numbers were so
    egregious that they had to be wrong. Sometimes things are as bad as
    they seem.

    1. James C. Sherlock Avatar
      James C. Sherlock

      Thank you.

      I’m just legitimately trying to get to the bottom of this. VDOE is assisting me. I think we’ll get there.

      We are finding find both success and failure in unexpected places.

      1. Figuring out what the successes are doing right and replicating that while identifying what the failures are doing wrong and fixing them can make the whole state a lot better place.

        Falls Church has had about the best school system in the state for a long, long time (at least since they graduated me and got me out of the system). They are a good place to look for successes, although moderate size, lots of money, an educated community and a long term commitment to excellence may make those traits less than portable.

        The rural systems that have been successful may be a more productive place to look. They have succeeded despite their environment rather than because of it. That makes them interesting.

        BR has been over the failures fairly extensively, so few surprises there. The trick is how to fix them.

      2. The recently employed Deputy Super in F’burg’s bio shows success in Rural school districts.

  4. LarrytheG Avatar

    As we know, “virtual” can and is a lot of different methods and techniques. Some are pretty good, good enough to be offered at the State level including a private provider and some are awful, not more than a lecture via a computer screen.

    Some schools had decent or good virtual instruction and some had terrible and so that could be an interesting correlation if you actually could well-characterize the kind of virtual instruction a school was providing.

  5. vicnicholls Avatar
    vicnicholls

    Excellent reading and analysis Capt.

Leave a Reply