by James C. Sherlock
We need all the help we can get assessing Virginia schools and producing actionable information to make them better.
The Standards of Learning exams show the results of poor learning, but do not identify actionable causes.
Directed to choose an additional measure of school quality by the federal ESSA (Every Student Succeeds Act) in 2015, Virginia, like most states, chose to use a school climate survey.
Virginia chose a hell of a good one developed in Virginia by University of Virginia scientists for the federal government. It was used here very successfully for several years.
Then Virginia Department of Education (VDOE) functionaries dumped it in 2020 for what were clearly political/philosophical reasons.
I have not seen any evidence that the new survey has been proven valid or reliable, even if one agrees with their woke philosophy. Frankly it could not have been validated in such a short span of time in the middle of COVID.
The new leadership at VDOE needs to return to the original.
Virginia’s pre-woke school climate survey was called by its developers Authoritative School Climate Survey (Authoritative School).
It has proven to be an instrument of improvement for those that have acted on its results. There are dozens of studies supporting its validity, the U.S. Dept of Education placed it on its list of approved surveys, and researchers around the world have adopted it.
The results of that survey in 2019 at Walker-Grant Middle School in Fredericksburg were predictive of dreadful chronic absenteeism in that school in 2020-21, and massive learning losses in 2021-22.
In 2020, the same survey found the learning climate far better at Fredericksburg’s James Monroe High. It accurately forecast that COVID-year attendance would be much better at James Monroe and the learning losses far less extreme than at Walker-Grant.
Then Virginia stopped using it.
An opportunity to change Authoritative School was provided when VDOE and DCJS in 2020 combined their two existing survey efforts into a single survey instrument and administration cycle.
Richmond bureaucrats scrapped the scientifically developed and validated Authoritative School for a new, untested one they “developed” in the middle of COVID in 2020. They have administered it since 2021.
That new one is Virginia School Survey of Climate and Working Conditions (Virginia School Survey).
It is woke, but it also appears useless for the purpose.
Pre-COVID Authoritative School Climate Survey results in Fredericksburg
- Walker-Grant Middle School 2019
- James Monroe High School 2020
The key scales presented in the reports in Authoritative School are scaled scores designed to make them easier to interpret. For each scale, raw scores for student and staff participants within a school were averaged to create school means for students and for staff. The school means across the state were transformed so that the state average is 10 and the standard deviation is 1.
As a result, schools with scores between 9 and 11 are in the average range. Schools with scores of 11 or higher are statistically above average and schools below 9 are statistically below average and show a need for improvement.
Walker-Grant Middle was awarded student scores that averaged 7.8. James Monroe 9.54.
The staff-awarded scores in Walker-Grant Middle averaged 9.28. At James Monroe 9.0. Both were in the lower quarter of the average zone, but they were in it. So, the staffs were relatively sanguine about their schools. Only those at James Monroe were proven relatively correct.
The huge disconnect between the staff and the students’ opinions of school climate at Walker-Grant was a big deal. The staff was clueless about how the students felt.
And the students were proven prescient in their assessments.
Operational results during and after COVID – same schools
So let’s see if the survey results predicted operational results:
Chronic absenteeism in the 2020-21 school year. The entire division turned in terrible results, but James Monroe was by far the best.
- Walker-Grant 73.6%.
- James Monroe 46.9%.
- Division as a whole 70.5%.
- State 11%
Chronic absenteeism in 2021-22. James Monroe approached state norms. Walker-Grant doubled the state average rate of chronic absenteeism.
- Walker-Grant 41%,
- James Monroe 24%.
- Division as a whole 36%.
- State 20%
Math SOL pass rates 2021-22. James Monroe was near state averages.
- Walker-Grant 27%
- James Monroe 62%
- Division 40%
- State 66%
Reading SOL pass rates 2021-22. James Monroe exceeded state averages.
- Walker-Grant 50%
- James Monroe 82%
- Division 55%
- State 73%
So, there can be no question what happened here.
Walker-Grant Authoritative School results showed that the middle school was in trouble before COVID hit, and that such a weak and chaotic school climate could not sustain the stress of COVID.
The Walker-Grant students nailed the problems there in the 2019 school climate survey. The staff at that school missed them.
The Authoritative School surveys used from 2013 to 2020 had been painstakingly and iteratively developed and extensively assessed as both reliable and valid. And they worked.
The surveys at both Walker-Grant and James Monroe were predictive. They provided actionable information for Fredericksburg schools.
The politics of school climate surveys.
Authoritative School found a place on the list of climate surveys validated by the U.S. Department of Education.
We have the ability to use those surveys to make informed decisions on which schools need improvement and in what areas, not just from test scores but from excellent school climate assessments.
But in 2020 VDOE decided not to use it any more.
The Richmond coup. School climate surveys are high-stakes measurements. Careers rise and fall based upon the outcomes. Strengths and weaknesses are exposed. In the best schools and divisions, the information they provide is used to improve the learning climate.
So, the results had better be valid and reliable.
All education researchers and statisticians are trained to know that surveys used for high-stakes measurement should be rigorously evaluated for reliability and validity, and that a change in the items means that reliability and validity must be re-established.
VDOE, with the lead on designing the Virginia School Survey, seems to have ignored the Standards of Educational and Psychological Testing which is the Bible developed by the American Educational Research Association and the American Psychological Association.
The team that met in Richmond to redefine the climate survey, either from home or masked at work, had no chance during COVID to design and test a survey that measured school climate. So, they just started using it.
They appear clearly to have tried to make the new survey more reflective of progressive values.
The Virginia School Survey questions document eliminated everything from Authoritative School that did not square with their sensitivities. And they added their own questions (items).
Fine – elections have consequences.
The bureaucrats were perhaps carrying water for the administration. And maybe a little of their own. But whatever they were trying to accomplish, they did not do it right.
Let’s examine some of the carnage:
- The word “fair” is used once in the Virginia School Survey. Agree/disagree:
The school rules are fair.
- That question replaced the original question in Authoritative School Climate Survey: Agree/disagree:
This school consistently has high expectations for student behavior with strict and fair discipline.
- The word strict is banished in the new survey.
- The word “discipline” is used once in the new survey. It is in a question about school resource officers, apparently the only adults who can be associated with discipline. In Authoritative School Climate Survey, “discipline” was used 35 times.
- “Disciplinary” — gone in the new one. In the original, “Disciplinary Structure” was a major scale.
- “Academic Expectations” was also a major scale in the original. There is no use of that term in the new student survey. Or even the term “expectations.” Dickens would be sad.
You get the point. VDOE and the Department of Criminal Justice Services (DCJS) took away an established tool in the middle of a crisis, showing at a minimum poor judgment and misplaced priorities.
Changing a reliable and valid survey in the middle of the COVID storm means that the schools are deprived of an instrument they could have used to measure the strengths and weaknesses of their schools and know what was needed to improve.
It is more than just unfortunate that we cannot compare the pre-COVID school climate survey results with the new ones administered starting in 2021. But we can’t. Apples and oranges.
The surveys are entirely different. But, also unfortunately, the results can be accessed on the same government web page as if they were coherent.
Elsewhere DCJS announces:
To reduce survey administrative burden, the Virginia School Survey combines two required survey efforts into a single survey instrument and administration cycle.
Like telling Mrs. Lincoln that her husband would be delayed returning from the theater.
Maybe the new leadership of DCJS and VDOE will consider returning to the valid and reliable survey that the feds paid UVa scientists to develop and test — in Virginia.
The one that worked in Fredericksburg.