by David Wojick
Loyal Baconeers know what the so-called “Honesty Gap” is, as we have been discussing it at length. It is heavily featured in a recent report from the Virginia Department of Education. The gap itself is the big difference between the numerical thresholds of “proficiency” used by Virginia versus the National Assessment of Educational Progress (NAEP) for certain test regimes. NAEP’s threshold is much higher than Virginia’s (and most other States).
The gap has nothing to do with honesty, just language. Virginia has two tiers of passing scores: “proficient” and “advanced.” NAEP has three: “basic,” “proficient” and “advanced.” What Virginia calls proficient aligns closely to what NAEP calls basic. Comparing Virginia students’ SOL pass rates at the proficient level with their NAEP pass rates at the proficient level is likening apples to organges.
So, where does “honesty” come in? This is where the story gets interesting, and in my view unpleasant. It turns out that “honesty gap” is hyperbolic rhetoric, spawned by an advocacy group.
The Honesty Gap ( HonestyGap.org) is a project of the Collaborative for Student Success, which is funded by a number of prominent left-wing foundations. The project home page immediately hits you with this grandiose pronouncement:
Parents deserve the truth. Historically, states have exaggerated the percent of students who are proficient – as demonstrated by the huge gaps that have existed between state NAEP scores and what states report as their proficiency rate.
This is truly dishonest. There is nothing untruthful, inaccurate or exaggerated in the proficiency ratings of Virginia and the other states. If anything, NAEP is using an extreme definition of proficient. Switching to a three-tier system would provide a bit more information, but how about five tiers, or twenty? The point is there is nothing dishonest about the Virginia system. Calling this an “honesty gap” is a hoax. If anything, Virginia’s threshold is more accurate.
It turns out that “proficient” has a fairly wide range of meanings. But after checking six different dictionaries I found the most common synonym was “competent.” Virginia’s use of the term to cover around 70% of SOL test takers is certainly consistent with this definition. They are competent in the subject, for example reading or math. Those who fail to pass SOL exams are less than competent. This all makes sense.
A more extreme definition that also occurs occasionally is “expert.” The NAEP definition is up there somewhere as well. Mind you, I seriously doubt that a third of the students are experts, in math for example. That claim would be ridiculous.
In fact NAEP’s threshold looks to be in a form of semantic limbo, floating somewhere between competent and expert. As I have written elsewhere, NAEP is a federal black box so there is no way to tell what they think they are doing. See my https://www.cfact.org/2022/05/18/ridiculous-federal-4th-grade-reading-tests/.
In short, there is no honesty gap and the claim that there is, is itself dishonest. In addition to this issue being semantic nonsense, it distracts from the real gaps that deserve our attention. These are the well documented differences in test levels among various demographic groups. I address this issue in the CFACT article referenced above. Our tests may be semantically biased against low income students. This is a serious issue.
David Wojick, Ph.D. is an independent analyst working at the intersection of science, technology and policy. He has been on the faculty of Carnegie Mellon University and the staffs of the U.S. Office of Naval Research and the Naval Research Lab.
