About the Governor’s New Metrics…

by James A. Bacon

Governor Ralph Northam presented Friday his “Forward Virginia” blueprint for reopening Virginia’s economy when the COVID-19 epidemic recedes. As Virginia sees progress in five key metrics, the Governor says, he will relax his emergency restrictions in phases. The first of four phases would allow some businesses to re-open with “strict safety restrictions.”

“We will move forward, but in a way that prioritizes public health and builds public confidence,” said Northam. “Businesses know that customers will return only when they feel that it is safe to do so. Our blueprint for the path forward is data-driven and provides clear guidance, so Virginians will know what to expect and understand how we will decide to when to lift certain public health restrictions.”

Adopting a data-driven approach sounds reasonable in the abstract. The devil. of course, is in the details. The logic behind the Governor’s metrics is not self-evident, and may require more explanation. According to the Blueprint, the five metrics are:

  • Downward trend: Percentage of positive tests over 14 days
  • Downward trend: Hospitalizations over 14 days
  • Increased testing and tracing
  • Enough hospital beds & intensive care capacity
  • Increasing & sustainable supply of PPE

Here’s what the Governor’s press release offers by way of explanation:

Virginia is looking at a wide range of public health data. The Governor emphasized that key indicators will include a 14-day downward trend in confirmed cases as a percentage of overall tests and in reduced COVID-19 hospitalizations. While hospitalization rates have largely stabilized in the Commonwealth, confirmed cases continue to rise. …

To ensure the continued safety of Virginians, the Commonwealth aims to test at least 10,000 individuals per day. Karen Remley, former Commissioner of Health and current co-chair of Virginia’s Testing Work Group, outlined a four stage approach to meet this goal prior to safe reopening. The expanded testing plan includes hiring contact tracers, who will support local health departments in identifying individuals who may be exposed to COVID-19 and helping them self-isolate.

First question: How did the Governor pick the 10,000 number? Do we need 10,000 tests daily to draw statistically valid inferences about the prevalence of the disease, or was this number plucked out of thin air because it sounds good? Neither the governor’s press release nor news accounts address that question.

Now, let’s discuss these metrics one by one.

Downward trend in percentage of positive tests over 14 days.

This graph shows the percentage of positive COVID-19 tests since mid-March.

The percentage is highly volatile but has been heading in a higher direction. While the percentage of positive tests may reflect the underlying prevalence of the virus, it is also influenced by extraneous factors such as the number of tests conducted, the protocol behind the tests, and who is conducting the test.

The Virginia Department of Health will have a testing capacity of 400 to 500 per day. The other 9,500 will have to come from Virginia hospitals and private labs. Will these diverse groups are use the same or different technologies? Will their testing protocols be the same? In the past, the high percentage of positive tests has been tied to the priority given to patients showing COVID-like symptoms, health care practitioners in hospitals, and patients in prisons, nursing homes and other high-risk groups. Does the VDH propose testing the same groups with the same protocols, or does it propose shifting to a random testing of the general population?

I have argued previously that the number of positive tests is a meaningless indicator of the virus’ spread, and the same applies to the percentage of positive tests. Furthermore, it strikes me as a metric of limited value. What matters is not the percentage of the population that has contracted the virus (unless we’re trying to calculate herd immunity, which the Governor has not mentioned) but the number of people so severely afflicted that they go to the hospital…. which brings us the the next metric.

Downward trend: Hospitalizations over 14 days.

What does “hospitalizations” mean? Are we talking about the trend in patients admitted that day? Are we referring to the number of hospitalized patients confirmed by tests to have the disease? Are we talking about the total number of confirmed patients plus those being tested for the disease?

We published the graph below in a separate post earlier today. The red line shows the number of confirmed + being tested patients. Although the number has crept higher, it is far less than the number of available beds, which numbered 5,343 in data published yesterday.

What significance is the statewide average anyway? The prevalence of the disease and the burden it poses on local hospitals varies widely by region. In Northern Virginia, hospitals are far more stressed than, say, in Southwest Virginia. Emergency measures that might be appropriate for one might be inappropriate for the other.

Increased testing and tracing

The Governor says he will not relax emergency measures he he sees “increased testing and tracing.” Specifically, he proposes hiring “contact tracers” who will “support local health departments in identifying individuals who may be exposed to COVID-19 and helping them self-isolate.”

The value of contact tracing is evident in the early stages of an epidemic when it may be possible to identify rare individuals with a disease, isolate them, and alert others who may have been exposed to them. But once the virus has broken out into the community — when literally thousands or tens of thousands have it — what’s the point of tracing? The epidemic reaches a point where it is moving too fast to trace. The Governor has not explained how he expects this measure will help to control the spread of the disease at this stage of its spread.

Finally, what metric will the Governor be looking at? The number of VDH contact tracers? The number of patients contacted? The number of potential exposures traced? If we select a measure, do we know what constitutes a number that would justify relaxing the shutdown?

Enough hospital beds & intensive care capacity

Of all the metrics, this is the easiest to measure and track. Refer again to the graph above. The green and blue lines show the number of ICUs and ventilators in use. They are heading down. Hospitals have stated loudly and clearly that capacity is not an issue.

Increasing & sustainable supply of PPE

Personal protective gear is delivered to Virginia in large but infrequent batches. According to the press release, Virginia has ordered 17.4 million N95 masks, 8.3 million surgical masks, 17.1 million gloves, 1.7 million gowns, and 1 million face shields in a joint contract with Maryland and Washington, D.C. A second shipment from Northfield Medical Manufacturing, scheduled to arrive Friday, was supposed to include three million titrile exam gloves, 100,000 N95 masks, 500,000 3-ply procedure masks, and 40,000 isolation gowns.

So, which of these items will be measured — N95 masks, surgical masks, gloves, or gowns? Or will VDH measure as “basket” of equipment? What if hospitals have an ample supply of one category but a shortage in another? 

Will the VDH metric encompass state supplies only, or will it include equipment that hospitals are able to acquire on their own? And how do we define “sustainable” anyway? Must the equipment be contracted for? Or must it be delivered and stockpiled?

Alternatively, why not use the Virginia Hospital and Healthcare Association metric? The VHHA reports the number of hospitals “experiencing difficulty obtaining or replenishing PPE in the next 72 hours.” This morning, the VHHA indicated that only two hospitals were having difficulty.

In summary, the Governor’s metrics are only conceptual at this point. Many, many details need to be worked out to give them meaning as decision-making tools. Once the Governor’s people do work them out, it would be very helpful if the VDH added them to its COVID-19 dashboard.