Uh, Oh…

by Dean Wortmier

Because the panacea bonfire of the “New Abbott Labs Test” is being stoked, I have been forced to revive some brain cells that have been comfortably soaking up rum for the past 8 years.

How much help is testing for CoV2 going to be in deciding to ‘Open Up’ Virginia?

Let’s use Abbott’s Emergency Use Authorization (EUA) application[1] for our Gold Standard.

From the tests that Abbott submitted to get the EUA, the 95% confidence interval for the Probability of Detection, Pd, of the virus is (94.0, 100), greater than 94 but less than 100.  Moreover, the document tells us that the 95% confidence interval for the Probability of a False Alarm, Pfa, is (0, 11).

Pretty impressive, well, impressive enough to secure a EUA to soak up $Billions of taxpayer money, but is really going to help with decisions to end the quarantine measures for Virginia? (Did you see how I kept it relevant to Bacon’s Rebllion’s blog?) Remember, we’re making an executive decision that could cost granny her life.

We need to know that, if I select someone from the population and his test returns a positive result, then the person is really sick (immune).

We need what is known as the conditional probability, P(A|B), read Probability of event A given event B has occurred. In our case, P(sick person | +test), or P(s|+).  Mathematically, this written as

P(s|+) = P(+|s)P(s) / P(+)

where P(+|s) is probability of getting a + from a sick person (that’s just Pd), P(s) is the probability the guy REALLY is sick/immune from CoV2, and P(+) is the probability of getting a positive result at all, either through a good detection or a false alarm, or P(+|s)P(s) +P(+|not s)P(not s).

We’re all set to go except we don’t know P(s), and P(not s) = 1-P(s). But we can estimate. The U.S. has 600,000 reported cases, and estimates for total sickened is 10x that, 6M, out of a population of 360M so,

P(s) =approx 0.1666… Meh, let’s call it 2%

Let’s grab some values, Pd = 97%, Pfa = 5%, P(not s) = 98% and this

P(s|+) = 0.97×0.02/(0.97×0.02 + 0.05×0.98) = 0.0194/(0.0194 + 0.049) = 0.0194/0.0684 = 0.2836

P(not sick|+) = 1.0-0.2836

Do you see the problem? The test sucks for any decisions to ‘Open Up’ since nearly ¾ of the time a non-immune person will be declared to be immune.

Dean Wortmier is the pen name of a retiree living in Hampton Roads. A former employee of government-contracting businesses, he spent 23 years as an adjunct professor in mathematics, physics and computer science, and published more than a dozen degrees in information theory.

[1] https://www.molecular.abbott/sal/9N77-095_SARS-CoV-2_US_EUA_Amp_PI.pdf

There are currently no comments highlighted.

69 responses to “Uh, Oh…

  1. I’ll be honest, I understood absolutely nothing of Wortmier’s mathematical analysis. Zero. I suspect that few readers will fare much better. But if anyone out there is statistically literate, I hope they will weigh in and either confirm or dispute his conclusion that a non-immune person will be declared to be immune 3/4 of the time.

  2. Well, I am at zero understanding too, but my uninformed wild guesses are:

    1/ He’s a brilliant guy who brilliance allows him to see what many other very smart dudes or chicks don’t see, a distinct possibility in history of science, so take him dead serious for now in the moment.

    2/ Or, the Rum caught up with, overwhelmed, and despoiled his brain cells irrevocably before his effort at their revival, and he’s doing the best his can under the circumstances, but it ain’t good enough.

    3. Or he’s a very smart guy who don’t like Abbot Labs for whatever reason, including perhaps he got out of bed this morning grumpy, or he’s a competitor of some sort.

    All of these comments of course are wild guesses on my part. Jim, perhaps you know more than you’re saying. Did he appear off the street, or from MIT or UVA perhaps?

  3. I distrust anybody not fully identified, and I suspect this is a pseudonym. That aside, the antibody test is the one that has more utility for determining whether somebody might be immune. I don’t think Abbott has marketed this test for any thing except “is this person infected?” and the key standard of success is no false negatives. A false positive will get you treated unnecessarily perhaps, but a false negative lets you out to infect others….

    And if I’m right, this whole attack on the test is misdirection. It’s like attacking a football player over his inability to drive 300 yard down the fairway.

    Tell us more about your contributor, Jim. This look suspiciously like the email address used by “Nancy Naive.”

    • And, you would be wrong… on at least one count. SEE below and https://www.abbott.com/corpnewsroom/product-and-innovation/abbott-launches-covid-19-antibody-test.html

      Now to address your comment more directly. What you see above is Bayes Theorem (or Rule) applied to random testing. The calculations are accurate for the assumptions. and the conclusion is true for any random testing with the presented probabilities.

      This is why so many companies stopped random drug testing that resulted in immediate dismissal. Somebody sued the crap out of a company and won by presenting this to the jury. Nowadays, they test twice and thrice and discipline is replaced with therapy and warnings.

  4. I agree on the identification and credentials.

    I’d also like to see a couple of references to similar discussions among
    others with this level of schooling.

    Finally, if anyone thought that we were going to have a 100% effective testing regime – I don’t know what they are smoking.

    We’re just not going to that level of absoluteness.

    and let me give a real-world example. If you decide you’re going to have two of something so you have a backup in case one fails – the chances of failure are actually higher with 2 than 1 but the point is you do have a backup also.

    Testing will not be a perfect answer to risk.

    It may well take two different kinds of tests to get to a 99% level but even if you don’t – having NO testing is far worse.

    It’s a conundrum with a trap for those who expect perfection and get less.

  5. Are these “ad feminam” or “ad hominem” attacks? Just asking.

  6. Well, I would say this. This device searches for the virus in a specimen, not the antibody, so it would identify whether, or not, a person was sick. Immunity is a different question, the author has made an error, but wait until the end of my comment.

    If a person presented at the hospital with symptoms, this machine would be a godsend. In short order, it would with very high probability (0.97) determine if COV2 treatment were in order. Definitely a life saver, unless the person were one of the 3% of missed detection. If it falsely declares that the person has COV2 when he doesn’t, well, assuming the cure isn’t too bad, no harm done.

    That said, the calculations are true for Random Sampling of the population for “persons who are currently sick”. If you spy 3 Virginians, and stoppeth one of three, then test him, he is more likely to be a false alarm.

    Okay, the mistake on immunity. Tree days ago, Abbott Labs announced an antibody test for which they are seeking an EUA. Given the level of testing done for the virus EUA on this device, and the results (40 out of 40 true positives, and 30 out of 30 true negatives) with the presented confidence intervals, then unless they test MANY more times, e.g., 200 or 300 tests, the 95% confidence intervals shouldn’t be any different.

    In determining if a patient who had COV2 is “safe” to go home and out into public with immunity, the new test is also a godsend.

    But again, to use it in Random Sample testing… it would still “suck” and the results in the article are true.

    • So the Abbott test in question does accurately report who has the virus (a godsend you said?), which is good, and Abbott has a second test in process (not yet approved via EUA?) for presence of antibodies. Who ever claimed that widespread use of the initial test for the presence of virus was the gold standard for opening the economy in the absence of the other test, the one for antibodies? Clearly they work together. Why couldn’t you use the first to test a random population for the presence of virus?

      I still think “Dean Wortimer” and NN (a.k.a. [email protected]) 1) are one in the same and 2) that person is shadow boxing with a straw man.

      • Nice, out a guy why don’t you? But still way off base.

        The difference is what you do with the test. If the person is sick, the apriori probability is not 2%. It goes way up. So yes, at the hospital it is a godsend.

        But suppose the Gub’na says, “We can use the antibody test, and if it is positive, we give you a green card and you return to normal life.”

        That it cannot do, biecause of the mathematics presented.

        OTOH, it can be used collectively to test hypotheses, like, oh say, “The percentage of population already infected is more than 50%”

        Here, the aggregate positive and negative results, together with the confidence intervals can determine if the hypothesis is likely true or not.

  7. Sherlock has been saying what Fauci said yesterday, that there is a gap between actual available test capacity and actual use of the tests, where the rubber meets the road. There may be testing in theory but its use is not optimized for various reasons. Fauci said many factors are inhibiting full utilization right now.

    I get the impression the author is trying to defend the Northam administration by saying the tests are not ready for prime time. I am not sure that it true. But Virginia is not necessarily a terrible laggard either.

    The feeling I have is that the problem is probably going to shift from not enough tests, to needing new medical procedures and more infrastructure to administer.

    • This problem is a 50-state problem not just a Virginia problem and it points out that the landscape is littered with lots of questions that if left unanswered mean there is likely NOT a uniform standard approach to testing and it will vary by state and perhaps even within a state.

      Is that something that will work? It might if the professionals know the differences with the tests and can adapt accordingly.

      But that does mean that you need people who have the background and education necessary to make such assessments… it’s not for the average blogger.

      • My earlier, simpler math illustration was ignorance + no data = nonsense, and ignorance + no data + political bias = total nonsense. I now posit special knowledge + selective data + malice + anonymity = dangerous nonsense.

        • Having worked much of my life with scientists and mathematicians – as well as Statisticians.. I got well used to the realization that what they did , did take years of schooling as well as the kind of intellect that it takes to master those fields.

          Those folks do not think like the rest of us…. and the ones I worked with could indeed get a missile to go 10,000 miles and hit a target.

          • “My earlier, simpler math illustration was ignorance + no data = nonsense, and ignorance + no data + political bias = total nonsense. I now posit special knowledge + selective data + malice + anonymity = dangerous nonsense.”

            But in order to tell for sure, you cannot just say, “I know more than anybody,” or “I feel it”, you must really know it.

            Ignorance + faith = victim

  8. I can only impart knowledge; I cannot provide understanding.

    • The deep distrust fed by your anonymity abides. Reveal yourself if you would be taken seriously.

      • Steve – do you also feel the same way about virginiagal2 ?

        • Over time I’ve come to meet some of the regulars. Don’t know that one. Yes, in general, I give less weight to anonymous comments. It’s Jim’s party, not mine. Clearly “Dean” NN knows the material, but sensing strong bias against Sherlock or Abbott or the industry.

          • Strong bias against blind faith, the condition, not the band. Would you feel the same of Alice Adddertongue, or Richard Saunders?

            What’s in a name? Would a rose by any other still not wither and die?

            There is no strawman in mathematics, just error or not, and error will out.

  9. The test in the post discussed isn’t an immune test. It’s a test for the virus. The immunity tests proposed for general use IIRC are typically IgG tests for the type of immune globulin released after you get better, not the immune globulin type created when you first have an immune response. The test in the link for this post is a test for the virus itself, not for any type of immune globulins.

    Those are different things, used for different purposes. Testing for the virus tests for active infections, the other tests for immune response.

    Re the math. In cases where actual positives are rare, false positives are often more common than actual positives. A widely discussed example of this is breast cancer screening in young women.

    But. It’s nice outside and I want to work in the garden, so I’m not going to check the math, but the general principle is correct. I’ll post a link to a discussion of the underlying principle though.

    Scientific American https://www.scientificamerican.com/article/weighing-the-positives/

    • I hear an echo, but you said it better, and provided an excellent explainer for the math. “Dean” NN’s point is clearer. Thanks. Of course, there are financial reason$ the breast cancer screening$ continue, and sadly the testing on this will also be driven by who pays and how much.

    • I think the main take-away here is that the “test” is more than one and they actually test different things – and the implications of the tests are not the simple yes/no that many are wanting.

      It’s just not that simple.

      One level up – is the idea of who we test and why and do we do contact tracing?

      For all the hand-wringing of unemployment, what I’m hearing is that we need an “army” of people to do the contact tracing. If we’re going to shower
      money down on people, how about a job to go with it?

      Simple case – a guy shows up for work in a restaurant – they do a test – he tests positive and is sent to isolate. In the meantime – contact tracers find out all he has been in contact with and tracks them down and gets them tested.

      The uncertainty with the tests leaves us with the reality that it won’t be a airtight process but even capturing 90% of the infected is going to bear fruit and mitigate the scope and scale of the number of infected.

      At this point, we are so divided and so differing on our understanding and views that we’re just running in place.

      • The main reasons for testing are, one, to determine who is and isn’t currently infected, and two, to determine who is immune. Actually, add three, who has ever been infected.

        Contact tracing looks for people who might be contagious and allows you to have them self isolate to break the chain of new infections. That reduces the R0. If you can push the basic reproduction value below one, it will eventually die out. You are looking for active infections. Used for contact tracing and treatment of infected people.

        IgG is looking for people who might now be immune. We still need a little more data on what antibody levels indicate protection. You’re looking for people who can freely work without having to worry about getting infected. Useful for getting people back to work safely. Valuable information.

        Third case. You’re also looking for anyone who has been infected to determine the actual incidence, which is a multiple of known cases. Actual multiplier is going to vary significantly in different areas. Cases are the sum of people who have active infections plus the number of people without active infections but who have antibodies showing they have had the disease.

        That’s useful for true mortality rate and estimating herd immunity if the numbers are high. You do need pretty high numbers for herd immunity.

        Back to gardening.

  10. johnrandolphofroanoke

    I vote Dean Wortimer to draw a Bacon’s Rebellion math diagram for our XMAS T Shirt. I want a front sleeve pocket too!

    • On my BR T-shirt, I want a mug shot of Dr. Irwin Corey on Letterman, Dec. 6, 1983 per YouTube on Front. On Back I want Dean Wortimer’s math formula. I also want a full collection of Bacon’s Coved-19 Horror Graphics T-Shirts starting with:

      This full collection captures BR’s full range of talents, and it will fill BR coffers to pay its growing stable of literary, artistic, comic, and STEM talent, along with its renowned team of investigative journalists.

  11. Contact tracing?

    Facilitated no doubt by cellphones and OnStar?

    Hmm, they follow your phone and watch your car with plate-readers, trace your spending through checks and credit cards, monitor your internet sites with cookies…

    The only free people in America don’t have cars, cellphones or computers, and spend only cash. Oh, undocumented workers.

    • Contact tracing is as old as the hills. It doesn’t require advanced technology. It’s basically where were you and who else was there.

      Technology increases effectiveness at the expense of privacy. But basic contact tracing is tried and true and should not be demonized.

      Edited to add a pretty good link. https://www.npr.org/sections/health-shots/2020/03/10/814129534/how-the-painstaking-work-of-contact-tracing-can-slow-the-spread-of-an-outbreak

      And now I’m really heading back to gardening.

      • Totally agree. If we could agree that contact tracing must be done and that all of us should do what we can to participate in it and the technology, we could shut down the virus and get back to work much faster than what we are doing right now which is btching and crying and arguing and blaming.

        The only thing I’d like to see done is an easy way to get it off your phone once we are at the other end of this. I do not want a permanent way for the govt to follow me…

      • Don’t mess with Virginiagal2.

      • Indeed – read this:

        ” The Fairfax County, Virginia, health department is investigating a tuberculosis outbreak at an area school after three people tested positive for the potentially fatal bacteria, officials said Thursday.
        The health department has contacted 430 people who may have been exposed and need to be tested as a result of exposure to the three, the officials said.”

        this was in 2013… and this is what VDH typically and normally does with infectious diseases in general… they track down the contacts…and get them tested and I believe they must be tested..

  12. Agree. All of these ladies drive you hard right types bonkers! Ha ha! You, too, Gillispie.

    • Not me, I love intelligent debate. You provided little, Peter. I have also learned something (which Larry has also done for me at times.) But I am put off by “Nancy Naive,” with a gmail account set up just to create the identity screen, then using yet another pseudonym to provide a full post. Not quite sure why Jim let that happen. But as noted, I am now the former contributing editor….

      • I’m a little befuddled by Steve’s attitude. There are a dozen or more folks here that use pseudonyms.. from “mom” to “crazy” to “lift” to “V N” and we’ve never heard this complaint before.

        what gives?

        • Not posting the main article, and using a second pseudonym to mask your other pseudonym? Gotta admit that’s a new one. Hey, I read the post, studied the point, read the other article VA Gal posted, and learned something useful. Whoever it is, when the snark is put aside, he or she has something to offer. Come out of the shadow.

          • but Steve, I have no clue who people like CrazyJD are either. For that matter, almost none of the posters are known to each other so what’s the big deal?

            How would any of us actually be able to know the identities of the others anyhow?

            For the folks who post the articles, yes, I prefer that they not be anonymous but I do not see the “connection” you apparently see and beyond that – we’re back to the folks who post comments here – a whole bunch of them are essentially anonymous… we only know what each of them is willing to tell us about themselves.

            Snark – yes… and good at it but as long as it stays within the posted rules… it’s okay. It’s just another “style” , biting and sardonic, yes..

            this almost feels like you are going after a specific person whose main difference from others is not anonymity but style.

          • CrazyJD I’ve met! VaConsumer I’ve known for years. This blogging game is a nice way to waste time on a down day, but actually rots the brain….back to Netflix.

          • Well Heckfire – clue the rest of us in! Who is CrazyJD and VaConsumer?

          • You have at your disposal the means to dig past the pseudonyms, peel the onion, and expose… what’s in the middle of an onion? Maybe that’s a bad example. Parfait! A parfait has layers.

            You have an email.

    • This is a pretty good article that makes pretty clear that the whole idea of antibody testing is not the dead simple “proof” that many want and the if, ands, and buts are wholly a scientific condunrum… translation: “it aint simple and the answers themselves are complex and “depend”.

      This is not what many people want and actually are abandoning science
      and taking to the streets, i.e. “forget science, it does not know the economy and it’s taking us to economic ruin”.

      And surely, if we cannot agree on how to deal with a pandemic, we surely will not be able to agree about climate threats either. Not a good feeling.

      In short, there is a real question, as a society, that we can successfully deal with threats to our well being. Right now, we apparently cannot, and we are ending up on different sides – not only stymied on what to do but also with respect to HOW to go forward.

      • The equation is pretty different if you test recovered patients. Risk of false positives quite low.

        The argument is more that general testing for immunity, meaning among the general population, is going to have a significant amount of error. Targeted testing for immunity, among people known to have been infected, has a far lower risk.

        Even testing people who had symptoms but not a confirmed infection would be far less likely to produce a false positive.

        It’s really just an argument for thinking about the limitations and risks of testing, and to test the right people. Not an argument not to test.

        • re: ” It’s really just an argument for thinking about the limitations and risks of testing, and to test the right people. Not an argument not to test.”

          correct. NO approach or testing regime is going to be 100% correct. That does not mean no approach is useful.

          Even tests for other diseases have testing uncertainties…

          But the science identifies and quantifies what they are.

          Think about it this way. You use a tire gauge to determine the air in your tires. It’s not 100% accurate. It has errors and the errors vary by the type and brand of the Gage. You don’t not check the air in your tires because of that, in fact, you’d be a dang fool to not check
          your air – even if it’s going to be 2-3 lbs off!

          Same with disease testing. Same with hurricane track predictions.

          All these differing aspects are the realm of science… That’s what people who have the academic background and career experience know how to do.

  13. “Corporations are people too, my friend.”

  14. Bacon’s Rebellion editorial policy: I ask that all contributors and commenters identify themselves rather than hide behind pseudonyms. I have relaxed that policy over time, especially for comments. Given the tendency of people to be more polite and respectful of others when using their real names, I may start enforcing the no-pseudonym policy more aggressively.

    • I endorse that message!

    • Perhaps when the EUA Application for the antibody device, which Abbott has submitted on or about 4-15-2020, becomes available I may rewrite this specific to the use of that device.
      At that time, I would extend the paper to include using wide level testing, based on the CI in the application, not to determine if Joe Sixpack obtains a useful test, but how Virginia can profit from such polling to test hypotheses concerning total infections, as well as updating estimates.
      One thing Virginia must do, and conservatives ain’t gonna like it, is forced responses. Select the sample and demand participation.

      At that time, I’ll submit under my name with CV.

    • All I’d ask is that you have a transparent and consistent policy that applies to all..

  15. I think too many people and i include myself take themselves snd this blog too seriously. This is NOT a paying job but some pretend it is. I also find it odd that newbies are trying to force new “rules”’down our throats. I have participated here for 15 years and gave blogged professionally elsewhere. If you can’t have fun st BR, i don’t want to participate. Thus is aBLOG, not a legislative tipsheet. Some of the recent posts are so down in the weeds i don’t bother deciphering them.

  16. for extent – the accuracy is not critical if the errors are consistent – across the geography…

    but to use as a test for employment – not so good if high error rate.

    but this also points out the “decentralized” approach – different players with different tests with different quality – and no standard, no baseline no designation of what to use it for – or not.

    There has to be a standard – and that standard has to address whatever error rate there is – and then ideally a second method to validate the results and reduce the error rate.

    This kind of thing goes on all the time with lab tests… you’ll see results that also say “method used”… and that basically informs the confidence level.

    If we have to do this by state – then it means each states top lab has to set up some approach that all the subordinate labs will follow.

    Might be interesting at some points to actually hear from the State labs on how they want to proceed.

  17. The author has given me permission to describe him thusly: Dean Wortmier is the pen name of a retiree living in Hampton Roads. A former employee of government-contracting businesses, he spent 23 years as an adjunct professor in mathematics, physics and computer science, and published more than a dozen degrees in information theory.

  18. My congratulations to the esteemed Dean Wortmier. Welcome to Bacon’s Rebellion.

  19. johnrandolphofroanoke

    My favorite Dean Vernon Wormer line:

    “Who dropped a whole truckload of fizzies into the varsity swim meet? Who delivered the medical school cadavers to the alumni dinner? Every Halloween, the trees are filled with underwear. Every spring, the toilets explode. “

  20. Speaking of pseudonyms, who is SGillispie? Who is he/she and what line of work is he/she in and is he/she part of a political party in Virginia?

  21. I do not see S Gillespie in this string. Jim introduced him to me, as well, and he’s a business guy working hard to keep his not-small payroll intact, for which you should praise his name. If he wrote regularly, you would discern his political leanings, but he doesn’t contribute material.

    I do think Dean NN will be a welcome addition and will watch for the first signed post. Having told us what not to trust, advice on how the state ought to proceed might be a good topic. Because as Week Six of the Great Captivity begins, it is clear it won’t hold long….

    • He came here the same way as NN – via comment.

      If some of you actually know these folks using pseudonyms including Jim and others here do not – is that okay?

      If it is, why are you so bound up over NN?

      When I click on his/her profile it brings us musingfromjanus.

      Some of what is going on here on the nursing home issue walks and talks like a Republican narrative. Is Mr. Sherlock and Mr. Gillepsi involved with the Republican party?

      I think that’s a fair question for ANYONE who is posting here – if they are actually involved in a political party beyond just as a voter.

  22. Perhaps we need another Bacon Bits luncheon in Richmond, where us pseudononymouses (pseudononymice?) may meet and greet one another in the flesh, so to speak, or at least with a COVID-compliant elbow bump, and shamelessly cast off outer pretensions to reveal the true personality that lies beneath. After such mutual disclosure, who needs names?

    • I agree, Ackbar. This time just us.

      Cockroaches run from the light. Games, stripped of the dark, are far harder to play. Most particularly illusions, banality, secrets, and passive aggressive behaviors. This place needs fumigation.

  23. only if costumes are mandatory…

Leave a Reply