clock menu more-arrow no yes mobile
A health care worker takes a blood sample from a woman at Covid-19 antibody testing center in Brooklyn, New York, on April 25.
Michael Nagle/Xinhua via Getty Images

Filed under:

Why even a super-accurate Covid-19 test can fail

It’s hard to find a disease that’s rare.

Umair Irfan is a correspondent at Vox writing about climate change, Covid-19, and energy policy. Irfan is also a regular contributor to the radio program Science Friday. Prior to Vox, he was a reporter for ClimateWire at E&E News.

Update, May 27: The Centers for Disease Control and Prevention has posted new guidelines on antibody tests warning when Covid-19 prevalence is low (around 5 percent), less than half of those testing positive will truly have antibodies.

The persistent shortfall in diagnostic and antibody testing for the Covid-19 coronavirus in the United States remains a major hurdle to understanding — and controlling — the pandemic.

Public health officials know that some infected people can spread the virus without showing symptoms, sometimes for weeks. That makes testing to identify and isolate the infected the most important way to slow the spread of the virus. But the shortage of tests means officials don’t yet know how many are still out there, or how many have already recovered.

It also turns out that the tests are not always that great. Both the tests for Covid-19 that look for active infections and the serological antibody tests (that can identify past infections) have had issues with accuracy.

Scientists at the Cleveland Clinic found that one popular rapid genetic test told users they didn’t have the virus when they actually did about 15 percent of the time. Meanwhile, researchers in California studied 14 antibody tests on the market, some that were granted emergency use authorizations by the Food and Drug Administration, and found that only three of them delivered consistent results. These findings add another layer of fog on top of an already murky situation.

The US remains desperate for more tests, and companies like Quest Diagnostics are now offering more flexible options like at-home Covid-19 antibody testing to meet the demand. Some health officials have warned, though, that this pressure has led to a surge of inaccurate tests on the market.

Yet the bigger challenge isn’t the tests themselves, but the entire endeavor of calculating the prevalence of a brand new infectious disease like Covid-19.

Understanding the tricky math at play here could help us better make sense of test results and get a grip on the pandemic. So let’s dive in.

The tricky trade-off behind testing for Covid-19

The fundamental problem is that even the best tests struggle to find something that’s rare. That’s true if you’re looking for a novel virus, a form of cancer, or a genetic illness. While there have now been more than a million confirmed cases of Covid-19 in the United States, it’s still a tiny share of the overall population. So people who are currently infected or who have survived the virus are still scarce.

To understand why testing for an uncommon virus like this can be so challenging, it helps to understand the theory behind it.

Tests are evaluated on two key metrics: specificity and sensitivity. Sensitivity is the likelihood that a test will detect your target. A highly sensitive test will have a low false negative rate. It’s especially good at turning up a positive result when your target is actually there.

Specificity is the likelihood that the test won’t be fooled by something other than the intended target. A test with high specificity will have a low false positive rate. It excels at producing a negative result when your target isn’t there.

These traits are often in tension. Making a test more specific can make it less sensitive and vice versa — If you cast a wider net, you can’t be as picky about what you catch. Advancing on both fronts is possible in some cases, but it’s difficult, and it’s almost impossible to make a test that’s perfectly sensitive and specific.

The prevalence of the disease influences the chances of a correct test result

For patients getting tested, the main concern is how to interpret the outcome: If I test negative with an RT-PCR genetic test, what are the chances I actually have the virus? Or if I test positive with an antibody test, does it actually mean I have the antibodies?

It turns out that the answers to these questions don’t just hinge on the accuracy of the test. “Mathematically, the way that works out, that actually depends on how many people in your area have Covid,” Eleanor Murray, an assistant professor of epidemiology at the Boston University School of Public Health, said.

However, it’s tough to figure out the total number of people who have been infected with Covid-19. With untold numbers of people spreading the virus and with many not showing symptoms at all, it’s hard to say how many people currently have the virus or have antibodies to it. There are estimates that can be drawn from data like hospital visits and deaths.

But oddly, one of the best ways to understand the value of an individual test result is to test a large group of people at random. You can estimate the spread of a virus in a population with a serological survey (a.k.a. serosurvey), even with a less-than-perfect test. Then you can account for false results using statistical techniques in your analysis.

“The thing with the population-based surveys is even if they are not perfect in terms of their accuracy and operating characteristics, you can adjust for that,” said Natalie Dean, an assistant professor of biostatistics at the University of Florida. “If the tests perform worse, there is going to be more uncertainty in the final answer. ... The end result is that you can kind of say that something is low [prevalence], but you can’t precisely say it’s 1 percent or 2 percent or 3 percent.”

These probabilities — the accuracy of the test and the prevalence of the virus in a population — can be tied together in a statistical formula known as Bayes’ theorem to show how to interpret a Covid-19 test. In short, Bayes’ theorem calculates the odds of one outcome given another related outcome. For example, given a positive test result, what is the probability that it’s true?

Bayes’ Theorem
Bayes’ Theorem

Let’s say we have a hypothetical Covid-19 test for antibodies that is both 99 percent sensitive — meaning almost all people with antibodies will test positive — and 99 percent specific, meaning almost all people who were never infected will yield a negative result.

If you test a group of 100 uninfected people, odds are one of them will still test positive even though they don’t have the virus. Conversely, if you test 100 people who were infected, it’s likely one of them will still test negative.

Now let’s presume the virus has a prevalence rate of 1 percent, so one person in 100 carries antibodies to it. If you test 100 random people and get a positive result, what is the chance that this person was truly infected?

Deborah Birx, the White House Covid-19 response coordinator, explained the answer at a press conference on April 20: “So if you have 1 percent of your population infected and you have a test that’s only 99 percent specific, that means that when you find a positive, 50 percent of the time will be a real positive and 50 percent of the time it won’t be.”

It may seem a little counterintuitive that a 99 percent specific test could yield a 50-50 result. But Bayes’ theorem shows that at a lower prevalence rate of the virus, false positives make up a larger share of the results. You can see how it plays out in the graph below showing the positive predictive value of the test:

Plot showing the performance of a hypothetical antibody test for Covid-19
The odds of a correct result from a hypothetical 99 percent-specific test for Covid-19 changes with the prevalence of the virus in the tested population.
Christina Animashaun/Vox

At the same time, as the prevalence rate increases, so does the likelihood of a correct result. If 10 percent of people in a population are infected, a positive test result is more than 90 percent likely to mean that antibodies are truly present.

“[W]hether 2% or 20% of tests come back positive makes a huge difference in how confident we are in any individual positive result being correct,” said Zachary Binney, an epidemiologist at Emory University, in an email. “A bit weird-feeling, I know, but it’s the truth.”

How to test better

To use and interpret these tests effectively, first remember what the test is actually detecting. With a genetic test like an RT-PCR test, you are detecting the genetic material of a virus, not necessarily the active virus itself. As researchers in South Korea discovered, the virus’s genetic material can linger even after the infection is extinguished, yielding a false positive test result.

A serological test looks for antibodies, which play a crucial role in the body’s immune response to Covid-19. It can take several days for an infected person to develop antibodies to SARS-CoV-2, the virus that causes Covid-19. So a negative result doesn’t necessarily mean that the virus isn’t present. On the other hand, a positive antibody test doesn’t always mean you are immune, nor does it mean that the virus has been defeated.

With a genetic test, the main fear is that a false negative could leave someone to spread the virus unwittingly, so it makes sense to emphasize sensitivity. That way, it will pick up more actual cases, with the trade-off that some positive results might not be correct.

With a serological test, a false positive could give someone a false sense of security for having survived the virus and lead to riskier behavior. So these tests should be weighted toward specificity. That way, they’re less likely to tell people they have the antibodies when they don’t (with the trade-off that some people with antibodies will get a negative test result).

Another thing to consider is how you are planning to use the information from the tests. If you are trying to diagnose an infection in an individual patient with an RT-PCR test, it pays to also consider other factors like symptoms and how the patient was exposed. It may be worthwhile to perform a second test at a later date or using a different method.

Similarly, serological testing can also be improved by repeated testing. “If we give you a second [different] test, the chances of having two random positives in a row is much lower than the chance of having one random positive,” Murray said. “If you actually are not positive, then the second test will be negative. Doing multiple tests is a way of getting a more accurate result for any individual.”

The White House now recommends doing two antibody tests per patient to get a reliable estimate of the spread of the virus, ideally with tests from two different manufacturers to minimize the risk of the same problem repeating itself.

Dr. Anthony Fauci and Dr. Deborah Birx speak with President Trump and Louisiana Gov. John Bel Edwards in the Oval Office on April 29.
Mandel Ngan/AFP via Getty Images

It’s also important to keep track of how prevalent the virus is in your population; it can change a lot from place to place. One controversial estimate of the prevalence of Covid-19 in Santa Clara County, California, found that up to 4.1 percent of residents had been infected, although the finding was not drawn from a random sample.

“I think the Santa Clara numbers, for example, are almost — not quite — meaningless, but very close to meaningless,” Harvard epidemiologist Marc Lipsitch said on a call with reporters Thursday. “Because almost all of it could be false positives.”

New York state, however, where prevalence is clearly much higher, is a different story. Gov. Andrew Cuomo said on April 23 that early results of a statewide serosurvey found that 13.9 percent of people in the state have antibodies to Covid-19. In New York City, that number rose to 21.2 percent of residents.

“The numbers coming out from a place like New York are probably very reliable — not perfectly reliable, but the most reliable,” said Lipsitch.

Despite all the caveats, testing remains one of the most valuable tools for controlling the Covid-19 pandemic. And right now, the biggest hurdle isn’t the accuracy of the tests but the scale at which they are deployed.

Many researchers agree that the United States needs to be testing millions of people per day, yet the country is struggling to test more than 200,000 in a day. And since a test is just a snapshot in time, many people, particularly in high-risk jobs, will have to be tested over and over. While more accurate tests will always be welcome, no matter what, we’ll need more of them.

Today, Explained newsletter

Living in an abortion ban state is bad for mental health

Future Perfect

How the US is preparing to fight — and win — a war in space

Down to Earth

Scientists strapped cameras to a bunch of polar bears. The footage is breathtaking — and alarming.

View all stories in Science

Sign up for the newsletter Today, Explained

Understand the world with a daily explainer plus the most compelling stories of the day.