Tests are the key to unlocking lockdown but how accurate are they?

By Jerome Burne

Distributing tests as fast as possible to discover who is now infected with the virus, has become a test of organisational machismo for the government – 100,000 promised by the end of last month. Apparently they hit the target but only by sleight of hand with the figures – something this government seems not averse to. 

But what we really need to know is whether the tests are any good. Tests are an essential and valuable part of medicine. They come in a reassuring yes/no format – I’m afraid you do have HIV, no you are not pregnant. 

But there is also a long history of disputed tests such as the mammogram. Apparently, a valuable use of technology to spot breast cancer early, it may damage more patients with unnecessary treatment than the number of cancers prevented.  Unreliable tests that generate too many false positives (reporting you have it when you don’t) or false negatives (saying don’t when you do) can be worse than nothing. 

The tests to tell if you have had the virus, and so are probably both immune and non-infectious, will rely on detecting the antibodies in the blood, that the body makes in response to infection. But it turns out that the antibodies to Covid 19 work in ways that are still mysterious. 

For example, even when you have them, it’s not clear how long you could be shedding the virus or how long they protect you or crucially, whether you could be re-infected. 

Just the science. But what is the science

The race to produce the first antibody test for Covid 19  has been won by Roche. It has had preliminary approval in the EU and USA and UK approval is imminent. Reports are describing it as accurate but so far there has been little if any debate about the science behind such tests. 

The claim is always that they are based on the science, but officials have been very reluctant to let anyone see what that science is – something else this government has form for. What is known about the new Roche product certainly raises questions, outlined below.

So to find out what might be going on behind the scenes, I turned to David Crowe, who had provided details for a post I did a couple of months ago about the early response to the virus in China. Crowe is a Canadian software and telecommunications engineer with a degree in mathematics and biology who has become an independent expert in 21st century global infections such as SARS, Ebola and flu. 

He described the response at the time as a ‘rush to judgement, based on the rapid application of an unproven test, made worse by the use of powerful unproven drugs with toxic side-effects on those who test positive.’ [read post]

Drugs with a high side effect toll

One of the drugs that he was concerned about was the antiviral ribavirin, already being rolled out against Covid 19. He pointed out that during the academic post-mortem of the response to the SARS epidemic in 2003, a World Health Organization expert panel had concluded that ribavirin, widely used during the epidemic, ‘caused the destruction of red blood cells (hemolytic anaemia) in one-third to two-thirds of patients and that 75% of them developed liver problems’.

This post is not about the safety or effectiveness of the many drugs being put forward as treatments for Covid-19, but I can’t resist noting that use of some of the others also seem just thrown in the mix.. 

For instance, the warnings that come with the antibiotic Fluoroquinolone include this: ‘Can cause serious or disabling side effects that may not be reversible,’ Would you really try that when Vitamin C is an alternative? [Read more]

But on to the antibody tests. To find out how reliable they might be, Crowe has been examining all antibody test documentation submitted to the US FDA (Food and Drug Administration) plus a series of antibody surveys of groups of people from around the world. Experts can find a referenced account of the finer details here. 

Food intolerance test can spot viral infection

The conventional account of antibodies often used to explain how vaccines work, is that about a week after you are infected, or get vaccinated, a general infection-fighting antibody called IgM is produced along with one that is more directly targeted called IgG. When the infection is over IgM gradually disappears, leaving IgG to keep you immune. 

Interesting note: IgG may be familiar to some readers as the marker for various food intolerances, which the body has responded to as if they were an infection. York Labs, which does intolerance testing, is said to be working on a home test for IgG triggered by the virus. 

Crowe’s research has thrown up findings that suggest that the picture is much more complicated than generally assumed. Logically you can’t produce specific antibodies against an infection before you have had it. But that is what was found in a Dutch study last year. 

Researchers were testing saved blood from 2019 and earlier and found that almost 14% of samples tested positive for Covid19 antibodies. The implication is that the test they were checking wasn’t as specific as they had believed. 

In the idealised model it’s assumed that you know when the infection began – so you can say how fast antibodies develop, but in the real world that is rarely known for certain. 

Antibodies don’t behave predictably

‘When antibody surveys are performed,’ says Crowe ‘the vast majority of people who test positive have no idea that they had previously been infected and cannot possibly be sure about the date. Thus, the incubation period for the virus is impossible to determine accurately, as well as the range of days after infection that IgM and IgG start to develop.’

Evidence for this comes from a test developed by the Wadsworth Centre in New York. This found the 40% of people had no detectable antibodies 11-15 days after symptoms had started, and even more patients had none between 16-20 days. So, at precisely the time people should be generating antibodies, there were actually more people without them, suggesting that the generation of antibodies doesn’t necessarily happen in a predictable fashion.  At the moment you just have to be cautious about accepting results.

Antibody tests are usually reported in a very definite way – Yes/No.  But all tests are based on a numerical scale, often presented to the user of the test as a change in the intensity of a colour. If you are above an arbitrary line – paler – your antibody response is weak, and you don’t have it. Below the line – darker – you do. There are no tests that are actually binary, positive/negative.

But an error in one manufacturer’s lab found that if a sample was repeatedly diluted so there was half as much of it each time, this didn’t make each step paler, indicating fewer antibodies. In some cases, the more dilutions, the more intense the colour became.  Chinese researchers tried to find a link between more severe or longer-lasting symptoms and more intense colour but there was no correlation. 

Infection that travels back in time

Even more worrying, Crowe found research showing that a positive antibody result didn’t necessarily mean that the antibody was created as the result of a Covid-19 infection. ‘Manufacturers always check their tests on blood from people with unrelated medical conditions,’ says Crowe. ‘People who are infected with quite different viruses or bacteria

‘Yet they can show up as positive for Covid. Examples include 10% of those with hepatitis B and 17% of those with a streptococcus infection. If you are testing hundreds of thousands or millions of people, that becomes a huge number who people who think they are safe but maybe they aren’t.’

Several of these background issues with antibody tests, in general, show up with the new Roche product. Crowe has been looking at the data that Roche has made available so far. He points out that, rather than reporting that you have an IgG antibody specific to Covid, it doesn’t distinguish between IgG, IgM and IgA.(another antibody protecting against infections).

So a positive test – you’ve got antibodies – could mean you’ve only got IgM, which disappears very rapidly, rather than IgG which is normally longer lasting. 

Roche test. The numbers are tiny

Then there are the false positives. Roche claimed that 100% of samples from symptomatic patients were positive after 14 days. Sounds promising. But they only tested 29 samples from a group of 69 patients. And there’s no guarantee that there aren’t multiple tests from the same person in that group (so the number of people tested after 14 days could be less than 29). Is that really sufficient for a test intended for millions?

By this stage of Crowes report, there was such a gulf between what he had found and what we were being told, I had to ask some follow-up questions. How could we be embarking on a massive and presumably hugely expensive testing program with so many unknowns?

DC: I still don’t know. I think public health officials had been trained on what to do and went on autopilot, and almost all journalists and politicians just fell into line.

JB: So do the senior people involved in the UK testing program, know it is all highly unreliable but sort of hope it will be OK?

DC: They have absolute total faith in the miracles of modern medicine. My most surprising revelation was that if an antibody survey says 10% of people are positive, there is absolutely no way to double-check the number. It could equally be 0% or 100%.

JB:What organisations buy the test? Are there lots of them from different manufacturers? Do they have to be passed as fit for purpose?

DC: It seems like everybody buys what they want. Different NHS trusts probably use different test kits. If a university does a study, that’s probably different. I haven’t seen any attempt to cross-validate (i.e. buy three test kits and see if the % positive is similar or, even better, that everyone who’s positive on one, is positive on all three and vice-versa).

JB: What organisation in the UK is rolling out these tests? What would they say if asked about these findings, suggesting they are effectively useless? 

DC: It looks like the UK is starting with a massive survey, which probably will take a while to complete:

This is UK science at its most prestigious. It’s led by the Department for Health and Social Care (DHSC) and the Office for National Statistics (ONS), ‘the study draws on the world-leading scientific expertise of the University of Oxford, backed by the proven testing capabilities of data science company IQVIA UK and the National Biosample Centre in Milton Keynes.

‘The only way to resolve these issues,’ says Crowe ‘is with a massive observational study that monitors people who don’t test as being infected at the start of the trial and also don’t have antibodies, showing they hadn’t been infected in the past. 

The vital experiment. Intrusive, Inefficient and Expensive

‘They would then have to agree to be tested frequently, giving a drop of blood and having a nasal swab every few days. This way it would be possible to track when an infection developed and when antibodies (IgG and IgM) appeared. That way you could create an accurate model of how an infection progresses in the body, which would also tell you what proportion of people, who are infected, don’t develop antibodies, and how many do develop antibodies even though they haven’t tested positive for the virus.

This experiment would be time-consuming, intrusive, inefficient (as most people may never become infected) and expensive. But considering the vast sums of money spent on COVID-19 research, quarantining and treatment, and the even more tremendous sums of money lost by a hobbled economy, and the assertion of our politicians that they follow the science (not the head lemming), this would surely be worthwhile.

Jerome Burne

Jerome Burne

Jerome Burne is the editor of HealthInsightUK. He is an award-winning journalist who has been specialising in medicine and health for the last 10 years and now works mainly for the Daily Mail. His most recent book “The Hybrid Diet” was written with nutritionist Patrick Holford, published 2018. Award: 2015: Finalist for 'Blogger of the Year' Medical Journalists' Association.

5 Comments

  • As I understand it IgM ‘mops’ up immediately during and after an infection, but is not specific, and levels fall rapidly, so it’s diagnostically irrelevant. Specific IgG antibodies to SARS-Cov2 increase, and most leading labs have now decided testing after 14 days from onset of infection gives the most accuracy. Standards are also being set to eliminate between-lab inconsistency. The earlier LFIA tests are too poor to be relied on but now Roche and Abbott Labs have been approved by PHE, and Siemens and Diasorin are making headway, each claiming between 99 and 100% specificity (no false negatives) and 97.6+% sensitivity, eg some false positives. In my post today I’ve discussed the value of these tests. https://www.patrickholford.com/advice/testing-testing-are-you-positive. I’d rather know than not since natural immunity is a good reason not to get vaccinated which I fear will be thrust upon us before adequate safety checks are completed.

    • (A) The ‘understanding’ of IgM does not appear to be supported by the evidence. Specifically nobody followed people long enough to see the “levels fall rapidly”.
      (B) There are no time series to show that IgG levels do increase in anything approaching a predictable fashion.
      (C) “14 days from infection” is meaningless when almost nobody knows their day of infection. Almost all the validation is done based on days from symptoms, which is meaningless for different reasons.
      (D) ‘Approved by PHE’ is pretty meaningless when bureaucrats are desperate to get tests out there.
      (E) Manufacturers know that they can’t get tests without claims of 99% accuracy or so. So [shock] their documents always claim 99% accuracy (or so). Maybe Volkswagen should be responsible for testing the gas mileage of their own vehicles?
      (F) The levels of antibodies in many places are measured at 5% or less, which makes no sense when PCR tests, an instantaneous look at infection, routinely are found in 10%-25% of people tested. Antibody tests are the sum of everyone infected in the past. RNA tests are just today. And no, they aren’t good at determining who to test, and often the choice of who to test is not based on individual risk (e.g. working in a factory with a seemingly large number of cases). The only exception is Wuhan where I calculate recent testing is coming in at 0.02% positive. If you believe that I have a bridge in Beijing to sell you.

      • You seem to dismiss every part of everything to do with tests, be it for infection or for antibody positivity. I’d be interested to know, if you were in charge, what would you actually recommend doing? Nothing? I’d be interested to know.
        More specifically, re your (pointB) critique since it is positive sars-cov2 antibodies which is the crux of the matter, the Oxford study (and they are not selling tests) https://www.medrxiv.org/content/10.1101/2020.04.15.20066407v2.full.pdf shows an extremely clear increase in IgG levels. This is visually obvious in Figure 2 D and Figure 3 A, after 14 days of infection and, in the limit of this study, up to 60 days for anyone who wishes to see. Are you saying this data is false? Your (C) critique that no-one knows when they were infected so this is ‘irrelevant’ doesn’t make sense to me. Whether one is actually, unknowingly infected on Monday, symptomatic on Wednesday, free of infection on Friday, the message is get tested any time after 2 weeks from when you think your infection started, not in the week of infection, for the best chances of having higher sars-cov2 antibodies. What is wrong with that?

  • No mention of the difference between neutralising antibodies (NAbs) and binding antibodies which is relevant to any discussion on the relevance of the tests.

    • Good point. Even if all the problems with antibodies are resolved, the question is whether the antibodies are effective or not. Rather than neutralization tests, it would be nice to have actual evidence. It’s also possible that we are missing something that is important, and people are immune even when negative on these tests. Determining real immunity is time consuming, you’d have to follow a lot of people for a lengthy time.

Leave a Reply to Patrick Holford


WP-Backgrounds by InoPlugs Web Design and Juwelier Schönmann