The Moderna Paper (Has More Prob's)

A potentially serious problem with the Moderna paper. Also: 95.4% N-positivity after Delta breakthrough infections.

I find reason to suspect a skew in the interval between “infection” and blood sample date in the Moderna N-antibody paper.

Meanwhile, a hiding-in-plain-sight Public Health England N-antibody paper that shows “bonkers” (science term) rates of N-antibody-positivity after breakthrough infection.

Seroconverters of the World

I don’t like to take an antagonistic approach to study interpretation, even when I think the study in question might be a bit, shall-we-say, psy-op-y.

If there’s something I think is “missing” - something I’d like to see, but that the authors don’t show - I try to assume that if I saw it, it wouldn’t totally upturn the study results; and further try not to jump the gun looking for reasons to prove that it would do so.

But in this case, after another two days’ of reflection and working with the data and the text, I think it is appropriate to rip the study in question “a new one.”

I think that if the interval between PCR-positive “infections” and sample date were shown in Follmann, et al., it would reveal that the placebo group had a lot of infections in February (due to seasonal infection patterns) and the injection group had a lot of infections in March (due to waning of infection efficacy). I think it would thus show that the “late-early breakthrough” infections were all sampled almost immediately after PCR+, before enough time had passed for the arrival of N-antibodies.

But do the authors plot these intervals, even though they surely had access to the data? No, that information is totally hidden from the reader’s view. The subject of the interval between PCR+ and blood sample is not even directly mentioned.

The analysis that leads to this suspicion has been inserted into a new footnote on the original post. It is also cross-posted below, and will be revised into a more reader-friendly format when I have time. For now, just know that there may be problems with the Moderna N-antibody paper and I have written a long, wonky analysis of why this is the case, which you can read if you feel like it.

But really, why bother with that, when…

N-ver Mind, N-ver Mind

It turns out there has been another paper online since October which shows almost universal N-positivity in a sample of Delta-era breakthrough infections, and 78% for Alpha breakthrough infections. (In fact, this paper may be the source of the UKHSA’s October “recent observations” comment to begin with! Funny.)

I have inserted a discussion of this paper in “The Hot Spot” since it fulfills the criteria for “the kind of N-antibody paper that would be useful” listed in that post.

The headline result: 62/64 Delta breakthrough infections are N-positive (including 1st-dosed only). Limiting to 2nd-dosed, 42/44 of Delta (almost same sample size as Follmann, et al.) and 63/71 of overall (larger sample size than Follmann, et al.) are N-positive. The Moderna antibodies paper is refuted (too early post-injection, too many likely false-positives, has the discrepancies I discuss below, and is not a true view of “breakthrough immune response”).

Whitaker, et al. Table 1. Further discussion in “The Hot Spot,” footnote 6

Donate-prompt before the dive into the weeds of the Moderna paper flaws:

If you anticipate deriving value from this post, please drop a few coins in your fact-barista’s tip jar. (It will go toward the heavily-strained Unglossed Orange Rectangle Budget.¹)

click button or visit https://ko-fi.com/unglossed

6 Days is (Not) a Long Time

Seriously, skip reading this unless you are somehow very interested in why I think this paper is critically flawed. Also, if not yet familiar with the paper and my previous discussion, see the former first:

The Moderna N-Antibodies Paper

Time of sampling can affect whether N antibodies will be present in donor blood (seroconversion). This can take several weeks post-infection. The authors make no comment on how this could affect the N-positivity rates for the Covid-vaccinated group, and do not provide a plot of PCR-to-sample(PDV)-intervals. This is a glaring oversight at best. At worst, it reflects an attempt to obscure an uneven skew in said intervals. In fact, there is abundant evidence suggesting such a skew.

Placebo N-seropositivity is too high in the -53-5 Day set.

Follmann, et al.² present their main analysis “stratified” according to the mean interval between illness-visit PCR-positive and individual patient unblinding:

Per the other analyses in Follmann, et al. N-positivity rates are also presented for blood samples taken on Day 57, after PCR-positivity or new N-positivity on Day 29, in Table 3. N-positivity rates are further shown for blood samples taken on Day 29 after PCR-positivity on Day 1, in Table 2. Samples are taken again at the unblinding (PDV), and N-positivity rates are higher.

This shows that the placebo group has not completed seroconversion 28-29 days after Day 1 and 29 PCR+. This is in line with normal immunological expectations.

For the bottom row, seroconversion rate increases ~10% for both groups after the Day 57 sample (i.e., by the time of PDV sample), demonstrating that seroconversion can take longer than 28 days post PCR+ in some cases.

Placebo seroconversion increases 6.5% after Day 29.

Returning to Table 3, for the 30 placebo-group “new PCR+ on Day 29” to be only ~60% N-seropositive on Day 57 is reasonable (as is for the vaccine group to be less so, given that the status of being “vaccinated” (having high anti-spike antibodies that temporarily reduce likelihood of infection) takes effect shortly after Dose 1, not Dose 2).

So if ~40% placebo group N-seronegativity is reasonable ~28 days after infection, why aren’t ~20% of the placebo group N-seronegative in the “-5-53 days PDV [serum sample day] PCR+” group?

Well, maybe because the sky-high seropositivity rates in Table 1 are some sort of “error.” But also, maybe because ~30 of that ~40% were false positives to begin with, given that Day 29 PCR testing was issued regardless of symptoms (as noted). However, an assumption of equal absolute counts of Day 29 false-positivity suggests that 30 * .3 is too high (since a maximum of 6 in the treatment group could be false-PCR+ on Day 29), so… that doesn’t help much.

But the final “maybe,” or more like a probably, is that:

The Days to PDV grouping in Table 1 is deceptive.

The authors imply that their decision to “stratify” results strictly by side-of-the-mean (half of infections before this relative-to-PDV-cutoff, half after) were to demonstrate similar seropositivity (apparently meaning, in the placebo group). So they are acknowledging that timing could be important, but taking the most narrow means of testing against that hazard possible. At the least it’s a very grade-your-own-homework-y approach; at worst it’s hiding a disparity in timing between the placebo and vaccine groups.

For the placebo group, the “5 to 53” days before the end of March would essentially describe “February and March, 2021.” These infections might skew more to February, just after the winter wave, accounting for the otherwise weirdly high N-seropositivity (so almost all in this group were infected long enough before PDV to generate antibodies; also note that 54-150 day infections might skew to January, explaining why the longer period does not have more infections, because it wasn’t really longer in “wave time”). For the vaccine group, infections during February and March would skew to March, as infection efficacy began to wane, or as the Canada-East Coast “mini spring 2021 wave” began to take off (which itself might have been driven by vaccine failure).

By not showing the actual plot of infection dates, the authors could be creating an intentionally deceptive comparison between a predominately two-month-post-PCR+ placebo set and predominately one-month-post-PCR+ vaccine set.

Another hint that this might be the case, and that something is “off” with the low N-antibody rate in the Covid vaccine group in this paper, comes from a little-noticed Public Health England study which may actually have been the source of the “recent observations” mentioned by the UKHSA (reviewed in “The Hot Spot,” footnote 6):

Whitaker, H. et al. “Nucleocapsid antibody positivity as a marker of past SARS-CoV-2 infection in population serosurveillance studies: impact of variant, vaccination, and choice of assay cut-off.” medrxiv.org.

In this, not only are samples after “possible late breakthrough, Delta” infections among the double-dosed almost universally (42 out of 44) N-seropositive, but samples after “probable early breakthrough, Alpha” infections are 78%. This is a lot more than the 32.1% rate for the -5-53-day group reported by the Moderna paper. Why the difference? Because Whitaker, et al. made sure to use sample-intervals that allowed time for seroconversion.

Whitaker, et al. Table 1. (annotated)

So it seems that a skew in sample timing (i.e., time from PCR-positive to individual participant unblinding) probably account for the aberrant results in some form (with possible high false positives in the -54-150 Covid vaccine infections group).

But “probably” doesn’t rule out the other “maybes.” Note that in the higher column of Table 3, Day 57 N-seropositivity rates are only 86.9% among the 61 placebo participants who were (newly) N-seropositive at Day 29. Since there is no PCR-false-positive hazard here, this effectively means that due to “random N-positivity loss,” a sample of 324 recently infected placebo participants shouldn’t exceed ~87% positive; however, I think that number is not accurate thanks to the smaller sample and possible bias for asymptomatic infections.

So, you be the judge. Personally, I’m totally sure there’s no reason the authors would be motivated to “err” on the side of an over-high N-positive placebo rate in Table 1.

My comment, at the preprint

I have posted a super-condensed summary of these complaints at the paper’s medxriv.org page. It is cross-posted here in the footnotes.³

If you derived value from this post, please drop a few coins in your fact-barista’s tip jar.

click button or visit https://ko-fi.com/unglossed

1. Do you think orange rectangles just grow on trees?

2. Follmann, D. et al. “Anti-nucleocapsid antibodies following SARS-CoV-2 infection in the blinded phase of the mRNA-1273 Covid-19 vaccine efficacy clinical trial.” medrxiv.org

3. http://disq.us/p/2ooains

The authors evince no apparent regard for the importance of the interval between PCR+ and serum sample (PDV), especially given the small number of presumed infections among the mRNA-1273-vaccinated in the main analysis, simply remarking

Anti-N seropositivity at the PDV was similar when stratified by median days from illness

Not for the vaccinated, it wasn't (50% vs 32% when stratified, Table 1). Did the authors merely lump both groups together to get around investigating why N-seropositivity was 18% lower in the -5-53 day vaccinated set?

In fact, the same stratification should have been expected to produce different N-positivity rates in the placebo group, had infections been evenly spaced in the -5-53 day interval. Since the authors find only 74% Day 29 N-positivity for placebo participants who are PCR+ on Day 1, and 60% on Day 57 for placebo who are PCR+ on Day 29, it's clear the placebo group isn't defying standard expectations about seroconversion not being instantaneous (bearing in mind a higher false positive rate on Day 1/29 due to screening, obviously) - until the main analysis, when suddenly there is no apparent penalty for near-PDV infections. So maybe there were almost no near-PDV infections in the placebo group (as in, infections skewed toward February due to seasonal patterns) while in the vaccine group the opposite was true (infections skewed to March due to the waning of infection efficacy)?

Thus, both the values for the placebo group and Covid-vaccine group suggest uneven time between PCR+ and PDV. The authors make no comment on this problem and *do not present* a plot of to-PDV-intervals for either group, even though they obviously had full access to that data. This is a glaring oversight at best.

It's not the only one. Have the authors never heard of false positives / base rate fallacy? I doubt it. So why isn't this taken into account, when comparing a group with a frequent outcome to a rare outcome? Among 14.5k participants in both arms, a mere 25 false PCR+ in both groups would be enough to render the main results way off the mark (Placebo: (100% x (1-((.066x648 - 25)/(648-25)) = 97.1% mRNA-1273: 100% x (1-((.593x52 - 25)/(52-25)) = 77.8%). Yet no consideration of the problem is made. The word "false" is not even in the text.