Post-market validation of a further two serological assays for COVID-19

Updated Report

Post-market validation of a further two serological assays for COVID-19

_

13th October 2020 Report prepared for:

Office of Health Protection, Commonwealth Government of Australia The Therapeutics Goods Administration (TGA) of Australia

Report prepared by:

Dr Katherine Bond
Ms Suellen Nicholson
Ms Tuyet Hoang
Dr Mike Catton
Professor Benjamin Howden Professor Deborah Williamson

 

Executive Summary

Here, we present results of our post-market validation of a further two serological assays for the detection of SARS-CoV-2 antibodies. Testing was undertaken on a cohort of stored serum prior to the COVID-19 outbreak in Australia, and on samples of serum collected from patients with SARS-CoV-2 infection confirmed by molecular testing.

Our findings for the ‘Lungene’ COVID-19 IgG/IgM Rapid Test Cassette and the Innoscreen COVID-19 IgG/IgM Rapid Test Device were that both met, or were within the acceptable confidence interval for, the performance characteristics as reported by the manufacturer in the instructions for use (IFU).

1. Introduction

This work continues the post-market validation work previously reported on 28th April, 2nd June, 10th August and 24th September 2020. Following the Initial laboratory responses and release of the viral whole genome sequence by Chinese investigators in early January 2020, there was initially a rapid development of serological assays for COVID-19.1–3 The most publicised serological tests for COVID-19 have been lateral flow immunoassays, also known as serological point of care tests (PoCT). The urgent need for diagnostic testing has meant that many test kits have had an expedited assessment from the Australian Therapeutic Goods Administration (TGA). As such, robust post-market validation of COVID-19 diagnostic kits that are listed on the Australian Register of Therapeutic Goods (ARTG) is essential.

Here, we present findings from a post-market validation study of two further serological PoCT (both listed on the ARTG), to supplement the reports dated 28th April 2020, 2nd June 2020, 10th August 2020 and 24th September. This brings the total number of assays evaluated to fifteen PoCT and one ELISA.

2. Methods

2.1 Establishment of patient cohorts and serum samples

In order to test sensitivity and specificity of the included lateral flow assays, a testing panel was developed consisting of the following three patient cohorts:

Sensitivity analysis

1. Serum from patients with SARS-CoV-2 detected by RT-PCR from upper and / or lower respiratory tract specimens.

Specificity analysis

  1. Serum from patients with infections with the potential for cross-reactivity in serological assays, namely (i) patients with respiratory viral infections, including seasonal coronavirus infections and (ii) patients with other acute infections (e.g. dengue; CMV; EBV).
  2. Serum from a representative sample of the Victorian population collected in 2018 and 2019 (‘pre-pandemic controls’).

All serum samples were obtained from a tertiary hospital (Royal Melbourne Hospital, RMH) or the state reference laboratory for virology (Victorian Infectious Diseases Reference Laboratory, VIDRL). Serum samples were aliquoted into 100uL aliquots for processing and storage at time of entry into the study.

2.2 Test descriptions

Fifteen lateral flow serological assays in total have been assessed, two were described in detail in report date 28th April, three were described in a report on the 2nd June, three were described in report dated 10th August, five were described in report dated 24th September and an additional 2 assays are described here. Common features are that:

  1. they are single use immunochromatographic lateral flow tests, for the detection of IgM and/or IgG in serum, plasma or whole blood
  2. the specific SARS-CoV-2 recombinant antigen(s) incorporated into the assay are not described in the IFU
  3. IFUs indicate that test results should not be used as the sole basis for clinical management decisions, requiring interpretation alongside clinical features and other diagnostic (molecular) assays

Immunochromatographic assays involve detection of anti-SARS-CoV-2 IgM or IgG antibodies through binding to immobilised recombinant antigen attached to colloidal gold, followed by detection of the conjugates by an anti-human IgM or IgG antibody. A control line is also incorporated, which measures adequacy of fluid flow along the test strip. Reported manufacturer reported characteristics are summarised in Table 3 and include details for assays described in previous reports. In general, with respect to the generation of reported performance characteristics limited information was supplied regarding:

Cohort

Characteristics

Purpose of samples

Total (samples / patients)

1

SARS-CoV-2 RT PCR-positive patients

Sensitivity analysis

50/49

2

Other non-COVID-19 infections

Specificity analysis

30/30

3

Pre-pandemic controls

Specificity analysis

70/70

  1. where validation samples were sourced from
  2. whether plasma, serum, whole blood or a combination of these were used for

    validation

  3. what proportion of patients included were confirmed by a result from RT-PCR
  4. what the time frame was for collection of samples post the onset of clinical

    symptoms.

Table 2: Reported performance characteristics of included serological assays according to manufacturer’s instructions for use

Assay

Sensitivity

Specificity

‘Lungene’ COVID-19 IgG/IgM Rapid Test Cassette (Hangzhou Clongene Biotech Co Ltd)

IgM: 66.7% [57.6, 74.91]
IgG: 90.24% [83.58, 94.86] IgM or IgG: 91.06% [84.56, 94.45] (n = 123)

IgM or IgG: 96.48% [91.97 98.85] (n = 142)

Innoscreen COVID-19 IgG/IgM Rapid Test Device (Innovation Scientific Pty Ltd)

IgM*: 97.1% [90.06, 99.65] IgG*: 94.3% [86.01, 98.42] (n = 70)

IgM#: 99.5% [97.2 99.99] IgG#: 100% [98.14, 100] (n = 197)

*Samples collected more than 7 days from symptom onset; # Samples collected within 7 days of symptom onset

2.2.2 RT-PCR

Patients with confirmed COVID-19 infection had SARS-CoV-2 detected using the Coronavirus Typing assay (AusDiagnostics, Mascot, NSW). This is a two-step, hemi-nested multiplex tandem PCR, with seven coronavirus RNA targets plus a proprietary artificial sequence as an internal control. In addition, all positive samples had SARS-CoV-2 detected at VIDRL where testing was first conducted using an in-house assay for the SARS-CoV-2 RdRp gene. If positive, subsequent testing for the SARS-CoV-2 E gene was performed, using previously published primers.4

2.2.3 MICRONEUTRALISATION ASSAY

The microneutralisation assay is an in-house assay performed in the Subbarao laboratory, based in the Doherty Institute, University of Melbourne. SARS-CoV-2 virus, initially isolated from a clinical specimen from a patient in Melbourne, Australia,5 is propagated in Vero cells, before being incubated with dilutions of test sera. This solution is subsequently inoculated onto a monolayer of Vero cells. Cell cultures are reviewed at five days, with cytopathic effect scored and compared between test and control wells. The ability of test sera to inhibit viral invasion and replication is reported as a titre, calculated by the Reed and Muench method, with titres above 40 considered positive. The assay has been validated against an initial panel of serum from SARS-CoV-2 PCR confirmed patients and a representative serum cohort from 2016 with the assay cut-off of 40 determined by a receiver operating curve (ROC) analysis.

2.3 Testing protocol

Testing of the lateral flow assays was performed in the Clinical Trials Research Laboratory in the Department of Pathology RMH, by three laboratory research technicians, all of whom had undergone previous training in the use of lateral flow assays. Testing was performed exactly as per the IFU using the previously described serum panel (Table 1) for the ‘Lungene’ COVID- 19 IgG/IgM Rapid Test Cassette (Hangzhou Clongene Biotech Co Ltd, sponsored by Ausliance Group Pty Ltd Lot numbers 2020030165 & 2020030166, with testing of discordant samples using kits from Sponsor APAC security Pty Ltd lot S202005501) and the Innoscreen COVID-19 IgG/IgM Rapid Test Device (manufactured and sponsored by Innovation Scientific Pty Ltd, lot number SR202006002).

For all testing, lateral flow test strips were read in duplicate, a third read was undertaken if the first two were discordant, with the third read taken as the final result. If number of devices allowed, any samples which had discordant results between lot number were tested a third time, with the third test result taken as the final result. All testing was undertaken in a blinded manner with results collated by an independent investigator at the conclusion.

2.4 Statistical analysis

Statistical analysis was carried out using GraphPad Prism (version 8.4.2). Binomial 95% confidence intervals (CI) were calculated for all proportions.

2.5

Sensitivity of the serological assays was calculated as the number of positive results for each component of the test, divided by the number of samples from patients with confirmed COVID-19 as determined by RT-PCR.
Specificity was calculated as the number of negative results for each component of the test, divided by the number of samples from patients without confirmed COVID- 19 as determined by RT-PCR and clinical end point (Cohort 2 and 3).

Ethics

Ethical approval for this project was obtained from the RMH Human Research Ethics Committee (RMH HREC QA2020052). This ethics approval allows for prospective serum collection following discharge from hospital, thus enabling longitudinal assessment of the performance of serological assays. Patients recruited into this project also provided specimens to assess the performance of plasma samples.

3. Results

3.1 Comparison of serological PoCT with RT-PCR

Serum samples tested in this analysis included 50 samples for the sensitivity analysis, and 100 samples for the specificity analysis (Table 1). Sensitivity findings according to time of collection relative to sample onset are reported in Tables 3 and 4. Summary performance characteristics, with respect to overall sensitivity, sensitivity for samples collected more than 14 days from symptom onset and specificity, can be found in Table 5.

Table 3: Comparison of the ‘Lungene’ COVID-19 IgG/IgM Rapid Test Cassette with RT-PCR for 49 patients with confirmed COVID-19 infection, stratified by days post-symptom onset.

Days post- symptom onset

Samples (n)

IgM detected (%) [95% CI]

IgG detected (%) [95% CI]

IgM or IgG (%) [95% CI]

4-8

6

0 (0.0) [0.0, 45.9]

2 (33.3) [4.3, 77.7]

2 (33.3) [4.3, 77.7]

9-14

6

4 (66.7) [22.3, 95.7]

3 (50.0) [11.8, 88.2]

5 (83.3) [35.9, 99.6]

15-20

6

3 (50.0) [11.8, 88.2]

6 (100) [54.1, 100]

6 (100) [54.1, 100]

21-30

16

8 (50.0) [24.7, 75.4]

15 (93.8) [69.8, 99.8]

15 (93.8) [69.8, 99.8]

>30

16

3 (18.8) [4.1, 45.7]

16 (100) [79.4, 100]

16 (100) [79.4, 100]

Total

50

18 (36.0) [22.9, 50.8]

42 (84.0) [70.9, 92.8]

44 (88.0) [75.7, 95.5]

CI = Confidence interval (Clopper-Pearson)

Table 4: Comparison of the Innoscreen COVID-19 IgG/IgM Rapid Test Device with RT-PCR for 49 patients with confirmed COVID-19 infection, stratified by days post-symptom onset.

Days post- symptom onset

Samples (n)

IgM detected (%) [95% CI]

IgG detected (%) [95% CI]

IgM or IgG (%) [95% CI]

4-8

6

3 (50.0) [11.8, 88.2]

2 (33.3) [4.3, 77.7]

3 (50.0) [11.8, 88.2]

9-14

6

4 (66.7) [22.3, 95.7]

3 (50.0) [11.8, 88.2]

4 (66.7) [22.3, 95.7]

15-20

6

5 (83.3) [35.9, 99.6]

5 (83.3) [35.9, 99.6]

5 (83.3) [35.9, 99.6]

21-30

16

14 (87.5) [61.7, 98.5]

16 (100) [79.4, 100]

16 (100) [79.4, 100]

>30

16

11 (68.8) [41.3, 89.0]

14 (87.5) [61.7, 98.5]

14 (87.5) [61.7, 98.5]

Total

50

37 (74.0) [59.7, 85.4]

40 (80.0) [66.3, 90.0]

42 (84.0) [70.9, 92.8]

CI = Confidence interval (Clopper-Pearson)

When only samples collected more than 14 days following symptom onset were considered, the sensitivity of the ‘Lungene’ COVID-19 IgG/IgM Rapid Test Cassette was 97.4% (95%CI: 86.2-99.9%) and the Innoscreen COVID-19 IgG/IgM Rapid Test Device was 92.1% (95%CI: 78.6- 98.3%).

The specificity of the ‘Lungene’ COVID-19 IgG/IgM Rapid Test Cassette was 98.0% (95%CI:93.0-99.8%) and the Innoscreen COVID-19 IgG/IgM Rapid Test Device was 96.0% (95%:CI:90.1-98.9%).

Table 5: Comparative performance of serological assays with RT-PCR, for 150 samples from 149 patients

Performance Characteristic

Sensitivity, all samples (%) [95% CI]

Sensitivity, >14 days# (%) [95% CI]

Specificity (%) [95% CI]

Test Assay

Lungene IgM

36.0 [22.9, 50.8]

36.8 [21.8, 54.0]

99.0 [94.6, >99.9]

Lungene IgG

84.0 [70.9, 92.8]

97.4 [86.2, 99.9]

99.0 [94.6, >99.9]

Lungene IgM or IgG

88.0 [75.7, 95.5]

97.4 [86.2, 99.9]

98.0 [93.0, 99.8]

Innoscreen IgM

74.0 [59.7, 85.4]

78.9 [62.7, 90.5]

99.0 [94.6, >99.9]

Innoscreen IgG

80.0 [66.3, 90.0]

92.1 [78.6, 98.3]

97.0 [91.5, 99.4]

Innoscreen IgM or IgG

84.0 [70.9, 92.8]

92.1 [78.6, 98.3]

96.0 [90.1, 98.9]

# Samples collected more than 14 days from symptom onset

3. Discussion

Here, we present results of our post-market validation of the ‘Lungene’ COVID-19 IgG/IgM Rapid Test Cassette and the Innoscreen COVID-19 IgG/IgM Rapid Test Device. The ‘Lungene’ COVID-19 IgG/IgM Rapid Test Cassette met all the stated performance characteristics with respect to sensitivity for IgM, IgG, IgM or IgG and for specificity. The Innoscreen COVID-19 IgG/IgM Rapid Test Device met the stated performance characteristics for IgG sensitivity. The sensitivity for IgM and the specificity were just lower than that stated in the IFU, however with overlapping confidence intervals.

Both assays performed well for convalescent samples collected greater than 14 days from symptom onset, with an IgM or IgG sensitivity of 97.4% (95%CI: 86.2-99.9%) for the ‘Lungene’ COVID-19 IgG/IgM Rapid Test Cassette and a sensitivity of 92.1% (95%CI: 78.6, 98.3%) for the Innoscreen COVID-19 IgG/IgM Rapid Test Device.

In summary, our data describe the performance characteristics of two further PoCT devices, the tested performance characteristics for both assays correlated well with those stated in the IFU.

4. Acknowledgements

We thank staff of the Pathology department at RMH, the Serology department of the Victorian Infectious Diseases Reference Laboratory (VIDRL) and the Hospital in the Home medical and nursing staff at RMH. We also thank patients and their families who have contributed to this study.

5. References

  1. Nisreen M.A. Okba et al. Severe Acute Respiratory Syndrome Coronavirus 2−Specific Antibody Responses in Coronavirus Disease 2019 Patients. Emerging Infectious Disease journal 26, (2020).
  2. Petherick, A. Developing antibody tests for SARS-CoV-2. The Lancet 395, 1101–1102 (2020).
  3. Wu, F. et al. Neutralizing antibody responses to SARS-CoV-2 in a COVID-19 recovered patient cohort and their implications. Pre-Print MedRxiv 20 doi:10.1101/2020/03/30.20047365.
  4. Corman, V. M. et al. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT- PCR. Eurosurveillance 25, (2020).
  5. Caly, L. et al. Isolation and rapid sharing of the 2019 novel coronavirus ( SARS -CoV-2) from the first patient diagnosed with COVID -19 in Australia. Medical Journal of Australia 212, 459–462 (2020).