Case Study: Point-of-care serological assays for SARS-CoV-2 in a UK hospital population: potential for enhanced case finding
Want the original PDF of this study? Download here!
Severe Acute Respiratory Syndrome coronavirus 2 (SARS-CoV-2) has become a global pandemic. Case identification is currently made by real-time polymerase chain reaction (PCR) during the acute phase and largely restricted to healthcare laboratories. Serological assays are emerging but independent validation is urgently required to assess their utility.
We evaluated five different point-of-care (POC) SARS-CoV-2 antibody test kits against PCR, finding concordance across the assays (n=15). We subsequently tested 200 patients using the OrientGene COVID-19 IgG/IgM Rapid Test Cassette and find a sensitivity of 74% in the early infection period (day 5-9 post symptom onset), with 100% sensitivity not seen until day 13. Specificity was 96%, but in validating the serological tests uncovered potential false-negatives from PCR testing late-presenting cases. A positive predictive value (PPV) of 37% in the general population precludes any use for general screening. Where a case definition is applied however, the PPV is substantially improved (95·4%), supporting use of serology testing in carefully targeted populations. Larger studies in specific patient cohorts, including those with mild infection are urgently required to inform on the applicability of POC serological assays to help control the spread of SARS-CoV-2 and improve case finding of patients that may experience late complications.
On 31 December 2019 alarmover a pneumonia-like viral illness caused by an unknown pathogen was first raised to the World Health Organisation. A genome sequence for the presumed causative agent since designated Severe Acute Respiratory Syndrome coronavirus 2 (SARS-CoV–2) was publically released by early January 2020,1 enabling molecular diagnostics to be rapidly developed. SARS-CoV–2 has now extensively spread and been designated a global pandemic.
Clinical presentation ranges from mild upper respiratory tract infection to severe pneumonia with acute respiratory distress syndrome.2 The burden on healthcare systems has been devastating, particularly for intensive care services.3 As of April 2020 there have been over 1 million cases with a case fatality ratio varying from 1—6%.4 The true case fatality ratio is thought to be closer to 1–2% with a considerable number of unrecognised subclinical cases likely to be found in the community.
Without the availability of serological testing, this has proven very difficult to accurately assess.2 While real-time PCR is available during the acute infection it is limited by laboratory and skilled operator requirements,5 and can only be used early in the symptomatic clinical course. Serologic testing for SARS-CoV–2 offers the potential for enhanced case finding for patients presenting later in the disease course, as well as enabling serosurveillance at a much wider level than molecular diagnostics allow.
Multiple lateral flow serology tests have recently been developed for rapid, point-of-care (POC) use, providing qualitative results within 15 minutes. Whilst many of these devices have their initial development in China, they require verification in the cohort intended for use before extensive uptake. We take five of these POC kits and evaluate their suitability and potential utility for use in a western European population that had experienced symptoms in line with the Public Health England (PHE) case definition for testing6 against the current gold standard of RT-PCR. We go on to take one assay forward for further evaluation (OrientGene COVID–19 IgG/IgM Rapid Test Cassettes), completing 200 assays and commenting on test performance metrics in a UK inpatient population.
Validation of serology POC tests
Fifteen patients (mean age 61 years, range 42-84 years, 8/15 female) underwent testing with each of the five assays. All patients had RT-PCR testing conducted between 2 and 10 days from symptom onset, except patient 11 who was first tested at day 13 (Figure 1; chronological relationship between symptom onset, presentation to secondary care, RT-PCR testing and antibody testing). Of the 75 tests, all were included in the final analysis. No test kit failures were observed. The control line was visible on all 75 test kits run at 15 minutes post-sampling.
Validation PCR positive patients:
Ten patients with positive RT-PCR SARS-CoV-2 results were tested with each of the five assays (total = 50). The same seven patients were reactive across all five serology tests (35/50; Figure 1) and were in concordance for reactive tests from day 8 onwards. One further patient was reactive using the SARS-CoV-2 total antibody test (Wantai) (1/50), reactive for IgM only with the COVID-19 split IgG/IgM Rapid Test Cassette (OrientGene) and SARS-CoV-2 split IgM/IgG One Step Rapid Test Device (Encode) at day 6 (2/50) but negative for the other two assays (2/50). The remaining tests were unreactive, all of which were at an earlier stage post symptom onset of day 5 or 6 (10/60).
Validation PCR negative patients:
Five patients with a minimum of one negative RT-PCR result (range 1-4 negative PCRs) were tested with each of the five assays (total = 25). The same four patients were unreactive across all five serology tests (20/25). The fifth patient was reactive using all five assays (5/25).
OrientGene COVID-19 IgG/IgM Rapid Test Cassette analysis
200 patients (mean age 66 years, range 18-96 years, 45% female) underwent testing. All 200 assays were included in the final analysis. Two test kit failures were observed (failure of solution to ascend chromotography paper), of which both were repeated with subsequent valid reads.
OrientGene analysis PCR positive patients:
150 patients had positive RT-PCR SARS-CoV-2 results. A total of 130/150 OrientGene COVID-19 split IgG/IgM assays were reactive. Of those tested at 5-9 days post-symptom onset (n=50) 37/50 were reactive (74% sensitivity, 95% confidence interval [CI] 59.7 to 85.4%). At days 10-14 post-symptom onset (n=50), 43/50 were reactive (86% sensitivity, 95% CI 73.3 to 94.2%). At greater than 14 days post-symptom onset (n=50) there were 50/50 reactive assays (100% sensitivity, 95% CI 92.9 to 100%) (Table 1b, 1c). Tests were uniformly reactive from day 13 onwards (n=67) (Figure 2). Reactive assays were observed with a single reactive IgM line (6/130), single reactive IgG line (5/130), or both reactive IgM and IgG lines (119/130) (Table 1a). The longest period post-symptom onset tested was at 40 days and demonstrated persistence of both an IgM and IgG reactive line.
Table 1. OrientGene point-of-care SARS-CoV-2 serology testing (n=200) among inpatients,
March-April 2020, London.
1(a) Control, IgM, and IgG line detection across different symptom durations, where C = control line
only, G = IgM line observed, M = IgM line observed.
1(b) Sensitivity and specificity across all 200 assays.
1(c) Sensitivity and specificity across different symptom durations.
OrientGene analysis PCR negative patients:
50 patients that were presumed negative were included. Assays on 48/50 negative patients were unreactive (specificity 96%, 95% CI 86.3 to 99.5%). Two patients demonstrated a reactive IgM line only.
POC assay performance characteristics
Re-testing of patients with reactive assays with a second test had demonstrable reproducibility (n=3). The tests with too little blood, or too much blood failed, with no reactivity seen on test lines in the former and no reactivity in test or control lines in the latter.
Where an estimated prevalence of 2.7% in the general population is applied the positive predictive value (PPV) is calculated at 37%,7 negative predictive value (NPV) 99.6%. When calculated for suspected cases meeting the PHE criteria using a prevalence of 48% (86/177 positive in those tested across the week of 23 March-29 March 2020) in our inpatient population at the same time point there is a PPV of 95.4% and NPV of 88.4%.
Serologic testing for SARS-CoV-2 has the potential to augment control strategies by improving our understanding of the true burden of infection during the current pandemic. While the small number of tests performed (in comparison to the estimated UK prevalence of 2.7% as of 28th March 2020)7 during this early validation study makes it difficult to draw high-powered conclusions, the results offer useful insights into the potential utility of POC serological testing at this early stage in the pandemic response. Key concerns exist around the reliability of serology testing and identifying any potential time points for appropriate use.
Our study raises significant concern for the reliance on serology testing in the initial infection phase. Analysis of the larger 200 patient cohort shows that when the initial patient group (days 5-9 post symptom onset) are considered, there is a sensitivity of 74%, with the potential therefore to miss a significant number of positive cases during their most infectious period.8 Practically, if considering potential use of the assays in the early phase of infection (as suggested by some of the assay manufacturers, and during which period self-isolation is advised by PHE) the sensitivity drops to 66% (n=32). It is not until the end of the second week from symptom onset that the test appears to demonstrate a more practical use. In the greater than 14 day since symptom onset group there was 100% sensitivity. The latest false negative was demonstrated on day 12 with reactive results produced in all tests taken thereafter (n=67) suggesting that there could be real utility in POC serology from 14 days onwards. Where PCR testing is currently limited due to demand this could therefore have a potential role in testing those healthcare workers who have already self-isolated because of possible infection in the last few months. In addition the assay may have a role in community testing where timelines in patients with mild symptoms already self-isolating would be less time critical. A role is not seen however for an impact on public health guidance around selfisolation practices during the infectious period. A PPV of 37% for the general population precludes any consideration of deployment of serology testing for general screening. Where a case definition is applied however, as is required for meeting the PCR testing criteria, the PPV is substantially improved(95.4%) and supports use of serology testing in carefully targeted populations. While limited to detection in targeted populations this would still enable a far better understanding of the infection fatality ratio then we are currently able to draw from predominantly hospital cases of infection. At a time when there is considerable pressure on healthcare services it is important to be able to identify SARS-CoV-2 negative patients, streamline pathways and help guide clinical decision making for differential diagnoses. In the larger, second half of the study, serological testing of ‘negative’ patients (n=50) demonstrated a specificity of 96% suggesting an additional role in supporting suspected inpatients that return negative PCR results. In particular this could help guide management of patients presenting outside the reliable detection period for PCR swabbing of the upper respiratory tract.8 If the ‘negative’ patients across both halves of the study (total n=55) are considered, three had reactive results. Given our selection of these ‘negative’ patients from among those who had an a priori clinical suspicion of SARS-CoV-2 infection but who were PCR negative (n=2/3), it is possible or indeed likely that these represented missed cases of SARS-CoV-2. One of the patients presented in extremis with acute respiratory distress-like features (Figure 1, patient 11), one with a prolonged symptom history prior to sampling for PCR, and one following inpatient hospital transfer after a prolonged stay due to a stroke and new onset of low-grade fever. Where PCR is currently the only diagnostic test in use, the ‘missed’ cases tested with PCR raise the possibility of supporting diagnostic capability via serology testing in specific cases. In particular, this may be beneficial in patients presenting during the late phase where PCR is less sensitive or where there is a protracted or unclear history of onset and clinical suspicion is supported by other findings, such as typical radiological changes.
When considering use in outpatient settings, it is vital that a suitable time period is established for a true negative result. Our study suggests that this point is to be found at day 13 post-symptom onset at the earliest, limiting the utility of these assays in enabling return to work or in informing additional self-isolation practices. There is however a real role in helping to identify those healthcare workers, and in fact the wider general population, that have been through a self-isolation period with presumptive infection as long as there is concomitant reinforcement of advice around the stringent infection, prevention control measures.
If widely employed, demand on suppliers for POC serological assays will be high and interchangeability of kits may therefore be required. The plurality of kits currently available makes it difficult to comment on individual assay performance at this early stage and it is therefore essential to evaluate performance across the range of kits available both against the gold standard (PCR) asself well as against each other as soon as possible. Of the five different assays we evaluated, there was direct concordance for detection of SARS-CoV-2 infection across seven of the PCR positive patients (35/38 of reactive results). All five kits failed to detect SARS-CoV-2 infection from two known early positive samples taken at day 5 and day 6 post-symptom onset. Our findings suggested that results consistently appeared reactive from only day 8 post-onset of symptoms onwards across all five kits (extended to day 14 onwards in the larger second half of our evaluation). We cite ongoing concern for their reliability during that early period and the significant impact that missing a positive infection could have on public health control measures.
All POC serology kits are qualitative and while there was general correlation with colour intensity as time from symptom onset increased, variance in intensity of positive results was noted. The test kit instructions comment clearly on expectation for varied colour intensity of positive results and advise that even very weak colour should be considered to be reactive. When considering the practicalities of widespread testing, clear instruction on interpretation will have to feature in standard operating procedures and should allow for comment on interpreting the colour intensity of reactive samples. Antibody levels analysed via validated laboratory based enzyme-linked immunosorbent assay (ELISA) platforms have suggested higher levels following severe compared to mild infections and cohort studies will now be needed to evaluate the extent of reliability for POC serology testing in mild community infection.9
Little in the way of conclusion can be drawn from the five test kits used to explore concerns around end-user difficulties. Re-sampling of reactive patients provided re-producible results and further investigation should be conducted to comment on reproducibility of POC serology kits. Of significant use however is the observation that tests completed against manufacturer protocol by transferring either too little, or too much whole blood into the well resulted in failure. Those tasked with carrying out testing must complete the test as instructed using the marked pipette in order to have reliable results. Failure to register a control line in these cases should avoid the mistake of reporting an inappropriate false result.
When comparing the utility of the SARS-CoV-2 split IgM/IgG tests (Core tests®, OrientGene, VivaDiag, Encode) versus the SARS-CoV-2 total antibody test (Wantai), the former have a potential capacity to provide greater information. Of the total reactive tests in the 200 cohort study (n=130), 119 showed both IgM and IgG reactivity. There were 11 reactive samples that differentiated between IgM (6/130) and IgG (5/130) (Table 1a). IgM/IgG reactivity was seen as early as day 5 post-symptom onset and as late as day 40 but there was insignificant data to make conclusions on the IgM/IgG relationship in SARS-CoV-2 infection at this stage. At present we have very little understanding of the immunological response to SARS-CoV-2 and the timeline for which IgM is likely to remain detectable. If SARS-CoV-2 has ongoing transmission among humans, detection of IgM may then have a role in detecting sentinel cases during subsequent periods of infection. Staged evaluation of IgM and IgG relationship in known SARS-CoV-2 positive patients over a prolonged period will be required to inform on the potential additional benefits, if any, of splitting out the IgM and IgG reactivity in POC assays. Cross-reactivity leading to false positive results is of considerable concern, both with human seasonal coronaviruses and zoonotic beta-coronaviruses, including SARS and MERS. Okba et al. used serum samples from 3 SARS-CoV-2 positive patients and found cross-reactivity with the SARS-1 S and S1 proteins, and less so but also with MERS-CoV S protein.9 The S1 protein appears to be more specific to the SARS coronaviruses, with cross reactivity not seen against an array of PCR positive samples for example seasonal human coronavirus OC43 as well as non-CoV pathogens, including EBV and CMV. Subsequent testing with IgA and IgG specific ELISA showed some cross-reactivity with OC43 PCR positive samples and highlights the importance of assay design, where perhaps multiple protein targets may be required to ensure specificity for SARS-CoV-2.9 This urgently needs investigation in the western European population before widespread role out of POC SARS-CoV-2 serology. The role of serology as an adjunct to PCR has also recently been reported, with a strong correlation seen between IgM and IgG and the severity of disease.10 Seroconversion for detection of IgM and IgG was seen at median day 12 and 14 respectively with possibility of detection much earlier in the clinical course and is supportive therefore of our findings of improving sensitivity through the first 14 days of infection.10
If shown to be successful in the community setting POC serology testing could have significant implications for tackling the pandemic in resource poor settings which lack the required infrastructure for centralised molecular techniques. The benefits of utilising point of care testing to improve the efficiency of case identification and resource allocation in such circumstances has previously been discussed during the Ebola epidemic and could therefore be of considerable use during the SARS-CoV-2 pandemic.11
The small number of assays evaluated in this study limits the degree to which conclusions can be drawn. Formal power calculations have not been conducted, but with a sensitivity of 100% at greater than 14 days, accompanied by a margin of error of only 0.04, this exploratory study could now inform larger scale appropriately powered studies to fully characterise the utility of these assays in specific population groups. Specifically, a greater number of samples would be required in order to comment reliably on sensitivity and specificity and in turn, once the evolving prevalence becomes clearer, the positive and negative predictive value of these tests in various groups. A false negative result could have considerable consequence and the low numbers do not fully account for this. The need for rapid initial verification of these assays to augment clinical and public health interventions in this pandemic does not yet enable comment on many aspects of the assays, including the IgM/IgG relationship in the weeks and months post-infection. While our cohort groups include immunocompromised patients, pregnant individuals, and elderly patients, we are unable to sufficiently comment on reliability in these groups. This study is limited to evaluating reactivity in symptomatic adults with moderatesevere disease only and response in children has not been investigated. All positive patients in this study were considered to have moderate-severe infection by nature of the admitting policy of the hospital and evaluation in those with mild disease symptoms is needed to comment further on the population level applicability of these assays.
SARS-CoV-2 has resulted in a global pandemic and is associated with considerable morbidity and mortality. Serology testing is easy-to-use at the point-of-care and its utility to support, and replace where appropriate, PCR could allow for a significant improvement in the understanding of the true infection burden. This will be vital to the control of further SARS-CoV-2 spread.
Serology testing appears to be unreliable during the early infectious period, however there may be a limited role for use of testing in the appropriate context, such as in any patient that is at least 14 days post-symptom onset. Further evaluation on a much larger scale is now necessary in order to reliably comment on the earliest stage post symptom onset at which testing can be utilised, as well as on its performance in different population groups. In the meantime, any use of rapid serological test kits must be interpreted with caution, particularly if symptoms are of 14 days or less duration.
A prospective single-centre study was completed at a 430-bedded acute London hospital during March 2020. Consecutive inpatients were selected for testing based on real-time PCR detection of SARS-CoV-2 (AusDiagnostics®, Australia) and stratified based on duration of symptoms. Patients were included if they were admitted to hospital with symptoms matching the PHE case definition for testing,6 and had an accurate history recorded of at least 5 days of symptoms.
Exclusion criteria included all patients under 16 years old. Mild infection not requiring admittance to hospital were excluded. Patients that were unable to provide an accurate history or those with vague onset of symptom duration were also excluded in order to accurately report time-lines. 15 SARS-CoV-2 CE marked split IgM/IgG antibody tests (Core tests®) were available for paired testing alongside 15 SARS-CoV-2 CE marked total antibody rapid tests (Wantai), 15 COVID-19 CE marked split IgM/IgG Rapid Tests (VivaDiag), 15 COVID-19 CE marked split IgG/IgM Rapid Test Cassettes (OrientGene), and 15 SARS-CoV-2 CE marked split IgM/IgG One Step Rapid Test Devices (Encode). Ten PCR positive and five PCR negative patients were selected and paired serology testing was carried out by the same three doctors (SJCP, AP, SJD) experienced in the use of rapid diagnostic test kits. In order to evaluate reliability of test reactivity in positive patients, five positive patients were evaluated between days 5-9 of symptom duration and five patients between days 10-14. Additionally, five patients thought to be true negative ((i) symptoms meeting the case definition lasting at least 5 days
but no more than 10 days at presentation, (ii) at least one negative SARS-CoV-2 PCR result, and (iii) an alternative clinical diagnosis for their symptoms) underwent paired testing in order to assess for false positive reactivity.
Each patient was tested by the same doctors, as per each product protocol. Test kits were then read at the designated time period as per individual protocol and a result recorded for both the appearance of a control line and the presence or absence of antibody test reactivity. All results were verified independently by two of the doctors.
Using the same inclusion criteria a further 200 patients were evaluated using the COVID-19 IgG/IgM Rapid Test Cassettes (OrientGene). Patient history from both the Emergency Department assessment and the admission clerking were carefully evaluated and patients excluded if there was doubt over exact symptom onset, including those patients with vague history of onset, presenting with confusion, or following early intubation with an incomplete history. Patients were stratified based on consecutive PCR positive results to one of 3 time interval groups since symptom onset, encompassing 83% of SARS-CoV-2 positive cumulative inpatients at the time of the study period end-point. 50 PCR positive patients were stratified to the 5-9 post-symptom onset group, 50 patients to the 10-14 day postsymptom onset group and 50 patients at greater than 14 days since symptom onset. An additional 50 patients were selected with a presumed negative diagnosis that did not meet PHE testing criteria, or had a fever with an alternative, non-respiratory primary diagnosis.
Five additional test cassettes were used to evaluate possible anticipated end-user difficulties. Three test cassettes were employed to retest a patient with a positive result to comment on reproducibility. One test cassette was completed with too little blood, failing to fill the pipette to the indicated line and one test cassette was completed with too much blood, by over-filling the well.
LSPM acknowledges support from the National Institute of Health Research (NIHR) Imperial Biomedical Research Centre (BRC) and the National Institute for Health Research Health Protection Research Unit (HPRU) in Healthcare Associated Infection and Antimicrobial Resistance at Imperial College London in partnership with Public Health England. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the National Institute for Health Research, or the UK Department of Health. We thank North-west London Pathology services for performing the diagnostic PCR tests as part of routine clinical care. EC acknowledges funding from the National Institute of Health Research (NIHR) Health Protection Research Unit in Healthcare Associated Infection and Antimicrobial Resistance and the Economic and Social Science Research Council.
SJCP, GWD, and LSPM designed the study methodology. SJCP, SJD, AP, and EC collated the data. All authors (SCJP, SJD, AP, EC, NM, JS, GWD, LSPM) reviewed the results and data analysis and contributed comments. SJCP drafted the initial manuscript with all authors (SCJP, SJD, AP, EC, NM, JS, GWD, LSPM) contributing significantly to revising this for submission. All authors (SCJP, SJD, AP, EC, NM, JS, GWD, LSPM) agreed on the final version for submission to the journal.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Assays were received with philanthropic support; we are grateful to Future Planet, Michelle Chan, Mr Gordon Buchanan and Effective Giving for their support during this pandemic.
Ethical approval and consent to participate
This study was a verification of a CE marked in vitro diagnostic test and the experimental protocol was approved by the Point of Care Diagnostics Committee at Chelsea and Westminster NHS Foundation Trust who also deemed that separate written informed consent was not required. The study was conducted in accordance with the Helsinki declaration.
Potential conflicts of interests
LSPM has consulted for bioMerieux (2013), DNAelectronics (2015-18), Dairy Crest (2017–2018), Umovis Lab (2020), received speaker fees from Profile Pharma (2018) and Pfizer (2018-2020), received research grants from the National Institute for Health Research (2013-2020), CW+ Charity (2018-2019), and Leo Pharma (2016), and received educational support from Eumedica (2016–2018). NM has received speaker fees from Beyer (2016) and Pfizer (2019) and received educational support from Eumedica (2016) and Baxter (2017). JS, the Editor-in-Chief of Oncogene has sat on Scientific Advisory Boards for Celltrion, Singapore Biotech, Vor Biopharma, Vaccitech, Heat Biologics, Lilly, Greenmantle and Benevolent AI, has consulted with Lansdowne partners and Vitruvian. He sits on the Board of Directors for BB Biotech Healthcare Trust. SJCP has received a research grant from the Scientific Exploration Society. EC has received speaker fees from bioMerieux (2019). All other authors have no conflicts of interest to declare.
Availability of data and materials
The data analysed during the current study and further details on the assays are available from the corresponding author (LSPM; email@example.com) on reasonable request, as long as this meets local ethical and research governance criteria.
- Zhang Y-Z. Novel 2019 coronavirus genome. Virological. Available from: http://virological.org/t/novel-2019-coronavirus-genome/319. Accessed on 28 March 2020.
- Baud D, Qi X, Neilsen-Saines K, Musso D, Pomar L, Favre G. Real estimates of mortality following COVID-19 infection. Lancet Infect Dis. 2020. doi: https://doi.org/10.1016/S1473-3099(20)30195-X.
- Public Health England. COVID-19: investigation and initial clinical management of possible cases. Updated 27 March 2020. https://www.gov.uk/government/publications/wuhan-novel-coronavirus-initialinvestigation-of-possible-cases/investigation-and-initial-clinical-management-ofpossible-cases-of-wuhan-novel-coronavirus-wn-cov-infection. Accessed on 30 March 2020.
- Flaxman S, Mishra S, Gandy A et al. Estimating the number of infections and the impact of non-pharmaceutical interventions on COVID-19 in 11 European countries. Imperial College London (30-03-2020) doi: https://doi.org/10.25561/77731. Accessed on 08 Apr 2020.
- Wolfel R, Corman VM, Guggemos W et al. Virological assessment of hospitalized patients with COVID-2019. Nature. 2020. doi: 10.1038/s41586-020-2196-x. Epub ahead of print.
- Okba NMA, Müller MA, Li W et al. SARS-CoV-2 specific antibody responses in COVID-19 patient. 2020. doi: https://doi.org/10.1101/2020.03.18.20038059. Accessed on 29 March 2020.
- Zhao J, Yuan Q, Wang Het al. Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019. Clin Infect Dis. 2020. doi: 10.1093/cid/ciaa344. Epub ahead of print.