Type I interferon pathway assays in studies of rheumatic and musculoskeletal diseases: a systematic literature review informing EULAR points to consider

Search strategy and eligibility criteria

A protocol for the SLR was developed and approved by the task force. Ovid Medline, Embase and Web of Science were searched for reports of IFN and RMDs up to October 2019. Search strategy is provided in online supplemental text 2–4. In addition to the RMD terms, papers were eligible for inclusion if they fulfilled the following criteria: (1) presented data on human patients with RMDs (with or without healthy controls); (2) design as cross-sectional; randomised control trials, case–control studies, non-controlled trials, diagnostic accuracy studies, cohort studies, intervention studies; (3) studies that described results on biological material derived from peripheral blood (ie, serum, whole blood, cell subsets); (4) written in English. Exclusion criteria were: (1) non-human studies; (2) conference abstracts, case studies, non-original articles such as editorials, review, opinion pieces, (3) articles that did not specify the type of IFN that the assay measures; (4) papers that did not describe their results as an assay, biomarker, test, score or similar in the abstract, (5) studies purely on IFN-I pathway genetics (see online supplemental text 5 for details). A minimum sample size was not considered within eligibility criteria as it could hid studies with more complex methods.

Titles and abstracts, followed by full-text screening was performed by two reviewers (AB and JR-C). The agreement between reviewers was high (>95%) and discrepancies were resolved by discussion or consultation with the convenor (EV).

Data extraction

An extraction template addressing all PICOs was developed. For the present report, the following fields were collected: method name, population studied, type of assay, material analysed, pathway element, detailed description of method and calculation of reported result, validity (including face validity, criterion validity, concurrent validity, discussed further below), reliability and feasibility. Association with clinical endpoints (ie, diagnostic accuracy) as well as assay responsiveness were analysed in a separate SLR (reference to SLR2). Due to the heterogeneity in methods and analyses reported, comparative statistical analysis or meta-analysis was not performed, and the results are presented in narrative form.

Interpretation of IFN assay validity

All assays included in SLR met face validity since the literature review filter implied a plausible role in the IFN pathway. For content validity, we described what aspects of the IFN pathway are measured in a description of principle of the assay. For criterion validity, no objective gold standard type I IFN pathway activation assay exists. We therefore described criterion validity as ‘evidence that assay measures IFN-I’ and four options were considered: (1) experimental stimulation of cells with IFN-I in vitro and demonstration of assay induction, (2) assays in patients receiving IFN-I therapy, (3) standard curves where appropriate, for example for an IFN-alpha ELISA but not IFN-inducible chemokines, (4) blocking with anti-IFN antibodies in vitro or as a therapeutic agent. For concurrent validity, we evaluated whether putative IFN-I assays were shown to correlate with other putative assay(s). For reliability, we sought evidence of reanalysis of samples in independent laboratories by the same method or repeat analysis of a sample at different times. Information on feasibility was provided based on review of the paper methodologies by the TaskForce panel as well as review of data in the paper.

Development of consensus terminology

After review of extracted data, terminology to describe aspects of type I IFN pathway, and assays designed to measure it, was developed by the task force. First, common themes with ambiguous concepts or that require harmonisation were identified. Next, definitions were produced by an iterative process until consensus was achieved.

Summary of RMDs and assays reported in the literature

Study selection is summarised in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses diagram (figure 1). A total of 10 037 abstracts were identified. A total of 276 of these reports fulfilled eligibility criteria and were used for data extraction. Some used more than one technique to measure IFN-I pathway activation. Hence, these 276 papers generated data on 412 methods.

Assays measured diverse aspects of the IFN-I pathway (figure 2). A summary of the assays used and their classification is in table 1. The most frequently studied RMD was SLE (n=204 reports), followed by RA (n=43), SS/pSS (n=41), SSc (n=32), myositis (n=29), antiphospholipid syndrome (APS) (n=6), multiple disease groups (n=44), other single RMDs such as AS, PsA, AAV, Behçet's disease, IgG4-RD (n=13). Sample sizes ranged from n=6 to n=1760.16 17 Types of assays also varied across RMDs (figure 3).

Description of assays reported in the literature

A detailed description of the principles of each assay is given in online supplemental text 6.

Immunoassays: IFN-I proteins (IFN-α and IFN-β)

Fifty-eight studies were identified for IFN-α and nine for IFN-β.

Content: Antibody-based methods to quantify IFN-I proteins in biosamples including ELISA, dissociation-enhanced lanthanide fluorescence immunoassay (DELFIA), multiplex assays and radioimmunoassays (RIA).

Criterion validity: These assays use specific antibodies, mostly commercial and validated by manufacturers, and most include controls and calibration curves. It is less clear whether a single sample and IFN subtype, such as serum IFN-alpha, is the most clinically relevant to a patient.

Concurrent validity: Concurrent validity was provided in 12/57 reports IFN-α and 4/9 for IFN-β against other assays such as expression of individual ISG,18 19 ISG expression scores,20 soluble IP-10 levels,21 flow cytometric measurement of SIGLEC1,22 bioassays,23 virus inhibition assays24 or reporter cell assays. Outcomes of these comparisons were highly variable. Some showed no clear association with IFN-α levels (ie, for IP-10, expression of individual genes),20 21 25 26 weak associations (IFN inducible gene expression, bioassay),20 21 or strong association (ie, between RIA and virus inhibition assays or SIGLEC-1 by flow cytometry or IFN-α and expression of individual ISGs).19 22

SiMoA (single molecule assay)

Eight studies were identified.

Content: A proprietary technology for ultrasensitive digital protein detection with a commercial assay for IFN-α (and some IFN-induced proteins).

Criterion validity: This was shown in six of eight studies. The study by Rodero et al showed that SiMoA could detect all subtypes of IFN-α, and excluded crossreactivity testing antibodies against recombinant IFN-β, IFNλ−1, IFNλ−2, IFN-ω and IFN-γ. This paper also described criterion validity showing that addition of anti-IFN-α antibody depleted the signal. Reproducibility was also confirmed. Matched plasma and serum samples tested for IFN-α showed strong correlation indicating a negligible influence of blood processing on IFN-α concentration and the ability to use either sample for retrospective patient screening. The potential implication of missing non-circulating sources was emphasised since intracellular IFN-I was detected in samples from patients with gain of function mutations in Stimulator of IFN Genes (STING) but not SLE despite high expression of ISGs in both. The IFN-α concentrations that are detected by SiMoA are often below the levels required to induce a cellular response in vitro.

Concurrent validity: This was reported in five of eight studies. SiMoA IFN-α measurements were highly correlated with a 6 ISG qPCR score as well as Nanostring and soluble Siglec-1.3 27 28 IFN-α protein by SiMoA correlated with cytopathic effect using Madin–Darby bovine kidney cells challenged with vesicular stomatitis virus.3 In some SLE cases, SiMoA detected IFN-α while a bioassay was negative, so the biological significance of the low IFN-α concentrations detected by SiMoA may require further confirmation.

Immunoassays: IFN-inducible proteins

Forty-two studies were identified.

Content: Immunoassays for soluble proteins encoded by ISGs in biosamples including ELISA, DELFIA, multiplex assays and radioimmunoassays.

Criterion validity: These assays do not evaluate IFN-I directly and production of some of the IFN-inducible proteins reported is known to be also induced by IFN-II or other inflammatory mediators. This is likely a particular problem if a single ‘IFN-inducible’ protein is reported. Of 42, 21 studies measuring chemokines cited papers showing they are IFN-I-induced. Only 3/42 studies provided experimental evidence. Bauer et al29 selected IFN chemokines whose transcripts were induced by IFN-α in a microarray study but did not check whether other IFNs or inflammatory mediators also induced these proteins. Thanarajasingam et al30 evaluated stimulation of whole blood including IFN-α, oligonucleotides with cytosine‐guanine repeats, Resiquimod (R848), LPS, IFN‐α+LPS and null (no stimulant) then measured cytokines/chemokines and IFN-α. Most of the other studies cite the evidence from Bauer et al.29 The IFN-inducible proteins included in assays are summarised in figure 4.

Concurrent validity: Of 42, 16 studies demonstrated concurrent validity against another test for IFN-I. 11/16 used an ISG expression assay.29–38 In two studies concurrent validity was tested between the IFN-inducible protein and two other techniques (flow cytometry for SIGLEC-122 39 as well as IFN-α protein21 26; and gene expression as well as a reporter cell assay).30 40

Flow cytometry

Twelve studies reported using flow cytometry to analyse IFN-I-inducible markers.

Content: Evaluation of IFN-inducible proteins, mostly on the cell surface, using flow cytometry. It allows cell-specific measurement that avoids artefacts due to changes in the cellular composition of the sample. The most common target was SIGLEC-1. Other markers reported were CD64 (FCGR1), MxA, IFITM1, PRKRA. Only monocytes can be analysed using SIGLEC-1 (CD169), so monocyte data represent the bulk of this literature.

Criterion validity: Of 12, 4 studies reported in vitro experiments showing stimulation of cells (PBMCs cells subsets or cell lines) with IFN-I and induction of the flow cytometry markers. Of 12, 9 studies cited evidence from the literature that the genes encoding the flow cytometry marker are IFN-inducible, or prior papers that report the marker as an IFN assay. These studies did not report whether other subtypes of IFN or other inflammatory mediators induced the flow cytometry markers.

Concurrent validity: Of 12, 5 studies demonstrated correlation with another IFN assay such as the same protein in serum by ELISA, the expression of the protein’s transcript by qPCR, against IFN-α protein measured by ELISA, or against IFN scores derived from qPCR assays.22 35 39 41–43 Some comparator assays compared ELISA or IFN Score results from unsorted blood with a specific cell population analysed by flow cytometry.

RNA microarrays

Microarrays were reported in 70 studies in total, with several differences in the methods section.

Content: A gene expression assay using probes that provide broad coverage of the transcriptome.

Of the 70 studies, 3 main methods of data analysis were used: (1) reporting an ‘IFN signature’, referring to the presence or absence of a cluster of ISGs, usually readily visible on a heat map; (2) an ‘IFN score’, usually referring to a continuous variable calculated from a predefined set of ISGs or (3) modules of ISGs, which allows the clustering of ISGs into two or more groups, and then a signature or score to be provided for each module or differences across modules are screened. The terms for these methods have not always been consistently used.

Of 70, 36 studies reported IFN signatures, 23/70 as IFN scores and 10/70 studies presented results as IFN modules. The most popular platform used was Affymetrix (39/70), followed by Illumina (15/70), Stanford microarrays (4/70), Agilent (3/70) or custom-made arrays. Microarrays were performed mainly using RNA from whole blood (41/70). Blood was collected using PAXgene (36/70) or Tempus tubes (5/70). Others used PBMCs or various combinations of isolated cell subsets.

Criterion validity: Of 70, 16 studies presented stimulation experiments showing induction of ISGs by IFNs-I or used public datasets to derive ISG sets. A few studies also reported ISG downregulation or IFN signature neutralisation after experiments or administration of IFN-targeted medications. The modular analysis provided data suggesting that modules may represent the relative abundance of type I versus type II IFNs. As for all gene expression assays described, a potential limitation when analysing unsorted cells is artefactual change due to changes of the cellular composition of the sample. Indeed, ISG expression differs between cell subsets and patients with RMDs often have cytopenias or expansion of cell subsets secondary to autoimmunity or immunosuppressive therapy.

Concurrent validity: Provided in 33/70 reports. Microarray results were validated against qPCR scores, individual ISGs expression, serum IFN-α, -β and IFN-inducible chemokines and a methylation array.

RNA-sequencing

Nine studies reported RNA-sequencing (RNASeq).

Content: Sequencing of the whole transcriptome providing qualitative and quantitative data on any RNA type.

Studies reported differentially expressed genes including ISGs in whole blood (3/9), isolated CD19+B cells (3/9), monocytes CD14+, pDCs, neutrophils (1/9 each subset). RNASeq was performed on Illumina.

Criterion validity: As for other gene expression methods. Stimulation with IFN-I was shown in one out of nine study, in another ISGs were derived from the Interferome database. ISG were not reported in subsets as for micro-arrays and qPCR studies.

Concurrent validity: provided in three ouf of nine reports, against chemokine score in serum,44 methylation profile of selected ISGs45 and qPCR.46

Nanostring

Five studies reported gene expression using Nanostring.

Content: A proprietary probe-based gene expression usually analysing 800 transcripts in manufacturer designed or customisable panels.

Nanostring was reported in in five out of nine studies to evaluate expression of ISGs in whole blood, PBMCs and CD19+B cells.

Criterion validity: as for other gene expression assays. Commercial predefined panels were provided by the manufacturer.

Concurrent validity: Provided in four out of nine studies where results were compared with serum protein IFN-α or SIGLEC1, IFN-I scores by qPCR or RNA-seq.28 39 46 47

IFN-I scores by qPCR

qPCR was reported in 122 studies with differences in the methods section.

Content: All qPCR studies report a set of real-time PCR techniques based on commercial or custom probes and primers for quantification of predefined ISG transcripts. Of the 122 studies, 5 main methods of data analysis were used: (1) 4/122 measured qPCR for genes encoding IFN proteins; (2) 30/122 reported expression of individual ISGs; (3) 82/122 studies reported an IFN Score—that is, expression of a set of ISGs as a continuous value; (4) 4/122 studies reported IFN-inducible chemokines as scores or individual genes; (5) 2/122 reported clusters of ISGs as a categorical signature.

Criterion validity: as for other gene expression assays. A total of 14/122 studies contained direct evidence based on experimental stimulation with appropriate positive and negative controls. The ISGs and reference genes chosen were highly variable between studies, and the rationale for these choices was not always given. A list of transcripts in scores used in RMDs is shown in figure 5. Sources and preanalytical features also differ across studies.

Some papers derived and used more than one score, related to the modular data from microarray studies or from unsupervised approaches (factor analysis or principal component analysis). Many studies reported correlations among ISGs, providing additional criterion validity. While many papers suggest these reflect balance of IFN-I and IFN-II, this was not demonstrated in any paper.46 48 48 49 49 50 50–54 However, when multiple scores were analysed, particular sets of ISGs, such as IFN-Score-B, show a stronger correlation with clinical endpoints than others in certain conditions. The metric properties of the scores, dynamic ranges and calculations to derive them may be other reasons that some sets of ISGs provide better clinical correlations, which also applies to other gene expression assays.

Concurrent validity: Provided for 43/122 studies, mainly by comparing new gene expression scores against previously published scores that use different ISGs. Several studies validate scores against levels of IFN-α protein measured by ELISA, SiMoA, etc); IFN-stimulated proteins (SIGLEC1, GAL9, G3BP); chemokines/chemokine scores; or expression of individual ISGs (MxA).

DNA Methylation arrays

Fourteen studies analysed an aspect of the IFN pathway using DNA methylation arrays.

Content: Analysis of methylation of genomic DNA to identify changes in actively transcribed genes

The material analysed included isolated cell subsets (most frequently CD4+T cells, but also CD8+T cells, CD19+B cells or neutrophils) as well as whole blood and PBMCs. The Illumina Human Methylation 450 BeadChip (HM450) was the most commonly method used for high-throughput human methylome analyses in 13/14 studies. Other studies used a newer Illumina assay based on the EPIC chip (Illumina Infinium MethylationEPIC BeadChip). Differential methylation was reported for ISGs as well as IFNA gene across all tested blood cell types and samples.

Criterion validity: These assays do not directly measure the transcription of IFNs or ISGs, nor their protein products so are only an indirect indication of activation of IFN-related pathways based on prior knowledge. The studies retrieved did not add any direct evidence to support their specificity.

Concurrent validity: Of 14, 10 reports presented concurrent validity. In three cases, results were validated against bisulphite sequencing, pyroseqencing and others looked at impact of hypomethylation on gene expression by qPCR (comparing to selected representative individual genes) or microarray results (IFN score) or RNA-seq. One study validated methylation results against IFN-α and -β protein levels in serum. Of 14, 2 studies validated their results against publicly available methylation datasets.

Reporter cell assays

Forty studies reported data from reporter cell assays.

Content: Quantitative analysis of the ability of IFNs in biosamples (ie, serum) to upregulate ISGs in a reporter cell line.

Various combinations of cells, biosamples and methods of readout have been published. Most studies analysed serum, with plasma in a smaller number. Samples were used to stimulate a reporter cell.

Two main groups of reports were identified. A first group of 31 studies used reporter cell assays with a qPCR readout. Most of these used a WISH cell line (human amnion) combined with patient serum (22 studies); 4 studies used HeLa cells; others used PBMCs from healthy controls, THP1 cells or HUVEC. The output readout was measured by qPCR using combination of 1–6 ISGs. The ISGs and reference genes chosen were highly variable between studies, and the rationale for these choices was not given. Results were usually presented as individual gene expression or IFN scores. The ISGs and reference genes chosen were highly variable between studies, and the rationale for these choices was not given. Results were usually presented as individual gene expression or IFN scores.

A second group of nine studies used measurement of output by luminescence or spectrophotometry. In this group, three studies used A549 cells (human lung carcinoma); two used novel HEK-Blue IFN-α/β and/or donor PBMCs; one used Hil3 (hepatoma cell line); one used U937 cells stably expressing an Mx1 promoter; one used U937-Mx1-Luc (containing luciferase reporter construct) and one used Fibroblast cells (FL). The results were presented in various formats such as: ‘Activity’ (titres, dilutions or arbitrary units) or concentration of IFN-α, etc.

Criterion validity: From both subcategories of reporter cell assays 14/29 and 2/9 gave experimental evidence on measuring IFN-I by using positive controls, inhibition with anti-IFN-I antibodies, stimulation with IFN-I or describing using a standard curve.

Concurrent validity: 8/29 and 5/9 studies, respectively, from these two groups presented concurrent validity. In most cases this was compared with IFN scores by qPCR measured in PBMCs or whole blood as well as against IFN-α protein levels in serum or plasma

Cytopathic effect assay

Eleven studies reported cytopathic effect assays.

Content: Measures the capability of IFN-I in a biosample to suppress the cytopathic effect of viral replication on a target cell in vitro. Various combinations of cells and viruses have been reported (discussed in online supplemental text 6). The most common combination is the human lung carcinoma cell line A549 challenged with encephalomyocarditis virus (EMCV).

Criterion validity: All studies used IFN-I standards and further three additionally used neutralisation of IFN with anti-IFN-α antibodies.

Concurrent validity: Three reports provided evidence of concurrent validity against SiMoA and RIA IFN-α protein measurement.

Plaque reduction assay

Content: Evaluation of the ability of IFN-I in a biosample to prevent cell killing by a lytic virus in vitro. This assay was used in n=7 RMD studies. Many different combinations of cells, patient sample types and methods of readout have been published. The cell line used most frequently was WISH in combination with serum and vesicular stomatitis virus (VSV) (n=5). Two other studies used fibroblastic cell lines (human foreskin fibroblasts (HFF) ormammary stromal fibroblastic (MSF) cell lines) with VSV. All these studies were published in 1970s and 1980s. The last study identified, from 2011, reported using VERO cell line in combination with serum and EMCV.

Criterion validity: All eight studies reported using IFN standards (recombinant IFN-α mainly, IFN-β and IFN-γ in some cases) as positive controls, as well including negative controls. A neutralisation assay with anti-IFN antibodies was also performed in three cases.

Concurrent validity: nNne of studies showed results of concurrent validity

IHC

A single study reported the results of whole blood stained for MxA as semi-quantitative intensity of staining. This assay shares similar considerations to other assays for single IFN-stimulated proteins.

Feasibility

Key considerations in feasibility provided by the task force members after reviewing manuscripts is provided in online supplemental table 1.

Definitions of terms

The terminology used to describe aspects of IFN-I pathway activation was not consistent in the literature eligible for this SLR. This includes the distinction between signatures and scores, abbreviations for IFN subtypes and even the use of the term ‘interferonopathy’ (used to indicate either monogenic diseases, but also to refer to any polygenic RMDs in which increased IFN pathway activation was observed, especially SLE). Having identified key areas with inconsistent nomenclature and reporting, consensus terminology was agreed through an interactive compromise process with reference to the existing evidence. Recommended terms relating to IFN-I reporting in basic and clinical research is presented in table 2.