Deep learning-based classification of erosion, synovitis and osteitis in hand MRI of patients with inflammatory arthritis

Patients

Patients contributing to the training/internal validation and external validation dataset were recruited at the outpatient clinic of the department of rheumatology and immunology of the University Hospital of Erlangen between January 2010 and July 2023 and were all part of the Erlangen imaging cohort, which has been described elsewhere.21 Briefly, patients with a diagnosis of RA fulfilling the 2010 American College of Rheumatology/European League Against Rheumatism classification criteria22 and patients with PsA fulfilling the classification criteria for PsA23 for whom at least one hand MRI was available were considered eligible for study participation. Patients with overlapping diagnoses of other immune-mediated inflammatory diseases were excluded from study participation.

For training/internal validation, hand MRI images from three different clinical studies were selected: the BAREBONE trial (Effect of Baricitinib Treatment on Peripheral Bone in RA; single-arm, prospective, phase IV trial to evaluate the effects of baricitinib on bone properties in active RA; NCT03701789), the EPos trial (Early PsA on treatment strategy; single-arm, prospective, phase IV trial to assess the effects of apremilast on bone changes in early PsA; EUDRACT (European Union Drug Regulating Authorities Clinical Trials): 2018-000335-27) and the ABEPSA trial (Abatacept Bone Effects in Psoriatic Arthritis; single-arm, prospective, phase IV trial to assess the effects of abatacept on bone changes in PsA; NCT04106804). In order to improve the algorithm by training it with a higher number of pathological findings and increase the variability of the training dataset, we also included a subset of hand MRIs performed at our hospital as routine clinical practice in patients with high disease activity (Disease Activity Score 28 (DAS28) score>5.1). For the external validation dataset, MRI images originated from the PARAJA cohort (Prospective cohort of PsA und RA patients treated with Janus Kinase inhibitors; a prospective clinical and imaging cohort of RA and PsA patients treated with JAK inhibitors; #19_18B) and from the clinical trial PSARTROS (Psoriasis-Arthritis & Bone Program; single-arm, prospective, phase IV trial to assess the effects of secukinumab on inflammatory and structural changes of the hands in PsA; NCT02483234).

The studies included scans at different timepoints for one patient with at least 3 months between scans. Additionally, each patient was only included in either one fold during cross-validation of the training/internal validation dataset or external validation. The association of each study to either dataset was according to their availability and release, as PARAJA and PSARTROS were available at a later date. This choice can be seen as random. Finally, demographic and clinical information were collected for all patients. A flow chart of the study design is available in online supplemental figure S3.

MRI methods

MRI scans of the hand were acquired on 1.5 T Magnetom Avanto and Aera systems (Siemens, Erlangen, Germany) for the training/internal validation and the external validation dataset. The MRI protocols were identical for both systems and consisted of five sequences: coronal T1-weighted (w) turbo spin echo (slice thickness 2 mm, repetition time (TR) 666 ms, echo time (TE) 12 ms, echo train length (ETL) 4, 384×252 matrix, flip angle 150, field of view (FOV) 158×240 mm), coronal T2w fat-suppressed (fs) (turbo inversion recovery magnitude (TIRM)—slice thickness 2 mm, TR 3500 ms, TE 60 ms, inversion time (TI) 160 ms, ETL 8, 448×358 matrix, flip angle 180, FOV 220×220 mm), axial T2 fs (slice thickness 3 mm, TR 4620 ms, TE 72 ms, ETL 12, 320×224 matrix, flip angle 180, FOV 150×150 mm), axial T1w fs contrast-enhanced (CE) (slice thickness 3 mm, TR 760 ms, TE 13 ms, ETL 3, 240×320 matrix, flip angle 150, FOV 150×150 mm) and coronal T1w fs images (slice thickness 2.5 mm, TR 571 ms, TE 11 ms, ETL 3, 512×384 matrix, flip angle 150, FOV 220×220 mm) after intravenous gadolinium injection (0.2 mL/kg) were used. Bone erosions were evaluated by using coronal T1w turbo spin echo and axial T1w CE sequences. The coronal and axial T1w fs CE images were assessed to evaluate tenosynovitis and periarticular inflammation along with axial images for confirmation of the findings on a perpendicular plane. T2w coronal and axial (fs TIRM) sequences were used to assess osteitis.

Image scoring, preparation and annotation

The region of interests (ROIs) included in the RAMRIS24 and in PsAMRIS25 scoring systems were used for our analysis. Detailed information on the ROIs can be found in online supplemental file 1. Each anatomical ROI was assessed manually by one expert reader (training dataset: either MYM or SB, rheumatologists with 4 years of MRI experience) blinded to all patient data except diagnosis. Agreement between the two readers was validated by a reader study for which a subset of MRIs were scored by both assessors (detailed information is reported in the dedicated paragraph; external validation dataset: MYM). Erosions, osteitis and synovitis were scored using the RAMRIS and PsAMRIS scoring systems according to OMERACT recommendations, which categorise pathologic changes on an ordinal scale for erosions (0–10), osteitis (0–3) and synovitis (0–3). As scores higher than three only occurred for erosions, we used an adapted score ranging from 0 to 3+, which included all scores≥3. Examples for each pathology and score are depicted in figure 1A.

To spatially define the ROIs, reference landmarks were manually added to each MRI image. A landmark corresponds to the 3-D centre of the anatomical ROI. Our approach used a custom landmark annotation tool. A graphical representation of this process is depicted in figure 1B. Initial experiments showed very high agreement between different annotators. For each MRI image, 23 and 24 landmarks for erosions and osteitis and 7 and 12 landmarks for synovitis were annotated for RA and PsA patients, respectively. Subsequently, 3-D ROIs were extracted automatically. For erosions and osteitis, a size of 30×30×30 mm was chosen for all ROIs except radius and ulna with 45×45×45 mm. For synovitis, all ROIs have a size of 45×45×45 mm except the radioulnar and radiocarpal ROIs with 30×30×30 mm.

Deep learning architecture for ROI scoring

For each MRI sequence type and each pathology, a separate ResNet-3D pretrained on a video classification task26 27 was trained for score prediction. In initial experiments, a Swin Transformer architecture28 with pretrained weights on a video classification task was compared with the ResNet-3D (online supplemental table S18). As only a small number of samples for our training was available, we did not observe a performance improvement when using the transformer architecture. As the transformer has higher computational costs, we decided to use the comparable ResNet-3D architecture. For automated prediction, only coronal MRI images (T1, T1 fs CE, T2 fs) were employed. The results of the sequence-specific networks were subsequently fused using a majority voting of individual sequence predictions. Additionally, we calculate a compound score for each patient by summing all lesion within one pathology to better estimate the overall correlation of predicted and actual disease activity. ROIs were individually normalised in the range (0, 1) and reshaped to the same size of 16×128×128 voxels. During training, random augmentations including brightness increase up to 20%, contrast increase up to 30%, rotation between −10° and +10°, flipping, 90° rotation and Gaussian noise were applied with a probability of 50% each. For all experiments, fivefold cross-validation was used, which results in a data split of 80% for training and 20% for testing. To prevent data leakage, each patient was only present in either of the sets. An additional external validation on an independent patient cohort was performed. All experiments were executed on either an A100, V100 or RTX3080 GPU (NVIDIA) using PyTorch V.1.13.1 and Python V.3.10.9. Each model was trained for 250 epochs with early stopping after 20 epochs of no improvement. The AdamW optimiser with default parameters was used with a learning rate of 0.0001. Weighted subsampling was applied for all tasks to handle class imbalance. The batch size was set to 10 for erosions and osteitis and 5 for synovitis. For the loss functions, cross-entropy loss was used for erosions and osteitis, while focal loss with a gamma of 0.7 was applied for synovitis. A slightly different training scheme including loss function and batch size was applied for synovitis, as the number of training data samples available in this case was smaller.

Impact of training data size on neural network performance

In order to gain insights into the influence of different training data sizes and into the variance within the study data, we investigated the performance of the neural network under different training set sizes in terms of area under the receiver operating characteristic curve (AUC) while keeping the test set unchanged.

Reader study

A reader study was performed with the aim of determining the inter-rater variability between expert readers. This indicates a natural upper bound for the automatic scoring. The reader study examined the agreement of experts with regard to classification performance of erosions, osteitis and synovitis based on the systematic evaluation of the same MRI scans. The human readers (MYM and SB) were blinded to all clinical and demographic information regarding the study patients. A subset of 30 RA patients from the BAREBONE trial was selected. To match the network scores, predictions higher than or equal to three were grouped within the class 3+.

Statistical methods

Patient characteristics were represented as summary statistics for continuous and categorical data. The selected metrics are commonly used in related work.15 18 The macro AUC measures how well a model distinguishes between positive and negative cases based on various threshold settings for each class separately, avoiding strong influence from the dominant class.29 30 The SD of the macro AUCs was calculated during a five-fold stratified cross-validation. We compare the accuracy of the classification performed by the algorithm against a reference macro AUC, which is the value expected in case of a random classification (ie, AUC≤0.5). We further computed the balanced accuracy, which calculates the average of recall and specificity for each class and compared the results against the value expected in case of a random classification of the four scores (0, 1, 2, 3+; ie, balanced accuracy≤0.25). Last, the weighted PR-AUC (Precision-Recall AUC) was calculated, which summarises the area under the precision-recall curve, that is, the trade-off between positive predictions and true positive rate depending on the class frequency. Summarising, macro AUC estimates how well the model separates classes, while balanced accuracy measures how well all classes can be predicted.

Regarding the reader study, Cohen’s kappa coefficients, balanced accuracy and macro AUC were computed to calculate the inter-rater agreement.

Patient characteristics

A total of 77 patients with RA and 35 patients with PsA were included in the study and provided MRI data for the training dataset. 63% of participants were women, the mean (SD) age of the training cohort was 54.1 (± 12.4) years, and the mean (SD) body mass index was 28.5 (±6.1) kg/m² (table 1). Disease activity was variable throughout the cohort with overall mean (IQR) TJC (tender joint count) of 2 (0–4) and SJC (swollen joint count) of 0 (0–2) and a mean (SD) DAS28 of 3.3 (1.3). Inflammatory activity on MRI was moderate and overall higher in patients with RA compared with PsA; the mean (SD) RAMRIS Score was 21.4 (3.5), while mean PsAMRIS (SD) was 7.2 (5.7). Detailed patients’ clinical characteristics and therapy as well as imaging features for each group are reported in table 1.

For the cross-validation dataset, MRI data of 211 scans were retrieved, corresponding to a total of 14 906 ROIs. Depending on the number of landmarks per lesion and the number of available MRIs, the total number of ROIs per lesion per MRI sequence ranged from a minimum of 1297 (T1 fs CE synovitis) to a maximum of 6388 (T1 osteitis) (table 2).

The validation cohort included another 75 patients who contributed with 220 scans. The number of ROIs ranged from 1452 (T1 fs CE synovitis) to 4416 (T1 erosion and osteitis) (online supplemental table S1). Compared with the training/internal validation cohort, PsA (n=62) was more prevalent than RA (n=13), but the age and sex distributions were similar. MRI signs of inflammation were less pronounced in the validation scans, with a mean (SD) RAMRIS Score of 14.7 (7.5) and a mean (SD) PsAMRIS Score of 4.8 (4.6). Detailed patient characteristics of the validation group are described in online supplemental table S2.

Automated scoring of erosions, osteitis and synovitis

The convolutional neural networks (CNNs) were trained separately for erosions, osteitis and synovitis and showed high performance as seen in table 3. To consider each class equally, independent of class imbalance, macro AUC is used as the main evaluation measure. Since the number of available ROIs is considerably different between MRI sequences, we present two evaluations: one considering all available MRI images, and a second considering only the cases where images are accessible for all MRI sequences (seq-equal) to avoid that the dataset itself is the reason for differences in performance between sequences. Further, we provide evaluations for voting ensembles as a combination of two or all used MRI sequences. Thereby, we combine the prediction probabilities, for example, for T1 and T2 fs, and average those to a combined probability vote.

The automatic scoring performed equally well for either using all available MRI images or seq-equal. Depending on the number of MRI images available, the best individual sequence and best voting ensemble changed. Across the five fold, the algorithm reached a mean macro AUC (mean and SD) of 92%±1% for erosion, 91%±2% for osteitis and 85%±2% for synovitis using majority voting with all available MRI sequences. Regarding erosion, the best individual MRI sequence is T1 with a macro AUC of 91%. For seq-equal, the sequences T1 and T1 fs CE both achieve a macro AUC of 90%. For osteitis, T1 fs CE achieves the highest macro AUC of 89%. Nonetheless, T2 fs provides a higher balanced accuracy of 57%. For seq-equal, the voting ensemble of T1 fs CE and T2 fs sequences achieve a macro AUC of 93% compared with 92% for majority voting. In the case of synovitis, T1 fs CE achieves the highest macro AUC for an individual MRI sequence of 84%. For seq-equal, a combination of T1 and T1 fs CE or all three sequences achieve a macro AUC of 87%. Detailed results on the CNN evaluation of erosions, osteitis and synovitis during cross-validation in the training/internal validation dataset are provided in online supplemental tables S3–5 and S6–8 for seq-equal.

The confusion matrices as shown in figure 2 provide detailed insight into correct and false predictions. For most cases, the main weight is distributed among the diagonal axis, which corresponds to correctly predicted scores. For synovitis, higher scores deviate from the diagonal indicating score underestimation.

Scoring of pathological changes in individual patients

To better access the general disease activity of a patient, the sums of the scores relative to each pathological change were combined for each hand. In figure 3, we present the correlation analysis between the CNN predictions of pathological changes and human scoring. We observed a high Spearman correlation (mean and SD) for the hand score of 90±2% for erosion, 78±8% for osteitis and 69±7% for synovitis indicating a strong agreement between the automated predictions and human scores on a patient level. For each pathological change, we performed a least-squares fit analysis, resulting in a slope of 1.05 and an intersection of 1.49 for erosion, 0.94 and 1.73 for osteitis and 0.71 and 3.37 for synovitis. These values provide insights into the relationship between the automated predictions and human scores for each specific pathological change. They indicate that the predictive capability for erosions is very high, for osteitis the predicted score is higher than the human scores and for synovitis lower scores are overestimated.

Performance of the algorithm in the independent validation cohort

During validation, we calculated the predictions using an ensemble of the respective networks trained for cross-validation. Using a completely disjunct dataset for validation, our trained networks achieved slightly lower results compared with cross-validation. Still, we could reach a macro AUC of 80% for erosions using an ensemble of all three sequences on the individual ROIs, as depicted in table 4. Here, the best individual sequence is T1 fs CE with a macro AUC of 79%. For osteitis the best performing sequence is T1 fs CE with a macro AUC of 86%. In the case of synovitis, again the majority voting achieves the best results with a macro AUC of 77% and the best individual sequence T1 with a macro AUC of 75%. Detailed results on the CNN evaluation of erosions, osteitis and synovitis in the validation dataset are provided in online supplemental tables S9–11. Confusion matrices of all combinations of MRI sequences can be found in online supplemental figure S1. Online supplemental figure S2 shows the summed score on the full hand of each pathology, which reaches a Spearman correlation of 65%, 73% and 34% for erosions, osteitis and synovitis.

Impact of training data size on performance

Decreasing the size of the patient cohort used for training the neural network significantly impacted the predictions for the test set. As shown in figure 4, we observed a positive correlation between the number of patients and the network’s predictive performance for each pathological change and MRI sequence. The random selection of the training data subset highlights the substantial influence of data variability within the training dataset. This impact becomes evident when assessing the scoring performance on a test set. In fact, when a random subset of images is selected for training, the obtained results might not reflect the data that is available during testing.

Reader study

The two readers achieved a Cohen’s kappa of 89% for erosion, 86% for osteitis and 84% for synovitis. The macro AUC between the readers is 97%, 94% and 90%, for erosion, osteitis and synovitis. The balanced accuracy for each task is 93%, 94% and 89%, respectively.