Reliability of paramagnetic rim lesion classification on quantitative susceptibility mapping (QSM) in people with multiple sclerosis: Single-site experience and systematic review

Background: Recent developments in iron-sensitive MRI techniques have enabled visualization of chronic active lesions as paramagnetic rim lesions (PRLs) in vivo. Although PRLs have potential as a diagnostic and prognostic tool for multiple sclerosis (MS), limited studies have reported the reliability of PRL assessment. Further evaluation of PRL reliability, through original investigations and review of PRL literature, are warranted. Methods: A single-center cohort study was conducted to evaluate the inter-rater reliability of PRL identification on quantitative susceptibiltiy mapping (QSM) in 10 people with MS, 5 people with clinically isolated syndrome, and 5 healthy controls. An additional systematic literature search was then conducted of published PRL reliability data, and these results were synthesized. Results: In the single-center study, both inter-rater and intra-rater reliability of per-subject PRL number were at an “Excellent” (intraclass correlation coefficient (ICC) of 0.901 for both) level with only 2-years lesion classification experience. Across the reported literature values, reliability of per-lesion rim presence was on average “Near perfect” (for intra-rater; Cohen's κ = 0.833) and “Substantial” (for inter-rater; Cohens κ = 0.687), whereas inter-rater reliability of per-subject PRL number was “Good” (ICC = 0.874). Only 4/22 studies reported complete information on rater experience, rater level of training, detailed PRL classification criteria, and reliability cohort size and disease subtypes. Conclusion: PRLs can be reliably detected both at per-lesion and per-subject level. We recommend that future PRL studies report detailed reliability results, including rater experience level, and use a standardized set of reliability metrics (Cohen's κ or ICC) for improved comparability between studies.