Validation of artificial intelligence spirometry diagnostic support software in primary care: a blinded diagnostic accuracy study

Objective and design The objective of the present study was to assess the discriminative accuracy of artificial intelligence (AI) software to identify COPD and other chronic respiratory diseases from primary care spirometry. This was a diagnostic study with blinded analysis. Methods Retrospective hand-held spirometry data from consecutive patients attending primary care clinics in Hillingdon (London, UK) between September 2015 and March 2019 were used. The index diagnosis was the “preferred” diagnosis determined by AI software (highest probability) using supervised random-forest machine learning to interpret raw spirometry data and basic demographics. The reference diagnosis was based on the consensus of expert pulmonologists with access to primary and secondary care medical notes and results of relevant investigations. Cross-tabulation of the index test results by the results of the reference standard for COPD and other respiratory disease categories provided the main outcome measures. Results In this primary care spirometry dataset from 1113 patients, 543 (48.8%) had a reference diagnosis of COPD. AI preferred diagnosis detected 456, achieving a sensitivity of 84.0% (95% CI 80.6–87.0%), specificity of 86.8% (83.8–89.5%), accuracy of 85.4% (83.2–87.5%) with area under curve (AUC) of 0.914 (0.896–0.930). AI preferred diagnosis identified 187 out of 249 patients with reference diagnosis of interstitial lung disease and 59 out of 107 patients with asthma, with AUCs of 0.900 (0.880–0.916) and 0.814 (0.790–0.836), respectively. Conclusion AI software achieved high sensitivity and specificity in identifying COPD using spirometry and basic demographic data and may support accurate diagnosis of COPD in primary care. AI software performed less well for other chronic respiratory disease categories.