Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes.
Cubuk C., Garrett A., Choi S., King L., Loveday C., Torr B., Burghel GJ., Durkie M., Callaway A., Robinson R., Drummond J., Berry I., Wallace A., Eccles D., Tischkowitz M., Whiffin N., Ware JS., Hanson H., Turnbull C., CanVIG-Uk None.
PurposeWhere multiple in silico tools are concordant, the American College of Medical Genetics and Genomics/Association for Molecular Pathology (ACMG/AMP) framework affords supporting evidence toward pathogenicity or benignity, equivalent to a likelihood ratio of ~2. However, limited availability of "clinical truth sets" and prior use in tool training limits their utility for evaluation of tool performance.MethodsWe created a truth set of 9,436 missense variants classified as deleterious or tolerated in clinically validated high-throughput functional assays for BRCA1, BRCA2, MSH2, PTEN, and TP53 to evaluate predictive performance for 44 recommended/commonly used in silico tools.ResultsOver two-thirds of the tool-threshold combinations examined had specificity of <50%, thus substantially overcalling deleteriousness. REVEL scores of 0.8-1.0 had a Positive Likelihood Ratio (PLR) of 6.74 (5.24-8.82) compared to scores <0.7 and scores of 0-0.4 had a Negative Likelihood Ratio (NLR) of 34.3 (31.5-37.3) compared to scores of >0.7. For Meta-SNP, the equivalent PLR = 42.9 (14.4-406) and NLR = 19.4 (15.6-24.9).ConclusionAgainst these clinically validated "functional truth sets," there was wide variation in the predictive performance of commonly used in silico tools. Overall, REVEL and Meta-SNP had best balanced accuracy and might potentially be used at stronger evidence weighting than current ACMG/AMP prescription, in particular for predictions of benignity.