Interobserver diagnostic variability at “moderate” agreement levels could significantly change the prognostic estimates of clinicopathologic studies: evaluation of the problem using evidence from patients with diffuse lung disease

Abstract 

Does interobserver diagnostic variability (IODV) influence the accuracy of prognostic estimates of clinicopathologic studies? “Best evidence” from usual interstitial pneumonia (UIP) and nonspecific interstitial pneumonia (NSIP) patients was used to investigate the effects of IODV. Systematic literature review identified studies of UIP and NSIP providing “best evidence.” Survival proportions from studies were compared using χ² and meta-analysis. Interobserver diagnostic variability was simulated in the data arbitrarily at 5% to 30% intervals. The various “diagnoses” were evaluated with κ, and χ² statistics were used to evaluate the interobserver agreement and compare survival proportions. The survival proportions for UIP and NSIP patients in 7 retrospective level III studies ranged from 11% to 58% and 39% to 100%, respectively. Analysis of simulation results with κ and χ² statistics showed that IODV greater than 10% resulted in significantly different survival proportion estimations. Interobserver diagnostic variability at moderate agreement levels significantly influences prognostic estimates. Evaluation and minimization of IODV in future clinicopathologic studies are indicated.

Keywords: Evidence based, Interobserver variability, Interstitial lung disease