Abraham Chan, Ilir Gashi, Sathish Gopalakrishnan, Karthik Pattabiraman and Kizito Salako. To appear in the Proceedings of the International Conference on Computer Safety, Reliability and Security (Safecomp), 2026. (Acceptance Rate: TBD). [ PDF | Talk ]
Abstract: In many safety-critical applications, binary classifiers detect undesirable operational states. Optimal adjudication schemes may provide an affordable means of constructing Bayes-optimal hybrid classifiers (i.e. a classifier configuration with the lowest expected cost of classification errors) by combining the outputs from multiple classifiers. However, statistical uncertainty poses significant challenges when applying such schemes, so that their use cannot be guaranteed to produce truly optimal configurations. We present statistical methods for estimating the extent to which various adjudication schemes, including optimal adjudication, fail to produce optimal classifier configurations. Applying these methods to two critical contexts—pneumonia diagnosis and concrete-defect detection—classical statistical bounds reveal how training-sample uncertainty, validation-sample uncertainty, and the relative cost of classification errors jointly constrain both the efficacy of adjudication schemes and the confidence that can be placed in the configurations they produce. So-called “smoothed” hybrid classifiers consistently give the lowest risk with the tightest optimality confidence bounds, particularly when the
relative cost of classification errors is very low/high.