Robustness — alternative aggregations

benchmark version 1.1.0 · held_out results v1.0.0 · splits v1.0/held_out, v1.1/multi_grid
updated 2026-09-29 13:18 UTC · git 7243f310ddbc

The same committed numbers, never re-measured — every value below is re-aggregated from the per-seed vectors already in result.json, under defensible alternative schemes (equal-weight-by-grid mean, median, leave-one-grid-out, seeded percentile bootstrap). A headline that survives all of them is robust to the aggregation choice; a rank-sensitive one is flagged, not hidden.

13of 24 scored task headlines rank-stable under every scheme

event_detection_global · balanced_accuracy ↑ (test set) rank-sensitive: winners ['gru', 'resnet']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.774 ± 0.0090.7860.766 … 0.781[0.765, 0.783]4 grids · 20 seed-values
gru0.837 ± 0.0340.8810.812 … 0.859[0.804, 0.867]4 grids · 20 seed-values
mlp0.792 ± 0.0160.7900.775 … 0.806[0.777, 0.806]4 grids · 20 seed-values
random_forest0.832 ± 0.0280.8100.799 … 0.848[0.808, 0.858]4 grids · 20 seed-values
resnet0.864 ± 0.0160.8730.857 … 0.874[0.849, 0.878]4 grids · 20 seed-values

fault_detection_global · balanced_accuracy ↑ (test set) rank-stable: random_forest wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.853 ± 0.0160.8670.844 … 0.870[0.837, 0.866]4 grids · 20 seed-values
gru0.839 ± 0.0160.8570.832 … 0.856[0.823, 0.852]4 grids · 20 seed-values
mlp0.829 ± 0.0180.8490.820 … 0.851[0.811, 0.842]4 grids · 20 seed-values
random_forest0.879 ± 0.0190.8900.861 … 0.897[0.862, 0.895]4 grids · 20 seed-values
resnet0.849 ± 0.0160.8640.842 … 0.866[0.834, 0.862]4 grids · 20 seed-values

fault_detection_line · balanced_accuracy ↑ (test set) rank-sensitive: winners ['random_forest', 'resnet']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.864 ± 0.0170.8770.853 … 0.882[0.847, 0.878]4 grids · 20 seed-values
gru0.855 ± 0.0190.8710.844 … 0.876[0.837, 0.871]4 grids · 20 seed-values
mlp0.836 ± 0.0170.8540.826 … 0.856[0.820, 0.850]4 grids · 20 seed-values
random_forest0.884 ± 0.0170.8840.867 … 0.899[0.869, 0.898]4 grids · 20 seed-values
resnet0.878 ± 0.0260.9060.867 … 0.906[0.854, 0.900]4 grids · 20 seed-values

fault_detection_local · balanced_accuracy ↑ (test set) rank-stable: resnet wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.868 ± 0.0130.8750.858 … 0.882[0.855, 0.879]4 grids · 20 seed-values
gru0.853 ± 0.0180.8640.845 … 0.863[0.836, 0.869]4 grids · 20 seed-values
mlp0.833 ± 0.0170.8500.823 … 0.854[0.817, 0.847]4 grids · 20 seed-values
random_forest0.883 ± 0.0160.8820.867 … 0.898[0.869, 0.897]4 grids · 20 seed-values
resnet0.891 ± 0.0220.9020.872 … 0.911[0.871, 0.911]4 grids · 20 seed-values

fault_grounded_global · balanced_accuracy ↑ (test set) rank-sensitive: winners ['cnn', 'gru', 'resnet']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.963 ± 0.0050.9690.961 … 0.969[0.958, 0.967]4 grids · 20 seed-values
gru0.955 ± 0.0090.9700.950 … 0.961[0.946, 0.962]4 grids · 20 seed-values
mlp0.954 ± 0.0080.9690.949 … 0.963[0.946, 0.961]4 grids · 20 seed-values
random_forest0.951 ± 0.0050.9550.948 … 0.956[0.947, 0.955]4 grids · 20 seed-values
resnet0.965 ± 0.0040.9690.963 … 0.969[0.962, 0.968]4 grids · 20 seed-values

fault_grounded_line · balanced_accuracy ↑ (test set) rank-stable: resnet wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.967 ± 0.0020.9700.966 … 0.970[0.965, 0.969]4 grids · 20 seed-values
gru0.963 ± 0.0050.9690.961 … 0.969[0.958, 0.967]4 grids · 20 seed-values
mlp0.964 ± 0.0050.9700.961 … 0.969[0.959, 0.967]4 grids · 20 seed-values
random_forest0.944 ± 0.0030.9450.941 … 0.945[0.941, 0.946]4 grids · 20 seed-values
resnet0.968 ± 0.0020.9700.967 … 0.970[0.966, 0.970]4 grids · 20 seed-values

fault_grounded_local · balanced_accuracy ↑ (test set) rank-sensitive: winners ['cnn', 'resnet']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.966 ± 0.0030.9690.965 … 0.970[0.963, 0.969]4 grids · 20 seed-values
gru0.953 ± 0.0100.9670.948 … 0.958[0.943, 0.961]4 grids · 20 seed-values
mlp0.960 ± 0.0040.9680.957 … 0.964[0.957, 0.964]4 grids · 20 seed-values
random_forest0.931 ± 0.0040.9360.927 … 0.934[0.927, 0.935]4 grids · 20 seed-values
resnet0.968 ± 0.0010.9690.967 … 0.968[0.967, 0.969]4 grids · 20 seed-values

switch_detection_global · balanced_accuracy ↑ (test set) rank-stable: resnet wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.747 ± 0.0190.7810.735 … 0.764[0.730, 0.764]4 grids · 20 seed-values
gru0.733 ± 0.0270.7250.721 … 0.738[0.706, 0.757]4 grids · 20 seed-values
mlp0.714 ± 0.0120.7110.700 … 0.722[0.703, 0.725]4 grids · 20 seed-values
random_forest0.770 ± 0.0440.7540.719 … 0.799[0.731, 0.811]4 grids · 20 seed-values
resnet0.830 ± 0.0230.8650.810 … 0.846[0.809, 0.851]4 grids · 20 seed-values

event_state_global · balanced_accuracy ↑ (test set) rank-sensitive: winners ['random_forest', 'resnet']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.699 ± 0.0140.7070.688 … 0.712[0.687, 0.712]4 grids · 20 seed-values
gru0.702 ± 0.0250.7400.689 … 0.724[0.679, 0.725]4 grids · 20 seed-values
mlp0.713 ± 0.0220.7060.689 … 0.731[0.694, 0.733]4 grids · 20 seed-values
random_forest0.766 ± 0.0400.7240.717 … 0.786[0.732, 0.805]4 grids · 20 seed-values
resnet0.760 ± 0.0330.7770.734 … 0.782[0.729, 0.789]4 grids · 20 seed-values

fault_category_global · balanced_accuracy ↑ (test set) rank-sensitive: winners ['cnn', 'gru']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.882 ± 0.0630.9570.855 … 0.957[0.821, 0.929]4 grids · 20 seed-values
gru0.882 ± 0.0680.9630.854 … 0.963[0.817, 0.933]4 grids · 20 seed-values
mlp0.879 ± 0.0680.9620.851 … 0.960[0.814, 0.929]4 grids · 20 seed-values
random_forest0.806 ± 0.0680.8840.761 … 0.885[0.743, 0.863]4 grids · 20 seed-values
resnet0.876 ± 0.0670.9570.848 … 0.956[0.811, 0.925]4 grids · 20 seed-values

fault_category_line · balanced_accuracy ↑ (test set) rank-sensitive: winners ['cnn', 'gru']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.890 ± 0.0610.9640.865 … 0.963[0.831, 0.934]4 grids · 20 seed-values
gru0.888 ± 0.0640.9640.861 … 0.965[0.826, 0.935]4 grids · 20 seed-values
mlp0.883 ± 0.0650.9630.856 … 0.961[0.821, 0.931]4 grids · 20 seed-values
random_forest0.799 ± 0.0620.8270.749 … 0.867[0.741, 0.852]4 grids · 20 seed-values
resnet0.886 ± 0.0640.9620.860 … 0.963[0.825, 0.934]4 grids · 20 seed-values

fault_category_local · balanced_accuracy ↑ (test set) rank-sensitive: winners ['gru', 'resnet']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.880 ± 0.0650.9580.853 … 0.959[0.818, 0.929]4 grids · 20 seed-values
gru0.881 ± 0.0650.9600.855 … 0.960[0.818, 0.929]4 grids · 20 seed-values
mlp0.873 ± 0.0660.9510.844 … 0.952[0.810, 0.922]4 grids · 20 seed-values
random_forest0.782 ± 0.0600.8070.725 … 0.843[0.726, 0.833]4 grids · 20 seed-values
resnet0.881 ± 0.0640.9550.853 … 0.958[0.819, 0.929]4 grids · 20 seed-values

fault_classification_global · balanced_accuracy ↑ (test set) rank-stable: resnet wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.622 ± 0.0320.6700.605 … 0.660[0.590, 0.649]4 grids · 20 seed-values
gru0.622 ± 0.0310.6690.602 … 0.656[0.592, 0.648]4 grids · 20 seed-values
mlp0.630 ± 0.0300.6760.615 … 0.666[0.601, 0.655]4 grids · 20 seed-values
random_forest0.637 ± 0.0320.6790.621 … 0.674[0.607, 0.662]4 grids · 20 seed-values
resnet0.692 ± 0.0610.7730.650 … 0.749[0.637, 0.746]4 grids · 20 seed-values

fault_classification_line · balanced_accuracy ↑ (test set) rank-stable: resnet wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.685 ± 0.0380.7060.649 … 0.721[0.651, 0.717]4 grids · 20 seed-values
gru0.645 ± 0.0320.6810.628 … 0.683[0.615, 0.671]4 grids · 20 seed-values
mlp0.645 ± 0.0340.6790.625 … 0.686[0.612, 0.673]4 grids · 20 seed-values
random_forest0.613 ± 0.0320.6610.597 … 0.651[0.583, 0.639]4 grids · 20 seed-values
resnet0.754 ± 0.0410.8050.717 … 0.783[0.717, 0.790]4 grids · 20 seed-values

fault_classification_local · balanced_accuracy ↑ (test set) rank-stable: resnet wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.681 ± 0.0390.6850.643 … 0.719[0.646, 0.715]4 grids · 20 seed-values
gru0.636 ± 0.0330.6780.621 … 0.674[0.605, 0.661]4 grids · 20 seed-values
mlp0.634 ± 0.0300.6750.619 … 0.670[0.606, 0.659]4 grids · 20 seed-values
random_forest0.594 ± 0.0340.6280.570 … 0.633[0.562, 0.623]4 grids · 20 seed-values
resnet0.754 ± 0.0420.8250.721 … 0.787[0.716, 0.792]4 grids · 20 seed-values

fault_origin_global · balanced_accuracy ↑ (test set) rank-stable: cnn wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.856 ± 0.0140.8670.846 … 0.871[0.843, 0.868]4 grids · 20 seed-values
gru0.850 ± 0.0150.8630.841 … 0.865[0.835, 0.862]4 grids · 20 seed-values
mlp0.841 ± 0.0190.8650.831 … 0.863[0.822, 0.856]4 grids · 20 seed-values
random_forest0.682 ± 0.0590.7590.629 … 0.727[0.629, 0.736]4 grids · 20 seed-values
resnet0.846 ± 0.0200.8660.834 … 0.869[0.827, 0.863]4 grids · 20 seed-values

fault_phase_global · balanced_accuracy ↑ (test set) rank-sensitive: winners ['cnn', 'random_forest', 'resnet']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.853 ± 0.0080.8650.846 … 0.859[0.846, 0.861]4 grids · 20 seed-values
gru0.844 ± 0.0100.8570.836 … 0.850[0.834, 0.853]4 grids · 20 seed-values
mlp0.839 ± 0.0090.8490.835 … 0.849[0.830, 0.846]4 grids · 20 seed-values
random_forest0.847 ± 0.0230.8390.822 … 0.865[0.827, 0.867]4 grids · 20 seed-values
resnet0.847 ± 0.0170.8670.836 … 0.862[0.832, 0.862]4 grids · 20 seed-values

fault_phase_line · balanced_accuracy ↑ (test set) rank-stable: resnet wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.877 ± 0.0080.8860.871 … 0.886[0.869, 0.883]4 grids · 20 seed-values
gru0.856 ± 0.0150.8740.850 … 0.872[0.842, 0.869]4 grids · 20 seed-values
mlp0.841 ± 0.0100.8540.836 … 0.853[0.831, 0.849]4 grids · 20 seed-values
random_forest0.848 ± 0.0230.8390.822 … 0.865[0.827, 0.868]4 grids · 20 seed-values
resnet0.891 ± 0.0130.8950.882 … 0.901[0.878, 0.901]4 grids · 20 seed-values

fault_phase_local · balanced_accuracy ↑ (test set) rank-stable: resnet wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.879 ± 0.0080.8830.873 … 0.887[0.871, 0.886]4 grids · 20 seed-values
gru0.841 ± 0.0120.8510.833 … 0.851[0.830, 0.851]4 grids · 20 seed-values
mlp0.840 ± 0.0090.8500.834 … 0.850[0.832, 0.847]4 grids · 20 seed-values
random_forest0.834 ± 0.0240.8390.807 … 0.851[0.812, 0.855]4 grids · 20 seed-values
resnet0.886 ± 0.0150.8980.879 … 0.900[0.871, 0.898]4 grids · 20 seed-values

switch_type_global · balanced_accuracy ↑ (test set) rank-sensitive: winners ['cnn', 'resnet']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.712 ± 0.0090.7260.706 … 0.722[0.704, 0.720]4 grids · 20 seed-values
gru0.713 ± 0.0160.7390.701 … 0.727[0.698, 0.727]4 grids · 20 seed-values
mlp0.677 ± 0.0240.7050.664 … 0.706[0.655, 0.697]4 grids · 20 seed-values
random_forest0.716 ± 0.0480.7290.664 … 0.753[0.673, 0.759]4 grids · 20 seed-values
resnet0.736 ± 0.0430.7910.703 … 0.772[0.696, 0.775]4 grids · 20 seed-values

event_classification_global · balanced_accuracy ↑ (test set) rank-stable: resnet wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn0.624 ± 0.0140.6320.615 … 0.639[0.611, 0.638]4 grids · 20 seed-values
gru0.645 ± 0.0180.6750.635 … 0.663[0.628, 0.661]4 grids · 20 seed-values
mlp0.557 ± 0.0170.5560.538 … 0.569[0.542, 0.573]4 grids · 20 seed-values
random_forest0.577 ± 0.0550.5680.515 … 0.618[0.529, 0.626]4 grids · 20 seed-values
resnet0.699 ± 0.0390.7610.671 … 0.727[0.662, 0.732]4 grids · 20 seed-values

fault_location_global · mae ↓ (test set) rank-stable: mlp wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn20.038 ± 0.92021.44719.270 … 20.794[19.251, 20.874]4 grids · 20 seed-values
gru19.770 ± 1.54322.64818.626 … 21.070[18.352, 21.209]4 grids · 20 seed-values
mlp19.056 ± 1.18120.69718.010 … 19.962[18.029, 20.129]4 grids · 20 seed-values
random_forest22.599 ± 0.93524.13121.912 … 23.540[21.751, 23.451]4 grids · 20 seed-values
resnet20.228 ± 1.06521.81619.352 … 21.156[19.277, 21.176]4 grids · 20 seed-values

fault_location_line · mae ↓ (test set) rank-stable: mlp wins under every scheme

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn19.655 ± 1.04520.08118.538 … 20.309[18.765, 20.634]4 grids · 20 seed-values
gru19.144 ± 1.37818.88717.563 … 20.010[18.026, 20.476]4 grids · 20 seed-values
mlp18.958 ± 1.44418.68117.306 … 19.912[17.764, 20.331]4 grids · 20 seed-values
random_forest22.182 ± 0.98523.59021.429 … 23.203[21.321, 23.046]4 grids · 20 seed-values
resnet19.863 ± 1.26920.50018.521 … 20.751[18.789, 21.084]4 grids · 20 seed-values

fault_location_local · mae ↓ (test set) rank-sensitive: winners ['gru', 'mlp']

baselinemean over grids ± t-CImedian over gridsleave-one-grid-out rangeseed-bootstrap 95% CIn
cnn21.499 ± 0.97322.98120.671 … 22.232[20.653, 22.382]4 grids · 20 seed-values
gru21.004 ± 1.13522.71620.028 … 21.806[20.010, 22.050]4 grids · 20 seed-values
mlp21.638 ± 0.80622.48420.870 … 22.288[20.936, 22.378]4 grids · 20 seed-values
random_forest23.846 ± 0.43924.49723.531 … 24.314[23.462, 24.225]4 grids · 20 seed-values
resnet21.567 ± 0.95223.07720.779 … 22.304[20.727, 22.457]4 grids · 20 seed-values