Task — fault_detection_local

benchmark version 1.1.0 · held_out results v1.0.0 · splits v1.0/held_out, v1.1/multi_grid
updated 2026-09-29 13:18 UTC · git 7243f310ddbc

tier 1 · fault_detection · local view · 2 classes · all cells measured

What this task is. Decide, from one 50 ms waveform window, whether a short-circuit fault is present. This is the core protection decision: a relay that misses a fault leaves it burning (missed-fault rate), one that trips on a healthy window disconnects customers for nothing (false-alarm rate). Both error types are reported alongside balanced accuracy because their costs are asymmetric in practice. Local view: one measurement cubicle — three currents and three voltages (6 channels), what a single protection relay physically sees.

How it's scored

Headline metric balanced_accuracy (↑ higher is better); full metric set: balanced_accuracy · macro_f1 · missed_fault_rate · false_alarm_rate · accuracy. Metrics are a frozen contract (src/evemtbench/evaluation/metrics.py); label derivation: src/evemtbench/tasks/labels.

Example window

example waveform window

event flt_1phg_incipient · grid double_line (adapt_grid/test split) · sample 2, window #46, cubicle 3 (max-RMS) · 50 ms / 480 samples @ 9.6 kHz · top: 3-phase current, bottom: 3-phase voltage · figure provenance in assets/manifest.json

Results (held_out)

fault_detection_local — every grid × baseline; click headers to sort
taskgridbaselineheadlinetestbenchmarkparamsfit/seedtrace
fault_detection_localcigre_mvmajoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_detection_localcigre_mvthresholdbalanced_accuracy ↑0.660 ± 0.000 (n=5)0.810 ± 0.000 (n=5)—2strace: output · eval-code · log · train-code · config · W&B
fault_detection_localcigre_mvrandom_forestbalanced_accuracy ↑0.836 ± 0.002 (n=5)0.983 ± 0.001 (n=5)—17strace: output · eval-code · log · train-code · config · W&B
fault_detection_localcigre_mvmlpbalanced_accuracy ↑0.772 ± 0.002 (n=5)0.958 ± 0.003 (n=5)1.6M2.9mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localcigre_mvgrubalanced_accuracy ↑0.825 ± 0.036 (n=5)0.975 ± 0.013 (n=5)52k2.7mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localcigre_mvcnnbalanced_accuracy ↑0.826 ± 0.006 (n=5)0.970 ± 0.004 (n=5)49k2.4mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localcigre_mvresnetbalanced_accuracy ↑0.831 ± 0.043 (n=5)0.969 ± 0.008 (n=5)506k8.2mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localdouble_linemajoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_detection_localdouble_linethresholdbalanced_accuracy ↑0.729 ± 0.000 (n=5)0.916 ± 0.000 (n=5)—2strace: output · eval-code · log · train-code · config · W&B
fault_detection_localdouble_linerandom_forestbalanced_accuracy ↑0.881 ± 0.001 (n=5)0.499 ± 0.000 (n=5)—16strace: output · eval-code · log · train-code · config · W&B
fault_detection_localdouble_linemlpbalanced_accuracy ↑0.850 ± 0.001 (n=5)0.981 ± 0.000 (n=5)1.6M1.1mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localdouble_linegrubalanced_accuracy ↑0.864 ± 0.017 (n=5)0.984 ± 0.004 (n=5)52k2.2mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localdouble_linecnnbalanced_accuracy ↑0.872 ± 0.014 (n=5)0.987 ± 0.001 (n=5)49k2.0mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localdouble_lineresnetbalanced_accuracy ↑0.883 ± 0.030 (n=5)0.989 ± 0.008 (n=5)506k7.5mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localtestgrid_110kvmajoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_detection_localtestgrid_110kvthresholdbalanced_accuracy ↑0.755 ± 0.000 (n=5)0.886 ± 0.000 (n=5)—2strace: output · eval-code · log · train-code · config · W&B
fault_detection_localtestgrid_110kvrandom_forestbalanced_accuracy ↑0.882 ± 0.001 (n=5)0.672 ± 0.002 (n=5)—15strace: output · eval-code · log · train-code · config · W&B
fault_detection_localtestgrid_110kvmlpbalanced_accuracy ↑0.847 ± 0.001 (n=5)0.982 ± 0.000 (n=5)1.6M1.1mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localtestgrid_110kvgrubalanced_accuracy ↑0.846 ± 0.063 (n=5)0.963 ± 0.035 (n=5)52k2.4mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localtestgrid_110kvcnnbalanced_accuracy ↑0.875 ± 0.014 (n=5)0.987 ± 0.002 (n=5)49k1.7mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localtestgrid_110kvresnetbalanced_accuracy ↑0.949 ± 0.016 (n=5)0.995 ± 0.004 (n=5)506k9.3mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localieee39majoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_detection_localieee39thresholdbalanced_accuracy ↑0.793 ± 0.000 (n=5)0.914 ± 0.000 (n=5)—2strace: output · eval-code · log · train-code · config · W&B
fault_detection_localieee39random_forestbalanced_accuracy ↑0.931 ± 0.001 (n=5)0.886 ± 0.000 (n=5)—15strace: output · eval-code · log · train-code · config · W&B
fault_detection_localieee39mlpbalanced_accuracy ↑0.863 ± 0.001 (n=5)0.984 ± 0.000 (n=5)1.6M1.2mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localieee39grubalanced_accuracy ↑0.879 ± 0.049 (n=5)0.979 ± 0.016 (n=5)52k2.7mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localieee39cnnbalanced_accuracy ↑0.899 ± 0.005 (n=5)0.987 ± 0.001 (n=5)49k2.6mtrace: output · eval-code · log · train-code · config · W&B
fault_detection_localieee39resnetbalanced_accuracy ↑0.902 ± 0.006 (n=5)0.989 ± 0.001 (n=5)506k10.2mtrace: output · eval-code · log · train-code · config · W&B

Metric definitions

headline & reported metrics for this view — the metric set is a frozen contract; each links to the exact function that computes it

metricdefinitionentry
balanced_accuracymean of per-class recall — the majority class cannot buy a good scoreevaluate()
macro_f1unweighted mean of per-class F1 (zero_division=0)evaluate()
missed_fault_rateFN / (TP+FN) — fraction of true faults the model missedevaluate()
false_alarm_rateFP / (FP+TN) — fraction of non-faults flaggedevaluate()
accuracyfraction of exact-match predictionsevaluate()

Reproduce one cell

PYTHONPATH=src python -m evemtbench.baselines.runner --task fault_detection_local --protocol held_out \
    --grid <grid> --baseline <baseline> --seeds 0 1 2 3 4