Task — fault_category_local

benchmark version 1.1.0 · held_out results v1.0.0 · splits v1.0/held_out, v1.1/multi_grid
updated 2026-09-29 13:18 UTC · git 7243f310ddbc

tier 2 · fault_category · local view · 3 classes · all cells measured

What this task is. Group the fault into its coarse category (e.g. single-phase-to-ground vs multi-phase) — the level of typing many protection schemes actually act on, between binary detection and the full nine-class vocabulary. Local view: one measurement cubicle — three currents and three voltages (6 channels), what a single protection relay physically sees.

How it's scored

Headline metric balanced_accuracy (↑ higher is better); full metric set: balanced_accuracy · macro_f1 · accuracy. Metrics are a frozen contract (src/evemtbench/evaluation/metrics.py); label derivation: src/evemtbench/tasks/labels.

Example window

example waveform window

event flt_2phg_shc · grid double_line (adapt_grid/test split) · sample 84, window #1932, cubicle 3 (max-RMS) · 50 ms / 480 samples @ 9.6 kHz · top: 3-phase current, bottom: 3-phase voltage · figure provenance in assets/manifest.json

Results (held_out)

fault_category_local — every grid × baseline; click headers to sort
taskgridbaselineheadlinetestbenchmarkparamsfit/seedtrace
fault_category_localcigre_mvmajoritybalanced_accuracy ↑0.333 ± 0.000 (n=5)0.333 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_category_localcigre_mvrandom_forestbalanced_accuracy ↑0.597 ± 0.005 (n=5)0.631 ± 0.009 (n=5)—9strace: output · eval-code · log · train-code · config · W&B
fault_category_localcigre_mvmlpbalanced_accuracy ↑0.636 ± 0.004 (n=5)0.655 ± 0.002 (n=5)1.6M32strace: output · eval-code · log · train-code · config · W&B
fault_category_localcigre_mvgrubalanced_accuracy ↑0.646 ± 0.006 (n=5)0.664 ± 0.002 (n=5)53k1.8mtrace: output · eval-code · log · train-code · config · W&B
fault_category_localcigre_mvcnnbalanced_accuracy ↑0.645 ± 0.016 (n=5)0.660 ± 0.004 (n=5)49k1.0mtrace: output · eval-code · log · train-code · config · W&B
fault_category_localcigre_mvresnetbalanced_accuracy ↑0.651 ± 0.016 (n=5)0.680 ± 0.039 (n=5)506k5.7mtrace: output · eval-code · log · train-code · config · W&B
fault_category_localdouble_linemajoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_category_localdouble_linerandom_forestbalanced_accuracy ↑0.807 ± 0.005 (n=5)0.606 ± 0.192 (n=5)—11strace: output · eval-code · log · train-code · config · W&B
fault_category_localdouble_linemlpbalanced_accuracy ↑0.951 ± 0.001 (n=5)0.985 ± 0.001 (n=5)1.6M52strace: output · eval-code · log · train-code · config · W&B
fault_category_localdouble_linegrubalanced_accuracy ↑0.959 ± 0.002 (n=5)0.992 ± 0.003 (n=5)53k1.7mtrace: output · eval-code · log · train-code · config · W&B
fault_category_localdouble_linecnnbalanced_accuracy ↑0.963 ± 0.005 (n=5)0.994 ± 0.005 (n=5)49k1.7mtrace: output · eval-code · log · train-code · config · W&B
fault_category_localdouble_lineresnetbalanced_accuracy ↑0.966 ± 0.002 (n=5)0.998 ± 0.001 (n=5)506k7.7mtrace: output · eval-code · log · train-code · config · W&B
fault_category_localtestgrid_110kvmajoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_category_localtestgrid_110kvrandom_forestbalanced_accuracy ↑0.772 ± 0.004 (n=5)0.637 ± 0.010 (n=5)—10strace: output · eval-code · log · train-code · config · W&B
fault_category_localtestgrid_110kvmlpbalanced_accuracy ↑0.945 ± 0.001 (n=5)0.985 ± 0.001 (n=5)1.6M52strace: output · eval-code · log · train-code · config · W&B
fault_category_localtestgrid_110kvgrubalanced_accuracy ↑0.960 ± 0.002 (n=5)0.991 ± 0.001 (n=5)53k1.6mtrace: output · eval-code · log · train-code · config · W&B
fault_category_localtestgrid_110kvcnnbalanced_accuracy ↑0.958 ± 0.004 (n=5)0.983 ± 0.005 (n=5)49k58strace: output · eval-code · log · train-code · config · W&B
fault_category_localtestgrid_110kvresnetbalanced_accuracy ↑0.954 ± 0.007 (n=5)0.984 ± 0.005 (n=5)506k4.3mtrace: output · eval-code · log · train-code · config · W&B
fault_category_localieee39majoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_category_localieee39random_forestbalanced_accuracy ↑0.951 ± 0.000 (n=5)0.997 ± 0.000 (n=5)—12strace: output · eval-code · log · train-code · config · W&B
fault_category_localieee39mlpbalanced_accuracy ↑0.959 ± 0.001 (n=5)0.993 ± 0.001 (n=5)1.6M29strace: output · eval-code · log · train-code · config · W&B
fault_category_localieee39grubalanced_accuracy ↑0.960 ± 0.001 (n=5)0.993 ± 0.001 (n=5)53k57strace: output · eval-code · log · train-code · config · W&B
fault_category_localieee39cnnbalanced_accuracy ↑0.955 ± 0.005 (n=5)0.984 ± 0.005 (n=5)49k28strace: output · eval-code · log · train-code · config · W&B
fault_category_localieee39resnetbalanced_accuracy ↑0.955 ± 0.004 (n=5)0.984 ± 0.006 (n=5)506k2.3mtrace: output · eval-code · log · train-code · config · W&B

Metric definitions

headline & reported metrics for this view — the metric set is a frozen contract; each links to the exact function that computes it

metricdefinitionentry
balanced_accuracymean of per-class recall — the majority class cannot buy a good scoreevaluate()
macro_f1unweighted mean of per-class F1 (zero_division=0)evaluate()
accuracyfraction of exact-match predictionsevaluate()

Reproduce one cell

PYTHONPATH=src python -m evemtbench.baselines.runner --task fault_category_local --protocol held_out \
    --grid <grid> --baseline <baseline> --seeds 0 1 2 3 4