Task — fault_origin_global

benchmark version 1.1.0 · held_out results v1.0.0 · splits v1.0/held_out, v1.1/multi_grid
updated 2026-09-29 13:18 UTC · git 7243f310ddbc

tier 2 · fault_origin · global view · 2 classes · all cells measured

What this task is. Decide whether the fault lies on the observed line or elsewhere in the grid — the selectivity question. A relay must trip for its own zone and stay quiet for faults it merely sees from a distance. Global view: every cubicle in the grid at once (width differs per grid), the wide-area upper bound on observability.

How it's scored

Headline metric balanced_accuracy (↑ higher is better); full metric set: balanced_accuracy · macro_f1 · accuracy. Metrics are a frozen contract (src/evemtbench/evaluation/metrics.py); label derivation: src/evemtbench/tasks/labels.

Example window

example waveform window

event flt_3ph_shc · grid double_line (adapt_grid/test split) · sample 272, window #6256, cubicle 3 (max-RMS) · 50 ms / 480 samples @ 9.6 kHz · top: 3-phase current, bottom: 3-phase voltage · figure provenance in assets/manifest.json

Results (held_out)

fault_origin_global — every grid × baseline; click headers to sort
taskgridbaselineheadlinetestbenchmarkparamsfit/seedtrace
fault_origin_globalcigre_mvmajoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_origin_globalcigre_mvrandom_forestbalanced_accuracy ↑0.581 ± 0.009 (n=5)0.574 ± 0.003 (n=5)—44strace: output · eval-code · log · train-code · config · W&B
fault_origin_globalcigre_mvmlpbalanced_accuracy ↑0.774 ± 0.020 (n=5)0.969 ± 0.021 (n=5)42.9M9.0mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globalcigre_mvgrubalanced_accuracy ↑0.802 ± 0.027 (n=5)0.940 ± 0.031 (n=5)117k7.6mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globalcigre_mvcnnbalanced_accuracy ↑0.811 ± 0.020 (n=5)0.919 ± 0.035 (n=5)124k8.6mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globalcigre_mvresnetbalanced_accuracy ↑0.778 ± 0.018 (n=5)0.944 ± 0.042 (n=5)602k3.1mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globaldouble_linemajoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_origin_globaldouble_linerandom_forestbalanced_accuracy ↑0.842 ± 0.002 (n=5)0.917 ± 0.003 (n=5)—26strace: output · eval-code · log · train-code · config · W&B
fault_origin_globaldouble_linemlpbalanced_accuracy ↑0.865 ± 0.003 (n=5)0.990 ± 0.005 (n=5)12.0M3.0mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globaldouble_linegrubalanced_accuracy ↑0.863 ± 0.007 (n=5)0.969 ± 0.017 (n=5)69k2.1mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globaldouble_linecnnbalanced_accuracy ↑0.867 ± 0.010 (n=5)0.976 ± 0.019 (n=5)68k2.6mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globaldouble_lineresnetbalanced_accuracy ↑0.881 ± 0.012 (n=5)0.975 ± 0.020 (n=5)530k6.9mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globaltestgrid_110kvmajoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_origin_globaltestgrid_110kvrandom_forestbalanced_accuracy ↑0.759 ± 0.006 (n=5)0.799 ± 0.017 (n=5)—34strace: output · eval-code · log · train-code · config · W&B
fault_origin_globaltestgrid_110kvmlpbalanced_accuracy ↑0.854 ± 0.007 (n=5)0.984 ± 0.016 (n=5)26.7M8.5mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globaltestgrid_110kvgrubalanced_accuracy ↑0.875 ± 0.016 (n=5)0.972 ± 0.014 (n=5)92k5.7mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globaltestgrid_110kvcnnbalanced_accuracy ↑0.886 ± 0.010 (n=5)0.974 ± 0.016 (n=5)94k5.1mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globaltestgrid_110kvresnetbalanced_accuracy ↑0.866 ± 0.026 (n=5)0.983 ± 0.019 (n=5)564k4.7mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globalieee39majoritybalanced_accuracy ↑0.500 ± 0.000 (n=5)0.500 ± 0.000 (n=5)—0strace: output · eval-code · log · train-code · config · W&B
fault_origin_globalieee39random_forestbalanced_accuracy ↑0.548 ± 0.005 (n=5)0.534 ± 0.006 (n=5)—3.1mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globalieee39mlpbalanced_accuracy ↑0.869 ± 0.003 (n=5)0.970 ± 0.003 (n=5)103.4M32.4mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globalieee39grubalanced_accuracy ↑0.858 ± 0.007 (n=5)0.973 ± 0.003 (n=5)211k21.4mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globalieee39cnnbalanced_accuracy ↑0.860 ± 0.006 (n=5)0.951 ± 0.024 (n=5)234k21.5mtrace: output · eval-code · log · train-code · config · W&B
fault_origin_globalieee39resnetbalanced_accuracy ↑0.859 ± 0.010 (n=5)0.961 ± 0.025 (n=5)744k16.3mtrace: output · eval-code · log · train-code · config · W&B

Metric definitions

headline & reported metrics for this view — the metric set is a frozen contract; each links to the exact function that computes it

metricdefinitionentry
balanced_accuracymean of per-class recall — the majority class cannot buy a good scoreevaluate()
macro_f1unweighted mean of per-class F1 (zero_division=0)evaluate()
accuracyfraction of exact-match predictionsevaluate()

Reproduce one cell

PYTHONPATH=src python -m evemtbench.baselines.runner --task fault_origin_global --protocol held_out \
    --grid <grid> --baseline <baseline> --seeds 0 1 2 3 4