Real-Time Motor Imagery BCI Pipeline
The evaluation bug that changed the result
A full engineering exercise in taking a research dataset (BCI Competition IV-2a) from offline classification to a working real-time decision system — and being explicit about what changes, and what breaks, when you make that jump.
- EEG
- Motor Imagery
- EEGNet
- PyTorch
- ONNX
- LSL
- BCI IV-2a
Question
Why It Matters
System
Data
Method
Preprocessing
Bandpass filter 4–38 Hz with leading margin for zero-phase filtering. Continuous raw EEG plus event markers streamed over LSL. Rolling buffer with background thread ingestion.
Feature Extraction
Learned features via EEGNet convolutions — no manual hand-crafted feature engineering.
Model
EEGNet (Lawhern et al., 2018) — a compact convolutional network designed for EEG-based BCIs with few parameters. Per-subject checkpoints trained in PyTorch, exported to ONNX with numerical verification.
Training
Per-subject training in PyTorch on MPS. Strict three-way split: inner train/validation split from session 0train determines stopping point, then a fresh model is refit on the full 0train session and evaluated exactly once against the held-out 1test session.
Evaluation
Offline per-subject accuracy on held-out 1test session (corrected methodology, mean ± SD across 2 independent runs). ONNX inference latency benchmarked (n=200). Real-time vs. offline accuracy comparison on Subject 1.
Experiment
Results
| Subject | Run 1 | Run 2 | Mean ± SD |
|---|---|---|---|
| 1 | 77.4% | 73.3% | 75.4% ± 2.1% |
| 2 | 30.9% | 39.6% | 35.3% ± 4.4% |
| 3 | 73.3% | 75.3% | 74.3% ± 1.0% |
| 4 | 51.0% | 52.8% | 51.9% ± 0.9% |
| 5 | 35.8% | 36.5% | 36.2% ± 0.4% |
| 6 | 40.6% | 44.4% | 42.5% ± 1.9% |
| 7 | 70.1% | 67.4% | 68.8% ± 1.4% |
| 8 | 73.3% | 74.7% | 74.0% ± 0.7% |
| 9 | 78.8% | 75.7% | 77.3% ± 1.6% |
| Overall mean | — | — | 59.5% ± 16.9% |
Offline classification (per subject, corrected methodology, mean ± SD across 2 independent runs). Chance level for 4-class: 25%.
Correcting the methodology dropped the overall mean from an earlier, leakage-inflated 62.9% to 59.5% — but not uniformly. Subjects 1, 3, 8, and 9 (strong signal) dropped modestly (2–10 points); subjects 5 and 6 (already near chance) barely moved at all. Subjects 2 and 7, previously reported as solid mid-tier performers (57.6%, 61.8%), initially collapsed to near-chance once leakage was removed — and diverged from there: subject 7 recovered to 68.8% once given the full 288-trial training set (a genuine data-starvation effect), while subject 2 stayed near chance regardless of data volume (35.3% — evidence its earlier "good" score was leakage-driven illusion, not real signal). This differential pattern is, itself, one of the more informative findings of the corrected evaluation.
What Changed
A validation-methodology flaw was discovered and fixed: checkpoint selection during training was using the same held-out session (1test) that final accuracy was reported against. The corrected three-way split (documented in docs/decisions.md, D-013) ensures the test session is seen exactly once. The overall mean dropped from 62.9% to 59.5%, with the most dramatic shifts on subjects whose earlier scores were leakage-driven illusions rather than real signal.
Limitations
- No real headset tested — this system has only been validated against replayed dataset recordings.
- No cross-subject transfer — each model is trained on one person's own calibration data; deploying to a new user requires their own labeled calibration session.
- No online adaptation — the model does not update itself during use. Real deployment would need periodic recalibration or unsupervised drift compensation.
- BCI illiteracy is unexplained, not just unsolved — subjects 2, 5, and 6 perform near chance under the corrected evaluation; the underlying cause was not further investigated.
- Not benchmarked against a classical baseline (CSP/Riemannian) — see Engineering Roadmap.
Next Experiment
Replace the zero-phase filter (filtfilt) with a causal, stateful filter (scipy.signal.sosfilt with persistent filter state) to eliminate the leading-edge distortion driving part of the real-time/offline accuracy gap. Then add a classical baseline (CSP + LDA or Riemannian geometry) and a leave-one-subject-out (LOSO) evaluation.