Autoencoder-Based Market Regime Filtering for Self-Adjusting Signal Weights

Day nine of February's vol spike, the regime filter approved every signal the strategy sent. On day one its reconstruction error had printed at 6.2x threshold. By day nine the same metric read 0.8x. Nothing about the market had calmed down. The filter had stopped being surprised, and that was the problem. How it works: a small autoencoder trained on bar-level features, its reconstruction error gating the strategy's signal weights, a retrain loop watching the model's own error distribution. The retrain is what disarmed the alarm. This is backwards from what you'd expect — the self-correction loop caused the failure. Mechanics first. Every bar compresses into six features: 20-bar realized vol, the 5-over-20-day return ratio, volume z-score, lag-1 autocorrelation of returns, high-low range ratio, 20-bar return skew. All stationary-ish, all computable from OHLCV. A StandardScaler goes over the training window only. Fitting it on full history leaks the future into normalization, and every threshold looks perfectly calibrated in-sample and useless live. I lost a month of backtests to exactly this. The autoencoder is 6 → 2 → 6, MSE loss, a few hundred epochs. The bottleneck of 2 is the point: it forces the autoencoder to learn which feature combinations co-occur. A bar with vol in the 90th percentile and volume in the 10th sits off the manifold. Reconstruction error catches that. A genuine vol regime — everything elevated together — is off-manifold in a different direction, and the residual vector points at which features failed (vol, usually, in my data). I don't act on that yet. It sits in the logs. Weight rule: w = clip(thr / err, 0.15, 1.0). The threshold comes from the 99th percentile of errors on a validation window the autoencoder never saw. Training errors are autocorrelated, so the in-sample distribution looks tighter than anything live will produce — the percentile has to come from held-out bars. Once error crosses the threshold, weight decays toward a floor of 0.15. The floor keeps small positions alive so the strategy can demonstrate whether its edge survived the regime. A hard zero makes recovery unfalsifiable: no fills means no evidence, and no evidence means no way back except by decree. import numpy as np import torch import torch.nn as nn

def make_ae(n=6, latent=2): # latent=2 is deliberately brutal: forces the autoencoder to learn # which feature combinations co-occur, not to memorize individual bars return nn.Sequential( nn.Linear(n, 8), nn.ReLU(), nn.Linear(8, latent), nn.Linear(latent, 8), nn.ReLU(), nn.Linear(8, n))

def fit_filter(train, val, q=0.99): # scaler stats from the train window ONLY — fitting on full history # leaks the future into normalization; thresholds then look perfect # in-sample and useless live mu, sd = train.mean(0), train.std(0) + 1e-8 tr = torch.tensor((train - mu) / sd, dtype=torch.float32) ae = make_ae(tr.shape[1]) opt = torch.optim.Adam(ae.parameters(), lr=1e-3) for _ in range(300): opt.zero_grad() loss = ((ae(tr) - tr) ** 2).mean() loss.backward() opt.step() with torch.no_grad(): v = torch.tensor((val - mu) / sd, dtype=torch.float32) # threshold from held-out errors: training errors are autocorrelated # and understate what live bars will produce val_err = ((ae(v) - v) ** 2).mean(1).numpy() return ae, mu, sd, float(np.quantile(val_err, q))

def signal_weight(ae, mu, sd, thr, bar): x = torch.tensor((bar - mu) / sd, dtype=torch.float32) err = float(((ae(x) - x) ** 2).mean()) # floor of 0.15 keeps small positions alive so the strategy can # re-prove itself after the regime; a hard zero makes recovery # unfalsifiable — no fills means no evidence return float(np.clip(thr / max(err, 1e-9), 0.15, 1.0)) The self-adjustment layer The threshold is a moving target. Every day the system re-estimates it from a rolling 250-bar window of live errors, lagged five bars. That lag matters more than it looks: today's shock must not sit inside today's threshold, or today's error can never breach it. The filter would be approving its own anomaly. Retraining has a separate trigger. The system compares the rolling median of live errors against the median from the training baseline; a ratio above 1.5 for five consecutive daily checks fires a retrain on the trailing window. Streaks decay by one on a quiet bar instead of resetting — five days of drift shouldn't be erased by one calm afternoon. This is where the self in self-adjusting actually lives. The autoencoder retunes its own threshold, watches its own error distribution, retrains its own weights, rescales the downstream signals — nobody touches it intraday. One external AI does enter the loop. When a retrain fires, the pipeline bundles the drift stats — median ratio, streak length, per-feature error contributions, retrain-window composition — and sends them through the OpenAI API to draft an incident note for Slack. The LLM has no write access; weights restore automatically regardless. I originally gated restores behind a human ack, then watched two triggers sit unacknowledged through a weekend while the strategy traded at floor weights into decent conditions. The note exists for forensics now. Whether the LLM adds anything beyond the per-feature table I already compute in numpy — sample of one, and I can't tell. I've been testing whether the same call can separate structural drift from a transient vol shock using those feature contributions. Accuracy so far: a coin flip, though now it's a cheap coin flip. Here's the catch. After the retrain, error drops and weights restore. The market didn't get safer; the autoencoder got familiar with it. Familiar and profitable are different properties, and the model can only measure the first one — it never sees PnL or slippage. High-vol-with-trend and high-vol chop reconstruct to nearly identical error profiles. A trend follower makes a quarter in one and gives it all back in the other. The error cannot tell them apart. Retraining restores familiarity. Permission is a separate decision, and weights come back only if the strategy's realized PnL under the new regime clears a floor. The small positions at 0.15 weight exist to generate exactly that evidence. Reconstructability and tradability each get a vote. class SelfMonitor: # monitors the filter's own error distribution: the market may change # freely, silent decay of the model itself is what this catches def init(self, baseline_median, ratio=1.5, patience=5): self.baseline = baseline_median self.ratio = ratio self.patience = patience self.streak = 0

def update(self, live_err):
    if live_err > self.ratio * self.baseline:
        self.streak += 1
    else:
        # decay, don't reset — one calm bar shouldn't erase
        # five days of accumulated drift evidence
        self.streak = max(0, self.streak - 1)
    return "retrain" if self.streak >= self.patience else "hold"

def rolling_threshold(errs, window=250, q=0.99, lag=5): # exclude the most recent bars: today's shock must not sit inside # today's threshold, or it can never breach it — the filter would # be approving its own anomaly return np.quantile(errs[-window - lag:-lag], q) Flat, dead-vol stretches break the filter from the other side. Error scale collapses everywhere, the 99th-percentile threshold ends up centimeters above the noise floor, ordinary bars breach it on rounding, and weights oscillate between 0.2 and 1.0 with no regime in sight. The tell is the threshold itself — when its own level drops an order of magnitude over a few weeks, stop trusting the filter until vol returns. Correlation drift is the second failure. The autoencoder learns feature correlations as they were during training; vol and volume decouple live, every bar reads off-manifold for weeks, and the filter goes dark while the strategy may be trading fine. False positives that cost real opportunity, dressed up as caution. The retrain window is a standing dilemma. Too short, and the trigger fires on a one-off shock that gets memorized as the new normal; too long, and a slow structural drift never shifts the median enough to fire. My patience sits at 5 checks, ratio at 1.5, because that configuration survived two quarters without embarrassing me. Calibration by survivability. I don't have anything principled here. Log per-feature reconstruction error next to every fill, starting tonight. If your worst drawdown clusters don't sit inside error breaches, the filter is decoration and you can delete it. The question I keep circling without answering: should the latent space include the strategy's own live hit rate? Feed performance into the filter and you close a loop nobody can untangle afterward — the filter starts defending its own past decisions.