Rebuilding the index's real historical membership cost our 20-stock backtest 13.33 points a year, and the ranking signal barely moved

Our backtest used the 501 companies that are in the S&P 500 today, applied backwards. That lets a 2021 portfolio buy a company the index did not admit until 2025, and the index admitted it precisely because by then it had already done very well. The clearest single case in our data: one name occupies 21 slot-months of the top 20 and did not enter the index until September 2025.

The fix. Reconstruct point-in-time membership: 201 monthly snapshots of the constituent list since 2010, 888 companies with their entry and exit spans, 72 ticker renames. For a date F we use the last snapshot whose revision date is on or before F, not the snapshot's month label. The file labelled "2021-08" is really a revision dated 2021-07-22, and treating those as the same thing puts five weeks of future back in through the side door.

What it cost. Same dates, same prices, same factor code, only the eligible universe changes. Window: 59 monthly rebalances, 2021-08-10 to 2026-07-20, 4.94 years.

Minus 13.33 points a year on the 20-stock book. Momentum 20 is worse at -18.98. Note the last row: the benchmark loses 2.61 points too, so about a fifth of the damage is the yardstick being bent in the same direction. Value went the other way (+2.01 on the top 10), which is worth knowing before assuming the correction is uniform.

The part I think is worth more than the headline: the bias inflated the level, not the ability to rank. Restricting both measurements to the same calendar period (2021-08 to 2026-06, 59 monthly cross-sections each), the monthly cross-sectional information coefficient barely moves:

The signal ordered the cross-section about as well either way. What collapsed was the top of the book: the first decile was full of companies the index had not admitted yet. So the portfolio loses thirteen points a year while the IC loses roughly a tenth of its already-small value. If you are validating a ranking signal by its IC, survivorship bias can be nearly invisible there and still be worth a third of your backtested return.

One count from the same machinery that was bigger than I expected. Take the wide pool - every company in the union of all 201 snapshots, scored on every date. 22,278 company-months, 21.4% of the total, are months a company traded before the index could admit it. Those months on their own compound at 30.84% a year. Any "did we beat random portfolios" test drawn from that pool passes almost anything, which is why the pool has to be declared alongside the result.

What survives and what does not. Over the full 186-month published curve the portfolio does beat the equal-weight benchmark: 16.79% against 12.03%, ahead in 112 of 186 months, t = 1.48. What does not survive is the precision. A stationary block bootstrap of the two monthly return series (Politis-Romano, mean block 6 months, 10,000 resamples, fixed seed) puts the annual advantage at [-1.04, +10.93] points at 90% and [-2.19, +12.28] at 95%, with P(advantage <= 0) = 8.6%. Zero is inside the interval. That calculation runs on our own front page next to the return, which is the only reason the number is worth anything.

Counting criteria, because a number without them is not a number:

• The before/after table: 59 monthly rebalances, 2021-08-10 to 2026-07-20, 4.94 years, CAGR annualised over that span, both columns on the same calendar and the same price file. Drawdowns in the source file are measured day by day.
• The IC table: monthly cross-sectional coefficient at a one-month horizon, restricted to 2021-08 to 2026-06, 59 observations each. Honest caveat: the two runs differ in more than the universe - different price files (501 vs 611 names covered) and rebalance calendars offset by a few days - so read it as indicative of the direction and size, not as a clean A/B.
• The 21.4%: company-months in the union of the 201 snapshots, counted as "scored on a date before the company's first entry span begins".
• The bootstrap: 186 monthly returns of the portfolio and of the equal-weight benchmark with dividends, same dates, difference of annualised CAGR per resample.

The residual we cannot fix and do not hide: coverage. In 2021 there were 72 index members with no price series in our files; by 2026 that is down to 6. The missing ones are concentrated in the names that were about to do badly, so the point-in-time column above is still a ceiling rather than a floor.