JEPA learns what to keep from what it sees, and uses that to predict what comes next.

But what about the details it can’t predict? Wouldn’t it be easier for the model to drop them?

Look at this example
Imagine a traffic light that changes color randomly. Its next color is unpredictable, but its current color helps predict whether a cyclist will stop or move. If we drop the light, the cyclist becomes harder to predict.

▪️ A recent research call this set-level view "predictive closure:" A feature’s value depends on its relationships with the other features the model represents.

Keeping the light makes the cyclist easier to predict, but adds an unpredictable color to the model’s targets. Dropping it removes that target, but loses a useful clue.

The researchers separate these two costs:

1. Aleatoric uncertainty: what remains uncertain even with the full current observation, like the light’s next random color.
2. Predictive leakage: useful information available in the observation but missing from the representation, like the current color if the model drops the light.

An unpredictable detail can therefore be worth keeping because of what it helps predict. Random noise that helps predict nothing else offers no such benefit.

In controlled experiments, unpredictable features became easier to recover from the representation as they became more useful for predicting others.

That doesn’t guarantee JEPA keeps everything important, but it explains why unpredictable doesn’t automatically mean disposable.