Post by @maugendre@hachyderm.io

2026-10-07 13:38 UTC

Why would that be (according to Steven Frank)? "The excess dimensions create large subspaces of equivalent training performance but varying generalization ability. Within these subspaces, learning dynamics preferentially reach simple smooth solutions that capture the general structure of the problem rather than its idiosyncratic details". Similarly for proteins: https://bsky.app/profile/john-mcbride.bsky.social/post/3mpycngozbs2j

Replies (1)

@maugendre@hachyderm.io 2026-10-07 14:03

How to predict if you do not know what the driver is? "Conditional Feature Importance (CFI; Strobl et al. (2008)) is a classical variable importance measure that accounts for the relationship between the studied feature and the others." "We want to avoid making strong assumptions about the relationship between inputs and outputs, as it can be complex. However, we assume that the relationship among the input covariates is simple, typically because they originate from the same generative process (e.g. a measurement device). Then, it can be much more efficient and accurate to study the relationship between the j-th covariate, Xj, given the rest, X−j, rather than directly regressing y on X−j. This approach is also advantageous in settings where labeling data is expensive, as there are typically more available samples for the input pair (X−j, Xj) than for the pair (X−j, y)." Angel Reyero-Lobo, Pierre Neuvial, Bertrand Thirion (2026). https://arxiv.org/abs/2501.17520 #stats #statistics #probabilities #machineLearning #featureSelection

Open ##5002472