Post #4147054

Benchmarks are useful when they measure what matters. As far back as 1975, Goodhart wrote “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes,” or as Marilyn Strathern memorably rephrased it, “When a measure becomes a target, it ceases to be a good measure.” The possibility of stealing test answers may be the ultimate expression of Goodhart’s law. (9/15)