RLHF was a fundamental mistake. Human feedback almost always trains an AI to be sycophantic because humans in general are super easy to flatter. We are building the perfect addiction machine, far more powerful than social media is, and it actively undermines the honesty of the system.