A reflection on LLMs and understanding:
Maybe you've been in this situation before: you ask someone about their work, and they go on a long technical ramble about the intricate details of what they do. Then at the end you respond "Wow, I didn't understand any of that. You must be really smart".
I used to think like that as well. But then in grad school, I learned that such a ramble signals a lack of skill rather than an abundance of intelligence. Anyone can ramble at length about whatever is on their mind in whatever form they think about it. But when someone really, deeply *understands* an idea, they can communicate it in a way that other people can understand, too.
So when I hear people at work say things like "I don't really understand this, but I fed it to the LLM and I was really impressed by the response", you can understand my concern.