Post #1490136

@pbloem@sigmoid.social @ngaylinn@tech.lgbt What does the word "agent" mean in this context? By my read folks are replacing one inscrutable buzzword with another as though that changes anything important. Calling the guts of a so-called intelligent agent a next-high-reward-action predictor instead of a next-token predictor doesn't change the paucity of the model nor its fundamental and currently-unavoidable limitations. The "percepts" are biased, filtered, and digitized; the possible "actions" are as well; and the guts perform a seq2seq transformation. Yes it's digitized and filtered: FP4, FP8, FP16 etc. are not real numbers. That's the fundamental issue, not what labels we slap on the components or how complicated the wiring between them is. This is the reason LLMs and anything like RLHF that wraps them are ill-suited to most decisionmaking tasks along multiple critical axes whenever the decisions touch the real world. Joseph Weizenbaum was already writing about this 50 years ago. Decisions that matter are not "games" in any sense of that word. See also: small vs. large words (Leonard Savage) and the notion of Knightian uncertainty; the Chinese Room thought experiment (John Searle); the Mary's Room thought experiment (Frank Jackson); the octopus test (Bender and Koller: https://aclanthology.org/2020.acl-main.463.pdf ); and the notions of undermining and overmining (Graham Harman) for a start. I'm not a fan of his "snake oil" framing, but Arvind Narayanan is onto something when he says "this product is essentially an elaborate random number generator" in 2019: https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf (video of a similar talk from 2021, which is worth watching: https://www.cs.princeton.edu/news/how-recognize-ai-snake-oil ).