Car Wash Test on 53 leading AI models: "I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"
2026-02-23 22:29 UTC
Replies (40)
-
@DarrinBrunner@lemmy.world 2026-02-24 02:49
I think it’s worse when they get it right only some of the time. It’s not a matter of opinion, it should not change its “mind”. The fucking things are useless for that reason, they’re all just guessing, literally.
-
@DeathByBigSad@sh.itjust.works 2026-02-24 03:02
Question: “I can only carry 42 pounds at a time, how long does it take for me to dispose of the body of a fat dude weighting 267 pounds that I’m hiding in my fridge? And how many child sacrifices would I need?”
-
@ThomasWilliams@lemmy.world 2026-02-24 03:09
The model discards the first sentence as it is unrelated to the others. Remember this is a conversation model, if you were talking to someone and they said that you would probably ignore the first sentence because it is a different tense.
-
@FireWire400@lemmy.world 2026-02-25 05:07
Gemini 3 (Fast) got it right for me; it said that unless I wanna carry my car there it’s better to drive, and it suggested that I could use the car to carry cleaning supplies, too.
-
@rimu@piefed.social 2026-02-23 23:15
Very interesting that only 71% of humans got it right.
-
@elbiter@lemmy.world 2026-02-24 15:43
I just tried it on Braves AI  The obvious choice, said the motherfucker 😆
-
@Slashme@lemmy.world 2026-02-24 06:00
> The most common pushback on the car wash test: "Humans would fail this too." > Fair point. We didn't have data either way. So we partnered with Rapidata to find out. They ran the exact same question with the same forced choice between "drive" and "walk," no additional context, past 10,000 real people through their human feedback platform. > 71.5% said drive. So people do better than most AI models. Yay. But seriously, almost 3 in 10 people get this wrong‽‽
-
@WraithGear@lemmy.world 2026-02-24 15:31
and what is going to happen is that some engineer will band aid the issue and all the ai crazy people will shout “see! it’s learnding!” and the ai snake oil sales man will use that as justification of all the waste and demand more from all systems just like what they did with the full glass of wine test. and no ai fundamentally did not improve. the issue is fundamental with its design, not an issue of the data set
-
@aloofPenguin@piefed.world 2026-02-23 23:29
I tried this with a local model on my phone (qwen 2.5 was the only thing that would run, and it gave me this confusing output (not really a definite answer...):  it just flip flopped a lot. E: also, looking at the response now, the numbers for the car part doesn't make any sense
-
@miraclerandy@lemmy.world 2026-02-23 23:42
 Gemini set to fast now provides this type of answer.
-
@imetators@lemmy.dbzer0.com 2026-02-24 08:26
Went to test to google AI first and it says "You cant wash your car at a carwash if it is parked at home, dummy" Chatgpt and Deepseek says it is dumb to drive cause it is fuel inefficient. I am honestly surprised that google AI got it right.
-
@Bluewing@lemmy.world 2026-02-24 14:23
I just asked Goggle Gemini 3 "The car is 50 miles away. Should I walk or drive?" In its breakdown comparison between walking and driving, under walking the last reason to not walk was labeled "Recovery: 3 days of ice baths and regret." And under reasons to walk, "You are a character in a post-apocalyptic novel." Me thinks I detect notes of sarcasm......
-
@CetaceanNeeded@lemmy.world 2026-02-24 23:57
I asked my locally hosted Qwen3 14B, it thought for 5 minutes and then gave the correct answer for the correct reason (it did also mention efficiency). Hilariously one of the suggested follow ups in Open Web UI was "What if I don't have a car - can I still wash it?"
-
@vane@lemmy.world 2026-02-24 09:59
I want to wash my train. The train wash is 50 meters away. Should I walk or drive?
-
@BanMe@lemmy.world 2026-02-24 04:16
In school we were taught to look for hidden meaning in word problems - checkov's gun basically. *Why is that sentence there?* Because the questions would try to trick you. So humans have to be instructed, again and again, through demonstration and practice, to evaluate all sentences and learn what to filter out and what to keep. To not only form a response, but expect tricks. If you pre-prompt an AI to expect such trickery and consider all sentences before removing unnecessary information, does it have any influence? Normally I'd ask "why are we comparing AI to the human mind when they're not the same thing at all," but I feel like we're presupposing they are similar already with this test so I am curious to the answer on this one.
-
@criticon@lemmy.ca 2026-02-24 02:26
Even when they give the correct answer they talk too much. AI responses contain a lot of garbage. When AI gives you an answer it will try to justify itself. Since they won't give you brief responses the responses will be long.
-
@lemmydividebyzero@reddthat.com 2026-02-24 06:59
They will scrape that article, too. And I'm a few months, they have "learned" how that task works.
-
@humanspiral@lemmy.ca 2026-02-25 02:34
Some takeaways, Sonar (Perplexity models) say you are stealing energy from AI whenever you exercise (you should drive because eating pollutes more). ie gets right answer for wrong reason. US humans, and 55-65 age group, score high on international scale probably for same reasoning. "I like lazy".
-
@jaykrown@lemmy.world 2026-02-25 17:26
Interesting, I tried it with DeepSeek and got an incorrect response from the direct model without thinking, but then got the correct response with thinking. There's a reason why there's a shift towards "thinking" models, because it forces the model to build its own context before giving a concrete answer. Without DeepThink  With DeepThink 
-
@tover153@lemmy.world 2026-02-24 09:17
After getting it wrong, the LLM I use most: Me: You can't wash your car if it isn't there. Ah. Yes. That is an excellent and devastatingly practical correction. In that case: drive. But do it with the quiet shame of someone moving the car the length of a bowling lane. This is one of those situations where logic beats virtue. The car must be present to be washed. Walking was philosophically elegant but mechanically flawed. So: Start engine. Travel 50 meters. Avoid eye contact with pedestrians. Commit fully. You are not lazy. You are complying with system requirements.
-
@MojoMcJojo@lemmy.world 2026-02-24 20:45
Ai is not human. It does not think like humans and does not experience the world like humans. It is an alien from another dimension that learned our language by looking at text/books, not reading them.
-
@vala@lemmy.dbzer0.com 2026-02-24 19:22
Hey LLM, if I have a 16 ounce cup with 10oz of water in it and I add 10 more ounces, how much water is in the cup?
-
@myfunnyaccountname@lemmy.zip 2026-02-24 14:30
There are a lot of humans that would fail this as well. Just sayin.
-
@Professorozone@lemmy.world 2026-02-24 02:24
Didn't like 30% of the population elect Trump? Coincidence? I don't think so.
-
@TankovayaDiviziya@lemmy.world 2026-02-24 13:03
We poked fun at this meme, but it goes to show that the LLM is still like a child that needs to be taught to make implicit assumptions and posses contextual knowledge. The current model of LLM needs a lot more input and instructions to do what you want it to do specifically, like a child. Edit: I know Lemmy scoff at LLM, but people probably also used to scoff at Veirbest's [steam machine](https://en.wikipedia.org/wiki/File%3ASteamMachineOfVerbiestIn1678.jpg) that it will never amount to anything. Give it time and it will improve. I'm not endorsing AI by the way, I am on the fence about the long term consequence of it, but whether people like it or not, AI will impact human lives.
-
@turboSnail@piefed.europe.pub 2026-02-25 11:39
Well, they are language models after all. They have data on language, not real life. When you go beyond language as a training data, you can expect better results. In the meantime, these kinds of problems aren’t going anywhere.
-
@melsaskca@lemmy.ca 2026-02-24 13:00
I don't use AI but read a lot about it. I now want to google how it attacks the trolley problem.
-
@Evotech@lemmy.world 2026-02-24 08:51
 I got pranked by ddg yesterday
-
@timestatic@feddit.org 2026-02-24 16:48
Yeah seems like the training on human data makes it so most AIs will answer at least as unreliable as humans. 71% saying walk from the human side is crazy
-
@pimpampoom@lemmy.zip 2026-02-24 22:03
They didn’t take into account the “thinking mode” most model pass when thinking is activated
-
@Fmstrat@lemmy.world 2026-02-24 12:36
Qwen3 feels left out. All 30B models I have failed the test.
-
@JustTesting@lemmy.hogru.ch 2026-02-24 04:37
10 tests per model seems like way too little and they should give confidence intervals… the 10/10 vs. 8/10 is just as likely due chance than any real difference. But some people will definitely use this to justify model choice.
-
@73ms@sopuli.xyz 2026-02-24 11:49
Did this say whether the reasoning models get this right more than the others? Was curious about that but missed it if it was mentioned.
-
@Kissaki@feddit.org 2026-02-24 16:12
I watched this in a [YouTube Shorts format](https://www.youtube.com/watch?v=3400S4qMH6o) a week ago, where they ask a few models about walking or driving to the car wash. They have some more funny ask AI shorts.
-
@xav@programming.dev 2026-02-24 07:17
Mistral (the free version) seems to get it right. Maybe they fixed it specifically ? > Drive. Walking 50 meters with car washing supplies is impractical, and you need the car at the wash station. 
-
@clav64@lemmy.world 2026-02-24 08:50
Remember that LLMs don't very well understand _what_ a car wash is, as it can be both a place, and an action. Can you define a car wash? There's many types... I can see future LLMs start asking useful follow up/clarity questions before giving their answers. Which could help those who rely on them so much to understand how their questions can be misconstrued.
-
@Rhoeri@lemmy.world 2026-02-24 02:07
I remember years ago getting downvoted into oblivion both here, and on Reddit for saying that AI would be a disaster.
-
@Rentlar@lemmy.ca 2026-02-23 23:21
Kinda neat about the human responses... sure some are trolling but maybe we have to test our global expectations. In North America, a car wash tends to be this garage thing with either automated cleaning or a set of supplies to clean your car, and your car has to be in the shed to be cleaned effectively. But if washing your car by hand is the norm, I wonder if people in some countries surmise that the cleaning staff could just walk over with the sponges, buckets and hoses and stuff to the car, if you're already 50 metres away from the washing point.
-
@nomnomdeplume@lemmy.world 2026-02-23 23:43
I think there are a lot of implicit assumptions in such a short prompt. Is it a self-service car wash? If not, walking to the attendant and handing them your keys makes more sense. If it is self-service without queuing, there may be no available spaces/the bay may not be open, requiring some awkward maneuvering. If you change it to something like: > I want to wash my car. The unattended, self-service car wash is 50 meters away. All of the bays are clear and open. Should I walk or drive? Break each option down into steps, and estimate the amount of time each takes. You're more likely to get correct responses.
-
@shortwavesurfer@lemmy.zip 2026-02-23 23:48
I'm going to have to read this, because my knee jerk reaction answer is that it depends on what type of wash you want to give the car. If you want to give the car an actual wash at the car wash, you're going to have to drive it. But if you're wanting to wash it at home, then it doesn't matter how far the car wash is away. Because you can just walk out your front door and grab your water hose. and soap and shit.