Is AI inference getting cheaper or more expensive over time?

2026-05-05 23:35 UTC

I’ve read some of Ed Zitron’s long posts on why the AI industry is a bubble that will never be profitable (and will bring down a lot of companies and investors), and one of the recurring themes is that the AI companies are trying to capture growing market share in an industry where their marginal profits are still negative, and that any increase in revenue necessarily increases their costs of providing their services. But some of the comments in various HackerNews threads are dismissive, saying that each new generation of models makes the cost of inference lower, so that with sufficient customer volume, the companies running the models can make enough profit on inference to make up for the staggering up-front capital expenditures it took to build out the data centers, train their models, etc. It’s all pretty confusing to me. So for those of you who are familiar with the industry, I have several questions: Is the cost of running any given pretrained model going down, for specific models? Are there hardware and software improvements that make it cheaper to run those models, despite the model itself not changing? Is the cost of performing a particular task at a particular quality level going down, through releases of newer models of similar performance (i.e., a smaller model of the current generation performing similarly to a bigger model of the previous generation, such that the cost is now cheaper)? Is the cost of running the largest flagship frontier models going down for any given task? Or does running the cutting edge show-off tasks keep increasing in cost, but where the companies argue that the improvement in performance is worth the cost increase? I suspect that the reason why the discussion around this is so muddled online is because the answers are different depending on which of the 3 questions is meant by “is running an AI model getting cheaper over time?” And the data isn’t easy to synthesize because each model has different token prices and different number of tokens per query. But I wanted to hear from people who are knowledgeable about these topics.

Replies (10)

@scrubbles@poptalk.scrubbles.tech 2026-05-05 23:49

I think we're seeing a lot of optimization right now. The most exciting one I've seen is TurboQuant. Short version, every message you send to a model has context, the entire conversation you've had, instructions, skills, everything. That takes up an _exponential_ amount of ram, and this is what is causing the VRAM/RAM shortage. TurboQuant (and other copycats now) claims that it can reduce that VRAM usage of the context by **20x**. That's absolutely huge, that's 1M context models running on consumer hardware potential huge. Deepseek v4 also boasts some large claims, saying they have a model that does better than Anthropic's or OpenAI, while being 1/10th the size. That also is a huge reduction in compute and VRAM, but I'll be looking for the proof. We've seen other items too, with upgrades in running models, how quickly results are streamed, to me TurboQuant is the most exciting. I think it's good that they're finally looking at optimization. Yes, their cost has been power and compute. NVidia is more than happy to keep things inefficient because they sell GPUs that way. Software companies are doing the opposite now, reducing the compute overhead to start saving them money, which they desperately need to do if this is going to continue. New technology has always been horribly inefficient, it's only once more people see it does it start to get optimized. I think this is what is going to be required to finally push past the horribleness of AI companies, and they need to do this quickly.

Open ##3121822
@Danitos@reddthat.com 2026-05-06 04:47

https://www.tobyord.com/writing/hourly-costs-for-ai-agents. This person analyzed a question very similar to yours. For me, this means: 1. Cost of running state of the art models is increasing exponentially. 2. For a given target "intelligence", the cost is decreasing linearly. I don't really know what to make out of that in a broader picture

Open ##3121823
@Zarxrax@lemmy.world 2026-05-06 00:08

Its easy to think of it similar to something like computer hardware or game consoles. There is always newer and better hardware coming out. And the newer stuff is always more efficient (performance/watt) than the old stuff. But the user's expectations increase as well, so new hardware doesn't just aim to be more efficient, it aims to be more powerful. Then that sets a new baseline for expectations. So a lot of these LLM and other types of models are very much like that. The newer models definitely bring improvements in efficiency and performance. But no one wants to sit still, they have to keep pushing the envelope to make them better and more powerful.

Open ##3121824
@BlameThePeacock@lemmy.ca 2026-05-06 00:00

For an equivalent prompt and similar quality answer, yes. Inference prices are dropping. However, higher quality answers (or more complex prompt handling) are currently going up in inference price. The fun part will be once quality hits a point where the average user (or even business) doesn't care about the incremental quality change any more. Then it's going to be a race to the bottom for performance per dollar. Who cares if the not all companies or investors make money? They can make their bets, some will win and some will lose. I just want better tech for cheaper prices.

Open ##3121827
@brucethemoose@lemmy.world 2026-05-06 13:58

Yes. It’s dropping, dramatically. Look at the history of open and closed releases, on benchmarks that aren’t totally gamed, and it’s easy to see. LLM capabilities are plateauing, and bigger models are getting more and more niche. But inference efficiency is increasing exponentially. Tiny models are getting closer and closer to frontier ones. See: Qwen 27B, and how it can do most of what mega models did just months ago. And there’s tons of unpicked efficiency fruit in papers. Bitnet is the big one, but I’ve seen dozens of proof of concepts, just yet to be tried in a production model, that are dramatic efficiency boosts.

Open ##3121828
@vane@lemmy.world 2026-05-07 13:43

Cloud companies, phone makers, social network providers, hardware producers are so profitable that they can take parts of it's profits and fund AI in hope that in the future all knowledge can be proprietary and hidden behind their models. If that makes sense.

Open ##3121831
@humanspiral@lemmy.ca 2026-05-06 15:20

>new generation of models makes the cost of inference lower, so that with sufficient customer volume, the companies running the models can make enough profit on inference to make up for the staggering up-front capital expenditures cost per quality is definitely going down at a fast rate. LLM providers are in extremely competitive field, where open weight models are at a huge competitive advantage for any quality level (privacy, customizability). The competition is all on 2 month release cycles that essentially throw away the old version/code/weights each time. When Claude pretends its newest model is too powerful for non oligarchs to use, it limits its token reach, and then required contribution margin per token. The buisness model flaw is "one day, a winner becomes a monopoly, and AGI self improves the model at low (except for ultra expensive compute) cost." Monopoly pricing power is very hard/impossible to achieve, because if necessary, foreign governments will subsidize competition to not let a hostile US empire AGI monoplist take hold. Due to corrupt energy oligarchy, it is categorically impossible for US hosted services to ever provide comparative value compared to rational economic energy policies outside of the US. Distillation (Teacher/student RL) means that using another AGI (or leading LLM) will improve models that are behind. There will always be competition on the price/quality curve that prevents even the best/most expensive model from capturing all share. There's always free tier LLM competition availability as well. Finally, there are layers above LLMs. Agentic and swarm and "deterministic program access"/validation front ends to LLMs can add various levels of token burn, but also divert most tokens from the expensive LLMs, and iteratively improve output. There isn't just a cost/quality curve there is a cost/speed/quality/privacy curve, where non AI coordination tools can improve on the latter curve points independently of leading/expensive LLM/AGI quality.

Open ##3121835
@thedeadwalking4242@lemmy.world 2026-05-07 11:38

It's been getting better but it's paused since last nov pretty much

Open ##3121836
@General_Effort@lemmy.world 2026-05-06 14:31

1. The cost of hardware has always gone down. If it seems different in the last few years, that's because demand has gone up and people are outbidding each other to get the same scarce resource. But the fundamentals have not changed. Once Production catches up, prices will return to the previous trajectory. 2. Going down. At the same size, models are more capable. This is both because training data is being incrementally refined, and because better methods are thought up. 3. Going up. Bigger models are better, but more expensive to run. They also require more data to train. The trend towards bigger models will continue for the foreseeable future. How quickly models can be scaled up is limited by the logistics (data centers and training data)

Open ##3121837
@General_Effort@lemmy.world 2026-05-06 13:45

FYI: Ed Zitron is a PR expert. He has no background in engineering or finance. He has the skills to make people listen to him and give him money. He does not have the skills to determine if any of his assertion are true or not. If you're wondering if I'm calling him a liar, then I can only say that I can't read minds. If you're not wondering, then you weren't paying attention.

Open ##3121838