Post #4208432

@peter Before Anthropic secured more capacity, I was suggesting to clients that they should hedge with a couple of extra open source models, because Anthropic API availability was genuinely trending towards barely 2-9s. It's better now, but outages are still common, and their availability is 99.5% on the API. Re: kind of work, I'm in the process of writing a blog post about this, but most "agentic" work, especially with RLMs, is very "reactive", meaning that you spend a lot of time hot-potato-ing tasks on an event queue: - user message arrives at queue, hit LLM, LLM wants to run a tool, LLM idle while tool executes, tool execution finishing prompts LLM again - LLM triggers durable workflow, workflow runs over seconds, minutes, hours, hits callback to LLM, LLM meanwhile idle All of this time spent idle is free capacity. I presume most APIs already handle this kind of batching and scheduling (if you've ever experienced Claude or Z.ai stuttering, I think this is a symptom of this batching and scheduling). Numbers wise, I obviously can't divulge company info, but the order of magnitude is somewhere around 10^5 conversations monthly. Horizontally scaling is a no-brainer, because adding another GPU is a drop in the bucket compared to previous token costs on API.