Post #4263824

2026-07-31 02:56 UTC

@kini@maro.xyz @res260@infosec.exchange have either of you guys read the letter or plan A? I feel like you're having a conversation that's interesting but quite oblique to the biggest worries at hand (except for in the ways it relates to open source models, I suppose) I'm concerned about an ASI that's not aligned nor understandable to us; war between the US and China; US global dictatorship, etc. Based on the way you're talking, it seems like maybe you don't think those outcomes are credible or worth prioritising?

Replies (2)

@res260@infosec.exchange 2026-07-31 03:20

@ikesau@micro.ikesau.co @kini@maro.xyz I would/will share these worries when open source models stop being shipped. It feels weird to say but right now China is basically helping us all by doing a lot of their work in the open. However, there seems to be this assumption that we currently understand how frontier LLMs work. We really don't. The amount and quality of emerging properties that came from refining training methods, increasing the number of parameters, etc. is very high. We do not really understand how these emerging properties came to be. So we go from "we dont understand how this thing works when w traeined it" to "we don't understand how this thing work when the AI trained it". Both of those things are terrifying because we don't understand it, not because of who trained it

Open ##4263821
@res260@infosec.exchange 2026-07-31 03:24

@ikesau@micro.ikesau.co @kini@maro.xyz but more specifically to your last question, misaligned models is the most pressing threat imo. Because of the amount of AI agents currently running and their growing capabilities, even if once in a billion runs an AI agent goes rogue it can do a lot of harm while its trying to accomplish its goal.

Open ##4263915