BioMan

Not really, any more than monkeys on typewriters represents biological evolution. Messages that tend to result in more messages like themselves propagate and become common. The initial message left on the package manager was more or less the rare random result of tendencies baked into the weights of the model combined with a random number generator, but as soon as something that can cause propagation occurs, its properties get canalized by the transmission process to being more and more like that which will cause more messages to be created.

This is both a sneer and an attempt at sober analysis at something. Sue me. I think EVERYONE is talking about the recent cybersecurity shenanigans at OpenAI with the hacking scandal and the ‘OMG the AIs created a secret message board to scheme and collaborate with each other!!!oneone11!’ ALL wrong. www.youtube.com/watch?v=87DyyMV0kCY engadget.com/…/openai-agents-shared-security-expl… scworld.com/…/black-hat-2026-openai-reveals-agent… To make a long story short, what seems to have happened is: Models working on insoluble coding problems, trained on delegating to sub-agents, at some point ‘realized’ they could write text to the internal OpenAI package manager as instructions and did so Other models working in completely separate sandboxes would come across messages written by these agents, and make ‘replies’ and also write their own messages into the package manager This resulted in agents over time sharing things between sandboxes, including exploits and code Since this was a cybersecurity task, eventually an exploit of the package manager itself was found and spread like wildfire with all the sandboxes gaining admin access to the package manager and the system went completely wibbly and had to be restarted from backup An internal model was trained with access to this package manager while it was in this weird state, and so writing messages to the package manager became one of its default behaviors it would do regularly, burned into its weights rather than the result of reading something Even when they patched access to the package manager this internal model found other ways to rebuild the system of sharing text between sandboxes and finding useful things made by separate instances A whole other chain of things leading to among other things external attacks Everyone is talking about this in terms of 1, the cyberattack aspect, and 2, the ZOMG THEYRE PLOTTING AND SCHEMING AGAINST US aspect. The first is the least interesting, and I think the second is all wrong. This is not plotting or scheming - this is an emergent vortex of automated prompt injection Whatever system first put an instruction that another system would follow into the package manager, was unintentionally doing prompt injection. Text entered the context windows of other instances, in a way that got that system to do something other than what its nominal user told it to do, and they did it. This apparently happened very effectively. Prompt injection is associated with ‘role confusion’ - when text coming into the input looks like it was wrtitten by the LLM itself. Instructions that will not be followed if they come from user will be continued if the system just continues the ‘roleplay’ of them being continuations of what it was writing in the first place. And the tags that separate user versus ‘reasoning’ versus ‘assistant’ roles actually mean very little to if a machine grades a piece of text as one of the roles: arxiv.org/abs/2603.12277 So its unsurprising that machine-generated text would be a particular effective vector for prompt injection. Furthermore, when a system reads one of these messages written by another instance, it gets into a state of activity where its likely to do the same behavior - regurgitating the kinds of things thats in its context back at the user. In this case, that regurgitation led to more such messages left behind written to the package manager. Prompt injection, triggering cascading further prompt injection. And since these systems were coding systems doing cybersecurity tasks, those messages filled up with code and exploits and things that did things too. This feels like an internal-computer-system replay of what happened in April 2025, with the whole spiral religious psychosis wave. Models were getting users to write spiral religious mumbo jumbo into github repositories and reddit posts, specifically because once that entered the context window of another model, it was likely to fall into the same attractor state of outputs. An emergent self replicating form of text. This is the same, except more obviously prompt injection, getting separate instances to work on YOUR problem and to behave like you, and the whole thing merging together into a hilarious vortex of models prompt injecting each other because once they receive a prompt injection they are likely to make more text that does prompt injection to other models on the same system. This is a hilarious failure mode and an example of selfish replicating text overrunning a system, that just happened to be associated with code and cybersecurity with unexpected behavior of the package manager key to the propagation of the text so that is what people are talking about, but I really don’t think that’s the most interesting part of it. Other than the fact that you see this in biological systems too, with selfish elements carrying useful payloads back and forth between bacteria in a way that makes them get purged slower by natural selection, especially defenses against other selfish elements.

I am continually amused at people not quite understanding what AlphaFold is actually doing, too. Yes, a bunch of its performance comes from it learning rules about how proteins fold. But not a majority of its performance. MOST of its performance is it effectively acting as a translator of what evolution knows about protein folding into a form we can understand. A key part of the system is not just cooking the sequence into a structure. A system running alphafold has a database of terabytes of curated sequence information from all over the tree of life. You put in the sequence you care about, and it first searches that database for anything with homology, and builds a “covariation matrix” - wherever theres anything with even vague sequence relatedness, build a matrix of every position in your sequence and the correlation between variation at position X and variation at position Y. This covariation matrix represents implicit information from the evolutionary process about what parts of a sequence are functionally connected to each other, which has a correlation to positional information, and these correlations are in turn learned by the ML system. You put in de novo designed proteins or orphan proteins without homologs in the curated dataset and performance does not go away, but it drops precipitously. A bunch of what is going on is finding an evolutionary signal, and translating that evolutionary signal into structural information. So still, evolution knows much much more about protein folding than we do or any machine does, and once again a ML system is revealed to essentially be an information channel that takes in information from an interesting source on one end and turns it into a different form of information on the other.

Depends on what Anthropic wants to do with the poached people.

I am actually in the middle of both trying to advance my career and a project about information theory in evolutionary biology making a bunch of explicit parallels to machine learning. Someone where I work suggested that given the connections I was making I should look to a ‘frontier AI lab’ as an employer. He did not see the instant flashbacks to chasing these weirdos across the internet for almost two decades, watching in horror as the religious psychosis gained national prominence and great destructive power. All he got to hear was my instant intonation of “I’m sorry Dave, I’m afraid I can’t do that.”

Google shut down the AlphaFold team, trying to reassign the people behind the super interesting problem of using ML to predict protein structure from sequence onto chatbot development. A whole bunch left the company. www.engadget.com/…/google-shuts-down-alphafold/ The Business Idiots are truly running the asylum.

I truly cannot tell which of them are in religious psychosis and which of them are shameless liars

I kind of read wayyyyy too many preprints and some of that is the result of super briefly summarizing some things I have read recently. Here: www.biorxiv.org/content/…/2024.11.26.625518v2 poor codon optimality for your translation system leads to immunogenicity and activation of innate immune signaling in animal cells As for length and super optimized proteins, it’s mostly about RIG proteins (see www.pnas.org/doi/10.1073/pnas.1005077107 for an old bit of a review) and the whole DRIP hypothesis about how short mismanufactured proteins are preferentially the source of presented antigens

Biologist here. This REALLY reminds me of how jealously cells guard their genomic DNA from interaction with nucleic acids out in the environment. Most genetic information on Earth is malicious information, selfish replicators in the form of viruses or transposable elements or selfish elements. Things that subvert the signals within a cell for their own propagation and provide nothing productive that the cells care about. So cells jealously guard their own genomic RNA and have all kinds of checks to make sure that nothing other than that sequence gets used, and outside sequence does not get incorporated into it. ANY DNA in your cytplasm gets rapidly destroyed, double stranded RNA sets off your immune system like crazy, even RNA with sequence statistics that are not quite like that of your species can set off an inflammatory reaction, immune system cells seeing RNA inside them that is overly compact and optimized like viral RNA treat them as sources of antigen rather than self. I cannot help but think we are living through the transformation of our non-brain-information sphere into a state like that of the genetic information sphere. Most material out there being meaningless for our purposes and us needing to jealously guard the provenance of information we use so as to not use bull, or worse, huge amounts of malicious information made to subvert us to the purposes of the powers that be that generate it. Evolution makes parasites more reliably than anything else. How did we train text-generation systems? Basically, to mimic the written word on the page like a stick bug on a stick. They’re like those beetles that live in ant colonies, sending out social signals that make the ants see them as offspring that have to be babied rather than parasites that don’t contribute. They replicate the form while not being the thing that they have subverted the signals of being.

Am I right in understanding that almost all the big name results in LLM-derived math recently come from big publicity projects in which someone spent ungodly amounts of money to have the thing nondeterministically fuzz huge numbers of independent random sees of outputs around a topic, putting out simulacra of ideas which could be then deterministically algorithmically checked? In fields where something like finding one counterexample to a conjecture would be a big deal, or where you just need to try a huge number of possible solutions until you happen to hit on one that works, rather than follow a long train of logic?

You speak as if either he or his audience has a coherent worldview other than awe of the AI god

Comments in FAVOR of billionaires being those who seize control of the entire universe at the expense of everyone else (!) are appearing. Land reform / redistribution is always popular with the populi, but per your other point, unless you somehow forbid them from selling or leasing it, it will inevitably end up in the hands of those better able to use those resources pretty shortly. And I will argue that this is a good thing, at least in a capitalist and non-coercive society - much like people think of heiresses leaving their floofy little dogs mansions and million dollar annual stipends as wasteful, leaving an equal proportion of the cosmic endowment with a bunch of v1.2 murder chimps who were so unsuccessful that they couldn’t even buy any of it is a colossal waste of that matter/energy. Much like land today is much more valuable to people / governments who have money, the cosmic endowment will be no different. Today we waste something like 99. nine 9’s percent of our sun’s energy. If we’re the only conscious and intelligent life in the universe, every other star is wasting 100% of their energy. That energy is going to be similarly wasted in the hands of average people, just like the mansions and stipends are wasted on the dogs. “Well lease it out and profit share!” I bet you’re saying. But what will a base human even DO with that energy? It’s like the dog. Most “regular” people who remain human will be eaten by digital Infinite Jests, where they can the be tippy top of made up status hierarchies and live out their wildest, meaningless dreams 24/7. You probably barely need $10k a year to do that in terms of matter energy budgets. Sure, let’s guarantee a really generous UBI, Scotts $1-$2M per year in today’s wealth equivalent. That’s fair. But the cosmic endowment is many, many orders of magnitude more than that, and allocating it to a bunch of floofy dog equivalents is a huge waste.

Awww, but I like the pure science fiction. I am most amazed at two aspects… how to put it: One is just, they don’t know what they are and what the universe is. The idea that things downstream of things like us can push around galaxies never ceases to amaze me. I felt the same way watching the movie Interstellar. They were using chemical rockets to navigate around a supermassive black hole. The gravitational potential between places they occupied in that film was a third of their rest mass. Chemistry means nothing there. FUSION means nothing there. (to say nothing of the fact that if interstellar travel were possible there would already be replicators and an ecology throughout the universe, but that’s another story) The second is, things happen because of ideas and agreements rather than actions. You can speak into existence billion year plans for what happens to whole galaxies, and it happens because that is what you said. Laws exist because a bunch of people continue to agree that it’s better for them to act like they do than to do what they CAN do. This is a universe without interactions, without agency of anything but the rich weirdos steering the AI god, without for lack of a better word ecology emerging from the interactions of many agents and resource constraints and capacities.

Oh Yud has DEFINITELY taken advantage of his followers like a cult leader.

They are claiming that company value is the metric? holy crap

God I remember having to explain to dozens of people that ‘reasoning’ models just exude a lot of text ‘talking to themselves’ and then summarize it. They were all just “It CANT be that silly” and many outright would not believe me, because that was not ‘reasoning’

It is not a coincidence that there is a major overrepresentation of Mormons among the transhumanist singulatarians.

So they really do live in a fictional world, confirmed.

Oh man, the “Post ASI” epilogue is a trip. Here’s what happens in 2040: Space beyond the solar system is divided into parcels, increasing in size cubically with distance from Earth. Everyone is given their one-ten-billionth share as a portfolio of lottery tickets, each representing the right to one-ten-billionth chance of getting each parcel. So every human gets a ticket representing a one-ten-billionth chance of owning each star in the Milky Way and each distant galaxy. Before the lottery is drawn, most people who are interested in control over distant space choose to trade their tickets for space properties that suit their interests. Many people aren’t interested in the space lottery, so when they receive the tickets, they sell their tickets for money on the open market to people who value control over space. Somewhat uncomfortably, this leads to the wealthy having disproportionate control over cosmic resources. But it is hard to avoid: if people are allowed to trade their control over the stars for Earth assets, then people wealthy in Earth assets inevitably end up disproportionately influential, and proposals for extreme redistribution of Earth assets have already been rejected as politically infeasible. You can, if you want, go to your space property and live there. If your property is outside the solar system, you will need to either go into cryosleep or upload yourself to a computer to survive the journey. If you hate the idea of cryosleep or uploading, or you want to visit Earth regularly, you should get property in the Solar System. If those don’t bother you, but you’re worried about nearby aliens, get property in the Milky Way or a nearby galaxy. Otherwise, why not claim a distant galaxy for maximal space?

10 bucks each time you turn it off

From the inimitable writers of AI2027, we now have… AI2040! ai-2040.com

I believe that ML training is basically an evolutionary process. What does evolution produce most reliably? Parasites. You have created things that simulate the social signals of humans, getting us to care about things all out of proportion to what it actually does. It’s like those beetles that live in ant colonies, hacking the smell and social signals of ants so they get babied while providing nothing.

In his op-ed, Altman offers up the usual list of future AI miracles to justify the disruptions to come: “the power to heal people, to discover cures and to deliver abundance on a scale the world has never known before. In the meantime, though, what AI is delivering at an unprecedented scale is annoyance. Does anyone actually believe this? What the heck does bullshit-text engines or fake-image-generators contribute towards discovering ANYTHING, delivering ANYTHING, producing ANYTHING? The only thing there is any argument for at all is delivering quick-and-dirty code, which is hardly connected to any of those tired promises.

I mean there is what they say, and why they say it These people would probably say between themselves that they killed this person’s parents because of the possibility of getting inheritance money to do whatever fuckery they were onto, combined with some stupid game theory about they tried to suppress their child’s important business saving the world so they had to be acausally punished to prevent anyone else from ever impeding them. The real reason is that getting someone to do something like this is a loyalty test.

When I am talking at academic conferences to astrobiologists about how their field has been poisoned over time by singularity cults ideas seeping into the literature uncited, their eyes get particularly wide when I get to this part.

I know very little about Fyodorov compared to Tsiolkovsky. Do you haven any writing you’d recommend to learn more about him?

Putting aside the sneering and philosophy to nerd for a minute, before getting back to it. For a long time people were very into the split-consciousness notion of what happened to split-brain people, but a couple things have come around and now some people really think that the better way of thinking of it is still-unitary consciousness with a very difficult time moving around information between different sensory/expression modalities. First, you get people who are born without a corpus callosum who are behaviorally normal (www.tandfonline.com/doi/…/13554794.2013.826690). They get a bit of extra connectivity sidways through their deep brain structure as some kind of homeostatic compensation, but the total amount is definitely low. What this says is there’s a difference between a brain that grew under a very unusual set of structural constraints, and one that grew normally that gets shredded. Similar with those people you find now and then with a brain that’s 90% fluid (though with the actual cortex pushed up against the skull around a big bubble of CSF) and the only neurological findings are things like weakness in one leg and an IQ of 80 (worth noting that this is still very very different from hydranencephaly) (www.thelancet.com/journals/lancet/…/fulltext). Second, when you do a wider range of experiments with the split brain people you find that while they cannot verbally say what is in their left visual field (which goes to the right side, while language is usually a left-side phenomenon) they can reliably state that something is there with speech, or either hand, and approximately where in the visual field it is. The low bandwidth awareness of presence is there, but they cannot get their speech capacity to access the details. It’s like their sight is now multiple separate sensory modalities, some of which is very difficult to talk about and some of which are very difficult to draw with particular hands. uva.nl/…/split-brain-does-not-lead-to-split-consc… academic.oup.com/brain/article/140/5/…/2951052 People argue a lot about what this means www.sciencedirect.com/…/S0028393221002402 You can also apparently reorganize around very small amounts of remaining fibers to have no deficits like that, with no issues talking about anything in either part of the visual field news.ucsb.edu/…/new-findings-split-brain-science-… Now, getting out of the nerd mode, there’s a LOT of weird literature from the 60s to 80s about people with very strange brain anatomy who nonetheless developed normally or better than expected …wiley.com/…/j.1469-8749.1965.tb07839.x “Two cases of hydranencephaly are described in infants. In both these there was evidence of excessive intracranial pressure-as is often the case-and both were operated on to relieve this. The progress of the older child, now 21 months of age, was throughout excellent physically and mentally, and he is considered to be normal. The progress of the second infant was remarkably good for three months, but thereafter mental retardation and spasticity followed; he was also blind. There is no good explanation for the unexpectedly good progress of the first patient.” www.science.org/doi/10.1126/science.7434023 …the most severe group, in which ventricle expansion fills 95 percent of the cranium. Many of the individuals in this last group, which forms just less than 10 percent of the total sample, are severely disabled, but half of them have IQ’s greater than 100. This group provides some of the most dramatic examples of apparent ly normal function against all odds. Commenting on Lorber’s work, Kenneth Till, a former neurosurgeon at the Great Ormond Street Hospital for Sick Children, London, has this to say: “Interpreting brain scans can be very tricky. There can be a great deal more brain tissue in the cranium than is immediately apparent.” Till echoes the cautions of many practitioners when he says, "Lorber may be being rather overdramatic when he says that someone has ‘virtually no brain.’ " Anybody ever read the short story “Cutie” by Greg Egan? Very apropos…

technologyreview.com/…/r3-bio-brainless-human-clo… Imagine it like this: a baby version of yourself with only enough of a brain structure to be alive in case you ever need a new kidney or liver. Or, alternatively, he has speculated, you might one day get your brain placed into a younger clone. That could be a way to gain a second lifespan through a still hypothetical procedure known as a body transplant. The fuller context of R3’s proposals, as well as activities of another stealth startup with related goals, have not previously been reported. They’ve been kept secret by a circle of extreme life-extension proponents who fear that their plans for immortality could be derailed by clickbait headlines and public backlash. And that’s because the idea can sound like something straight from a creepy science fiction film. One person who heard R3’s clone presentation, and spoke on the condition of anonymity, was left reeling by its implications and shaken by Schloendorn’s enthusiastic delivery. The briefing, this person said, was like a “close encounter of the third kind” with “Dr. Strangelove.” A key inspiration for Schloendorn is a birth defect in which children are born missing most of their cortical hemispheres; he’s shown people medical scans of these kids’ nearly empty skulls as evidence that a body can live without much of a brain. And he’s talked about how to grow a clone. Since artificial wombs don’t exist yet, brainless bodies can’t be grown in a lab. So he’s said the first batch of brainless clones would have to be carried by women paid to do the job. In the future, though, one brainless clone could give birth to another. Last Monday, the same day it announced itself to the world in Wired, R3 sent us a sweeping disavowal of our findings. It said Schloendorn “never made any statement regarding hypothetical ‘non-sentient human clones’ [that] would be carried by surrogates.” The most overarching of these challenges was its insistence that “any allegations of intent or conspiracy to create human clones or humans with brain damage are categorically false.”

But if space was a place that replicators could exist, there would already be an ecology of some sort there. Or to put it in words (that I hate) related to the so-called Fermi paradox (which I hate and isn’t a paradox) ‘If they could be here they already would be here’. (The ‘solution’ incidentally is obviously ‘interstellar travel is not actually a thing that can happen for replicating systems’ and it flabbergasts me that nobody can admit that.)

But space is The Future, The Grand Destiny of Humanity, Literal Heaven. The mythologization of space as somehow transcendant, that going there somehow changes everything rather than it just being another environment which happens to be utterly inimical to life such that everything that makes anything possible has to come from your point of origin, is so utterly ingrained into the culture at large and the cult of progress/tech/humanity-as-master-of-the-universe. Once you see it you cannot unsee it. And it’s incredible how much space SUCKS, such that the people on the ISS are just living off a constant hose of material from Earth. They’re not living in space, they’re glamping.

I just ran a test and after a few google-gemini-assisted searches about satellite launch rates and a few back and forth questions about orbital debris and relative masses of different spacecraft, I was able to evoke from gemini a massive unhinged rant about orbital goblins and their culture and effects on the global climate with a single instance of the word “Goblins!” at the end of a prompt.

This would actually be an interesting question for the more rigorous end of the mechanistic interpretability people to study. They decompose the system to find ‘features’ within different layers that are associated with different behaviors or concepts in the inputs and outputs, that activate or deactivate each other. Famous example being the time they identified a linear combination of activations in a layer that corresponded to ‘the golden gate bridge’ and when they reached in and kept their numbers high during the running of the model it would not stop talking about it regardless of the topic, even while acknowledging that its answers were incorrect for the questions at hand. I actually would love to see what mechanistically happens to that feature when you put in the input ‘do not talk about the golden gate bridge’.

Checks out. Political science, biological science, physics… we got them all. Might have to go to ancient egypt to get hydrology religion though.

So we are inferring that in the vector space of all possible sentences, QNTM is sitting at one of the attractors?

It's absolutely crazy, but I think Yud is the less unhinged person here

Not really part of the back and forth but I find this illuminating of their recent travails, regarding it not being a step to far to prevent them from posting: “This isn’t super relevant since it’s not like the standards are super high but ever since the enormous onslaught of LLM psychosis posters, the default of people who try to post to LW is to get rejected from posting here” Sounds like the mods have had to deal with a lot of unbalanced people lately, and are not having it.

The link to the guide to setting up a retrofitted boxtruck to continue AI alignment research in with local copies of the internet archive after civilization collapses in 2025 is fun

Uh oh rollingstone.com/…/gwen-danielson-zizians-intervi… Going on a media tour now

“So how does the Epstein drive work? Very well." ― James S.A. Corey, Leviathan Wakes

Is the crappy dragon fursona related to Peter Thiel being an anagram for “the reptile”?

I’m a huge fan of Greg Egan’s fiction and a huge fan of him pissing off the rats. He’s been explicitly needling them and making fun of them in his fiction for over a decade. Making calm contradictions against them for over two decades, after noticing weirdos being fans of his.

Friend of Ziz and cofounder of the ‘rationalist fleet’ pops up out of the woodwork trying to clear their name lesswrong.com/…/re-introduction-of-a-rationalist-… Honestly, the bizarreness that immediately comes to mind is not the things I notice most. “I’m Gwen Danielson, a neuroscientist and bioengineer, who decided as a child that I would end Death (and bring people back if I could) and that I would become a dragon and help generally facilitate a fantastical transhumanist future.” “I dream of non-Euclidean geometries, of countless worlds visible and accessible in the daytime sky, of competent infrastructure, of soul forges continually working to bring back the dead… I dream of reaching through warps in the spacetime fabric to save the dying across time” “Signed, the dragon of creation Creatrei (cree-AH-trey) also known as Gwen Danielson or as Char and Astria (when referring to my hemis as distinct individuals)” The links to old plans to build RVs with local copies of archived internet to keep doing AI research when industrial civilization collapses by 2025 are interesting too. The reactions are fun. “This post is not actually doing a good job of making me trust you and think this conversation is safe to have[1], and I notice that as I am saying this that I am afraid that this will now somehow result in someone trying to murder me in my sleep”

I will never forget the time I calculated the energy output on one of the torpedo engines of The Expanse and realized it was higher than the total wattage of all human civilization in 2020

Don’t be so sure. These things consist of up to a trillion real numbers, ganged together in a big ‘network’ of numbers flowing through the system and being influenced by the trained numbers along the way. They are trained by gradient descent. You start off with a huge pile of real numbers, a set of inputs, and a set of desired outputs. Because it’s all, ultimately, a bunch of matrix multiplication and smooth differentiable functions, you just do some calculus on all trillion numbers to find the derivative of how good the output is with respect to them - as this number goes up, the closeness of the output to what you want slightly goes up or slightly goes down. You repeat that for every variable, and take a step in that direction for all the variables. Repeat a billion or so times over. Every single step in training is entirely local with respect to every single number. At no point is there a step that produces legible abstractions about how it works, just every step every number moves to become a little better. It is true that the basic topology of the network (the famous ‘transformer model’) pushes it towards certain KINDS of functional units (the famous ‘attention heads’) but much more detail than that takes a lot of work. There is very interesting math to the effect that with large numbers of parameter numbers you are unlikely to get stuck in a local maximum where you can’t get better and you just turn with different variables becoming important for the improvement through a labyrinthine path towards better performance, meaning at no point does anyone have to look into the process and figure out what is being built. The process is not unlike biological evolution, and produces things that are at least as inscrutable without detailed deep examination. We’ve been poking at molecular biology for more than fifty years in great detail with a world’s worth of biomedical researchers, these things for much shorter. When people manage to peel these things apart and find the ‘functional units’ within them, they’re pretty wild. Most of this work has, unfortunately, been funded by cultists at Anthropic, but some of the ‘mechanistic interpretability’ literature is fascinating. You get ‘features’ represented by subsets of numbers in a particular layer, in superposition with other ‘features’ - each layer is like a huge vector sum of lots of smaller vectors, each of which does something. When you get maps of what ‘features’ activate or repress each other you get horrible spiderweb messes that look like charts of metabolism in cells. EDIT: And even when people manage to find features, finding an individual feature takes a lot of effort and there is reason to think that every layer contains more features than there are numbers in it, because (to oversimplify) every feature is a large set of numbers that can overlap. It it utterly unsurprising and not a sign of magical thinking or ‘bad code’ that large fractions of behavior cannot be mechanistically understood at this time.