Post #4227192

@scott@sfba.social @danielcornell@mastodon.social It was a test run in which the LLM was explicitly prompted to perform some pen testing and find vulnerabilities, and they didn't do it on an airgapped system but on one connected to a web proxy. It may have been on purpose, it may have been just overconfidence, but the LLM hacked the proxy to gain direct Internet access and tried to hack Huggingface to cheat on the pen test. It absolutely sounds like something a machine learning system would do. Just like that one artificial neural network that some researcher wanted to train to sort lists (to see whether it would rediscover an existing sorting algorithm or evolve a new one), but it actually learned to delete lists because that's the fastest way to arrive at a sorted list, since the empty list is always "sorted". It even learned how to hack its own machine to delete the test data it was supposed to sort. Any AI model will always try to produce something that technically fulfills the requirements, but it won't necessarily play by the rules. And if you give it any chance, it will lie and cheat, just because it can. And it doesn't even need a mind or consciousness for that.