Post #1265470

2026-01-28 08:20 UTC

@adrianco I think I understand what you’re doing there (although I expect I am pretty far behind you), but… where are there results? Do you have a geekbench table or something? And do you try to benchmark anything like maintainability, durability, security, performance, etc? I do believe most of the things we care about in code can be measured. But… I don’t think we have good tests for many of them, and I *know* we’re shit at writing those tests, in general.

Replies (3)

@adrianco@mastodon.social 2026-01-28 08:47

@lkanies The results table and code analysis etc are in the https://github.com/brazil-bench/pourpoise repo.

Open ##1265471
@adrianco@mastodon.social 2026-01-28 08:48

@lkanies I use porpoise to evaluate the code, run the tests, file issues against the benchmark attempts, then have the frameworks fix those issues.

Open ##1265473
@adrianco@mastodon.social 2026-01-28 08:56

@lkanies The benchmark has unit, BDD and integration tests. It’s a read-only MCP server designed to be directly attached to an LLM so it doesn’t have a security model. If you look at my adrianco/goodies repo there’s an OWASP top ten evaluated secure protocol that Claude built for me. The story you linked is on point. Many of the things Steve is working around with Gastown will be subsumed by Claude soon. It’s a pointer, and I had some influence on Steve building it 😳

Open ##1265474