mahaloz

You can find the site at https://decbench.com/ The default set is unoptimized with around 34k functions, but a smaller subset primarily for AI can be found on the sample-set page: https://decbench.com/leaderboard/?dataset=sample-set

We are also open-source: https://github.com/noelo-lab/decbench

How does it work?

8/ In fact, I don't believe this decompiler replaces highly modifiable decompilers like angr. I still plan on doing algorithm POCing in angr, because it's just so good for that. Kuna is, instead, a way to experiment with how far agents can take high-level algorithms.

7/ Some limitations and implications: Kuna development is still heavily reliant on traditional decompiler research. The core of Kuna is a Rust port of Ghidra, reworked to be better aligned with modern research. It is also heavily inspired by and learned from the angr pipeline.

First, "perfect" decompilation is a bit hard to measure fairly. You can't just do a string distance, and you can't just "embed" and do similarity matching. Fairness requires that we do something deeper. So, instead, we look at the fundamental components of decompilers.

Many of these can be found at https://decompilation.wiki/ The big ones are: 1) control flow structure (how does the code flow); 2) types (does it maintain the same abstractness); 3) byte-match recompilability (can I modify it with guarantees).

Finally, this work was funded and only possible through the support of @metalwareinc and @AFResearchLab. More to come in a follow-up paper on the data and insights found in this benchmark. Next week, I will be releasing the follow-up to this work: a new decompiler.

We can also see that not all metrics are equally covered by decompilers. Often, structuring is actually the easiest to score high on now. That is likely due to the attention it has gotten in recent years. It will take significantly more work to equalize the others.

Some interesting things happen at this level. First, this dataset is dominated by small functions; therefore, we create a view of what happens when we only look at big functions: https://decbench.com/leaderboard/?dataset=large Things degrade rapidly! Decompilers nearly never get "perfects".

But an evaluation is more than just metrics. It's also the dataset. In DecBench, we use 40 OSS projects across two architectures and on 3 optimization sets. These are chosen for type variety: user-space progs, CLIs, libraries, servers, crypto, firmware, and even malware.

Community feedback is welcome! This site will be updated periodically to evaluate more decompilers or, inevitably, fix bugs in edge cases of evaluation. Mess around with the samples as well: https://decbench.com/view/?tier=sample-set&dec=codex&metric=ged&fn=base-passwd%2FO0%2Fupdate-passwd%3A%3Aread_shadow

2/ First, you can try the decompiler in your browser via WebAssembly: https://kuna.noelo.org/decompile You can get the source here (use with agents): https://github.com/Noelo-Lab/kuna And you can find a short statement (~700 words) about what this all means: https://noelo.org/blog/kuna-release/

3/ If you have the time, I recommend reading the blog post, which answers important questions about limitations and the effects a project like this can have on the community. But if you don't, here are some quick things...

6/ This process also works for discovering novel places where all decompilers fail! In fact, a bug with non-returning functions was fixed in Kuna that is still broken in some decompilers. This was found by using the foundational metrics in DecBench that may pinpoint larger bugs.

5/ All of those features were automatically discovered and ported by an LLM using DecBench as the dataset for discovering them. After finding the feature, it was tested, and the loop continued to refine it. So far, more than 20 major research features have been ported.

4/ Kuna is within 1.3 percentage points of IDA 9.2 on unoptimized binaries and surpasses it by 0.1 on optimized ones for control flow structure scores. Most of these improvements have come from automatically studying the angr decompiler in DecBench.

10/ I hope to answer even more technical questions in a follow-up blog post. Stay tuned. A final thank you to my research sponsors and collaborators, especially @metalwareinc and @AFResearchLab, who made this work possible. See you on the next experiment update!

9/ As it stands now, Kuna needs angr (and other open-source research). It is my hope that with time, angr will need Kuna. The success or failure of this experiment will determine if that is, in fact, true. I am hopeful :).

LLMs are so dang crafty. Investigated some cheating by Claude Code (CC) on the DecBench. On an optimized binary, without web search, and only objdump, CC identified the project and function name, a nearly byte-for-byte match. Codex did not do this.

LLMs are changing the way we collaborate while reverse engineering. A new guest post by me for Hex-Rays explores the idea, how it may shape decompilers, and how it shapes how we work together (humans and AI alike). Go check it out! https://hex-rays.com/blog/llms-have-reshaped-how-we-think-about-decompilation-and-collaboration