mastodon
4.6.6
A professional fixer.
I've been iterating with Claude+systing to convert a networking library we use from tokio-rs to io_uring, and man tokio-rs is really fast entirely because it fucks up Linux trying to be smart. io_uring is slow because it's so efficient and great, so every single stupid decision that's ever been made in the Linux kernel networking stack shows up in all of its glory and you get to eat a 40% performance hit while tokio-rs with its 6000 threads busy looping in epoll_wait just scream.
Wrote a blog post about the recent developments with systing. I’ve been adding a ton of features around enabling better Claude Code integration and it’s been very useful.
https://josefbacik.github.io/kernel/systing/debugging/2026/02/23/systing-1.0.html
I’ve been putting systing on the training runs a lot recently and that’s been super educational. I pivoted systing away from perfetto first to DuckDB in the last months because there’s always way more data than a human can easily analyze. Giving Claude a SQL interface to a whole bunch of data really speeds up the iteration loop for performance analysis and fixing. Perfetto is still very much integrated into systing, but more of a “sanity check what Claude is telling me” sort of thing.
When building a BPF based application, it’s best to strip the build steps down as much as possible, because building in Docker or some other constrained environment is quite annoying if you can’t get to bpftool or something. Commit your skeletons and your bindings (if you’re in Rust), and use libbpf/libbpf-rs to do the linking for you. Spoken by a guy who has been letting Claude stumble through trying to build a Docker image for systing for a day.
I did a talk about systing at All Systems Go. First time doing the conference, I had a great time! Unfortunately I had poor time management skills so spent a while talking about general tracing and debugging strategies and less time actually showing how I use systing. I’ll be recording a walkthrough of an example debugging session when systing next week to give a more concrete example of how to use it. The talk is on YouTube if you want to watch https://youtu.be/XLLWg3T7A-s?si=Wdh8iW7Awdy4G62b
To continue to beat this dead horse, I literally just used the Link: tag in a patch that was committed to fix a WARN_ON() to see if it was part of a series. It was, and now I can make sure the whole thing is backported. Just because some people don't find it useful doesn't mean it's useless.
Today is my last day at Meta. This has been the best team I’ve ever been on, and I’ve been on some great teams. Next week I start a new chapter, I will be joining Anthropic to help them scale out their infrastructure and put my decades of kernel and systems experience to use. I will be stepping back from kernel development as my primary job for the first time in my career. I’m sad to leave my colleagues, but I’m excited to try something new and see where it takes me.
You know we wouldn't need Link: and other hacks if our development process took advantage of any of the modern development tools that have come out in the last 20 years...
If anybody else is having odd trackpad behavior on their 13 inch Framework laptop, I think I figured it out (at least for me). If I've got it on my lap and I'm typing, sometimes It's in an awkward angle and the pressure for my palm is flexing the case slightly so that the trackpad won't "click", so it gets stuck. Take your hands off the laptop and press around on the trackpad until it clicks to unstick the mouse.
A thing I'm quite tired of seeing in technical spaces are exceptions being made for engineers because they are "smart". Widely known as the brilliant asshole.
This is code. It can be difficult, doing it well is even more difficult. It's also table stakes for this job. Especially when it comes to OSS. We are not so unique that we can't be replaced with another coder. I have replaced myself with very competent engineers at Meta in my work with btrfs.
What makes "smart" engineers is their ability to work with other people. This fantasy that "ideas" are how we communicate and the best idea wins, the most technical argument wins, is simply not true. You get your code in because you know how to communicate. Does that include technical arguments? Of course it does. But if you bring that technical argument to the table with "I can't believe you were so stupid to do it this other way, you should all thank me for being here" you are going to be far less successful.
That doesn't make you smart. To me, "smart" includes all the things necessary to accomplish your task as quickly and efficiently as possible. Part of that for us is coding, but the larger part is communication and the relationships we build with each other by being consistent with how we communicate and treat other people.
I work with the smartest people in the world. They are smart because they can code. They are smart because they are kind and gracious with their communication. They are smart because they build community in the work that they do.
If your community is a developer of 1 and nobody wants you around, it doesn't matter how good of a coder you are. You have failed at one of the core tenants of your job. In the real world, with real stakes, real bosses, real accountability, you would be fired. And that would be the correct outcome.
The power of OSS is the fact that it's many developers working on a thing. We all witness the power of this every day, but still cling to this fantasy that it's one smart asshole that keeps the whole thing together.
We are all replaceable in OSS. That's the beauty of it. It will outlast every once of us.
Systing has grown a variety of features over the last month, the coolest thing is Jonathan Wiepert’s integration of Strobelights BPF based python stack support. You can now get perf-style python stack traces with systing. This integrates into the whole stack, so if you have native code the python calls into you will get the python part and the native stack combined properly.
Some other more advanced features have been added. The ability to specify a trigger event and then continuously trace until the event occurs was added for finding rare events. Systing will buffer N seconds of trace and dump the trace on the trigger event, keeping the trace file small.
I’ve also fleshed out the event context configuration so you can specify arguments to collect from trace events or kprobe/uprobe to more information in the trace. This is very basic right now as it only allows for one argument and it attaches the value to the event name, but this will be fleshed out later to be included in a context track and allow for multiple values.
You can clone and build from the repo, there’s always changes going in. With the python stack support being merged I feel comfortable moving towards distributed packaging for wider distribution.