@darrenmoffat@mastodon.social @JdeBP@tty0.social @ska@social.treehouse.systems I've been waiting until I had some time and a keyboard to reply to this, since it won't be short. Before getting into the question, I'll first explain the problem from my perspective. Well problem*s*: 1) PAM interacts with PAM applications (that's sshd in this case) using a callback "conversation function". You give PAM a function pointer inside the application, and if it wants to talk to the user (eg to ask for a password), it'll call that function with a pointer to an array[0] of structs that say things like "ask for the password and don't echo keystrokes". As far as the application is concerned, it calls pam_authenticate() or whatever, and that call blocks, then if it needs to PAM magically calls back into the app's conversation function. In sshd the call stack looks something like: sshd main -> sshd event loop -> sshd auth kbdint-> pam_authenticate() -> ???? -> sshd_pam_conv. 2) PAM modules get loaded into the address space of the application, and can do pretty much anything without restriction. The application has zero insight into or control over those things. From the application's perspective, any PAM call (eg pam_authenticate, pam_setcred, pam_session, pam_chauthtok) requiring interaction *blocks*. If interacting with your user is simple (eg a blocking read from /dev/tty) then this is OK. In sshd, the pam_authenticate call blocks until it's done, but in order to be done it needs to interact with the user, and to do that it needs to run the event loop, because it needs to send the prompt, then process replies, all while doing various other things needed by the SSH protocol. The first attempt to deal with this was to have the conversation function call the event loop *again*. This was pretty icky. Another attempt to deal with this came from FreeBSD, where they put the sshd event loop and the PAM calls in different threads. This solved the blocking problem, but introduced a new one: all of sshd, PAM and every PAM module are all loaded into the same address space, so they all need to be thread safe. I'm not sure if FreeBSD could guarantee that for their PAM stack and modules, but we were not at all convinced that all of those would be true on all platforms. This lead @damienmiller@hachyderm.io to take the FreeBSD code, but shim the pthread_ functions to make it use *processes* instead of *threads*. This solved the shared-address space problem, but introduced new ones. Remember how I said PAM modules could do whatever they wanted? Some of them do things like stash secrets in auth to be used later, or set random supplementary GIDs in the calling process. Since the auth part was now in a separate process, this state wasn't propagated. That lead some OS vendors to re-enable the threading code and disabling the split process again. I think the define was USE_POSIX_THREADS. I changed that to the current UNSUPPORTED_POSIX_THREADS_HACK to make it clear that anyone using it was on their own. Over the years we looked at other solutions such as swapping stacks using makecontext/setcontext (which would likely work well but were deprecated by POSIX) or tricks like sigaltstack (which would probably also work and would be portable but hacky) but changing it would be a large undertaking, and would require *years* of debugging, support and maintenance. It's mostly only Damien and me maintaining this, we both have full time jobs, neither of which are this, and so far neither of us have had the time or inclination to embark on anything like this. So where do we go from here? Well the sshd process model has change significantly since then, so it might be possible to do better now, but again, so far neither of us have had the time or inclination to embark on that. If PAM implementations could agree on and implement a non-blocking equivalent of conversation functions (similar to AIX's native authenticate[1]), we might switch and even drop support for the old functions, but again, that'd be a large undertaking. So that's where we are now. Hope that helps explain how we got here. [0] or an array of pointers, depending on the PAM implementation. But that's a separate problem. [1] https://www.ibm.com/docs/en/aix/7.1.0?topic=authenticate-subroutine