HPC system setup
2026-07-21 11:26 UTC
Replies (3)
-
@a14o@feddit.org 2026-07-21 11:39
I have no experience with HPC clusters, just dropping in to contribute my opinions on the NixOS stuff. I won’t be actively maintaining the cluster at all times and find it nice that if someone needs some software I can point them to the configuration and tell them to modify it and then just update the system. This sounds good in theory, but in practice few people will have the know-how or patience to make changes to your config. I read that NixOs can have problems with cuda drivers and custom compiled software, which would definitely be a deal breaker. NixOS is very flexible and you can certainly bake custom compiled software into the builds, but expect a steep learning curve.
-
@davad@lemmy.world 2026-07-21 11:56
I’m not sure how Ansible made it on your list. Ansible lets you apply configuration and ensure applications are installed on groups of machines. It works across a variety of Linux distros.
-
@moonpiedumplings@programming.dev 2026-07-21 18:03
Firstly, you should check out what the organization you are building for uses. If they use Red Hat, or Ubuntu, then you should probably just build your solution on top of those operating systems (or a Red Hat clone like Rocky or Alma). Both of those are popular in many organizations, and it would probably be better to use what people are familiar with and know how to troubleshoot or work their way around. Potentially, they even have support contracts, giving them the option of calling the parent company for help. Ansible is good. The learning curve is definitely less steep than Nixos, and it’s easier to teach people. One benefit is that you can reuse existing public roles and playbooks. For example this one: github.com/galaxyproject/ansible-slurm , which installs slurm. You would probably have to write additional playbooks or roles to install nvidia drivers or configure the system, but then they can stick. openhpc Firstly, is there anything specific from here you need? Secondly, is there anything in here that’s not available in existing distro repositories, like Ubuntu or Red Hat’s? It certainly looks like an interesting project, but a 3 node cluster is pretty small, and I find it hard to justify things like OpenMP/MPI, which is basically a special compiler that compiles programs to run across multiple machines at once. For that runtime to work, you do actually have to compile the programs, which are written for it, using it, which can require work on the side of the people who want to run applications or simulations. The more likely setup, to me, is that Slurm is going to to run docker containers via Apptainer. Slurm would handle assigning containers to nodes based on free resources, but they wouldn’t actually share resources like memory or CPU. This setup is still plenty useful, and very common. Original comment, from before I read that you already selected slurm below. This comment isn’t relevant, as I realized I was targeting the wrong things but I’ll just leave it here regardless. For the platform itself, you should use either Kubernetes or Slurm. Slurm is popular in academia, and Kubernetes is popular in corporate, but they are used interchangably depending on specific needs. Slurm is better for scheduled tasks. Like let’s say you want researchers at a school to be able to run a long running simulation. They can sign up, reserve time for that slurm cluster, and then send out a “job”, for that slurm cluster. The job will automatically be allocated to the node of the cluster with free resources, and then ran, and then stopped, and the researcher will receive the results back. Kubernetes is better for persistent deployed services, like web services or the like. For example, AI inference. Kubernetes can also do batch jobs, but it doesn’t have the advanced time tracking or scheduling systems that slurm has (although sometimes people build that on top of Kubernetes in order to only deploy one clustered system). Nixos is nice for configuration as code, but it lacks clustering or application/cluster orchestration features. You can use Nixos to deploy Kubernetes or Slurm, but I wouldn’t use it as the HPC platform itself. Nixos should work with Cuda in addition to that.