Post #3090492
2025-12-18 16:16 UTC
@geerlingguy@mastodon.social Slightly strange:
A few minutes in there's a comment on how RDMA can only be enabled on 3 of the ports because of a naming conventions? CC @never_released@mastodon.social
That seems extremely strange to me.
Suspect it's time to update this post: https://gist.github.com/FCLC/6e0f0e79e9d4f5740573f09d7579eb72
Even with that limitation, you should still have 3 more TB5 ports on the M3Ultra devices.
Suspect the switch clustering side can be addressed using these: https://shop.atto.com/products/thunderlink-eu-model-5102-dual-port-thunderbolt%E2%84%A2-5-to-qsfp28-lc-optical-100gb-ethernet-adapter?_pos=7&_fid=69a565b99&_ss=c
Turns a pair of TB5 ports into a pair of ...
Replies (2)
-
@fclc@mast.hpc.social 2025-12-18 16:16
@geerlingguy@mastodon.social @never_released@mastodon.social ... QSFP 100 Gbe ports. Assuming bus bandwidth isn't the issue causing the RDMA enabled port limitations, the clustering setup then becomes 2 ports/device to upstream backhaul switching, and each Mac being connected All2All within a 4 device topology
-
@geerlingguy@mastodon.social 2025-12-18 16:42
@fclc@mast.hpc.social supposedly more may be possible. I need to ask more about that now that I got the video out the door