For my setup, I already have (4) CX3 non-pro cards. I needed to get a single CX4 card in order to support the mlx5 driver that is now bundled with macOS and my macs can be on the network now via thunderbolt.
I am interested in setting up a simple point to point network at home where the following topology diagram gives you a good idea of what I'm working with:
Yes I have a lot of slots here for NICs I have not obtained yet, anyway, as you can see I am thinking of making a poor man's switch out of the threadripper over here. And I'll be able to load up the rest of its PCIe lanes with NVMe storage. Unfortunately the fastest systems also have GPUs in them that occupy the large PCIe slots so only the 5600G and my two X99 systems (Yeah i don't recall the model number on that Xeon) will be able to benefit from having greater than 4 PCIe gen 3 lanes for getting full bandwidth out of these NICs. But that is okay.
I am interested in learning how to go about implementing RoCEv1 which I understand to be a layer 2 protocol and I think that the limited needs of my home network will make it so that I can "get away with" RoCE v1 to achieve RDMA acceleration. Please correct me if I'm wrong but layer 3 and routing doesn't seem like something I'll need or want with something like this.
I still have lots of things I do not understand and am not familiar with.
I am interested in setting up a simple point to point network at home where the following topology diagram gives you a good idea of what I'm working with:
Code:
┌──────┐ ┌────────┐ ┌──────────┐ ┌────────┐
│M1 Max├──TB3──┼─MCX455─┼─┐ │ TR 1950X │ │R9 5950X│
└──────┘ └────────┘ └─┼──MCX354──┼───┼─MCX354 │ ┌──────────┐
┌────────┐ │ │ └────────┘ │R7 5800X3D│
│R5 5600G│ ┌─┼──MCX354──┼───────────────┼──MCX353 │
┌───────┐ │ ??????─┼──┘ │ │ ┌──────────┐ └──────────┘
│ 5820K │ └────────┘ │ │ │ E5-2696? │
│ ?????─┼───────────────┼──??????──┼──┼──?????? │
└───────┘ └──────────┘ └──────────┘I am interested in learning how to go about implementing RoCEv1 which I understand to be a layer 2 protocol and I think that the limited needs of my home network will make it so that I can "get away with" RoCE v1 to achieve RDMA acceleration. Please correct me if I'm wrong but layer 3 and routing doesn't seem like something I'll need or want with something like this.
I still have lots of things I do not understand and am not familiar with.
- Is RoCE a thing that you turn on, provided both ends of a link have supporting equipment (typically entailing both NICs and switch support it, in my case, both NICs) and then all of a sudden the regular ethernet connections between the machines become RDMA enabled? If it is not this simple to reason about, then what would be a sensible way to reason about it?
- What software would I use to implement bridging to implement switching on the threadripper node? If I work out how to enable RoCE would bridge-utils/brctl Linux software bridge then gain RDMA capability and allow for CPU offload for traffic that traverses the threadripper?
- Is there RoCE support for Windows (which would be running on the 5800X3D system) on CX3 drivers?
- Obviously one thing to consider is to not load up NICs in the threadripper, get a 40Gbit switch instead. I think the main concern here is just that it seems I would need to heavily modify a switch to make it not sound like a jet engine in my office. Yes if I do resurrect all these systems to make this frankenstein network and compute cluster of all my systems, it will be kind of loud and warm and having an enterprise switch might not be unreasonable to add. But I am interested now in if there is something useful/edifying I could do along the above lines. The fact that it gives my threadripper a sentimental second chance at life is a factor yes.
Last edited:
