Infiniband or iSCSI

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

MBastian

Active Member
Jul 17, 2016
382
127
43
Germany
Currently I am planning an update to my NAS and have a JMCD NAS12S4 on the way to replace my old case as I need two more disks in my ZFS pool.
Through a lucky windfall I can get six or even more 3.2 GB SAS SSDs for free so I pondered if it wouldn't be cool to have some seriously fast SAN storage to my workstation and it's VMs. To make that work I would need to replace the current ITX Motherboard I have in my old setup to gain the necessary x4 PCI 3.0 slot for a NIC.
Now the issue: For ASPM to work on my NAS box I can't buy ConnectX-3 cards as they do not support it. I could buy a ConnectX-4 card but they are rather expensive, especially if they are Infiniband capable cards.
Since this is just fun and games for me ... literally as I plan to put my Steam, Gog and Epic libraries on a volume on the SSD pool so I can use them on my Windows VM and also when I boot native Windows. I am unsure if having 40 GBit Infiniband is really worth it, maybe an iSCSI device over a 25 or 40 Gbit Ethernet link would be good enough?
Anyone knows a good enough el Cheapo version that supports ASPM? Both my main Workstation and the NAS Box are in the same room. The rest of the House can work with 1Gbit Ethernet or WIFI 6e.
 
Last edited:
hello,
1) if you buy connecx4 (not lx version) it can be reflashed to vpi mode (ethernet+infiniband)
2) on any connectx (3/4/5 pro/lx/ex etc) you can use RDMA over ethernet (RoCE v1/v2) on iSCSI (iser)
3) as i remember correctly connectx does not support aspm
 
  • Like
Reactions: MBastian
ASPM does not work on ConnectX period. The Intel X710 is one of the few cards which work well.
 
  • Like
Reactions: nexox
Thanks for the input. I never had contact with SAN components so I am a bit overwhelmed.

2) on any connectx (3/4/5 pro/lx/ex etc) you can use RDMA over ethernet (RoCE v1/v2) on iSCSI (iser)
While Infiniband sounds fascinating I think iSCSI is fine for my purpose. I think two ConnectX-4 LX dual port cards connected back to back with Balance-RR for 50G would be the best/cheapest option?

ASPM does not work on ConnectX period. The Intel X710 is one of the few cards which work well.
As far as I could gleam from different sources most ConnectX-4 cards do sort of support ASPM, in the sense that they do allow the system to go to C3 or lower. Some people complain that it only works in PCI slots provided by the chipset not the CPU but that is ok for me. I do not mind the ~12W the card itself takes.
The x710 does not have RoCE but several offloading features and it is more expensive. Anyone has some input how much load it puts on the CPU at 25g or 50G?
If I where running my server and workstation 24/7 I'd consider x710, but I do not need to modify or write new files most of the time. I've set up a read-only mirror of my families most frequently used data on a single 5TB USB disk connected to my M920x acting as Firewall and internal services host.
 
Last edited:
Some people complain that it only works in PCI slots provided by the chipset not the CPU but that is ok for me. I do not mind the ~12W the card itself takes.
hello,
yes that is
please take into account:
1) connectx4-lx is more energy efficient i.e.
- from offical docs Specifications
- from user experience Connectx-4 Lx 2x 25GbE with ASPM support + idle power consumption measurements
2) connecting via chipset can utilize just 32 Gbps (theoretically), on practical perspective you will have about 25-27Gbps, so there are no advantages with lacp/balance-rr
 
Last edited:
  • Like
Reactions: MBastian
2) connecting via chipset can utilize just 32 Gbps (theoretically), on practical perspective you will have about 25-27Gbps, so there are no advantages with lacp/balance-rr
Thanks, I totally ignored that. At least in Europe a ConnectX-4 LX card starts at 49€ per piece and a 3m SFP28 Copper DAC is at 30€.

Edit: On Aliexpress prices are lower due to a sale event. As far as I remember fake Mellanox cards aren't a thing, so it should be safe to order from a seller with good rep?
The card I linked above comes with a low profile bracket. Ok, so I just order two cheap full height brackets ... then I saw the prices. Are people insane? Nearly 20€ for a piece of metal if you want to buy inside the EU?!
 
Last edited:
Some ConnectX cards have ASPM. My Supermicro-branded Cx4 Lx dual 25GbE cards support it but only on newer firmwares. My CX354 dual 40GbE Mellanox-branded card says it is supported but disabled. Also some CPUs seem to be able to do lower C-states without working ASPM.

Ironically I also bought one of the Supermicro 25GbE Broadcom cards because it explicitly says it supports ASPM in the spec sheet, but it didn't.
 
  • Like
Reactions: MBastian and nexox
Edit: On Aliexpress prices are lower due to a sale event. As far as I remember fake Mellanox cards aren't a thing, so it should be safe to order from a seller with good rep?
almost all of my cards (about 20) are from ali, and 4-5 from ebay, connectx-3/4/5/6
the same about transceivers and fibers, as well as switches
 
  • Like
Reactions: MBastian