Does anyone know 40+ Gbps RoCe V2 Nic below $100 ?

hyltcasper · Dec 4, 2020

Hi, I found ConnectX 3 cards from very cheap $100 in local seller.

But I need RoCe V2, only pro has the feature.

These cards are looking same. What happens if we flash pro firmware to ConnectX 3. Does it works like 3 pro?

Edit: I am looking a card with RoCe V2 40+ Gbps below $150 including shipping.

i386 · Dec 4, 2020

The conntect-x3 is using a slightly newer controller/asic.
Some people already tried to flash the pro firmware without succes, the cards didn't work after the flashing.

Rand__ · Dec 4, 2020

And $100 for a CX3 is not cheap... those go for $20 or less if you're willing to crossflash

hyltcasper · Dec 4, 2020

i386 said:
The conntect-x3 is using a slightly newer controller/asic.
Some people already tried to flash the pro firmware without succes, the cards didn't work after the flashing.

Thank you for quick information. What about connectx 4 firmware?

Rand__ said:
And $100 for a CX3 is not cheap... those go for $20 or less if you're willing to crossflash

I don't know what is crossflash. Which card go for $20 and where? I have the first day in this area.

Can I get Nvme-oF and Roce v2 in $100 budget? I want to setup all NVME storage clusters with Microsoft S2D.
Or is the Roce v1 enough for that?

Rand__ · Dec 4, 2020

There are many threads here for crossflashing, one is https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/

And maybe it helps if you provide your intended setup? Host to host? Cluster (-> Switch!)? $100 total, per NIC ?...

hyltcasper · Dec 4, 2020

Rand__ said:
There are many threads here for crossflashing, one is https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/

And maybe it helps if you provide your intended setup? Host to host? Cluster (-> Switch!)? $100 total, per NIC ?...

Host to host, very short distance, 35 cm max. I will read the thread.

Rand__ · Dec 4, 2020

Then the important question is what S2D requires ... RoCE v1 (Cx3) or 2 (CX3 pro or newer).
The o/c it depends on the speed you want...

hyltcasper · Dec 4, 2020

Rand__ said:
Then the important question is what S2D requires ... RoCE v1 (Cx3) or 2 (CX3 pro or newer).
The o/c it depends on the speed you want...

S2d doc recommends iwarp.
Mentions roce and not mentions v2
Kepler 47 azure hci project is build on thunderbolt 3.
Some certified supermicro azure hci servers has mellanox ConnectX 4
I don't know which is better

Rand__ · Dec 4, 2020

Better is what fits the budget i guess...

IWarp is a Chelsio implementation which works without switch support.
RoCE usually needs switches to support this (not sure how it is with host 2 host but might work)

You're not the first to try this though...

RDMA via RoCE 101 for Storage Spaces Direct (S2D)

S2D, Storage Spaces Direct, Storage Spaces,SBS, SBSC, Technology, System Builder Tips, Hyper-V, Hyper-V Failover Clustering, Windows Server

blog.mpecsinc.ca

Our ebay development S2D cluster – jtpedersen.com IT made simple

Most seem to utilize the Pro's though.

Totally depends on the budget in the end (newer might be better supported, older is cheaper but potentially more complicated to setup)

i386 · Dec 4, 2020

Rand__ said:
IWarp is a Chelsio implementation which works without switch support.

IWarp is the official, standard implementatation for rdma over ethernet, but nobody except Chelsio implemented it. Not even intel who helped to create that standard

kapone · Dec 4, 2020

- Microsoft/SMB3 RDMA works great over RoCE v2.
- ConnectX-3 Pro RoCE v2 works great...even without official switch support (i.e. DCB and PFC. Hell, I'm running RoCE v2 through the ICX 6610 with no issues)
- Connect X-3 Non pro is a waste of time, if RoCE v2 is important to you. It'll never work.
- iWarp...once upon a time, there was Chelsio...and they came up with it...and then...it kinda died.

p.s. NVMEoF...all flash NVME storage clusters...S2D...and you're worried about the cost of Network Adapters?? Your storage is gonna be 100x the cost of nics.

hyltcasper · Dec 5, 2020

kapone said:
- Microsoft/SMB3 RDMA works great over RoCE v2.
- ConnectX-3 Pro RoCE v2 works great...even without official switch support (i.e. DCB and PFC. Hell, I'm running RoCE v2 through the ICX 6610 with no issues)
- Connect X-3 Non pro is a waste of time, if RoCE v2 is important to you. It'll never work.
- iWarp...once upon a time, there was Chelsio...and they came up with it...and then...it kinda died.

p.s. NVMEoF...all flash NVME storage clusters...S2D...and you're worried about the cost of Network Adapters?? Your storage is gonna be 100x the cost of nics.

I don't know I need RoCe V2 or not need. Do i need a switch i dont know that too. I already obtained nvme disks. I just need to mirror disks between clusters to get high available virtual machines. I prefer S2D because I have free Microsoft Datacenter licence.

My budget is $150 including shipping per pic. Shipping is very expensive to Turkey except from China.

Rand__ · Dec 5, 2020

Dont forget that you also need a matching cable (0,5m or so)

hyltcasper · Dec 5, 2020

What about HP 620QSFP28 for $90? I didn't understand single port 4 x 25G. How it works?

hyltcasper · Dec 5, 2020

Rand__ said:
Dont forget that you also need a matching cable (0,5m or so)

I assume passive cheap DAC cable works well.

Rand__ · Dec 5, 2020

hyltcasper said:
What about HP 620QSFP28 for $90? I didn't understand single port 4 x 25G. How it works?

Thats a special card and not likely to work (https://www.servethehome.com/hpe-620qsfp28-4x-25gbe-single-qsfp28-port-ethernet-adapter-review/)

and re cable, well some cards are picky, some are not. depends on what you get. Just make sure to match the form factor (qsfp/+/28)

hyltcasper · Dec 5, 2020

Rand__ said:
Thats a special card and not likely to work (https://www.servethehome.com/hpe-620qsfp28-4x-25gbe-single-qsfp28-port-ethernet-adapter-review/)

and re cable, well some cards are picky, some are not. depends on what you get. Just make sure to match the form factor (qsfp/+/28)

Sorry, I didn't understand. Why it doesn't work?

i386 · Dec 5, 2020

Did you change the title of this thread?
825110-B21 can be had for ~$100, it's a hpe branded connect-x 4 single port edr/100GBE vpi nic

kapone · Dec 5, 2020

hyltcasper said:
I don't know I need RoCe V2 or not need. Do i need a switch i dont know that too. I already obtained nvme disks. I just need to mirror disks between clusters to get high available virtual machines. I prefer S2D because I have free Microsoft Datacenter licence.

My budget is $150 including shipping per pic. Shipping is very expensive to Turkey except from China.

That's like saying...I want to go from NY to LA...but I don't know if I want to fly/drive/walk/crawl...

Each method (fly/drive/walk/crawl) has it's own intricacies, pros/cons and details that need to be worked out. Just because you have a free license of Window Server Datacenter Edition doesn't mean anything either.

Do you understand what it takes to setup S2D properly?

Network Adapters are only one component of that. There's a lot more to it than just buying NICs.

hyltcasper · Dec 6, 2020

kapone said:
That's like saying...I want to go from NY to LA...but I don't know if I want to fly/drive/walk/crawl...

Each method (fly/drive/walk/crawl) has it's own intricacies, pros/cons and details that need to be worked out. Just because you have a free license of Window Server Datacenter Edition doesn't mean anything either.

Do you understand what it takes to setup S2D properly?

Network Adapters are only one component of that. There's a lot more to it than just buying NICs.

i386 said:
Did you change the title of this thread?
825110-B21 can be had for ~$100, it's a hpe branded connect-x 4 single port edr/100GBE vpi nic

I decided to go with RoCe V2, I changed thread title after this.

Does anyone know 40+ Gbps RoCe V2 Nic below $100 ?

Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Member

Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

Member