Mellanox ConnectX-3 2-Port MCX354A-FCBT (56Gb IB / 40GbE) $150

[dswartz]

I'd guess: getting a decent IB switch and use IPoIB and/or SRP and/or RDMA for the storage and data replication (or whatever)?

 

[T_Minus]

IPoIB w/ISCSI was my plan unless there's a better way

 

[dswartz]

Depending on your setup, SRP might be preferable. Better performance and less CPU overhead...

 

[T_Minus]

Depending on your setup, SRP might be preferable. Better performance and less CPU overhead...

Thanks, I'll check into that I saw ESXI supports SRP, not sure my Connectx2 VPI do, have to check

 

[dswartz]

ESXi you need to delete the vmware-provided drivers and load some community VIBs. Google for esxi + infiniband srp and you should find what you want. The connectx-2 do in fact support SRP (it's more a question of the host having drivers...)

 

[T_Minus]

ESXi you need to delete the vmware-provided drivers and load some community VIBs. Google for esxi + infiniband srp and you should find what you want. The connectx-2 do in fact support SRP (it's more a question of the host having drivers...)

Haha, much thanks, glad you told me that I would have been screaming at it I'm sure!!

 

[markpower28]

SRP for IB or iSER for iSCSI. Both works for ESXi 5.5/6.0

IPoIB w/ISCSI was my plan unless there's a better way

 

[dswartz]

My bad: when I mentioned community VIB that is to get an IB subnet manager that will run on the vsphere host (which you might or might not care about...)

 

[markpower28]

you may want to looking at the following post
https://forums.servethehome.com/index.php?threads/vsphere-1-8-3-iser-benchmark.5468/
https://forums.servethehome.com/index.php?threads/connectx-2-and-esxi-6-0.5382/#post-46823
https://forums.servethehome.com/index.php?threads/ib-napp-it-and-srp.5248/#post-45649

 

[Dajinn]

What about Server 2012 and SMB shares, no iSCSI?

 

[dswartz]

No idea, don't use them, sorry...

 

[godless]

Seller has posted another 60 of these, in case you missed out the first time (like me).

Mellanox Connectx 3 FDR Infiniband 40GIGE MCX354A FCBT Low Profile | eBay

 

[PnoT]

What about Server 2012 and SMB shares, no iSCSI?

SMB 3.0+ and SMB Multi-channel blow iSCSI out of the water.

I set my Hyper-V hosts up with 1 card in each host:

1st port IB with a subnet for SMB/RDMA traffic
2nd port 10GbE on my management subnet

Use smb constraints to keep traffic from running across my management subnet

 

[Dajinn]

SMB 3.0+ and SMB Multi-channel blow iSCSI out of the water.

I set my Hyper-V hosts up with 1 card in each host:

1st port IB with a subnet for SMB/RDMA traffic
2nd port 10GbE on my management subnet

Use smb constrained delegation to keep traffic from running across my management subnet

What do you mean your "management subnet"? Are you using a QSFP to SFP+ cable to chain switches?

 

[PnoT]

What do you mean your "management subnet"? Are you using a QSFP to SFP+ cable to chain switches?

Yep these work great: Network Adapter - Newegg.com

 

[Dajinn]

Yep these work great: Network Adapter - Newegg.com

That's interesting, would that work on my dell powerconnect 5448 combo SFP ports you think?

 

[Chuckleb]

I have a box of those, good deals every so often on eBay. They convert the 40/56 to 10 very nicely. Almost no increase in depth of adapter.

 

[PnoT]

That's interesting, would that work on my dell powerconnect 5448 combo SFP ports you think?

It's just an adapter so if you have the right transceivers to work with your switch it should be fine but I've only tried it with DAC in my setup. I think Chuckleb has used them with success as well.

 

[Aluminum]

Just fyi, one of the regular $35 sellers of those QSFP{SFP+ adapters on ebay accepted $25/freeship best offers on orders of >1.

 

[Smalldog]

The eBay seller isn't taking $150 anymore. I BO'd 4 @ $150 each, and they countered with $200 each.