Recommended 10gbe host to host (ESXi) setup

[yu130960]

I have done as much reading as I can and am looking for a current recommendation on all the parts that I need to hook up two ESXi hosts (directly without the need for a 10gbe switch).

I am looking at these Intel 10gbe cards but for a short connection (1 to 3 feet) between hosts I am totally unclear as to what the cheapest most effective cabling solution.

Any pointers so I can grab something on ebay would be great. I am wanting to stick to Intel as I want to be as close to plug and play as possible.

 

[pyro_]

you should be able to use a passive sfp+ to spf+ cable to connect the two together.

 

[mervincm]

I have been down this road. In the end there were so many gotchas that I decided to avoid it, and picked up a switch that had 2 10GIGE ports for 300$

My advice would to to first set it up with 1 GBE cards ecaxtly how you hoped to use 10GBE. once you are positive it works the way you think it will, hope it will, then spend the money to buy the cards.

PS I bought ESXi supported broadcom 10GB cards with transceivers etc. All I had to go was add a 15$ LC fibre cable from monoprice.

 

[yu130960]

I have been down this road. In the end there were so many gotchas that I decided to avoid it, and picked up a switch that had 2 10GIGE ports for 300$

My advice would to to first set it up with 1 GBE cards ecaxtly how you hoped to use 10GBE. once you are positive it works the way you think it will, hope it will, then spend the money to buy the cards.

PS I bought ESXi supported broadcom 10GB cards with transceivers etc. All I had to go was add a 15$ LC fibre cable from monoprice.

Which switch are people buying with 2 10gbe ports? I currently am using a HP ProCurve 1810G - 24 GE, J9450A and set ESXi up to trunk 2 gbe back to the switch.

 

[mervincm]

https://forums.servethehome.com/ind...k-crs226-24g-2s-in-desktop-rm-rackmount.3769/

 

[bds1904]

Quanta LB4M HP Brocade Open Source Switch 90 Day Warranty 100 Tested | eBay
Brocade BR 1020 10Gbps Fcoe CNA PCIe Dual Port Card w Full Height Bracket | eBay
Brocade 57 0000075 01 10GB 850nm SWL SFP XBR 000180 | eBay
LC LC 3M Duplex 10 Gigabit 50 125 Multimode Fiber Optic Cable OM3 Aqua 10GB | eBay


Complete solution, 1 switch, 2 cards and 2 brocade sfp's & 2 patch cords. $427 total, and you get a 48port GBe switch too.

The brocade 1020's work great with esxi.

Sounds like stepping up to the LB4M will simplify your network too. Single 10Gb link from each server to the LB4M and all your other devices plug right into the 1Gb ports on the LB4M.

 

[yu130960]

Thanks for the list on a complete solution.

 

[mrkrad]

You will find it is difficult to reach 10gbe from one VM with esxi 5 Regardless if you go server to server or server to switch! There's nothing wrong with going server to server direct though!

 

[Scott Laird]

$427 total? $159 for an LB4M (with 2xSR SFP+), 2x$49 for Mellanox 10Gbe cards, 2x$25 for SFP+ for the cards, and 2x$10 for fiber. That's barely over $300.

 

[yu130960]

Would this work for a direct ESXI <--> ESXI setup? Total $120

Mellanox Technologies MC3309130-002 Passive Copper Cable 1x Sfp+ $21
MELLANOX MNPA19-XTR CONNECTX 2 10GbE ETHERNET NETWORK SERVER ADAPTER $49 x 2

 

[Scott Laird]

Probably, if ESXI supports that NIC.

 

[Marsh]

I used Mellanox MNPH29C-XTR ConnectX-2 EN Dual-port SFP+ 10GbE PCI-E 2.0 Adapter
Each ESXI or Windows 2012r2 server have a dual port Mellanox EN card. With dual port cards, I have a 3 nodes ring.

 

[firenity]

Quanta LB4M HP Brocade Open Source Switch 90 Day Warranty 100 Tested | eBay
Brocade BR 1020 10Gbps Fcoe CNA PCIe Dual Port Card w Full Height Bracket | eBay
Brocade 57 0000075 01 10GB 850nm SWL SFP XBR 000180 | eBay
LC LC 3M Duplex 10 Gigabit 50 125 Multimode Fiber Optic Cable OM3 Aqua 10GB | eBay

I would also go with this combination.
However, you can get Brocade compatible SFP+ transceivers from Fiberstore for 25 $ each (confirmed working in the Brocade 1020 thread).

 

[Marsh]

I used best offer for Mellanox MNPH29C card, brought 24 cards at $35 each. Found some Mellanox SFP+ cables around $15 each
Total 2 x $35 + 2 x $15

 

[yu130960]

I used best offer for Mellanox MNPH29C card, brought 24 cards at $35 each. Found some Mellanox SFP+ cables around $15 each
Total 2 x $35 + 2 x $15

I think I am going to go the same route. What is the difference between MNPH29C-XTR and MNPH29D-XTR?

 

[Marsh]

I don't own any MNPH29D model, MPNPH29C and D both are in ConnectX-2 family. I guess D is better than C model.
I got my cards from Ebay seller liquid8technology.com
I think he / she are not willing to accept $35 per card now, but ok with $45. Earlier this year, I purchased 8 cards each time. 3 different transactions, but each time there is at least one DOA card. Seller always replace the card or issue refund ( when they do not have anymore in stock ) promptly.

 

[weust]

Switch wise a Zyxel GS1910-24 might be interesting too?
Rather cheap at 130 Euro's including taxes here in The Netherlands.
Has four SFP+ ports (shared with four RJ45 ports) and is silent due to no fans.

XGS1910/GS1910 Series 24/48-port GbE Smart Managed Switch | ZyXEL

I have this one as home, and although I don't use SFP+ cabling yet, I like it.
Web based management or serial.

 

[firenity]

Switch wise a Zyxel GS1910-24 might be interesting too?
Rather cheap at 130 Euro's including taxes here in The Netherlands.
Has four SFP+ ports (shared with four RJ45 ports) and is silent due to no fans.

XGS1910/GS1910 Series 24/48-port GbE Smart Managed Switch | ZyXEL

The GS1910-24 has no SFP+ ports, the XGS1910-24 however has 2.

 

[weust]

Ah, my bad. When I bought mine it wasn't for those ports.
So didn't reaaly looked into it, though I thought I read about it.

Just 1GBIC then. Or Gbic, not sure which it is.