ConnectX-2 EN cards do a good job at offload. They are high performance 10gbe nics. Unfortunately they do not support RDMA over Ethernet. You need connectX-3 for that.
$30.29 OBO - HP 10GB Mellanox ConnectX-2 10GBe SFP+
- Thread starter h0tw1r3
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
That's fine. RDMA is a bit "overkill" for my home servers... not that 10GbE isn't
!I was just worried that I'd run into the same performance wall I'm seeing with the two (albeit older) ConnectX cards I'm using with IPoIB now; it definitely seems to be a CPU-load issue. I should have skipped IPoIB and jumped straight to "real" 10GbE; IPoIB isn't that great, IMO.
I might jump on these and see how low Liquid8 is willing to go: Mellanox MNPH29D XTR Connectx En Dual Port 10GIGABIT Network PCIe Card | eBay (Too bad they're not low-profile and I can't seem to find any low-profile brackets anywhere; I'll have to mod the brackets. Not really a problem though!)
Could always zip tie them to the chassis
Cannot speak for the ConnectX-3's but even on the ConnectX-2 IPOIB is still a pig, especially in ESXi if you are doing eIPOIB. I moved to straight 10Gbe for that reason (SRP seems to have gone away from most things and iSer isn't well supported)
Cannot speak for the ConnectX-3's but even on the ConnectX-2 IPOIB is still a pig, especially in ESXi if you are doing eIPOIB. I moved to straight 10Gbe for that reason (SRP seems to have gone away from most things and iSer isn't well supported)
Out of curiosity, what kind of SFPs were you using? I tried both a passive DAC cable and some generic fiberstore ones, connected to my Mikrotik switch and they didn't want to work. Same SFPs work fine in the ConnectX-2 ENs . Could be pebkac too
For us, thinner cabling between racks, longer runs. DAC would be great but running 40-50 is bulky.
@Entz - Followup. We got our transceivers in today, everything works great. These are what we used. We also used some ER single mode ones and that worked in the adapters as well.
10GBASE-SR SFP+ MMF 850nm Transceiver-$18
So the Fiberstore parts work well with the Mellanox EN cards as well as the converter.
Mine came in too. JDSU PLRXPL-SC-S43-22-N SFPs seem to work fine. I haven't tested up to 10Gb yet as I maxed out the SSD read speed in my workstation.
Similar reasoning to what Chuckleb said - I'm running 25 feet between workstation and server. I think an SFP cable would have been more expensive and definitely bulkier than the SFPs + patch cable. I also tend to move things around a lot, so being able to change the patch cable is nice.Any reason to purchase transceivers and fiber separately instead of either pre-made passive copper or active cables if length isn't a concern (short runs only)? (Sorry for all the questions... learning is fun!
I use Infiniband to to connect my LabESX hosts to an iSCSI NAS (SAN?). I am currently just doing host-to-host-to-host, but am going to expand to an IB switch at some point in the coming months. Some are doing true 10GbE networking, I chose IB because it was a cheap way to get ultra-fast, low-latency connectivity in the home. I bought three used dual port ConnectX cards and cable sets at the close of 2013 ... the whole setup cost me less than $250.
My day job has turned into something that more resembles Service/Project management than the stuff I truly love to do, so I use my lab gear at home to scratch the engineering itch and stay current with technology. At some point in 2015 I plan on leaving my current position for contract/consulting work, but after seven years in a role that has become less and less technology focused I don't feel quite comfortable going mercenary, so I am using the lab to polish my skill-set. That, and I just love to build, tear down, and tinker. I believe my wife is going to be thrilled when I finally land a position that allows me to indulge my true passion so I stop building these data centers in our home.
What do all you people connect these cards too? Is there a "relatively" cheap switch that can handle more than 2 10GbE SFP+ connections without needing to spend a rediculous amount of money?
I'm trying to figure out how I could interconnect 3 nodes from my Dell C6005 server (all running ESXi) and my NAS without needing to install 3 x 2 port cards...
I'm trying to figure out how I could interconnect 3 nodes from my Dell C6005 server (all running ESXi) and my NAS without needing to install 3 x 2 port cards...
I have a ConnectX-2 EN connected to Amazon.com: MikroTik CRS226-24G-2S+IN Cloud Router Gigabit Switch, 24x 10/100/1000 Mbit/s Gigabit Ethernet with AutoMDI/X, Fully manageable Layer3, RouterOS v6, Level 5 license: Computers & Accessories
And a small storage server on the other 10GbE port.
thats an extremely interesting switch/router. That would only solve 1/2 of my connections though (1 for my NAS, 1 for 1 ESXi box, leaving 2 ESXi boxes with a single NIC each).
Currently, I have three C6100 nodes (two ESXi nodes and one node for storage (rewired for 6x3.5" in the front of the C6100)) connected in a triangle fashion with ConnectX-2 dual-port cards. ESXi vSwitches won't function as bridges, unfortunately, so the connection between the ESXi nodes is on a separate subnet and is used for vMotion traffic only. I just got everything installed yesterday, in fact. VM storage over 10GbE is amazing.
4+ devices is tricky without a switch, unfortunately. I wish there were more affordable 10GbE switch options!
That's really interesting - I totally missed that in searching around for an inexpensive 2 SFP+ switch. Is it fanless? And is that an LCD screen on it?
Double affirmative regarding your questions.
It is fanless and it is a LCD screen
I am using a MikroTik CRS 226 in the livingroom,
and a DGS 1510-28X in the basement and another one on the attic.
I have OM3 running between the switches but as I have just one storage system supplying 700MS/s sequential its more or less because of the fun of it.
I am still wondering why the DGS 1510-28x is 360€
D-Link DGS-1510-28X, 28-Port, smart managed in Netzwerk LAN/Modems: Switches | heise online Preisvergleich
but $788
D-Link SmartPro DGS-1510-28X - switch - 28 ports - managed - desktop, rack-mountable Overview - CNET
that is very strange.
Could be cheaper to import them to the US, than buying one there
It is fanless and it is a LCD screen
I am using a MikroTik CRS 226 in the livingroom,
and a DGS 1510-28X in the basement and another one on the attic.
I have OM3 running between the switches but as I have just one storage system supplying 700MS/s sequential its more or less because of the fun of it.
I am still wondering why the DGS 1510-28x is 360€
D-Link DGS-1510-28X, 28-Port, smart managed in Netzwerk LAN/Modems: Switches | heise online Preisvergleich
but $788
D-Link SmartPro DGS-1510-28X - switch - 28 ports - managed - desktop, rack-mountable Overview - CNET
that is very strange.
Could be cheaper to import them to the US, than buying one there
Thanks! I see they have a rackmount version as well - do you happen to know if that one is fanless too?
Seems to be fanless, too.
I bought the desktop model by mistake, so I am not quite sure.
But according to this thread: https://forums.servethehome.com/index.php?threads/mikrotik-crs226-24g-2s-in-desktop-rm-rackmount.3769/
it is fanless, too
I bought the desktop model by mistake, so I am not quite sure.
But according to this thread: https://forums.servethehome.com/index.php?threads/mikrotik-crs226-24g-2s-in-desktop-rm-rackmount.3769/
it is fanless, too
Both are fan less desktop model has the LCD on top, rack-mount on the front. LCD is a touchscreen as well, but don't expect a phone like experience, the display and touch sensor are much lower quality than any mobile phone.
One thing you need to be aware of with this switch is that it does not support dynamic LACP port bonding. apparently the hardware switch chipset supports it, but the OS does not.
Also ... well ... It's not exactly a polished OS. I don't mean to be too negative, many features and low price, but ... well .. I always get the "beta" quality feel.
If I could dynamic LACP bond to it, and it had 4 SFP+ I would be very happy (2 ESXi hosts, 1 storage server, 1 workstation)
One thing you need to be aware of with this switch is that it does not support dynamic LACP port bonding. apparently the hardware switch chipset supports it, but the OS does not.
Also ... well ... It's not exactly a polished OS. I don't mean to be too negative, many features and low price, but ... well .. I always get the "beta" quality feel.
If I could dynamic LACP bond to it, and it had 4 SFP+ I would be very happy (2 ESXi hosts, 1 storage server, 1 workstation)