The microtek OS is the weirdest I have ever use. I have the rack mounted MIKROTIK CRS226-24G-2S+RM Cloud Router and its fanless. This seems a pretty good price if one is in EU/UK - D-Link DGS-1510-20 - 20-PORT GIGABIT SMARTPRO SWITCH - STACKABLE 2 SFP 2 10G... | eBay? - I have a 48 port version, bit funny with respect to the SFP compatibility with my Gnodal though.
$30.29 OBO - HP 10GB Mellanox ConnectX-2 10GBe SFP+
- Thread starter h0tw1r3
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Hi,
I just ordered two of these cards. I plan to use one for an ESXi host and one for Linux host. I'm still fairly newb to the world of 10GbE, and would love to have you guys give me some clarity on the types of modules and cables that I should get for these cards.
Can someone provide me some examples, pros/cons of different types of modules and cables? So far, from browsing on eBay, I see copper Twinax DAC modules+cable and 10G-BASE-SR fiber modules using LC cable. Am I even on the right track with this observation?
Also, I cannot find OSX driver for this card. Does anyone know if one exist?
Thanks a bunch.
I just ordered two of these cards. I plan to use one for an ESXi host and one for Linux host. I'm still fairly newb to the world of 10GbE, and would love to have you guys give me some clarity on the types of modules and cables that I should get for these cards.
Can someone provide me some examples, pros/cons of different types of modules and cables? So far, from browsing on eBay, I see copper Twinax DAC modules+cable and 10G-BASE-SR fiber modules using LC cable. Am I even on the right track with this observation?
Also, I cannot find OSX driver for this card. Does anyone know if one exist?
Thanks a bunch.
The only real difference is cost over distance. If your not looking to transmit that far, then a Twinax cable will be fine (albeit, they are a lot more expensive, but are also more resillient to rough environments). If your planning on moving it around a lot, Twinax is better, if you plan to stick and forget it, LC-LC fiber is fine.
A 5M cable of twinax on amazon is $92 (but you don't need to buy SFPs, this is priced on amazon.com)
A 5M LC-LC OM3 10G fiber cable is $10 (from Fibercables.com for example)
Lastly, if also depends on what your connecting those two cards to, to each other, both will work, to a switch, now you need to order two twinax cables, and/or 4 SFPs + cables.
A 5M cable of twinax on amazon is $92 (but you don't need to buy SFPs, this is priced on amazon.com)
A 5M LC-LC OM3 10G fiber cable is $10 (from Fibercables.com for example)
Lastly, if also depends on what your connecting those two cards to, to each other, both will work, to a switch, now you need to order two twinax cables, and/or 4 SFPs + cables.
Any one have the instructions for reflashing the HP branded ones back to Mellanox firmware?
I have firmware version 2.9.1000 and I have horrible performance. I'm testing with jperf (iperf with gui). I get around 100MB's and a high of 180MB's It's all over the place really. This is with a Quanta LB4M switch. The switch and both clients see a 10GBe Link. I'm not using Jumbo frames. Using Copper 3m SFP+ DAC. Would like to update the firmware on both the Windows 7 machines before digging any further. And Yes both card's are in 8x PCI-E v2 slots.
Could be a switch issue, never had a managed switch before.
Thanks,
Levi
I have firmware version 2.9.1000 and I have horrible performance. I'm testing with jperf (iperf with gui). I get around 100MB's and a high of 180MB's It's all over the place really. This is with a Quanta LB4M switch. The switch and both clients see a 10GBe Link. I'm not using Jumbo frames. Using Copper 3m SFP+ DAC. Would like to update the firmware on both the Windows 7 machines before digging any further. And Yes both card's are in 8x PCI-E v2 slots.
Could be a switch issue, never had a managed switch before.
Thanks,
Levi
Are you sure that the are electrically 8x as well as physically? Are you sure that the MB is able to use full 8x in that slot with your current configuration. I can't recall where, but there is a way to see how many lanes are currently active in the card. This performance level is exactly what I saw till I moved my card to a slot that was able to actually provide 8x.
I'm experiencing the same problem as Levi is. Both of the cards are in x8 slots electrically. The max I'm getting is only 150 Mb/s. This is connected to a IBM G8124. I tried direct connection and got the same results. The connection is between a Server 2012 R2 and a Windows 7 machine.
I was able to get 700MB's Transfer speed with these card when I had build my own 10Gb router using Linux so I know it's not the slots. It's a MTU issue some where. I can't go higher then 1500 or I get packet fragmentation and that's what I think the problem is. I think their is a bug in the firmware regarding MTU that's why I'm trying to flash it but when I try flashing the firmware in windows I get an error because it's LSI firmware and HP rebranded Card.
I haven't had time to dig into this issue for us. Have you tried a test tool like Microsoft NTTtcp? Here is a write up on how to use it, also in his guide is how to optimize the Melanox for windows. Confessions Of A 10 GbE Network Newbie – Part 2: Test Toolkit | Cinevate – Tools for Filmmakers and Photographers . Also a tool called iperf, and a gui for it called jperf. I like NTTcp though because it reports problems like packet Fragmentation. Which is my problem. Have you had any luck flashing the firmware? I will start a new thread when I have more time to tinker.
I BELIEVE this is the way to do it... PLEASE PLEASE PLEASE be aware that you are doing this on your own, and it could completely destroy your card. I tried it on 2 cards, and both are still detected after the flash, but this is in my TEST box and I haven't had a chance to actually verify that the cards still work since my only 10GbE connection is in "production" lab use.... For all I know, performance could be worse after the flash. Additionally, I haven't figured out how to backup the existing firmware yet, so you are on your own if you have to roll-back.
AT YOUR OWN RISK:
- Download and extract the management tools: Mellanox Technologies
- Download and extract the firmware for your card: Creating a Customized Firmware Binary - Mellanox Technologies
- I BELIEVE our cards are MNPA19-X[ST]R...but please check!
- I used firmware version 2_9_1200.
- Install the management tools (I used Fedora, but it should work in other OSes including Windows)
- Fedora requires gcc, rpm-build and kernel-devel to be installed, and gcc-c++ and zlib-devel if you plan on installing the linux drivers later
- See http://www.servethehome.com/custom-firmware-mellanox-oem-infiniband-rdma-windows-server-2012/ for a starting point if using Windows.
- Grab the device ID using mst:
- mst status
- you will see a list of MST devices, and most likely 2 entries per card...I used the _crX entry, not the pciconflX entry
- mst status
- Burn the firmware
- mlxburn -dev /dev/mst/mt26448_pci_cr1 -fw /path/to/firmware/ConnectX2-rel-2_9_1200/fw-ConnectX2-rel.mlx -conf /path/to/ini/ConnectX2-rel-2_9_1200/MNPA19_A1-A3.ini -allow_psid_change
- Check the firmware version (assuming you have the drivers installed as well):
- ethtool -i enp4s0
I would be curious to see if performance is better, assuming someone is brave enough to actually flash their card. Mine were extras, so...
I have spare cards but don't have time to try it out atm. I will give it a shot this weekend. Thanks for the procedure.
I flashed one of mine (and lost my notes; took three hours to do and find a verifably correct image, five minutes to do, one second to forget to save the goddamn text file, so thank you) which gave me RDMA a while back, and to the best of my recollection, it worked, but when I reinstalled it this week, despite showing enabled in Powershell, it was not doing RDMA transfers (other side was a ConnectX-3, so no issue there) most obviously because I was tracking my speed in Task Manager/Resource Monitor.
Also, despite some peaks at 6/7Gbps, I was usually seeing 170MB/s on a single SMB Live Migration, but in the past 4+Gbps has been my usual speed (also not a disk issue; I was able to push the transfer higher by doing multiple Live Migrations, but I stayed at 800-1500Mbps EDIT: *per VM migration* e.g. 4Gbps with 3 or 4 running). Most of my VMs are 10-20GB, so it's an annoyance there, but I do have one 220GB VM which also does 500KB/s in db writes, so that is... problematic, not least of all because it has my UBNT mFi server, several months of electrical power usage logs... and acts like a "soft" circuit breaker, cutting mains to UPSes upon surges (like my fridge spiking at 800W or running at 450W for 15 minutes for no obvious reason) which protects 2/3rds of my LAN.
I just installed two more of these, and probably will add my fourth and final card in, so first off, thank you again for the notes, but yes, this just became more important to me, especially as I moved off my Brocade 1020s specifically to get RDMA... also I'd love to get rid of the "QOS (ETS) capability is missing" error in my logs.
So if people could mention which (and where they got) firmwares that are doing well in TCP and SMB3+RDMA performance here, I'm sure I wouldn't be alone in being very grateful.
Also, despite some peaks at 6/7Gbps, I was usually seeing 170MB/s on a single SMB Live Migration, but in the past 4+Gbps has been my usual speed (also not a disk issue; I was able to push the transfer higher by doing multiple Live Migrations, but I stayed at 800-1500Mbps EDIT: *per VM migration* e.g. 4Gbps with 3 or 4 running). Most of my VMs are 10-20GB, so it's an annoyance there, but I do have one 220GB VM which also does 500KB/s in db writes, so that is... problematic, not least of all because it has my UBNT mFi server, several months of electrical power usage logs... and acts like a "soft" circuit breaker, cutting mains to UPSes upon surges (like my fridge spiking at 800W or running at 450W for 15 minutes for no obvious reason) which protects 2/3rds of my LAN.
I just installed two more of these, and probably will add my fourth and final card in, so first off, thank you again for the notes, but yes, this just became more important to me, especially as I moved off my Brocade 1020s specifically to get RDMA... also I'd love to get rid of the "QOS (ETS) capability is missing" error in my logs.
So if people could mention which (and where they got) firmwares that are doing well in TCP and SMB3+RDMA performance here, I'm sure I wouldn't be alone in being very grateful.
Last edited:
@PigLover are you sure about the CX-2 (just EN, no Pro, VPI, etc.)
http://www.mellanox.com/related-docs/prod_software/ConnectX-2_RDMA_RoCE.pdf
I swear I've had it working with RDMA to one of my CX-3s
@spyrule I certainly use -need- more than 1GbE. Aside from allowing me to kick VMs around much faster which helps me with my electrical constraints, my data is roughly organized into 3 VHDXs, one for IT/Admin stuff to keep my LAN/WAN running at roughly 2TB, one for media I didn't make (books, music, etc) which is about 1.5TB, and then content I did make, which is just under 4TB and will soon expand.
Put simply, the difference in a D2D transfer of those 3 VHDXs or checksummed file copy is cut from 18-24 to 6-8 hours, and I have plenty of improvements to make in (disk) storage speed.
Regardless, in a failure/RAID rebuild situation, that difference could be huge -- or catastrophic.
This doesn't include iSCSI volumes used for "local" incremental backups using the simple Windows native client.
Topology is roughly two Dell 5524s connected/stacked at 7 meters with a pair of VERY thick HDMI cables (20Gbps bidi per cable), giving me four 10GbE uplink ports. Two are for workstations, two are for VM/storage hosts. A fifth machine will go in for D2D backup in direct connection to the server with my SAS expander box.
http://www.mellanox.com/related-docs/prod_software/ConnectX-2_RDMA_RoCE.pdf
I swear I've had it working with RDMA to one of my CX-3s
@spyrule I certainly use -need- more than 1GbE. Aside from allowing me to kick VMs around much faster which helps me with my electrical constraints, my data is roughly organized into 3 VHDXs, one for IT/Admin stuff to keep my LAN/WAN running at roughly 2TB, one for media I didn't make (books, music, etc) which is about 1.5TB, and then content I did make, which is just under 4TB and will soon expand.
Put simply, the difference in a D2D transfer of those 3 VHDXs or checksummed file copy is cut from 18-24 to 6-8 hours, and I have plenty of improvements to make in (disk) storage speed.
Regardless, in a failure/RAID rebuild situation, that difference could be huge -- or catastrophic.
This doesn't include iSCSI volumes used for "local" incremental backups using the simple Windows native client.
Topology is roughly two Dell 5524s connected/stacked at 7 meters with a pair of VERY thick HDMI cables (20Gbps bidi per cable), giving me four 10GbE uplink ports. Two are for workstations, two are for VM/storage hosts. A fifth machine will go in for D2D backup in direct connection to the server with my SAS expander box.