RDMA with 40GB Ethernet Mellanox ConnectX3 Cards

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Chuntzu

Active Member
Jun 30, 2013
383
98
28
I am having some problems very reminiscent of my early struggles with ConnectX2 cards and RDMA. I have a Dell z9000 32 port 40GBE switch with some really old FTOS firmware which does not have DCB. But from my understanding that dcb in not specifically needed for RDMA to function just need to enable dcb in windows rmda to work with smb direct. I have gotten RDMA to work with my infiniband switch just trying to get it to work with the ethernet switch now. Any advice would be appreciated.

PS - this may be a stretch, but could anyone help me acquire a new ftos or firmware for my switch? I tried and failed since the switch is out of contract. I would appreciate any help getting the firmware, I can figure out the flashing from the documentation.
 
  • Like
Reactions: Lance Joseph
May 8, 2015
142
24
18
33
I am having some problems very reminiscent of my early struggles with ConnectX2 cards and RDMA. I have a Dell z9000 32 port 40GBE switch with some really old FTOS firmware which does not have DCB. But from my understanding that dcb in not specifically needed for RDMA to function just need to enable dcb in windows rmda to work with smb direct. I have gotten RDMA to work with my infiniband switch just trying to get it to work with the ethernet switch now. Any advice would be appreciated.

PS - this may be a stretch, but could anyone help me acquire a new ftos or firmware for my switch? I tried and failed since the switch is out of contract. I would appreciate any help getting the firmware, I can figure out the flashing from the documentation.
There's a guy on ebay that has a bunch of white force10 z9000's listed that I spoke with. He claims to have the latest 9.7 firmware. Could be worth a shot to hit him up and see if he's willing to send you the bin file. Even if it's for a small fee. FWIW, if the version you're running is the same as all the others I've seen on ebay (something like 8.3 i believe) it's so outdated and so insecure that I wouldn't want it connected to my network at all.
 
  • Like
Reactions: Chuntzu

Lance Joseph

Member
Oct 5, 2014
82
40
18
I am having some problems very reminiscent of my early struggles with ConnectX2 cards and RDMA. I have a Dell z9000 32 port 40GBE switch with some really old FTOS firmware which does not have DCB. But from my understanding that dcb in not specifically needed for RDMA to function just need to enable dcb in windows rmda to work with smb direct. I have gotten RDMA to work with my infiniband switch just trying to get it to work with the ethernet switch now. Any advice would be appreciated.

PS - this may be a stretch, but could anyone help me acquire a new ftos or firmware for my switch? I tried and failed since the switch is out of contract. I would appreciate any help getting the firmware, I can figure out the flashing from the documentation.
I started a similar project around the time that you first posted this.
I've three systems, each with a Connectx-3, connected to a Z9000.
My aim is similar to yours - to play with Storage Spaces Direct!

My Z9000 is running the following firmware:
Stack-unit 0 FLASH BOOT 8.3.11.5[boot] 8.3.11.5
If you do end up getting the latest firmware, I'd very much appreciate a copy.

Are you only interested in SMB direct at this time or are you looking to toy with SSD?
Were you also interested in the new hyperconverged solution, utilizing SSD with Hyper-V?
I'm still trying to find out what types of configurations are possible that avoid single points of failure.
 

Chuntzu

Active Member
Jun 30, 2013
383
98
28
I started a similar project around the time that you first posted this.
I've three systems, each with a Connectx-3, connected to a Z9000.
My aim is similar to yours - to play with Storage Spaces Direct!

My Z9000 is running the following firmware:
Stack-unit 0 FLASH BOOT 8.3.11.5[boot] 8.3.11.5
If you do end up getting the latest firmware, I'd very much appreciate a copy.

Are you only interested in SMB direct at this time or are you looking to toy with SSD?
Were you also interested in the new hyperconverged solution, utilizing SSD with Hyper-V?
I'm still trying to find out what types of configurations are possible that avoid single points of failure.
Yeah s2d is what I'm interested in. I'm using 40gb infiniband right now but it would be nice to only have the one fabric. So far no luck on the firmware
 
May 8, 2015
142
24
18
33
Anyone who's used these switches care to chime on on transceiver or cable compatability? I have one coming in at the end of the week. Supposedly it has the latest firmware on it. If this is the case I'll try and dump it and host it up somewhere for forum members.
 

Lance Joseph

Member
Oct 5, 2014
82
40
18
Anyone who's used these switches care to chime on on transceiver or cable compatability? I have one coming in at the end of the week. Supposedly it has the latest firmware on it. If this is the case I'll try and dump it and host it up somewhere for forum members.
I'd like a copy of the firmware if you don't mind ;)
You should be able to upload it here as a file to avoid self-hosting.

I'd tried numerous 40G QSFP+ cables with the Z9000, all of which worked fine.
Part numbers (according to the Quanta T5016-LB8D into which they're now connected) :
FCBG410QB1C10-FC (Dell / Finisair)
FCBG410QB1C50-FC (Dell / Finisair)
MC2210310-010 (Mellanox)
CAB-Q-Q-7M (Arista)
 
  • Like
Reactions: rockitlikeithott