Help with Direct Connect on MCX354A cards

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Cipher

Member
Aug 8, 2014
159
15
18
53
I have two boxes running Windows Server 2019. Both servers have the same Mellanox MCX354A-FCBT ConnectX-3 card which have been updated to the latest firmware (2.42.5000) and latest driver (5.50.51000).

At this point, I'd like to directly connect these cards (ie no switch) so I can use the 40/56GBs connection to copy files between the servers. I'm not that familiar with these cards, or their connection options, so I was hoping for some guidance on the easiest way to get this up and running. Does someone have a link to a guide or a list of steps I need to complete. For context, here are some of the questions I have:

1) What cable/transceiver should I be using?
2) Do I need to connect both ports on these cards to achieve the highest speed or is only one port needed?
3) How does one choose between the InfiniBand or Ethernet connection options on these cards?
4) Is there a connection option preference when running these cards in Windows Server?
5) Is there any software configuration I need to setup for a direct connection? What about for using technology like RoCE?
6) Once everything is configured, how do I ensure certain network activity (eg copying files between servers) uses the Mellanox cards rather than the built in Intel Ethernet NIC?
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
I have two boxes running Windows Server 2019. Both servers have the same Mellanox MCX354A-FCBT ConnectX-3 card which have been updated to the latest firmware (2.42.5000) and latest driver (5.50.51000).

At this point, I'd like to directly connect these cards (ie no switch) so I can use the 40/56GBs connection to copy files between the servers. I'm not that familiar with these cards, or their connection options, so I was hoping for some guidance on the easiest way to get this up and running. Does someone have a link to a guide or a list of steps I need to complete. For context, here are some of the questions I have:

1) What cable/transceiver should I be using?
2) Do I need to connect both ports on these cards to achieve the highest speed or is only one port needed?
3) How does one choose between the InfiniBand or Ethernet connection options on these cards?
4) Is there a connection option preference when running these cards in Windows Server?
5) Is there any software configuration I need to setup for a direct connection? What about for using technology like RoCE?
6) Once everything is configured, how do I ensure certain network activity (eg copying files between servers) uses the Mellanox cards rather than the built in Intel Ethernet NIC?
1) Get a passive qsfp+ dac cable for distances between 0.5~10metres, active dac cables or transceivers + om3/om4 fibers for > 10metres
2) Just one port is required to get a 40gbe link
3) There is a mellanox device in the devicemanager under system devices which allows you to select the protocoll
4) IMO ethernet becuase it doesn't require a subnet manager running :D
5.1) Not really
5.2) You need to install the DCB feature and smb multichannel
6) Use the route command and give the route a low metric/cost
 
  • Like
Reactions: Cipher

Cipher

Member
Aug 8, 2014
159
15
18
53
Thank you, i386. Those answers really help.

The one item that is still confusing is #6 as I'm not really sure what you meant. I'd like to use the Mellanox cards whenever I'm transferring files between both servers and then use the 1Gb Ethernet NIC's in both servers for all other networking tasks like accessing the internet or accessing these servers from various clients (eg Laptops, Tablets). Is this possible and can it be done automatically without me having to specify which networking approach should be used?
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
Put the server-to-server connection on a different IP subnet, no gateway required. Use this (different subnet) IP to setup your transfers. Everything else should work without any specific configuration.
 
  • Like
Reactions: Cipher

Cipher

Member
Aug 8, 2014
159
15
18
53
Hi Kapone,

I'm not a networking expert so I just wanted to ensure I'm understanding this. Here is my current setup:

Server 1
Ethernet Port 1
-IP: 192.168.2.40
-Subnet: 255.255.255.0
-Gateway: 192.168.2.1

Mellanox Port 1
-IP: 192.254.202.59
-Subnet: 255.255.0.0
-Gateway: <BLANK>

Server 2
Ethernet Port 1
-IP: 192.168.2.41
-Subnet: 255.255.255.0
-Gateway: 192.168.2.1

Mellanox Port 1
-IP: 192.254.207.54
-Subnet: 255.255.0.0
-Gateway: <BLANK>

For the Ethernet ports, those values were assigned by the DHCP server in my Router. For the Mellanox ports, those values were defaulted at some point as I didn't assign them myself. If I'm understanding what you are saying, then I would go into the TCP/IPv4 properties for the Mellanox ports and add something similar to the following static values:

Server 1
Mellanox Port 1
-IP: 10.0.0.1
-Subnet: 255.255.0.0
-Gateway: <BLANK>

Server 2
Mellanox Port 1
-IP: 10.0.0.2
-Subnet: 255.255.0.0
-Gateway: <BLANK>

If this is correct, how would these servers choose which path to take for the following tasks:

1) Open Windows Explorer and copy/move files from one server to the other via a mapped drive
-I would always want the faster Mellanox direct link to be used for this task rather than the slower Ethernet network.

2) Open Windows Explorer and copy/move files from one of these servers to my laptop via a mapped drive
-This would have to use the Ethernet ports since my laptop is connected to the same network using an IP address from the DHCP in the router.

Thank you
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
For #1, map the drive using \\10.0.0.xx\sharename on either server. Don't use the machine name or DNS name. That ensures that, "that" traffic will always go over the Mellanox port.

For #2 - Correct. No special configuration needed.
 
  • Like
Reactions: Cipher

Cipher

Member
Aug 8, 2014
159
15
18
53
Thanks, Kapone. Just tested with an extra 40Gb cable I had lying around and it works perfectly.