I have an SX6012 (MLNX OS) and starting to be desperate to get it up and working. The switch seems not to switch at all — which is, frankly, quite an issue with a switch
The desired setup is as follows: I have a 4-node Supermicro server, each with a ConnectX-4 in a SIOM slot, plus a ConnectX-3 in a PCIe slot. CX4s are connected to ports 1 to 4, CX3s to ports 9 to 12. Ports 5 to 8 are unoccupied. Cables are Mellanox for 56 Gbit Ethernet. All cards are in Ethernet mode. WinOF-2 drivers installed for CX4, but CX3 cards use Microsoft drivers. OS is Windows Server 2019 Datacenter with roles and features needed for hyper-converged infrastructure. Using WAC to setup the cluster, but am stuck on network connectivity.
I am using two VLANs — 48 for CX4s and 58 for CXs. Switch configuration has VLAN interfaces defined, each with an IP address assigned (192.168.48.1, 192.168.58.1 respectively). Ports are members of VLAN. All cards are configured in the OS with the respective VLAN number. MTU in the OS is set to 9000.
All ports are configured as 56 Gbit and shown as so in Windows. To avoid issues with 56 Gbit, I downgraded speeds on ports 1 and 2 to 40 Gbit but to no avail.
Ping doesn't work. I do have firewall exceptions and servers can ping each other using 1 Gbit connections.
TCP doesn't work. I am using iPerf to run a test data stream. Again, works via 1 Gbit.
ARP resolution doesn't work — using arp-ping.exe, won't resolve anything.
So far, all tests on CX4s only.
Switch OS version: PPC_M460EX 3.6.1002 2016-06-09 20:24:26 ppc
System profile is vpi-single-switch.
Current running switch config:
All ports are green in MLNX OS web interface:
All networking config as shown in WAC:
Any help will be greatly appreciated.
The desired setup is as follows: I have a 4-node Supermicro server, each with a ConnectX-4 in a SIOM slot, plus a ConnectX-3 in a PCIe slot. CX4s are connected to ports 1 to 4, CX3s to ports 9 to 12. Ports 5 to 8 are unoccupied. Cables are Mellanox for 56 Gbit Ethernet. All cards are in Ethernet mode. WinOF-2 drivers installed for CX4, but CX3 cards use Microsoft drivers. OS is Windows Server 2019 Datacenter with roles and features needed for hyper-converged infrastructure. Using WAC to setup the cluster, but am stuck on network connectivity.
I am using two VLANs — 48 for CX4s and 58 for CXs. Switch configuration has VLAN interfaces defined, each with an IP address assigned (192.168.48.1, 192.168.58.1 respectively). Ports are members of VLAN. All cards are configured in the OS with the respective VLAN number. MTU in the OS is set to 9000.
All ports are configured as 56 Gbit and shown as so in Windows. To avoid issues with 56 Gbit, I downgraded speeds on ports 1 and 2 to 40 Gbit but to no avail.
Ping doesn't work. I do have firewall exceptions and servers can ping each other using 1 Gbit connections.
TCP doesn't work. I am using iPerf to run a test data stream. Again, works via 1 Gbit.
ARP resolution doesn't work — using arp-ping.exe, won't resolve anything.
So far, all tests on CX4s only.
Switch OS version: PPC_M460EX 3.6.1002 2016-06-09 20:24:26 ppc
System profile is vpi-single-switch.
Current running switch config:
Code:
##
## Running database "boldbrick-dc-3"
## Generated at 2020/12/29 00:24:08 +0100
## Hostname: switch02
##
##
## Running-config temporary prefix mode setting
##
no cli default prefix-modes enable
##
## Port configuration
##
port 1/1 type ethernet force
port 1/2 type ethernet force
port 1/3 type ethernet force
port 1/4 type ethernet force
port 1/5 type ethernet force
port 1/6 type ethernet force
port 1/7 type ethernet force
port 1/8 type ethernet force
port 1/9 type ethernet force
port 1/10 type ethernet force
port 1/11 type ethernet force
port 1/12 type ethernet force
##
## Interface Ethernet configuration
##
interface ethernet 1/1 mtu 9000 force
interface ethernet 1/2 mtu 9000 force
interface ethernet 1/3 mtu 9000 force
interface ethernet 1/3 speed 56000 force
interface ethernet 1/4 mtu 9000 force
interface ethernet 1/4 speed 56000 force
interface ethernet 1/5 speed 56000 force
interface ethernet 1/6 speed 56000 force
interface ethernet 1/7 speed 56000 force
interface ethernet 1/8 speed 56000 force
interface ethernet 1/9 mtu 9000 force
interface ethernet 1/9 speed 56000 force
interface ethernet 1/10 mtu 9000 force
interface ethernet 1/10 speed 56000 force
interface ethernet 1/11 mtu 9000 force
interface ethernet 1/11 speed 56000 force
interface ethernet 1/12 mtu 9000 force
interface ethernet 1/12 speed 56000 force
interface ethernet 1/1 switchport mode hybrid
interface ethernet 1/2 switchport mode hybrid
interface ethernet 1/3 switchport mode hybrid
interface ethernet 1/4 switchport mode hybrid
interface ethernet 1/9 switchport mode hybrid
interface ethernet 1/10 switchport mode hybrid
interface ethernet 1/11 switchport mode hybrid
interface ethernet 1/12 switchport mode hybrid
##
## VLAN configuration
##
vlan 48
vlan 58
interface ethernet 1/1 switchport access vlan 48
interface ethernet 1/2 switchport access vlan 48
interface ethernet 1/3 switchport access vlan 48
interface ethernet 1/4 switchport access vlan 48
interface ethernet 1/9 switchport access vlan 58
interface ethernet 1/10 switchport access vlan 58
interface ethernet 1/11 switchport access vlan 58
interface ethernet 1/12 switchport access vlan 58
vlan 48 name "DC 40 GbE A (primary)"
vlan 58 name "DC 40 GbE B (secondary)"
##
## STP configuration
##
spanning-tree port type edge default
##
## L3 configuration
##
ip l3
ip routing vrf default
interface vlan 48
interface vlan 58
interface vlan 48 ip address 192.168.48.1 255.255.255.0
interface vlan 48 mtu 9000
interface vlan 58 ip address 192.168.58.1 255.255.255.0
interface vlan 58 mtu 9000
##
## DCBX PFC configuration
##
dcb priority-flow-control enable force
dcb priority-flow-control priority 3 enable
interface ethernet 1/1 dcb priority-flow-control mode on force
interface ethernet 1/2 dcb priority-flow-control mode on force
interface ethernet 1/3 dcb priority-flow-control mode on force
interface ethernet 1/4 dcb priority-flow-control mode on force
interface ethernet 1/9 dcb priority-flow-control mode on force
interface ethernet 1/10 dcb priority-flow-control mode on force
interface ethernet 1/11 dcb priority-flow-control mode on force
interface ethernet 1/12 dcb priority-flow-control mode on force
##
## LLDP configuration
##
lldp
##
## Network interface configuration
##
no interface mgmt0 dhcp
interface mgmt0 ip address 192.168.8.3 /24
##
## Network interface IPv6 configuration
##
no interface mgmt0 ipv6 address autoconfig default
no interface mgmt0 ipv6 enable
##
## Other IP configuration
##
ip name-server 192.168.8.21
ip name-server 192.168.8.22
ip name-server 8.8.8.8
ip route 0.0.0.0 0.0.0.0 192.168.8.1
hostname switch02
ip domain-list corp.boldbrick.com
##
## Other IPv6 configuration
##
no ipv6 enable
All networking config as shown in WAC:
Any help will be greatly appreciated.