Need help with starting 10gb setup

TeleFragger

Member
Oct 26, 2016
151
19
18
48
any help greatly appreciated... warning... n00b at this ...


Ok so I'm new to this and I have a test lab at work.

I have 4 esxi 6.7 boxes - will get to them later
a windows 10 pc (dell precision 5810)

Our guys were getting rid of a mess of hardware so I grabbed it and figured time to play around.
I stuck a mellanox card in the win10 box and drivers installed, albeit, MS drivers. but it seems to be good?
I also connected a cable to the switch and the light on the switch lit up but in win10 the NIC says cable unattached.

switch - Flextronics F-x430066 8 Port 4x SDR Infiniband
cable - Mellanox Mcc4l30-300 Microgigacn Latch 0.3 M Infiniband Cable
cards - HPE InfiniBand 4X DDR Conn-X PCI-E G2 Dual Port HCA (483514-B21)

so few questions...
what is needed on win10 side other than what I have done?

if I put another card in a secondary pc, and just attach both to the switch, just give static IP's and that is it?

so out of the manual... I get the green light with just the cable from pc to switch and nothing else to switch...

The GREEN LED indicator to the left of each port will light when the physical connection is established (that is, when the unit is powered on and a cable is plugged into the port with a functioning port plugged into the other end of the connector)

in Windows 10 dev manager - shows using Mellanox drivers signed by Microsoft
ipoib6x.sys - 4/25/2018 ver. 5.50.14643.0

C:\>mst status -v
MST devices:
------------
mt26418_pci_cr0 bus:dev.fn=01:00.0
mt26418_pciconf0 bus:dev.fn=01:00.0

C:\>mlxfwmanager -d mt26418_pci_cr0 --query
Querying Mellanox devices firmware ...
FATAL - Can't find device id.
FATAL - Can't find device id.
Device #1:
----------
Device Type: N/A
Part Number: --
Description:
PSID:
PCI Device Name: mt26418_pci_cr0
Port1 MAC: N/A
Port1 GUID: N/A
Port2 MAC: N/A
Port2 GUID: N/A
Versions: Current Available
FW --
Status: Failed to open device
 

TeleFragger

Member
Oct 26, 2016
151
19
18
48
I learned a lot and have them working in win10 and esxi 6.7 as Ethernet... caveat is that I can only get the top port working and the bottom wont. even though ibstat states that there are 2 ports, windows sees 2 ports but plug cable in and no link lights.. ughhhh
since I wasn't getting any replies here, I also posted elsewhere.. hope it isn't against the rules to post this and if so mod plz remove link...

starting to play with 10gb - Mellanox and need help


if you want to see all that ive done and where ive gone...
 

billc.cn

Member
Oct 6, 2017
44
9
8
To get infiniband to work, you will need a fabric manager and an infinite amount of time to set the whole network up...
 
  • Like
Reactions: Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,076
5,094
113
I am going to give a bit of perspective here. You can get a 10GbE switch inexpensively. You can also hey SFP+ 10GbE cards inexpensively.

Ethernet is significantly easier to setup and integrate than IB. At 56Gbps or higher IB has some advantages. Anything slower I would just do Ethernet.
 
  • Like
Reactions: TeleFragger

TeleFragger

Member
Oct 26, 2016
151
19
18
48
I am going to give a bit of perspective here. You can get a 10GbE switch inexpensively. You can also hey SFP+ 10GbE cards inexpensively.

Ethernet is significantly easier to setup and integrate than IB. At 56Gbps or higher IB has some advantages. Anything slower I would just do Ethernet.
so I am not doing IB at all.. I am doing IP and got it all going. Albeit not totally how I want as I want to use a switch but for now...

you say
You can get a 10GbE switch inexpensively. You can also hey SFP+ 10GbE cards inexpensively.
can you elaborate with part numbers and prices so I can see what "inexpensive" is?

so far I have $0 into it doing a pier to pier setup. My ESXI 6.7 box will only see these cards as ethernet, which is fine with me and it is working with my freenas box as well.

I wouldn't mind upgrading to sfp+ but as stated... define inexpensive as everything I have seen (for 5 pc's) will run me about $500 at a minimum and my budget is maybe $150