Mellanox ConnectX-3 Cards show as cable unplugged but I could have done almost anything wrong.

[Deano]

So it's my first network that wasn't plugged in via a telephone cable or a Cat Cable and my first server (Unraid - I'll call it Upstairs Computer) I'm really enjoying having a file server and VM's to do my video production remotely from downstairs without making Downstairs Computer basically unusable. Except all my Traffic, both file and internet is running through my poor Router because my ConnectX-3 cards show as cable unplugged.

Downstairs is a PRIME Z690-A｜Motherboards｜ASUS USA and it's in the lowest slot
Upstairs is a https://www.msi.com/Motherboard/Z370-SLI-PLUS/Specification and it's in the lowest slot

All Dust Caps removed and everything plugged in in a much cleaner environment than this might look

Q1. I don't know if this is my problem already because i didn't check the specifications before i brought the cards:
2x Mellanox CX354A ConnectX-3 FDR Infiniband 40GbE PCIex8
Prolabs OM3-24C-MPO/F-13.72M Fibre Optic Cable
Finisar FTL414QL2C-E5 QSFP 4X14Gpbs FDR Transceiver

Q2. I don't know if i've plugged them in right, It's quite different from putting a Cat Cable straight into the motherboards IO Cutout location, but it's such a simple thing I couldnt exactly find a lot of videos on it. I found a picture here Assemble the Optical Fiber InfiniBand Cables - Sun Datacenter InfiniBand Switch 36 HTML Document Collection for Firmware Version 2.1 but that doesnt mean i couldn't have plugged it in upside down. the label on my Finisar's is on the top when plugging it into a desktop tower case

Q3. Will the network display cable disconnected even if everything is plugged in correctly, but i haven't made a connection to the second computer? or is cable disconnected more of an immediate problem for each computer? I made downstairs IP 10.10.20.197 and subnet 255.255.255.0 rest blank, and upstairs 10.10.20.198/24

Both cards were flashed and everything on the same windows 10 same day

running mstconfig -d 03:00.0 q on Upstairs gives me:
Device type: ConnectX3
Device: 03:00.0

Configurations: Next Boot
SRIOV_EN False(0)
NUM_OF_VFS 8
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 True(1)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 PXE(1)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 True(1)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 PXE(1)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)

Downstairs mlxconfig -d /dev/mst/mt4099_pci_cr0 query gives the exact same results as above

running flint -d mt4099_pci_cr0 query full on downstairs gives:
Image Type: FS2
FW Version 2.42.5000
FW Release date 5.9.2017
MiC Version 2.0.0
Config Sectors 1
PRS Name cx3-1_MCX354A_fdr_09v.prs
Product Version 02.42.50.00
Rom Info: type=PXE Version=3.4.752
Device ID:4099
GUIDS: lots of numbers
Macs Lots of numbers
VSD: -blank here-
PSID MT_1090120019



added a couple of pictures here, honestly i'm ready to try anything, nothing is too foolproof for me

 

[alaricljs]

The close up looks upside down and not fully inserted. Typically you can see the little hook-catches on one side of the module and those go toward the pcb of the card as you insert it.

 

[alaricljs]

Oh, and if you can slide the module back out without pulling the plastic loop then it wasn't in all the way.

 

[Deano]

The close up looks upside down and not fully inserted. Typically you can see the little hook-catches on one side of the module and those go toward the pcb of the card as you insert it.

Wow! if it's just that, then that would be incredible, once my VM has finished encoding tonight's videos and I wake up i'll spin down the Unraid Array and try to plug everything in again the other way up. Here's hoping! (and hoping I didn't do any damage)

 

[Deano]

Oh, and if you can slide the module back out without pulling the plastic loop then it wasn't in all the way.

Oh wow, that could be a very important both for now and if i get into this further and get some more PC's, Switches and Network cards, the pulling-on-it check! That's good design, Mellanox seem very clever.

 

[Deano]

Thoroughly checked the Transceiver both for how far it's plugged in and if it's the right way up, pulling back on it won't release it, only the tab does, and trying to turn it up the other way won't even let it plug in half as far, it just stops. The transceiver also makes a good clicking sound when it's in as far as it is, and won't go forwards or backwards.
So I'm back to square one, with one thing checked off.