ESXi 6.7u3 home build went boom - need help troubleshooting

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

voip-ninja

Member
Apr 5, 2019
36
2
8
Hey everyone.

I started putting together new ESXi 6.7 update 3 based home server platform last weekend and have slowly been turning things up.

Intel E2278G CPU
Supermicro X11-SCA-F motherboard
32GB ECC RAM
LSI 9211-8i HBA in IR
2 x 1TB Samsung EVO 860 SSD as boot disk volume (Hardware RAID 1)
Intel 540-T2 2 x 10Gb NIC
Silverstone 400 series case

Now installing ESXi was no big deal. I built a couple of VMs (one Windows 10 Pro for Blue Iris and one Ubuntu 18 for Plex server) and I also added iSCSI storage from my Synology NAS over dedicated 10Gb direct connection.

Initially things working ok other than a few minor issues (can't get rid of chassis intrusion alarm on IPMI, etc.)

This morning I was itching to try to pass through the Intel GPU to the Plex VM and that's when things went sideways.

I put ESXi into maintenance mode, set the Intel GPU to pass through and rebooted the box. When the box came up it had no datastore available, both the iSCSI and integrated 2X1TB RAID were missing. Basic CLI troubleshooting didn't help much with figuring out what was going on as there wasn't even datastore listed that I could mount.

I changed the GPU back (probably didn't have anything to do with the issue but who knows) and rebooted the box again. This time the LSI RAID came back but the iSCSI still gone. Checked a few things and this time decided to do a full power down of the server and back up... this time it comes up and the LSI HBA and iSCSI both gone!

I am scratching my head trying to figure this out. Am I going to have to reboot my NAS to get my iSCSI back up every time ESXi needs a restart? If so this is not cool.

What could explain the intermittent nature of the HBA not being available? Is it that after a software reset the HBA is in a bad state? Do I have to do a hard power down or some other action every time? The card was fairly recently manufactured so I did not bother to update the firmware on it... is it possible that is the issue?

Right now I have flipped from a lot of excitement about getting this new environment up and running to lots of concern that this new environment might not be stable enough for my home.... where it needs to run with lots of up time and provide Plex, surveillance recording and a few other critical services to my family.

Any help is appreciated.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Well Firmware updates of Bios and HBA can't hurt.
Anything in the System Event Log?
Board new or used (potentially run CMOS reset) ?

iSCSI is mounted via ESXi or via Bios Level? If ESXi then try mounting manually instead of rebooting when down - not sure if ESXi auto-reconnects as it does for NFS since I dont use it
 

voip-ninja

Member
Apr 5, 2019
36
2
8
Well Firmware updates of Bios and HBA can't hurt.
Anything in the System Event Log?
Board new or used (potentially run CMOS reset) ?

iSCSI is mounted via ESXi or via Bios Level? If ESXi then try mounting manually instead of rebooting when down - not sure if ESXi auto-reconnects as it does for NFS since I dont use it
Hi Rand thanks for replying.

I wasn't even aware that iSCSI could be set to mount at the BIOS level, I have it configured to mount in ESXi.

The board is new and was already on the latest BIOS that Supermicro came out with in August. I updated IPMI to try to resolve a couple of nuisance issues (updating did not solve them).

I will investigate the version of FW on the HBA... it's an older model and has had ESXi support baked in since at least the 4.0 days so I'd be somewhat surprised if a slightly out of date FW version is causing the problem.

Which log in ESXi would provide the most info? I've reviewed several of them and see what I assume to be issues but I'm not enough of an ESXi ninja to know how to proceed.

Code:
0:00:00:00.000 cpu0:1)WARNING: Serial: 787: Serial port com2 failed during initialization: Failure
0:00:00:04.692 cpu0:2097152)WARNING: VMKAcpi: 307: \_SB_.PCI0.LPCB.UR12: skipping IRQ 10 conflict
0:00:00:04.693 cpu0:2097152)WARNING: VMKAcpi: 318: \_SB_.PCI0.LPCB.TIMR: skipping GSIV 0 conflict
0:00:00:04.698 cpu0:2097152)WARNING: VMKAcpi: 307: \_SB_.PCI0.SIRC: skipping IRQ 14 conflict
0:00:00:04.726 cpu0:2097152)WARNING: Chipset: 396: Bus 12 (06) is already defined
0:00:00:04.735 cpu0:2097152)WARNING: PCI: 225: Non-PF mem (0x0 - 0xfff) out of range
0:00:00:04.735 cpu0:2097152)WARNING: PCI: 452: 0000:00:15.1: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) - out of resources on parent: Root
0:00:00:04.735 cpu0:2097152)WARNING: PCI: 475: 0000:00:15.1: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) status: Already exists
0:00:00:04.741 cpu0:2097152)WARNING: PCI: 225: Non-PF mem (0x0 - 0xfff) out of range
0:00:00:04.741 cpu0:2097152)WARNING: PCI: 452: 0000:00:1e.0: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) - out of resources on parent: Root
0:00:00:04.741 cpu0:2097152)WARNING: PCI: 475: 0000:00:1e.0: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) status: Already exists
0:00:00:04.747 cpu0:2097152)WARNING: PCI: 678: 0000:00:1e.0: Unable to free BAR[0] (MEM64 f=0x4 0x0-0x1000): Not found
2020-02-06T15:43:01.774Z cpu11:2097654)WARNING: Keyboard: 777: Ignoring registration attempt: External USB driver already registered.
2020-02-06T15:43:01.774Z cpu11:2097654)WARNING: VMKAPICore: 2472: Failed to register external keyboard driver, type: 1
2020-02-06T15:43:02.156Z cpu15:2097704)WARNING: etherswitch: PortCfg_ModInit:910: Skipped initializing etherswitch portcfg for VSS to use cswitch and portcfg module
2020-02-06T15:43:02.456Z cpu7:2097629)WARNING: ScsiPath: 8938: Adapter Invalid does not exist
2020-02-06T15:43:02.456Z cpu12:2097631)WARNING: PCI: 1209: 0000:00:14.0 is nameless
2020-02-06T15:43:14.782Z cpu2:2097804)WARNING: ScsiScan: 1633: Failed to add path vmhba1:C0:T0:L0 : Not found
2020-02-06T15:43:14.786Z cpu2:2097804)WARNING: ScsiScan: 1633: Failed to add path vmhba1:C0:T1:L0 : Not found
2020-02-06T15:43:15.146Z cpu3:2097859)WARNING: Tcpip: 1308: failed to unset gateway (error = 0x31)
2020-02-06T15:43:15.253Z cpu6:2097865)WARNING: FBFT not enabled
2020-02-06T15:43:23.610Z cpu0:2097567)WARNING: NFS: 1227: Invalid volume UUID 5e3c289c-0918b602-1544-90e2bad6ccbc
2020-02-06T15:43:33.153Z cpu8:2098674)WARNING: PCI: 189: 0000:01:00.0: Bypassing non-ACS capable device in hierarchy
2020-02-06T15:43:33.156Z cpu8:2098674)WARNING: PCI: 189: 0000:02:00.0: Bypassing non-ACS capable device in hierarchy
2020-02-06T15:43:33.158Z cpu8:2098674)WARNING: PCI: 189: 0000:02:00.1: Bypassing non-ACS capable device in hierarchy
2020-02-06T15:58:04.605Z cpu9:2098736 opID=f3e5628e)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-06T16:02:41.968Z cpu6:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2020-02-06T16:02:41.968Z cpu6:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:qbert.Target-1.264d3b34dc TPGT: 1 TSIH: 0]
2020-02-06T16:02:41.968Z cpu6:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 172.16.100.1:38093 R: 172.16.100.2:3260]
2020-02-06T16:06:07.904Z cpu11:2099751)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2020-02-06T16:06:07.904Z cpu11:2099751)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:qbert.Target-1.264d3b34dc TPGT: 1 TSIH: 0]
2020-02-06T16:06:07.904Z cpu11:2099751)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 172.16.100.1:38093 R: 172.16.100.2:3260]
2020-02-06T16:06:07.909Z cpu11:2099719)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014054429d2dcd6beed4c99db100d9" state in doubt; requested fast path state update...
2020-02-06T16:06:08.325Z cpu11:2099719)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014054429d2dcd6beed4c99db100d9" state in doubt; requested fast path state update...
2020-02-06T16:06:09.313Z cpu11:2099719)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014054429d2dcd6beed4c99db100d9" state in doubt; requested fast path state update...
2020-02-06T16:06:10.299Z cpu11:2099719)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014054429d2dcd6beed4c99db100d9" state in doubt; requested fast path state update...
2020-02-06T16:06:10.934Z cpu7:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2020-02-06T16:06:10.934Z cpu7:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:qbert.Target-1.264d3b34dc TPGT: 1 TSIH: 0]
2020-02-06T16:06:10.934Z cpu7:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 172.16.100.1:50207 R: 172.16.100.2:3260]
2020-02-06T16:06:21.515Z cpu7:2099751)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2020-02-06T16:06:21.515Z cpu7:2099751)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:qbert.Target-1.264d3b34dc TPGT: 1 TSIH: 0]
2020-02-06T16:06:21.515Z cpu7:2099751)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 172.16.100.1:50207 R: 172.16.100.2:3260]
2020-02-06T16:06:21.519Z cpu1:2097483)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:633: vmhba64:CH:0 T:0 L:1 : Task mgmt "Abort Task" with itt=0x120 (refITT=0x11d) timed out.
2020-02-06T16:06:21.519Z cpu1:2097483)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014054429d2dcd6beed4c99db100d9" state in doubt; requested fast path state update...
2020-02-06T16:06:24.545Z cpu6:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2020-02-06T16:06:24.545Z cpu6:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:qbert.Target-1.264d3b34dc TPGT: 1 TSIH: 0]
2020-02-06T16:06:24.545Z cpu6:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 172.16.100.1:28668 R: 172.16.100.2:3260]
2020-02-06T16:10:50.625Z cpu8:2099751)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2020-02-06T16:10:50.625Z cpu8:2099751)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:qbert.Target-1.264d3b34dc TPGT: 1 TSIH: 0]
2020-02-06T16:10:50.625Z cpu8:2099751)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 172.16.100.1:28668 R: 172.16.100.2:3260]
2020-02-06T16:10:50.629Z cpu8:2099716)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014054429d2dcd6beed4c99db100d9" state in doubt; requested fast path state update...
2020-02-06T16:10:51.295Z cpu8:2099716)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014054429d2dcd6beed4c99db100d9" state in doubt; requested fast path state update...
2020-02-06T16:10:52.294Z cpu0:2099708)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014054429d2dcd6beed4c99db100d9" state in doubt; requested fast path state update...
2020-02-06T16:10:53.488Z cpu0:2099708)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014054429d2dcd6beed4c99db100d9" state in doubt; requested fast path state update...
2020-02-06T16:10:53.742Z cpu0:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2020-02-06T16:10:53.742Z cpu0:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:qbert.Target-1.264d3b34dc TPGT: 1 TSIH: 0]
2020-02-06T16:10:53.742Z cpu0:2099751)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 172.16.100.1:44124 R: 172.16.100.2:3260]
2020-02-06T16:22:58.198Z cpu5:2099212 opID=2e3b25d3)WARNING: NFS: 1227: Invalid volume UUID naa.60014054429d2dcd6beed4c99db100d9:1
2020-02-06T16:22:58.216Z cpu5:2099212 opID=2e3b25d3)WARNING: NFS: 1227: Invalid volume UUID 5e3c3d61-5d59f700-f224-90e2bad6ccbc
2020-02-07T07:54:52.018Z cpu5:2097805)WARNING: ScsiDeviceIO: 1564: Device naa.600508e0000000000a2512763418d506 performance has deteriorated. I/O latency increased from average value of 3436 microseconds to 70177 microseconds.
2020-02-09T09:09:14.629Z cpu0:2098687 opID=f042d621)WARNING: PCI: 189: 0000:01:00.0: Bypassing non-ACS capable device in hierarchy
2020-02-09T09:09:14.631Z cpu0:2098687 opID=f042d621)WARNING: PCI: 189: 0000:02:00.0: Bypassing non-ACS capable device in hierarchy
2020-02-09T09:09:14.634Z cpu0:2098687 opID=f042d621)WARNING: PCI: 189: 0000:02:00.1: Bypassing non-ACS capable device in hierarchy
0:00:00:00.000 cpu0:1)WARNING: Serial: 787: Serial port com2 failed during initialization: Failure
0:00:00:04.692 cpu0:2097152)WARNING: VMKAcpi: 307: \_SB_.PCI0.LPCB.UR12: skipping IRQ 10 conflict
0:00:00:04.694 cpu0:2097152)WARNING: VMKAcpi: 318: \_SB_.PCI0.LPCB.TIMR: skipping GSIV 0 conflict
0:00:00:04.698 cpu0:2097152)WARNING: VMKAcpi: 307: \_SB_.PCI0.SIRC: skipping IRQ 14 conflict
0:00:00:04.727 cpu0:2097152)WARNING: Chipset: 396: Bus 12 (06) is already defined
0:00:00:04.735 cpu0:2097152)WARNING: PCI: 225: Non-PF mem (0x0 - 0xfff) out of range
0:00:00:04.735 cpu0:2097152)WARNING: PCI: 452: 0000:00:15.1: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) - out of resources on parent: Root
0:00:00:04.735 cpu0:2097152)WARNING: PCI: 475: 0000:00:15.1: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) status: Already exists
0:00:00:04.741 cpu0:2097152)WARNING: PCI: 225: Non-PF mem (0x0 - 0xfff) out of range
0:00:00:04.741 cpu0:2097152)WARNING: PCI: 452: 0000:00:1e.0: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) - out of resources on parent: Root
0:00:00:04.741 cpu0:2097152)WARNING: PCI: 475: 0000:00:1e.0: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) status: Already exists
0:00:00:04.747 cpu0:2097152)WARNING: PCI: 678: 0000:00:1e.0: Unable to free BAR[0] (MEM64 f=0x4 0x0-0x1000): Not found
2020-02-09T09:18:20.808Z cpu8:2097688)WARNING: Keyboard: 777: Ignoring registration attempt: External USB driver already registered.
2020-02-09T09:18:20.808Z cpu8:2097688)WARNING: VMKAPICore: 2472: Failed to register external keyboard driver, type: 1
2020-02-09T09:18:21.504Z cpu3:2097663)WARNING: ScsiPath: 8938: Adapter Invalid does not exist
2020-02-09T09:18:21.504Z cpu0:2097665)WARNING: PCI: 1209: 0000:00:14.0 is nameless
2020-02-09T09:18:22.127Z cpu5:2097737)WARNING: etherswitch: PortCfg_ModInit:910: Skipped initializing etherswitch portcfg for VSS to use cswitch and portcfg module
2020-02-09T09:18:34.803Z cpu4:2097837)WARNING: ScsiScan: 1633: Failed to add path vmhba1:C0:T0:L0 : Not found
2020-02-09T09:18:34.806Z cpu4:2097837)WARNING: ScsiScan: 1633: Failed to add path vmhba1:C0:T1:L0 : Not found
2020-02-09T09:18:35.269Z cpu7:2097567)WARNING: Device: 1462: Failed to register device 0x4304fba54070 logical#swdevroot#com.vmware.iscsi_vmk0 com.vmware.iscsi_vmk (parent=0x34684304fba543f8): Already exists
2020-02-09T09:18:35.279Z cpu7:2097567)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:18:37.469Z cpu5:2097912)WARNING: FBFT not enabled
2020-02-09T09:18:45.829Z cpu1:2097567)WARNING: NFS: 1227: Invalid volume UUID 5e3c289c-0918b602-1544-90e2bad6ccbc
2020-02-09T09:18:50.027Z cpu13:2098445)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:18:50.457Z cpu2:2098502)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:18:50.565Z cpu4:2098527)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:18:50.930Z cpu9:2098538)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:18:55.288Z cpu5:2098730)WARNING: PCI: 189: 0000:01:00.0: Bypassing non-ACS capable device in hierarchy
2020-02-09T09:18:55.290Z cpu5:2098730)WARNING: PCI: 189: 0000:02:00.0: Bypassing non-ACS capable device in hierarchy
2020-02-09T09:18:55.292Z cpu5:2098730)WARNING: PCI: 189: 0000:02:00.1: Bypassing non-ACS capable device in hierarchy
2020-02-09T09:18:55.421Z cpu5:2098730)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
0:00:00:00.000 cpu0:1)WARNING: Serial: 787: Serial port com2 failed during initialization: Failure
0:00:00:04.693 cpu0:2097152)WARNING: VMKAcpi: 307: \_SB_.PCI0.LPCB.UR12: skipping IRQ 10 conflict
0:00:00:04.694 cpu0:2097152)WARNING: VMKAcpi: 318: \_SB_.PCI0.LPCB.TIMR: skipping GSIV 0 conflict
0:00:00:04.699 cpu0:2097152)WARNING: VMKAcpi: 307: \_SB_.PCI0.SIRC: skipping IRQ 14 conflict
0:00:00:04.727 cpu0:2097152)WARNING: Chipset: 396: Bus 12 (06) is already defined
0:00:00:04.736 cpu0:2097152)WARNING: PCI: 225: Non-PF mem (0x0 - 0xfff) out of range
0:00:00:04.736 cpu0:2097152)WARNING: PCI: 452: 0000:00:15.1: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) - out of resources on parent: Root
0:00:00:04.736 cpu0:2097152)WARNING: PCI: 475: 0000:00:15.1: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) status: Already exists
0:00:00:04.742 cpu0:2097152)WARNING: PCI: 225: Non-PF mem (0x0 - 0xfff) out of range
0:00:00:04.742 cpu0:2097152)WARNING: PCI: 452: 0000:00:1e.0: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) - out of resources on parent: Root
0:00:00:04.742 cpu0:2097152)WARNING: PCI: 475: 0000:00:1e.0: Failed to add BAR[0] (MEM64 f=0x4 0x0-0x1000) status: Already exists
0:00:00:04.748 cpu0:2097152)WARNING: PCI: 678: 0000:00:1e.0: Unable to free BAR[0] (MEM64 f=0x4 0x0-0x1000): Not found
2020-02-09T09:55:47.806Z cpu8:2097688)WARNING: Keyboard: 777: Ignoring registration attempt: External USB driver already registered.
2020-02-09T09:55:47.806Z cpu8:2097688)WARNING: VMKAPICore: 2472: Failed to register external keyboard driver, type: 1
2020-02-09T09:55:49.102Z cpu11:2097737)WARNING: etherswitch: PortCfg_ModInit:910: Skipped initializing etherswitch portcfg for VSS to use cswitch and portcfg module
2020-02-09T09:55:49.739Z cpu2:2097663)WARNING: ScsiPath: 8938: Adapter Invalid does not exist
2020-02-09T09:55:49.739Z cpu4:2097665)WARNING: PCI: 1209: 0000:00:14.0 is nameless
2020-02-09T09:56:01.773Z cpu9:2097837)WARNING: ScsiScan: 1633: Failed to add path vmhba1:C0:T0:L0 : Not found
2020-02-09T09:56:01.777Z cpu9:2097837)WARNING: ScsiScan: 1633: Failed to add path vmhba1:C0:T1:L0 : Not found
2020-02-09T09:56:02.235Z cpu7:2097567)WARNING: Device: 1462: Failed to register device 0x4304fba54070 logical#swdevroot#com.vmware.iscsi_vmk0 com.vmware.iscsi_vmk (parent=0x33aa4304fba543fd): Already exists
2020-02-09T09:56:02.245Z cpu7:2097567)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:56:04.434Z cpu1:2097912)WARNING: FBFT not enabled
2020-02-09T09:56:12.794Z cpu3:2097567)WARNING: NFS: 1227: Invalid volume UUID 5e3c289c-0918b602-1544-90e2bad6ccbc
2020-02-09T09:56:16.954Z cpu4:2098445)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:56:17.381Z cpu0:2098503)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:56:17.491Z cpu2:2098527)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:56:17.849Z cpu14:2098538)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
2020-02-09T09:56:22.174Z cpu4:2098730)WARNING: PCI: 189: 0000:01:00.0: Bypassing non-ACS capable device in hierarchy
2020-02-09T09:56:22.176Z cpu4:2098730)WARNING: PCI: 189: 0000:02:00.0: Bypassing non-ACS capable device in hierarchy
2020-02-09T09:56:22.179Z cpu4:2098730)WARNING: PCI: 189: 0000:02:00.1: Bypassing non-ACS capable device in hierarchy
2020-02-09T09:56:22.307Z cpu4:2098730)WARNING: VisorFS: 1093: Attempt to setattr non sticky dir/file from tar mount
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
iSCSI in Bios or card bios needs a NIC that support it, not sure if the 540 does it, been a while that I used them.

System Event Log is the Log in IPMI, not esxi, but o/c those are relevant too, usually it would be vmkernel.log.

The above reminds me of early attempts to set bifurcation (sharing of pci devices) - are you sure you have not set too many devices to be passed through? Or wrong the ones :p? Auto added ones you did not take out?


Else - is this a known good x540 or a cheap knockoff?
For some reason after boot the network was not working ok (thus no iSCSI) - now it can either be the NIC or the pcie distribution/resource handling.
If you cant get it to run shortly I'd say open a case with Supermicro, they don't know much ESXi (and likely wont cover that part), but its a hardware change that induced the issue and maybe they have a recommendation.
 

voip-ninja

Member
Apr 5, 2019
36
2
8
I've made a bit of headway on this last night as I couldn't sleep and was working on it.

I moved the server into its permanent rack location, booted it up and (knocks on wood) it mounted the internal SSD RAID set no issue. I'm now a bit suspicious that something with the way it was restarted maybe caused it to not mount.

Now, the iSCSI piece gets a bit more interesting. I was able to ping no issue between the NAS and the 10Gb card which are on a dedicated crossed over connection. I then started looking into why there was no volume to mount on the ESXi host and saw that it had no targets for iSCSI configured. When I set it up I auto-populated the targets but then removed all targets other than the one on the cross-over connection because I didn't want the storage traffic going to the switch that's handling the rest of my network.... I also wanted to make sure I could run a 9,000 MTU between the devices with no issues.

So, for whatever reason the iSCSI mount issue was related to ESXi losing the targets to the share.

Looking into it a bit further it seems that iSCSI with NAS is known for being somewhat unreliable in this regard. I set up an NFS share on the Synology and that seems to be working.

I dug a bit further into the GPU pass through piece and unfortunately it seems that the shiny new CPU I just spent $480 on is not yet supported for GPU passthrough. At least not that I could discover. This is disappointing as I waited a really long time to get ahold of this GPU specifically to be able to do pass-through to Plex.

I am using both PCIE v3 card slots (one for NIC and one for HBA) so I'll have to see if there is a compatible NIC that will fit in one of the short slots on the motherboard. Otherwise looks like I'm SOoL on getting that part of my new build going at least until a driver/patch to ESXi makes it happen.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
just looked at the board - there sure is a lot of pcie sharing going on...
Have you set the x16 slots to auto or hardcoded 8/8 (if you can?)

And given the fact that neither of your cards actually needs a pci-e v3 x8 slot as both are pcie- v2 devices you could for a x4 replacement.
eg PCIe 3.0 x4 10G Dual Copper Server Adapter-Intel X550AT2 Chip-X550-T2 Compatible (no endorsement, first thing i found with a x4 connector)

Have not found any good x4 SAS Raid card though :/
 

voip-ninja

Member
Apr 5, 2019
36
2
8
Yes there are a lot of things shared between PCIE slots and for example the M.2 slots.

I bought 2 of the Intel cards since they were on the Synology compatibility list and got copper DAC cable for them since it's inexpensive (total cost for everything $200). I want to keep power consumption on this setup low (which is why I was going to use the Intel integrated GPU) and I really only needed the GPU so I could start to handle transcode of 4K video.

Now that these new CPUs are coming online I hope that the next ESXi update/patch will add support for it.

I suppose there could also be some option in the UEFI that I missed that enables pass-through. Right now though ESXi definitely doesn't see it... I have only scratched the surface so I might try to hit up the hardware forum to see if anyone has more information on using this graphics core with ESXi.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
You might need to make sure that the bios uses the BMCs GPU as primary so ESXi will use that and not the igpu or you can't pass that through...
 

voip-ninja

Member
Apr 5, 2019
36
2
8
You might need to make sure that the bios uses the BMCs GPU as primary so ESXi will use that and not the igpu or you can't pass that through...
Thanks, I had not thought of that, just assumed that the VGA port on the mobo would be using the integrated video.