ESXi crash with AOC-SLG3-2E4 and Intel DC P3600 Passthrough

[K D]

They are not getting recognized as flash though (?)

They are. The one 750 I tried to mark as capacity shows iscapacityflash as 1. The rest show as isssd 1


Sent from my iPhone using Tapatalk

 

[K D]

If I were you I would still check if they had a gpt label. That can also cause issues.

As far as I can see hey don't. Partedutil does not list any partitions.

Is there any specific command I can run to verify?


Sent from my iPhone using Tapatalk

 

[Rand__]

They are. The one 750 I tried to mark as capacity shows iscapacityflash as 1. The rest show as isssd 1

Hm I see, sorry. Must have been looking at the wrong drive, sorry.

PartedUtil should be able to list/set the label

 

[whitey]

Images from my vSAN readiness notes/pdf I put together when I was still testing vSAN back in the day.





See if those help at all. Probably a gold bugget/tidbit in there for ya'! :-D

 

[K D]

Thanks. Will check these out. Started a new assignment that is sucking up all my time with regular travel and I'm not able to spend much time on this. The plan for this weekend is to somehow get the nodes to work. I have the direct attach cables also ready to check out.

 

[K D]

The disks have no partitions on them. This is the error I get with any combination of SSD .

[root@esxd2:~] esxcli vsan storage tag add -d t10.NVMe____INTEL_SSDPE2MW800G4___
__________________CVCQ5301001F800FGN__00000001 -t capacityFlash
[root@esxd2:~] vdq -q
[
   {
      "Name"     : "t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ5301  001F800FGN__00000001",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "1",
"IsCapacityFlash": "1",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ5452  009D800FGN__00000001",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "1",
"IsCapacityFlash": "1",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "t10.NVMe____INTEL_SSDPEDMD400G4_____________________PHFT5363  00TT400BGN__00000001",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "1",
"IsCapacityFlash": "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "t10.ATA_____SuperMicro_SSD__________________________SMC0515D  90717B6M4194",
      "VSANUUID" : "",
      "State"    : "Ineligible for use by VSAN",
      "Reason"   : "Has partitions",
      "IsSSD"    : "1",
"IsCapacityFlash": "0",
      "IsPDL"    : "0",
   },

]
[root@esxd2:~] esxcli vsan storage add -s t10.NVMe____INTEL_SSDPEDMD400G4_______
______________PHFT536300TT400BGN__00000001 -d t10.NVMe____INTEL_SSDPE2MW800G4___
__________________CVCQ5301001F800FGN__00000001
Unable to add device: Failed to reserve disk t10.NVMe____INTEL_SSDPE2MW800G4____  _________________CVCQ5301001F800FGN__00000001 with exception: Failed to write VS  AN intent log Failed to write partition
[root@esxd2:~]

 

[Rand__]

I assume you don't have another U2/pcie converter you could try?

 

[K D]

I assume you don't have another U2/pcie converter you could try?

Tried with 2 different cards and 6 different drives. No issues with AIC. Only with U.2. I should be getting the cables 8639 to 2.5 ssd cables delivered today but I won't be able to test them till the weekend.

I suspect it is an issue with the backplane's compatibility. I've not been able to find a single supermicro SKU which uses this backplane with this card. There is a newer version of this card that uses Oculink instead of 8639 that I found last night. Will see if I can buy one of those to test.

 

[Rand__]

Nah I meant a single adapter card like this one just for testers.
Cheap Funtin Alternative from China - $12

But If you still could not try without backplane then this should come first as this is probably the only thing i do differently


I that backplane officially good for nvme? I thought that a regular SAS3 backplane is missing the extra nvme pins - i.e. its not working electrically even if the connectors fit physically.
I know @AJXCR was pondering general direct attached SAS3 nvme compatibility this in another thread a while back but I think the outcome was the above.

 

[K D]

Nah I meant a single adapter card like this one just for testers.
Cheap Funtin Alternative from China - $12

But If you still could not try without backplane then this should come first as this is probably the only thing i do differently


I that backplane officially good for nvme? I thought that a regular SAS3 backplane is missing the extra nvme pins - i.e. its not working electrically even if the connectors fit physically.
I know @AJXCR was pondering general direct attached SAS3 nvme compatibility this in another thread a while back but I think the outcome was the above.

Yeah. This backplane is officially certified for NVME - BPN-SAS3-826A-N4. It has 8 SAS and 4 Hybrid ports. Has a total of 3 black SAS connectors and 4 White NVME connectors.

I stocked up on U.2 drives at a premium over AIC hoping to use this backplane . I need 4 drives per host ( Using your setup of 2 drive groups). I'll order a couple of those funtin clones as well.

 

[Rand__]

Nice - hope you get it going

Edit: Looking at the BPN it makes me wonder why the nvme drives are at the PSU side where they get the least aamount oof cooling (due to no fan).

Or maybe thts only the case in my boxes where the PSU is usually mostly idle and not blowing much air...

 

[K D]

The cases that use these backplanes in the supermicro ultra skus have 4 fans instead of 3. The PSU is mounted under the motherboard.

 

[Rand__]

Didnt look like that here, thats why I was wondering

Supermicro | Products | SuperServers | 2U | 6028R-TDWNR

 

[K D]

I did not see that one. I was looking at the Ultra Servers. Example Supermicro | Products | SuperServers | 2U | 6029U-E1CR4

 

[K D]

Checked with 8639 cables directly connected to the SSD from the card and still have the same issues. It works with 2 AICs but not with an AIC and a 2.5 mm drive . I get the same error. I am stumped.

[root@localhost:~] esxcli vsan storage add -s t10.NVMe____INTEL_SSDPEDMD400G4D_HHHL_NVMe_400GB____CVFT44530044400BGN_
_00000001 -d t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ5301001F800FGN__00000001
Unable to add device: Failed to reserve disk t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ5301001F800FGN__00000001 with exception: Failed to write VSAN intent log Failed to write partition
[root@localhost:~]

 

[K D]

OK... I'm unable to create a VMFS partition on the disks nor format them. The AIC 3700 does not have any issues. It's just the 2.5 inch drives.

[root@localhost:~] dd if=/dev/zero of="/vmfs/devices/disks/t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ53
01001F800FGN__00000001" bs=512 count=34 conv=notrunc
dd: /vmfs/devices/disks/t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ5301001F800FGN__00000001: Input/output error
[root@localhost:~] dd if=/dev/zero of="/vmfs/devices/disks/t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ53
01001F800FGN__00000001" bs=512 count=34 seek=1562824334 conv=notrunc
dd: /vmfs/devices/disks/t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ5301001F800FGN__00000001: Input/output error
[root@localhost:~] partedUtil mklabel /vmfs/devices/disks/t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ530
1001F800FGN__00000001 msdos
Error: Input/output error during write on /dev/disks/t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ5301001F800FGN__00000001
WriteNewPtable: Unable to commit to device /vmfs/devices/disks/t10.NVMe____INTEL_SSDPE2MW800G4_____________________CVCQ5301001F800FGN__00000001

 

[Rand__]

Do you have another board to test with?
My card is running on a x10 board - maybe the x11/bios is not working properly with this card?
Similar in other OS's?
Can you get support from SM?

 

[K D]

Having similar issues with Windows 2012 R2 as well. I will check it out with an X10 board. Don't know why I didn't think of that.

 

[K D]

Worked like a charm on a X10DRH-iT board. I guess this card is not compatible with the X11 boards. I have an updated version of this card that is certified for X11 boards on order. It'll be here by the end of this week.

 

[T_Minus]

So SuperMicro AOC-SLG3-2E4 (NVME AIC) does not work, just making sure I'm following the parts from original post to now?

Have you tried a Funtin adapter instead to see?