Figured i'd post this for people who are looking for info on this riser and X12DPU migration from X11DPU.
So I got this X12DPU-6 system and was building it inside a different chassis (in SYS-6029U-E1CR4T, based on 829U-10 chassis, X11DPU was there previously), its been a long process. Many standoffs dont match and there are no holes corresponding to X12DPU in this chassis so i have at least 3 or 4 removed and missing.
X10DRU-i+, X11DPU and X12DPU all use same PSU sets. PWS-1K02A-1R, PWS-1K62A-1R and PWS-2K08A-1R are all fine and have same pinouts.
I installed this AOC-SMG3-2M2-U-NI22 riser that came with it(true supermicro pn is AOC-SMG3-2M2-U, the -NI22 suffix is from the fact it's nutanix OEM labeled), and put 2 nvmes there and I couldnt see them in the system. I figured I'd check in bios and there it was, yes. You could only configure it as JBOD or raid1 or raid0(but no direct access for smart-log data, nvme sensors are reported though via standard sysfs interface).
Then I noticed it has this USB(micro USB) connector. I attached it to the system, figured out I need to switch no 115200, 8N1 no soft/hard control and there was a text console. Very dangerous one. My drives were empty I didn't care, i started typing various characters and it was outputting them back as
and so on.
all responses from the console are prefixed with ticks number since controller boot.
When I typed T it executed secure erase on 0x1 without asking lol
s - secure erase on drive 0x0
t - secure erase on drive 0x1
u - secure erase on drive 0x2
v - secure erase on drive 0x3
if drive is installed secure erase procedure looks liek this:
# 15 seconds later host requests drives info?
# 57 seconds later it's done
i can confirm on the host the ext4fs that was on this first drive(VD) is gone but it was there and mounting before i pressed 's':
i didn't experiment much but raw readout performance of a clean(secure erased) samsung 990 pro is at ~3GB/s as reported by pv.
this is how the drives installed into controller are represented in OS:
note how they are from same nvme0 but two namespaces.
temperatures per drive are exposed via sysfs but no smart data is. or maybe i dont know the commands for this drive. specifically some USB-SATA bridges have special command prefix for accessing SMART of underlying drive.
M2SSD is probably max of sensors 2 and 3(2 = port0, 3 = port1 i presume), +1 or 2 degrees. or it just updates slowly.
MRVL has to be the controller chip temp. It's sensor 1.
other keys i found:
g prints this
i prints this
usually prints this when blkid requested
Some other commands I found: R is probably to read some registers/memory, it prints
Pretty interesting. But then i typed r00000001 and it totally yelled at me, exception occured, printed liux-like kernel dump, with registers and stalled. NVMe VD it was supposed to rpesent was no longer responding. Btw any command to controller in the past printed something in console but not anymore.
if you stall the controller i found no way to reset it outside the power cycle. i tried telling OS via sysfs to reset it, remove it but it never came back. of course the exposed NVMe fell off first.
that's about it. not experimenting anymore for now.
So I got this X12DPU-6 system and was building it inside a different chassis (in SYS-6029U-E1CR4T, based on 829U-10 chassis, X11DPU was there previously), its been a long process. Many standoffs dont match and there are no holes corresponding to X12DPU in this chassis so i have at least 3 or 4 removed and missing.
X10DRU-i+, X11DPU and X12DPU all use same PSU sets. PWS-1K02A-1R, PWS-1K62A-1R and PWS-2K08A-1R are all fine and have same pinouts.
I installed this AOC-SMG3-2M2-U-NI22 riser that came with it(true supermicro pn is AOC-SMG3-2M2-U, the -NI22 suffix is from the fact it's nutanix OEM labeled), and put 2 nvmes there and I couldnt see them in the system. I figured I'd check in bios and there it was, yes. You could only configure it as JBOD or raid1 or raid0(but no direct access for smart-log data, nvme sensors are reported though via standard sysfs interface).
Then I noticed it has this USB(micro USB) connector. I attached it to the system, figured out I need to switch no 115200, 8N1 no soft/hard control and there was a text console. Very dangerous one. My drives were empty I didn't care, i started typing various characters and it was outputting them back as
Code:
[0000000040]uart rx hex: [0x71] ascii:[q]all responses from the console are prefixed with ticks number since controller boot.
When I typed T it executed secure erase on 0x1 without asking lol
s - secure erase on drive 0x0
t - secure erase on drive 0x1
u - secure erase on drive 0x2
v - secure erase on drive 0x3
if drive is installed secure erase procedure looks liek this:
Code:
[0000051900]Secure erase on drive 0x0 start !
[0000051900]issue format with secure erase! please wait ...
[0000051900]uart rx hex: [0x73] ascii:[s]
Code:
[0000051915]admin op 0xe2
[0000051915]simple api cdw10 [0xe1 0x0 0x0 0x0 0x0]
[0000051915]simple api callback op e1 sub 0 cmd_status 0x0
[0000051915]simple api callback, type XOR_REQUEST_DMA !op e1 sub 0 src 0xe0461000
[0000051915]admin op 0xe2
[0000051915]simple api cdw10 [0xe1 0x0 0x0 0x0 0x1]
[0000051915]simple api callback op e1 sub 0 cmd_status 0x0
[0000051915]simple api callback, type XOR_REQUEST_DMA !op e1 sub 0 src 0xe0461000
[0000051915]admin op 0xe2
[0000051915]simple api cdw10 [0xe0 0x0 0x0 0x0 0x0]
[0000051915]0xe1248158 : 0x3f
[0000051915]Option Rom 1 size=0x13000(Checksum U32=0x31edf8b0)
[0000051915]UEFI VERSION[0.0.00.0003]
[0000051915]0xe1248158 : 0x3f
[0000051915]simple api callback op e0 sub 0 cmd_status 0x0
[0000051915]simple api callback, type XOR_REQUEST_DMA !op e0 sub 0 src 0xe0461000
[0000051915]admin op 0xe2
[0000051915]simple api cdw10 [0xe2 0x0 0x0 0x0 0x0]
[0000051915]======= VD 0 [0000051915]is found in ExpTable 0, [0000051915]and Gen exactly match.
[0000051915]simple api callback op e2 sub 0 cmd_status 0x0
[0000051915]simple api callback, type XOR_REQUEST_DMA !op e2 sub 0 src 0xe0461000
[0000051916]admin op 0xe2
[0000051916]simple api cdw10 [0xe3 0x2 0x0 0x0 0x0]
[0000051916]simple api callback op e3 sub 2 cmd_status 0x0
[0000051916]simple api callback, type XOR_REQUEST_DMA !op e3 sub 2 src 0xe0461000
Code:
[0000051957]Secure erase on drive 0x0 complete!
Code:
mount -o ro /dev/nvme0n1 /mnt/tmp/
mount: /mnt/tmp: wrong fs type, bad option, bad superblock on /dev/nvme0n1, missing codepage or helper program, or other error.i didn't experiment much but raw readout performance of a clean(secure erased) samsung 990 pro is at ~3GB/s as reported by pv.
this is how the drives installed into controller are represented in OS:
Code:
# nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 Marvell_NVMe_Controller 1 4.00 TB / 4.00 TB 512 B + 0 B 10001051
/dev/nvme0n2 Marvell_NVMe_Controller 2 4.00 TB / 4.00 TB 512 B + 0 B 10001051
/dev/nvme1n1 ..... my other drive 1 3.20 TB / 3.20 TB 512 B + 0 B ........temperatures per drive are exposed via sysfs but no smart data is. or maybe i dont know the commands for this drive. specifically some USB-SATA bridges have special command prefix for accessing SMART of underlying drive.
Code:
# sensors nvme-pci-5100;ipmitool sdr | grep -E 'M2|MRV'
nvme-pci-5100
Adapter: PCI adapter
Composite: +66.8°C (low = -273.1°C, high = +95.8°C)
(crit = +99.8°C)
Sensor 1: +66.8°C (low = -273.1°C, high = +95.8°C)
Sensor 2: +44.9°C (low = -273.1°C, high = +95.8°C)
Sensor 3: +38.9°C (low = -273.1°C, high = +95.8°C)
MRVL Temp | 67 degrees C | ok
M2SSD Temp | 45 degrees C | okMRVL has to be the controller chip temp. It's sensor 1.
other keys i found:
g prints this
Code:
[0000001370]uart rx hex: [0x67] ascii:[g]
[0000001371] --- raid_create RAID mode 0 SBS 0x400, disk cnt 0, ns cnt 1, pOrgReq 0x0
[0000001371]>PD_Disk[0] Valid=3, State 1 (DID 0, size 0xd1c0beb0)
[0000001371]>PD_Disk[1] Valid=3, State 1 (DID 1, size 0xd1c0beb0)
[0000001371]>PD_Disk[2] Valid=0, State 0 (DID 0, size 0x0)
[0000001371]>PD_Disk[3] Valid=0, State 0 (DID 0, size 0x0)
Code:
[0000051582]======= VD 0 [0000051582]is found in ExpTable 0, [0000051582]and Gen exactly match.
[0000051582]======= VD 1 [0000051582]is found in ExpTable 1, [0000051582]and Gen exactly match.
Code:
[0000001452]admin op 0xe2
[0000001452]simple api cdw10 [0xe1 0x0 0x0 0x0 0x0]
[0000001452]simple api callback op e1 sub 0 cmd_status 0x0
[0000001452]simple api callback, type XOR_REQUEST_DMA !op e1 sub 0 src 0xe0461000
[0000001452]admin op 0xe2
[0000001452]simple api cdw10 [0xe1 0x0 0x0 0x0 0x1]
[0000001452]simple api callback op e1 sub 0 cmd_status 0x0
[0000001452]simple api callback, type XOR_REQUEST_DMA !op e1 sub 0 src 0xe0461000
[0000001452]admin op 0xe2
[0000001452]simple api cdw10 [0xe0 0x0 0x0 0x0 0x0]
[0000001452]0xe1248158 : 0x3f
[0000001452]Option Rom 1 size=0x13000(Checksum U32=0x31edf8b0)
[0000001452]UEFI VERSION[0.0.00.0003]
[0000001452]0xe1248158 : 0x3f
[0000001452]simple api callback op e0 sub 0 cmd_status 0x0
[0000001452]simple api callback, type XOR_REQUEST_DMA !op e0 sub 0 src 0xe0461000
[0000001452]admin op 0xe2
[0000001452]simple api cdw10 [0xe2 0x0 0x0 0x0 0x0]
[0000001452]======= VD 0 [0000001452]is found in ExpTable 0, [0000001452]and Gen exactly match.
[0000001452]simple api callback op e2 sub 0 cmd_status 0x0
[0000001452]simple api callback, type XOR_REQUEST_DMA !op e2 sub 0 src 0xe0461000
[0000001452]admin op 0xe2
[0000001452]simple api cdw10 [0xe3 0x2 0x0 0x0 0x0]
[0000001452]simple api callback op e3 sub 2 cmd_status 0x0
[0000001452]simple api callback, type XOR_REQUEST_DMA !op e3 sub 2 src 0xe0461000
Code:
[number]r and waits for input number of register i presume. if you type eight zeroes it will print
[code]reg 0x0 = 0xe59ff018if you stall the controller i found no way to reset it outside the power cycle. i tried telling OS via sysfs to reset it, remove it but it never came back. of course the exposed NVMe fell off first.
that's about it. not experimenting anymore for now.