@dragonme
Sure no problem.
I have been where you were (using local disks as RDM), but I learned that it's actually really important (in regards to smart and error detection) to use passthrough. Thereby you should be able to prevent a failure before it occurs.
I have to be honest, I tried to picture the setup in my head but I am not exactly sure if I understood you correctly when you explained how the server is put together.
I can understand you got the S5520hc board, which unfortunately just has a single SATA controller. I am using a S2600CP in one of my boxes which has both a onboard SAS and an onboard SATA controller. I looked at the intel page and I found some expansion options for a raid controller...
Intel® Integrated Server RAID Module AXXRMS2LL040 Product Specifications <- this one could be an option, although it does not have "jbod" mode listed. however, this is the controller you could use for ESXi-based storage and then passthrough the onboard sata controller to your ZFS VM. This module has 4 ports but there's other options too:
Intel® Server Board S5520HC Product Specifications
What version of ESXi 6.5 are you running? In the Windows vSphere Client you are able to do it under Storage -> Devices. I have not seen an option for this in the ESXi HTML5 client yet. In the vSphere Web Client (vCenter only), you can go to the Host -> Configure -> Storage Devices -> Select your device -> All Actions -> Detach.
For the CLI-way, ther eare commands to do this through SSH or in the ESXi-shell. Be sure you detach the correct device. You should not be able to detach one which is already in use, but you never know, it can cost valuable data and time if you mess it up.
First, get a list of the devices:
esxcli storage core device list <- Will list all your devices with a lot of info. You need the ID which is right above the "Display Name" field, example:
Code:
[root@esxi01:~] esxcli storage core device list
t10.NVMe____INTEL_SSDPE2MD020T4C____________________CVFT4280002C2P0KGN__00000001
Display Name: Local NVMe Disk (t10.NVMe____INTEL_SSDPE2MD020T4CCVFT4280002C2P0KGN__00000001)
...
Status: on
...
Note that the "Status" shows as "on"
Now, I have a NVMe disk I wanna detach, so use the device name "t10.NVMe____INTEL_SSDPE2MD020T4C____________________CVFT4280002C2P0KGN__00000001"
Now, again, be sure you aren't using the device for anything. It should fail if it's in use, but again, you never know.
So, to then detach the device, in this example my NVMe drive:
Code:
[root@esxi01:~] esxcli storage core device set -d t10.NVMe____INTEL_SSDPE2MD020T4C____________________CVFT4280002C2P0KGN__00000001 --state=off
It should not return any message if successful. If unsuccessful it might say "device/system busy" or similar.
If you do another "esxcli storage core device list" command again, note the "Status" line..
Code:
t10.NVMe____INTEL_SSDPE2MD020T4C____________________CVFT4280002C2P0KGN__00000001
Display Name: Local NVMe Disk (t10.NVMe____INTEL_SSDPE2MD020T4CCVFT4280002C2P0KGN__00000001)
...
Status: off
...
In the ESXi HTML5 Client you should see that it is now showing the status as "Unknown" since the disk is now detached.
Do a rescan of all storage in the ESXi HTML5 Client or do it through esxcli as well...
Code:
[root@esxi01:~] esxcli storage core adapter rescan --all
Now you should be able to remove the disk and insert a new one. Do a rescan again after inserting the new one and it should show up as online. If the new one doesn't show up as online, you can manually attach it with (using my NVMe device as example again)
Code:
[root@esxi01:~] esxcli storage core device set -d t10.NVMe____INTEL_SSDPE2MD020T4C____________________CVFT4280002C2P0KGN__00000001 --state=on
I hope this helped.