TLDR: 9300-8i card looked to be the cause of the issue. 9211-8i card works!
Ok latest update of the following methods from mentioned above but also some other options that I tried:
3. Test 2 HGST 8TB drives with 9300-8i card
- What type of cable would required to connect the 8TB drives directly? Would I want a reverse, forward, or sideband breakout?
Supermicro eStore - Networking / SAS / SATA Cables
- Connect 2 HGST 8TB drives to 9300-8i breakout cable
- Test with both baremetal Freenas install and ESXI Freenas VM.
This option exhibited the exact same behavior as seen before. This also excluded the backplane even being connected. Tried multiple drives, swapping cables around, and even updating to various firmware versions but could not fix. Never got to testing ESXI as it failed under baremetal Freenas setup.
6. Test 2 HGST 8TB drives with 9211-8i card
- Connect 2 HGST 8TB drives to 9211-8i breakout cable
- Test with both baremetal Freenas install and ESXI Freenas VM.
This option worked!!! I did multiple transfers back and forth from the network share and did not get any errors. Also checked to see if UDMA errors increased under SMART for the drives and they stayed the same (UDMA was up and over 200 errors on one drive). I did not test under ESXI as I wanted to do a baremetal test with option 7 below first.
7. Test 24 HGST 8TB drives with 9211-8i card and BPN-SAS3-846EL1.
- Connect all 24 HGST 8TB drives to BPN-SAS3-846EL1 connected to 9211-8i card.
- Test with both baremetal Freenas install and ESXI Freenas VM.
Had initial issues with getting the new system to even boot just like I had with the 9300-8i card. Put 9211-8i into another system and erased rom and just installed P20 firmware only. Put 9211-8i card back into new server and booted no issue.So I decided to do a back to normal setup. Started with baremetal Freenas and was able to create a network share with 4 x 6 vdev (6 drives per vdev) and 1 pool. Baremetal had no issues with network transfers. ESXI bootup and passthrough of the device went fine and tested good also!
If I had to venture a guess, I believe that the firmware for the 9300-8i still has issues with latest Freenas/FreeBSD OS. Maybe in a few years it will work but for now I am probably going to move on. Hopefully this helps anyone else that may come across this though.