I reinstall the system with omnios-r151020-b5b8c75 and re-create the zpool
Config:
1. 12T zpool , 4vdevs * 2 * SATA3TB on Dell H310 HBA
2. 3 SSDs on system :
PLEXTOR-256M6+PLEXTOR-M6 on SAS1064E
PLEXTOR-256M5 on Intel SATA
3. DELL R520
SD.conf:
sd-config-list =
"ATA PLEXTOR PX-256M6", "physical-block-size:4096",
"ATA PLEXTOR PX-128M6", "physical-block-size:4096",
"ATA PLEXTOR PX-256M61.01", "physical-block-size:4096",
"ATA PLEXTOR PX-128M61.01", "physical-block-size:4096",
"ATA PLEXTOR PX-256M5", "physical-block-size:4096",
"ATA QEMU HARDDISK ", "physical-block-size:4096",
"IET VIRTUAL-DISK ", "physical-block-size:4096",
"NETAPP LUN ", "physical-block-size:4096";
Found something interesting , Maybe it's related to the silent fail of removing slog:
1. cannot force Plextor-256M6 or Plextor-128M6 to ashift=12:
no matter adding the log to zpool after I created it , or adding the log as I create the zpool , zdb -C always shows the ashift of slog is 9
2. I can force the Plextor-256M5 on Intel SATA to ashift=12
Now I'm using the Plextor-256M5 as slog device. I can add and remove slog so far . will try more as soon as the data restored.