Minimal hardware for napp-it all-in-one?

[leecallen]

I posted in a different thread: I have been running SmartOS on a SuperMicro motherboard for 6+ years, with spinning disks. I have decided I want to re-architect my ZFS/NAS solution using OmniOS and napp-it, and preferably AiO so I can put all my VMs on the same server and get local disk speed.

My current implementation is big and noisy (and kind of expensive) and I want to go small / lean / quiet / cheap this time. Ideally mini-itx with SSD.

I have given a lot of thought to the ECC issue and I am going to forego ECC. It really limits the motherboard (and therefore form factor) options and drives up the cost, esp to get a CPU with enough speed to handle my VMs. I think I understand the risks: a very small chance of a memory corruption that could be saved to disk, possibly corrupting the filesystem. FWIW my disk usage is read intensive, so I think that reduces the risk.

By foregoing ECC I can get a bunch of cores and lots of memory in a mini-ITX form factor, at a modest cost.

gea, I have been going through your AiO and Examples documents and I have some questions.

I want to avoid getting a RAID adapter in IT mode - I did that for my current system, ended up buying a used adapter on eBay, and the first one was defective - the whole thing cost me a couple of weeks.

You mention USB boot is an option but you seem to strongly prefer SSD, even SSD over USB. Why? Is USB flash disk practical?

If I do one of the above, can I then utilize the motherboard SATA for all of my pool disks? Or would I not have access to pass-thru mode in that configuration?
Or could all of my pool disks be NVMe, so I can avoid the complication of getting a RAID adapter in IT mode?

Thanks in advance for any & all advice!

 

[gea]

Very small cases are itx but this limits your choices regarding expandability (more disks, HBAs, faster nics. Very small cases are often quite loud due the small 40mm fans. You have much more options when you choose the slightly larger uATX format and a small case for uATX boards with larger fans.

A system without ECC has a very small chance of undetected/undetectable data corruptions due ram errors on write (bad data with correct ZFS checksums, happens with a statistical rate). Not very likely but possible. In the end a system with ECC but without ZFS has a higher propability of data corruptions so ZFS without ECC is better than others with but ZFS + ECC is the best choice.

SAS HBAs allow to attach many disks with a higher datarate than Sata (2x12G instead 6G), more robust transfers with cable length up to 10m instead 1m and full duplex transfers. It is quite usual to get HBAs like a BroadCom 9300 with IT or IR firmware (both are ok) for cheap ex https://forums.servethehome.com/ind...-lsi-sas3-controller.40506/page-3#post-387262 With many disks or backplanes, SAS is the first choice.

USB boot (using USB SSD, avoid sticks) is possible but I would not. Better use a 100 GB M.2 NVMe to boot ESXi and for the storage VM and pass-through Sata or other NVMe.

For a cheaper homeserver with ECC, I would prefer an AMD system. They offer ECC support even with cheaper desktop systems. Asrock is known for working ECC, check their specs. Some AMD (ex with graphics) must use AMD Pro CPUs. Care about min 64GB RAM capability and ESXi nic support (at best use current boards with Intel nics). Only newer OSX VMs > v12 will not work with AMD,

https://pg.asrock.com/mb/index.asp#AM4

 

[leecallen]

USB boot (using USB SSD, avoid sticks) is possible but I would not. Better use a 100 GB M.2 NVMe to boot ESXi and for the storage VM and pass-through Sata or other NVMe.

gea thank you for your thorough response. I just want to clarify though: If I use a M.2 boot drive, could I use SATA or NVMe for all of my pool disks? I mean, can those both be passed through to the OmniOS VM?

 

[leecallen]

I mean the motherboard SATA

 

[gea]

If you boot from an m.2 nvme you can passthrough onboard sata and other nvme (at least mostly, some nvme are only usable as vdisk)

 

[leecallen]

If you boot from an m.2 nvme you can passthrough onboard sata and other nvme (at least mostly, some nvme are only usable as vdisk)

Thank you, that's all I need.

 

[leecallen]

If you boot from an m.2 nvme you can passthrough onboard sata and other nvme (at least mostly, some nvme are only usable as vdisk)

Gea, I have built my napp-it AiO server (hardware), installed ESXi, and created the OmniOS VM using your template. I am having some trouble with the next step: passthrough of the SATA controller or disks.

I want to try two options. In both cases I am booting from M.2 NVMe - that's where ESXi & OmniOS are installed. And I have 3 SATA3 SSDs.

Plan A: using the motherboard SATA controller. Can I passthrough either the SATA controller or the individual SATA disks?

In ESXi, under Hardware, I see "Intel SATA controller" but the Passthrough column shows "Not capable". I tried adding the SATA controller to /etc/vmware/passthru.map and that did not help.

If I can't passthrough the controller can I pass the disks? I don't see the disks at all under the Hardware list (but BIOS Setup sees them).

Plan B: using an LSI 9300-8i flashed in IT mode, firmware version 12. I was able to set this up for Passthrough successfully, I think. How can I find out if OmniOS sees it? I don't have the disks connected to it yet -- I am waiting for a cable.

The reason I want to try both options: My little mini-ITX motherboard has only one PCI slot, and I have other plans for it. So ideally I would use the motherboard SATA controller.

Thank you.

 

[gea]

Gea, I have built my napp-it AiO server (hardware), installed ESXi, and created the OmniOS VM using your template. I am having some trouble with the next step: passthrough of the SATA controller or disks.

I want to try two options. In both cases I am booting from M.2 NVMe - that's where ESXi & OmniOS are installed. And I have 3 SATA3 SSDs.

Plan A: using the motherboard SATA controller. Can I passthrough either the SATA controller or the individual SATA disks?

ESXi supports individual disk pass-through only via SAS controller. There are "unsupported" options that may work or not.

In ESXi, under Hardware, I see "Intel SATA controller" but the Passthrough column shows "Not capable". I tried adding the SATA controller to /etc/vmware/passthru.map and that did not help.

Sata passthrough is not officially supported. Often it works, in your case not.

If I can't passthrough the controller can I pass the disks? I don't see the disks at all under the Hardware list (but BIOS Setup sees them).

Yes with an SAS HBA

Plan B: using an LSI 9300-8i flashed in IT mode, firmware version 12. I was able to set this up for Passthrough successfully, I think. How can I find out if OmniOS sees it? I don't have the disks connected to it yet -- I am waiting for a cable.

Should work. This is the default setup.

The reason I want to try both options: My little mini-ITX motherboard has only one PCI slot, and I have other plans for it. So ideally I would use the motherboard SATA controller.

The two alternatives
- Boot ESXi from Sata, try the NVMe for passthrough
- Assign the disks to ESXi and create virtual disks. No smartchecks but nearly the same performance.