PCIe SSDs on older server question

[katit]

Update. Well, not as I wanted, will need suggestions.

Main questions: Can I uses those disks as bootable? Or thats it? Is storing VMs on them will be beneficial and running host on old SATA?
I guess my Q is how critical disk speed for Hyper V host OS? Or it's only important for actual VM storage?

Is this setup how you imagined it would be? I mean I have nice fast 3.8T mirror. Just not system bootable...

During install windows wouldn't install on this drive. And it wouldn't partition more than 2TB. So I can't use it as bootable?



I am not sure where I should see it but BIOS doesn't see those either


I booted from old SSD (SATA) and windows does see those disks. I was able to partition but using MBR

And then results - do those speeds look correct to you? One picture of old system Intel SSD and another picture of those U.2 disks

 

[nexox]

I guess we didn't cover the boot-from-NVMe issue, but that's pretty unlikely, so far as I've seen it's required modded BIOS and those aren't available or possible for every motherboard. It's been quite a long time since I messed with it in Windows, but your boot drive doesn't have to store the OS files, just the bootloader, which can then run everything else off whatever storage you like - whether it's worth doing that on your system is up to you, I probably wouldn't bother for a VM host, but maybe for HyperV it's different.

Your benchmarks look a bit low, but I don't remember exactly which SSDs you got, and most of them are going to require multiple threads to really show their potential.

 

[katit]

I guess we didn't cover the boot-from-NVMe issue, but that's pretty unlikely, so far as I've seen it's required modded BIOS and those aren't available or possible for every motherboard. It's been quite a long time since I messed with it in Windows, but your boot drive doesn't have to store the OS files, just the bootloader, which can then run everything else off whatever storage you like - whether it's worth doing that on your system is up to you, I probably wouldn't bother for a VM host, but maybe for HyperV it's different.

Your benchmarks look a bit low, but I don't remember exactly which SSDs you got, and most of them are going to require multiple threads to really show their potential.

Maybe someone else responds on Hyper-V, but I am pretty sure it doesn't really matter where hypervisor is. It just boots up and runs, why would it use disk and especially, why would it use it in a way where performance matters? At least I don't think it's a big deal.

Low benchmark...

Does it matter which PCI slot(s) I use? I have blue ones, black ones, short ones and longer ones I placed disks in black ones (one of them longer) but numbers from tests is the same. Does it really matter?

And bonus Q. If I was picking decent, reliable and not necessary big SATA SSDs for HOST OS, which one that would be? I don't think I need more than 256G

 

[nexox]

Does it matter which PCI slot(s) I use? I have blue ones, black ones, short ones and longer ones I placed disks in black ones (one of them longer) but numbers from tests is the same. Does it really matter?

Choose slots that connect to CPUs, the only bad choice would be one that goes through the PCH, which may be only PCIe 2.0, and even if it's 3.0 it has to go through more hops to get to the CPU. All the slots should be at least x4 so that's not an issue.

If I was picking decent, reliable and not necessary big SATA SSDs for HOST OS, which one that would be? I don't think I need more than 256G

I personally like the Intel S3xxx series for this, they're rock solid and come in reasonably small sizes, I boot quite a few personal machines off 200 or 400GB S3700s and have abused hundreds of S3500s at work. The newer versions (S3x10 and S3x20) are also pretty good, but at least the S3520 models use TLC NAND, I much prefer MLC if a drive doesn't need high capacity.

 

[katit]

Choose slots that connect to CPUs, the only bad choice would be one that goes through the PCH, which may be only PCIe 2.0, and even if it's 3.0 it has to go through more hops to get to the CPU. All the slots should be at least x4 so that's not an issue.


I personally like the Intel S3xxx series for this, they're rock solid and come in reasonably small sizes, I boot quite a few personal machines off 200 or 400GB S3700s and have abused hundreds of S3500s at work. The newer versions (S3x10 and S3x20) are also pretty good, but at least the S3520 models use TLC NAND, I much prefer MLC if a drive doesn't need high capacity.

Sounds like it doesn't matter which one from what they say:

 

[CyklonDX]

to boot from nvme you need to be on uefi mode, as bypass to boxes that do not have functional uefi mode,
many people create usb boot sticks (that handle the initial pass boot to nvme disk that's not visible in boot devices)

hyper-v is bad in terms of numa assigment; you lack control on which numa node the vm is going to be assigned to;
you can only hope its going to sit on same pcie slot native to that cpu. (I recommend kvm, and pinning the cores on the cpu where the nvme and its resources are numa native to for best performance - anything going over qpi links going to slow down)

(also the mobo very likely does support bifurcation - it may only be hidden for this motherboard bios - bios editor tool may help to uncover those - as other S2600 boards families do support it.)

 

[katit]

to boot from nvme you need to be on uefi mode, as bypass to boxes that do not have functional uefi mode,
many people create usb boot sticks (that handle the initial pass boot to nvme disk that's not visible in boot devices)

hyper-v is bad in terms of numa assigment; you lack control on which numa node the vm is going to be assigned to;
you can only hope its going to sit on same pcie slot native to that cpu. (I recommend kvm, and pinning the cores on the cpu where the nvme and its resources are numa native to for best performance - anything going over qpi links going to slow down)

(also the mobo very likely does support bifurcation - it may only be hidden for this motherboard bios - bios editor tool may help to uncover those - as other S2600 boards families do support it.)

That would be a black belt type of kung-fu Modifying BIOS, etc.

I don't want to have USB boot stick. I want to have mirrored boot drive so I don't lose host OS. To me it seems logical. If I have issue with drive, I just replace one and keep on going.

Hyper-V: I know enough to be dangerous. It works for us. I did play with ncp-ng and others, but all comes back to familiarity issue. I am not admin full time. With hyper-V I know how to backup/restore/manage so it's easy pass for us. Performance been OK for what we do.

 

[CyklonDX]

I don't want to have USB boot stick. I want to have mirrored boot drive so I don't lose host OS. To me it seems logical. If I have issue with drive, I just replace one and keep on going.

You can get really small ones ... when plugged into the back - its almost like it doesn't exist.


for bios you can try those (sometimes menu's are just hidden behind a flag - and you can edit settings for it from the booted system itself)

Award BIOS Editor - Browse /awdbedit/awdbedit-1.0 at SourceForge.net

Awdbedit is a BIOS editor for Award BIOSes. It supports unpacking, replacing, and repacking of components within a BIOS image, and allows settings…

sourceforge.net

GitHub - BoringBoredom/UEFI-Editor: Aptio V UEFI Editor: an alternative to AMIBCP

Aptio V UEFI Editor: an alternative to AMIBCP. Contribute to BoringBoredom/UEFI-Editor development by creating an account on GitHub.

github.com

 

[katit]

You can get really small ones ... when plugged into the back - its almost like it doesn't exist.
View attachment 36005

for bios you can try those (sometimes menu's are just hidden behind a flag - and you can edit settings for it from the booted system itself)

Award BIOS Editor - Browse /awdbedit/awdbedit-1.0 at SourceForge.net

Awdbedit is a BIOS editor for Award BIOSes. It supports unpacking, replacing, and repacking of components within a BIOS image, and allows settings…

sourceforge.net

GitHub - BoringBoredom/UEFI-Editor: Aptio V UEFI Editor: an alternative to AMIBCP

Aptio V UEFI Editor: an alternative to AMIBCP. Contribute to BoringBoredom/UEFI-Editor development by creating an account on GitHub.

github.com

For boot USB what I mean I dont' know how to do that, and I don't understand how I can have it replaceable in case this USB drive goes bad.

 

[CyklonDX]

Look for clover nvme boot - there will be plenty guides
(here's just one example of million out there Booting from an NVME SSD on an older PC using Clover )

In terms of replacement/failover - you can create yourself 2-3 usb boot disks with same image for your server, and just put unused ones in a bag (maybe even inside your box - just make sure bag won't melt/transfer electricity or have multiple ones connected at the same time.

 

[nexox]

I have personally had far too many USB sticks flake out, especially sitting in the back of a server getting warmed up by all the exhaust air, to consider them for anything critical. There are, however, some industrial rated type that I have come up with recently that claim a 70C max operating temperature, which I might use if I had to. I have a couple of these on the way to use with my 3D printers, shipping has been very slow, but the price and specs are right: APACER EH163-M 16GB NANO USB THUMB DRIVE USB 3.1 - NEW - MESSAGE FOR BULK DEAL | eBay

 

[Tech Junky]

I know enough to be dangerous.

This sums up this thread completely.

 

[katit]

This sums up this thread completely.

But cmon, I am trying to learn, this is what those threads for I didn't claim being sysadmin.

 

[CyklonDX]

I have personally had far too many USB sticks flake out, especially sitting in the back of a server getting warmed up by all the exhaust air

Most motherboards including this one have usb port inside. - with nice airflow if i may add.
Those small ones are quite safe in terms of temps - you can go with usb 2.0 which has even higher temp resistance, and generate lower temps overall when powered. (if you do not need one of the front usb's one could also use that - keep in mind sometimes they are used for console)

 

[katit]

for bios you can try those (sometimes menu's are just hidden behind a flag - and you can edit settings for it from the booted system itself)

Award BIOS Editor - Browse /awdbedit/awdbedit-1.0 at SourceForge.net

Awdbedit is a BIOS editor for Award BIOSes. It supports unpacking, replacing, and repacking of components within a BIOS image, and allows settings…

sourceforge.net

GitHub - BoringBoredom/UEFI-Editor: Aptio V UEFI Editor: an alternative to AMIBCP

Aptio V UEFI Editor: an alternative to AMIBCP. Contribute to BoringBoredom/UEFI-Editor development by creating an account on GitHub.

github.com

I tried first utility but it wouldn't open BIOS file I have:

 

[katit]

I also browsed through BIOS options and not sure if anything from there looks like I can use?

Not sure what it this, but seems like it sees something in PCI? I don't know what it means and it needs me to enter "Path for boot option"



Also, not sure if any of this need to be changed.




And is this option relevant to what I am doing with drives?

 

[CyklonDX]

I tried first utility but it wouldn't open BIOS file I have:

Thats ami/aptio bios, not award. (should use 2nd link)

(here are some others)

GitHub - ADLINK/bios-utility: AMI BIOS Utility of Setup menu configuration

AMI BIOS Utility of Setup menu configuration. Contribute to ADLINK/bios-utility development by creating an account on GitHub.

github.com

K7Jo

BIOS Editor - AionSolution

Since the BIOS Hacking 101 article and the BIOS thread in this section, are still getting steady traffic, I thought I had better post some links on where to find them, until I can get the updated article up. AMI AMIBCP V7.60.04 (DOS) http://bios.rom.by/ROMutils/AMI/amibcp/AMIBCP_7_60_04.zip This...

www.aionsolution.com

 

[nabsltd]

For boot USB what I mean I dont' know how to do that, and I don't understand how I can have it replaceable in case this USB drive goes bad.

The USB does nothing but have an EFI boot partition that chain loads the NVMe boot sector. You write the drive once, and it never has anything written to it again, so the chances it will go bad are very slim.

If you are worried, though, buy a 5-pack of USB drives and write each one with boot forwarder. Test each one, then leave one plugged in, and run a zip tie through the other 4 and hang them from the back of the chassis.

 

[katit]

You got me convinced. I already ordered pair of S3500s but I'd like to try bootable USB stick.

Is there any tutorial for making one for Windows server?
I found this, but it's not the same: How to Create a Windows Server 2022 Bootable USB

I already made USB for Windows server install, but it takes it and creates bootable partition on SSD.

Any How-To on how to make such bootable USB so I can install actual OS on different driver?
And how to clone this USB?

 

[nexox]

I think this is what you're looking for: https://linustechtips.com/topic/153...rted-motherboards-with-clover-a-simple-guide/

I didn't read all the details but it seems to accomplish boot from USB with the OS installed on NVMe.