booting pfsense on 5018A-FTN4 server?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83
Is using QAT that important for you?
OpnSense has QAT on the roadmap for 22.7/End of July 2022 release
 

Kaishi

Member
Mar 7, 2011
49
2
8
Is using QAT that important for you?
OpnSense has QAT on the roadmap for 22.7/End of July 2022 release
QAT isn't the priority right now. Just getting this system to boot _any_ POSIX OS at all would be a step in the right direction.

From there I can decide what to do about one OS vs another.

Are you suggesting I should consider OPNsense instead of pfsense?
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83
QAT isn't the priority right now. Just getting this system to boot _any_ POSIX OS at all would be a step in the right direction.

From there I can decide what to do about one OS vs another.

Are you suggesting I should consider OPNsense instead of pfsense?
My guess is that OPNSense is unlikely to cure the underlying issue you're having with bios/mobo but it won't hurt do a quick comparison between these two similar projects (opnsense forked from PFSense) and make your own call.
 
Feb 19, 2021
45
29
18
I boot the pfsense installer with no issues. Installer sees the disks. I use the guided setup and configure a ZFS mirror on the two disks, and MBR partition scheme for BIOS, but otherwise leave all the defaults. Install completes, I remove USB, and reboot.

/QUOTE]

have we ever tried a standard non zfs filesystem? Windows isnt trying to boot a zfs filesystem....
 
Feb 19, 2021
45
29
18
These are in production everywhere with pfsense installed baremetal.

Revert all bios settings to factory.

Start with one ssd just one.

Install using the entire disk.

Please...
 

Kaishi

Member
Mar 7, 2011
49
2
8
These are in production everywhere with pfsense installed baremetal.

Revert all bios settings to factory.

Start with one ssd just one.

Install using the entire disk.

Please...
Literally one of the first things I tried. No improvement. I tried it again last week just for fun. No change.

Since thing I've confirmed that my system will install and boot Windows 10. It refuses to boot any POSIX OS including pfsense / opnsense. It doesn't want to boot anything without secureboot being enabled. I don't have an explanation. I've asked all the folks I know about this problem and we haven't managed to resolve it.

If you think it's an easy fix, please by all means, explain.
 

Kaishi

Member
Mar 7, 2011
49
2
8
Loaded optimized defaults.

Made sure secureboot was disabled. Deleted keys so system was in setup mode.

Rebooted into pfsense installer thumb drive.

Installed pfsense on a single disk (ada1).

Finished install. Rebooted.

No OS found, not even the copy of Windows 10 on the other disk (ada0).

If I enable secureboot and reset the keys to default, then it will boot windows 10.

No other OS works. I have screenshots to prove this.
 

Rain

Active Member
May 13, 2013
276
124
43
Or maybe the board only looking for the backup boot file (bootx64.efi) rather than a configured one.
If you haven't confirmed this to be the case, I'd wager that's the issue. I reviewed the rest of the thread and didn't see any other mention of it.

I've seen quite a few boards and products, especially products released when UEFI was still "new...ish", that refuse to boot to anything other than EFI/boot/bootx64.efi (technically the removable media bootloader path). All ISOs (or images) you'd possibly burn to removable media have EFI/boot/boot.efi and/or EFI/boot/bootx64.efi and you're not having any trouble booting those. I think Windows installs a shim to EFI/boot/bootx64.efi (to combat broken UEFI implementations like this) which is likely why you're not having any issues booting Windows after installing it.

Perform the pfSense installation (or some other *nix). Drop to a shell at the end of the installation (or reboot into a live *nix environment and mount the EFI partition). Copy the installed bootloader efi to EFI/boot/bootx64.efi (on the EFI partition). Reboot and see if it boots.
 
Last edited:
  • Like
Reactions: BoredSysadmin

Rain

Active Member
May 13, 2013
276
124
43
Following up with this since I was curious, I installed pfSense 2.6.0 in a VM in UEFI mode. Interestingly, it did install the bootloader to EFI/boot/bootx64.efi. It is still worth doube-checking, though. The installer may install the bootloader to a different location if it detects it's being installed on a VM. Likewise, "pfSense Plus", which I believe you're using, may do it differently as well.

After the pfSense installation completes, chose "Yes" to the question about dropping to a shell. Run the following commands to get the EFI partition mounted and look around (replace /dev/da0p1 with the correct device/partition):

Code:
# gpart list | less
(find the partition with "type: efi")
# mkdir /target
# mount_msdosfs /dev/da0p1 /target
# cd /target/efi
# ls
boot
# ls boot
BOOTx64.efi startup.nsh
If the .efi is in another location, create a boot directory and copy the .efi to boot/BOOTx64.efi.

Code:
# mkdir boot
# cp pfsense/pfsense.efi boot/bootx64.efi
(pfsense/pfsense.efi used as an example; it will likely differ)

It may be worth checking with a simple Debian installation as well. By default, it shouldn't boot because Debian will not install the bootloader to the removable media path. After installing, if you reboot into the installer and go into rescue mode, you can fix it without using a shell. See "Force grub-efi installation to the removable media path" @ UEFI - Debian Wiki
 
Last edited:

Kaishi

Member
Mar 7, 2011
49
2
8
If you haven't confirmed this to be the case, I'd wager that's the issue. I reviewed the rest of the thread and didn't see any other mention of it.

I've seen quite a few boards and products, especially products released when UEFI was still "new...ish", that refuse to boot to anything other than EFI/boot/bootx64.efi (technically the removable media bootloader path). All ISOs (or images) you'd possibly burn to removable media have EFI/boot/boot.efi and/or EFI/boot/bootx64.efi and you're not having any trouble booting those. I think Windows installs a shim to EFI/boot/bootx64.efi (to combat broken UEFI implementations like this) which is likely why you're not having any issues booting Windows after installing it.

Perform the pfSense installation (or some other *nix). Drop to a shell at the end of the installation (or reboot into a live *nix environment and mount the EFI partition). Copy the installed bootloader efi to EFI/boot/bootx64.efi (on the EFI partition). Reboot and see if it boots.
I can try that. But also, I did try something to that effect back when I made the post you quoted. However it didn't seem to help. But then I might have done something wrong. I'm not super familiar with these subtle nuances. In my experience, BIOS+MBR worked, and now, UEFI+GPT works. Anything during the transition is sorta confusing.
 

Kaishi

Member
Mar 7, 2011
49
2
8
Following up with this since I was curious, I installed pfSense 2.6.0 in a VM in UEFI mode. Interestingly, it did install the bootloader to EFI/boot/bootx64.efi. It is still worth doube-checking, though. The installer may install the bootloader to a different location if it detects it's being installed on a VM. Likewise, "pfSense Plus", which I believe you're using, may do it differently as well.

After the pfSense installation completes, chose "Yes" to the question about dropping to a shell. Run the following commands to get the EFI partition mounted and look around (replace /dev/da0p1 with the correct device/partition):

Code:
# gpart list | less
(find the partition with "type: efi")
# mkdir /target
# mount_msdosfs /dev/da0p1 /target
# cd /target/efi
# ls
boot
# ls boot
BOOTx64.efi startup.nsh
If the .efi is in another location, create a boot directory and copy the .efi to boot/BOOTx64.efi.

Code:
# mkdir boot
# cp pfsense/pfsense.efi boot/bootx64.efi
(pfsense/pfsense.efi used as an example; it will likely differ)

It may be worth checking with a simple Debian installation as well. By default, it shouldn't boot because Debian will not install the bootloader to the removable media path. After installing, if you reboot into the installer and go into rescue mode, you can fix it without using a shell. See "Force grub-efi installation to the removable media path" @ UEFI - Debian Wiki
I will try this out this evening. I'm not using a VM. This is a bare-metal install.

The confusing part is that the Motherboard's SATA configuration only lists the SATA Disks as being present when SecureBoot is Enabled. Without SecureBoot, while in the UEFI, the SATA settings show [Not Installed] for the SATA ports I'm using (0 and 1).

If I enable SecureBoot, everything behaves as I would expect, and Windows 10 cooperates perfectly. But only Windows 10.
 
  • Like
Reactions: Rain

Rain

Active Member
May 13, 2013
276
124
43
I will try this out this evening. I'm not using a VM. This is a bare-metal install.
Right; I tested in a VM, though. I just wanted to be clear it may be different in my VM test vs on baremetal.

The confusing part is that the Motherboard's SATA configuration only lists the SATA Disks as being present when SecureBoot is Enabled. Without SecureBoot, while in the UEFI, the SATA settings show [Not Installed] for the SATA ports I'm using (0 and 1).
That's definitely weird. When CSM is disabled (ie UEFI is forced), regardless of Secure Boot, SATA disks shouldn't show up in the Boot Device order; that's normal. However, they should still appear in Advanced > SATA Configuration. Especially if the SATA controller(s) are set to AHCI mode (and not IDE/RAID)

One follow-up question: If you boot into the built-in EFI shell with Secure Boot disabled, are SATA drives/partitions listed?
 
Last edited:

Kaishi

Member
Mar 7, 2011
49
2
8
Right; I tested in a VM, though. I just wanted to be clear it may be different in my VM test vs on baremetal.

That's definitely weird. When CSM is disabled (ie UEFI is forced), regardless of Secure Boot, SATA disks shouldn't show up in the Boot Device order; that's normal. However, they should still appear in Advanced > SATA Configuration. Especially if the SATA controller(s) are set to AHCI mode (and not IDE/RAID)

One follow-up question: If you boot into the built-in EFI shell with Secure Boot disabled, are SATA drives/partitions listed?
I don't mean in the boot order either. I mean literally the SATA controller's section of the motherboard's UEFI. The SATA is provided by the C2758 itself, if I understand correctly.

The disk themselves show up as [Not Installed]. And thereby cannot be used within the boot order (according to the motherboard, there aren't any disks). And yet, when I run an OS installer (any OS, Ubuntu to Windows 10, whatever) the disks show up just fine. But when that first reboot after the install occur, Nothing. Because the Motherboard still doesn't see the disks, and thereby doesn't see the GPT / MBR and any bootcode.

It sees nothing. (This is my answer to your question in the post above, you can see no partitions and no disks, nothing at all. There are 2 SATA SSDs connected, with partitions and bootable code on them, but not according to the firmware.)

If I enable SecureBoot, then the disks show up, but the only OS I've managed to get to boot is Windows 10. pfsense doesn't have SecureBoot support because FreeBSD itself doesn't have support. If I go into the EFI Shell, I can see the partitions on the disks.

To summarize: SecureBoot ON = things work as anticipated, but only Windows can boot.
SecureBoot OFF = no SATA devices can boot. USB boots fine but that doesn't get me very far.

EDIT: I added photos, showing what I'm talking about.
 
Last edited:
  • Like
Reactions: Rain

Kaishi

Member
Mar 7, 2011
49
2
8
These are in production everywhere with pfsense installed baremetal.

Revert all bios settings to factory.

Start with one ssd just one.

Install using the entire disk.

Please...
Just for you, I did exactly this, a second time. No difference whatsoever. Same weird anomalous behavior. I haven't tried removing the CMOS battery but that's about the last thing I can even conceive of (maybe some firmware-level soft-fuse for it having booted in SecureBoot mode before?)
 

Rain

Active Member
May 13, 2013
276
124
43
The disk themselves show up as [Not Installed]. And thereby cannot be used within the boot order (according to the motherboard, there aren't any disks). And yet, when I run an OS installer (any OS, Ubuntu to Windows 10, whatever) the disks show up just fine. But when that first reboot after the install occur, Nothing. Because the Motherboard still doesn't see the disks, and thereby doesn't see the GPT / MBR and any bootcode.

It sees nothing. (This is my answer to your question in the post above, you can see no partitions and no disks, nothing at all. There are 2 SATA SSDs connected, with partitions and bootable code on them, but not according to the firmware.)

If I enable SecureBoot, then the disks show up, but the only OS I've managed to get to boot is Windows 10. pfsense doesn't have SecureBoot support because FreeBSD itself doesn't have support. If I go into the EFI Shell, I can see the partitions on the disks.
Well, that is indeed interesting.

If the disks don't show up in the EFI shell that definitely shows the issue: When Secure Boot is disable the UEFI doesn't load/read/initiate the SATA controller and/or disks (and/or isn't loading EFI AHCI driver).

With that, unfortunately I'm out of ideas. Did you verify SuperMicro sent you a different replacement board and didn't just return your "broken" board when you RMA'd it?

I'd definitely try removing the CMOS battery (better yet, use the CMOS Clear jumper as well!) if you haven't already.

If the board allows you to install Secure Boot keys and you're hell-bent on using it, you could self-sign the bootloader you wish to use and install your own key.
 

Kaishi

Member
Mar 7, 2011
49
2
8
Well, that is indeed interesting.

If the disks don't show up in the EFI shell that definitely shows the issue: When Secure Boot is disable the UEFI doesn't load/read/initiate the SATA controller and/or disks (and/or isn't loading EFI AHCI driver).

With that, unfortunately I'm out of ideas. Did you verify SuperMicro sent you a different replacement board and didn't just return your "broken" board when you RMA'd it?

I'd definitely try removing the CMOS battery (better yet, use the CMOS Clear jumper as well!) if you haven't already.

If the board allows you to install Secure Boot keys and you're hell-bent on using it, you could self-sign the bootloader you wish to use and install your own key.
Different serial number than the one I started with. Different MAC addresses too.

I can try the CMOS clear jumper as well. I haven't, because I have to open the enclosure which means unracking it again.

The board does allow you to install your own keys. I have never signed a bootloader and hesitate to wind up in a funky state, where some OS update might brick my system.

I think Netgate wrote custom firmware for this board, as they resold these but with a badge and 32GB of eMMC attached in some manner.

EDIT: I'm sure you can understand why I've had to take breaks from fighting with this system. Same behavior on 2 different boards. 3 sets of SSDs with 3 different controllers.. Hundreds of hours wasted.