PXE booting live Debian

[Jelle458]

Hello all,

We are trying to implement a PXE boot for an image we use on up to 30 machines. The image is Cedar Drive Eraser, which runs on Debian, the ISO is a live Debian.

I have installed Ubuntu on a Dell R640, and DHCP, TFTP, HTTP is all working. In fact my work laptop can boot into Cedar with no issues using legacy boot.

The tricky part is that the servers we want to boot Cedar are also Dell R640 equipped with a 4-port NIC. My work laptop only has 1 NIC, while these servers have the 4.

I get the boot menu, and I can see on the PXE server that it fetches initram and vmlinuz just fine, but when it comes to initram and needs the .squashfs file it just dies.

The initram prompt tells me now that there is no connection at all. If I set up IP routing manually it works. I can use wget and the .squashfs file is downloaded, but when I end up in the initram prompt ip route is empty.

I have tried a thousand different ways to get this server to just get the damn filesystem, but it just won't unless I manually set it up, which isn't viable at all for us.

A picture showing how I can set up the IP route and address to make it work:


The wget command waits for some time before the next line comes, no errors must mean that the server was able to download the squashfs file (1.3GB), and suddenly pinging works after setting up IP.

I am very sure that my issue here is the APPEND line in the boot menu. I have tried various ones:

LABEL cedar-drive-eraser-live
    MENU LABEL Boot Cedar Drive Eraser Live
    KERNEL /live/vmlinuz
    APPEND initrd=/live/initrd boot=live ip=:::::eno1:dhcp fetch=http://192.168.100.1/cedar/filesystem.squashfs

This one did not work at all. I also tried setting IPAPPEND 2 at the top, and using this:

DEFAULT menu.c32
PROMPT 0
TIMEOUT 300

IPAPPEND 2

LABEL cedar-drive-eraser-live
    MENU LABEL Boot Cedar Drive Eraser Live
    KERNEL /live/vmlinuz
    APPEND initrd=/live/initrd boot=live ip=dhcp BOOTIF=${BOOTIF} fetch=http://192.168.100.1/cedar/filesystem.squashfs

I see using BOOTIF that the MAC is being parsed, so the BOOTIF works, but it still can't get the squashfs file.

I have now spent almost a full day figuring this out, I desperately need help here. I hope someone out there can tell me what's going on. ChatGPT is a hot pile of crap for this. It keeps going in circles and has been of no use at all, except for a major headache.

Please, someone help

 

[kapone]

Not sure if this will help...

Security Guide for Oracle Exadata

In this procedure, you configure Secure Eraser to run automatically when you reboot the nodes.

docs.oracle.com

 

[Jelle458]

Sadly it didn't give me much to go on. I tried a few of the options that made sense but no luck there

 

[petree77]

The 4-port NIC in those servers, is it by chance a broadcom NIC? You might also check on your switch(s) and confirm that the ports you're using on the switch are configured for portfast. If not when the OS driver starts link can drop and force the port to take 60 seconds to come back up, and DHCP usually times out before this occurs.

 

[Jelle458]

You are correct! The NIC is Broadcom. The switch however is an older 16-port very dumb gigabit rack switch, no possible management whatsoever.

The funny part here with the DHCP, is that we currently use WipeOS for wiping, but we are changing to Cedar. WipeOS provides a box that has a WAN port, and a LAN port. Through the box only port 80 and 443 is open (probably for uploading wiping certificates to the cloud) but the rest is closed once you use the LAN port.
The WipeOS box also has DHCP, and functions as a PXE server, so we have always booted WipeOS on these R640 servers. We build these server to be 100% identical, and we want to scale up to almost 100 servers, and our ISO and R2V3 certifications won't allow us to use USB, so we have to PXE boot.
Cedar does not provide any PXE solution, but their software is so good. They have also helped me with the PXE boot until it worked fine on my laptop. They can't reproduce my issue here, because they do not have the hardware.

So, if the servers work with WipeOS, and my laptop works with my setup, but the servers do not want to work with my Ubuntu, I can only see it's because the server NIC needs something that I am not giving in the APPEND line.

I am considering wiping Ubuntu, because someone else pointed me to netboot.xyz, but also the thought of deleting this after so much time seems sad.

Does anyone have experience with netboot.xyz?

 

[jode]

Hello all,

I get the boot menu, and I can see on the PXE server that it fetches initram and vmlinuz just fine, but when it comes to initram and needs the .squashfs file it just dies.

The initram prompt tells me now that there is no connection at all. If I set up IP routing manually it works. I can use wget and the .squashfs file is downloaded, but when I end up in the initram prompt ip route is empty.

I have tried a thousand different ways to get this server to just get the damn filesystem, but it just won't unless I manually set it up, which isn't viable at all for us.

The wget command waits for some time before the next line comes, no errors must mean that the server was able to download the squashfs file (1.3GB), and suddenly pinging works after setting up IP.

I am very sure that my issue here is the APPEND line in the boot menu. I have tried various ones:

LABEL cedar-drive-eraser-live
    MENU LABEL Boot Cedar Drive Eraser Live
    KERNEL /live/vmlinuz
    APPEND initrd=/live/initrd boot=live ip=:::::eno1:dhcp fetch=http://192.168.100.1/cedar/filesystem.squashfs

This one did not work at all. I also tried setting IPAPPEND 2 at the top, and using this:

DEFAULT menu.c32
PROMPT 0
TIMEOUT 300

IPAPPEND 2

LABEL cedar-drive-eraser-live
    MENU LABEL Boot Cedar Drive Eraser Live
    KERNEL /live/vmlinuz
    APPEND initrd=/live/initrd boot=live ip=dhcp BOOTIF=${BOOTIF} fetch=http://192.168.100.1/cedar/filesystem.squashfs

I see using BOOTIF that the MAC is being parsed, so the BOOTIF works, but it still can't get the squashfs file.

I have now spent almost a full day figuring this out, I desperately need help here. I hope someone out there can tell me what's going on. ChatGPT is a hot pile of crap for this. It keeps going in circles and has been of no use at all, except for a major headache.

Please, someone help

Is the Broadcom drive compiled as a module or compiled into the kernel?

 

[Jelle458]

I really do not know. How would I figure that out?

 

[nexox]

From the initramfs prompt run `lspci -d <broadcom nic pci id> -vvv` to find the driver name then `lsmod` to see if the driver is listed as a module, if it's not then it is compiled in

 

[Jelle458]

I will check on monday, thank you!

 

[jode]

I will check on monday, thank you!

If the Broadcom driver is loaded as a module, you will need to check if the module is available at PXE boot time. This means that the kernel needs to be loaded with an initramdb, which needs to contain this specific module.

If the driver is compiled/linked into kernel, look for another root cause