Help With PCI configuration on R720

[Dionysos]

Hello everyone,

At our kids' charity association, we've set up an old DELL R720, upgraded to the latest firmware, as a Home LAB to teach IT skills to kids. Here's our current setup on the R720:

- Dual CPU: Xeon E5-2660 (95W)
- 80 GB RAM
- Dual PSU 1100W

- (01:00.0, 01:00.1, 01:00.2, 01:00.3) 1/10GbE internal NIC: BCM57800
- (02:00.00) Internal LSI SAS2308 (PERC H710 MINI D1) reflashed in IT mode
- (05:00.00) RASIER 3 / Slot 06 (CPU01) -> VIA IEEE1394 card
- (06:00.00) RASIER 3 / Slot 07 (CPU01) -> NVMe PCIE3.0 x4



The server is running Proxmox 8. We've passed through the SAS LSI 2308 and the NVMe to a TrueNAS Core install, while the VIA 1344 is passed through a Debian install.

Everything is running smoothly – the 8 disks (4 HDD + 4 SSD) attached to the SAS work perfectly in TrueNAS, as well as the NVMe (used as L2ARC). We're achieving 8.5+ Gb/sec on the NIC using Jumbo frames, reaching 850GB/sec reading from files cached on the NVMe.

However, the issue arises when we try to add a GPU (an old Nvidia GT640/gk107) in the RASIER 02 Slot04 (CPU02). The system no longer recognizes the internal LSI SAS2308, even in the BIOS, but recognize the GPU.

We've consulted the official DELL documentation on how PCI slots priority work in the R720, but we're struggling to understand why this is happening. Without the GPU, everything uses the lanes on CPU01, and when we add the GPU, it's on the lanes that should be managed by CPU02.

(42:00.0) when the GPU is plugged on the SLOT 04 (RAISER02 PCI3.0 x 16)

Any help or clues would be highly appreciated.

Thanks!

 

[mrpasc]

Dell servers often expect GPUs to be connected to CPU1 first. So did you try to swap your VIA IEEE1394 card with the GPU?

 

[Dionysos]

Dell servers often expect GPUs to be connected to CPU1 first. So did you try to swap your VIA IEEE1394 card with the GPU?

thanx for the answer ,
i swap the GPU on the slot 06, now the GPU is not recognized, but all the rest is working.
however ipmi shows that the slots are populated


ipmitool -I lanplus -H host -U user -P passw sdr type 0x13


PCIe Slot1 | 90h | ns | 7.1 | Disabled
PCIe Slot2 | 91h | ns | 7.1 | Disabled
PCIe Slot3 | 92h | ns | 7.1 | Disabled
PCIe Slot4 | 93h | ok | 7.1 |
PCIe Slot5 | 94h | ok | 7.1 |
PCIe Slot6 | 95h | ok | 7.1 |
PCIe Slot7 | 96h | ns | 7.1 | Disabled
I/O Channel Chk | 03h | ns | 34.1 | No Reading
PCI Parity Err | 04h | ns | 34.1 | No Reading
PCI System Err | 05h | ns | 34.1 | No Reading
PCIE Fatal Err | 18h | ns | 34.1 | No Reading
Chipset Err | 19h | ns | 34.1 | No Reading

 

[ericloewe]

Just to be clear, without any passthrough, what are the results exactly? Xeon E5 v1 is pretty early for that sort of thing and rough edges abounded.

Your reading of the R720 manual seems correct, at first glance.