HP T740 esxi host

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
W

WANg

Well-Known Member
Jun 10, 2018
1,282
946
113
45
New York, NY
csp-guy said:
Hi,

I350 is sr-iov capable, that is why i wrote the previous post.

The realtek network controller is only visible in esxi after patch, but it is not sr-iov capable.
Click to expand...
Yes, I know the I350 itself is capable of SRIOV (that's why I was looking for one with my t730). But is the Hypervisor hardware capable of working with it? On my t730 the Mellanox CX3 that I installed is SRIOV capable, but it won't work on my t730 in ESXi 6.5 since it won't do PCIe ACS. That's why you'll need to look at the PCI devices to see if it's SRIOV capable. Does it say the I350 is capable?

 
W

WANg

Well-Known Member
Jun 10, 2018
1,282
946
113
45
New York, NY
csp-guy said:
Yes, it does, i made a screenshot,
Click to expand...
Okay, but if you click on the card it does give you the option to enable SRIOV on top, right?

For example, on my t730, the Mellanox CX3 can support SRIOV and passthrough, but the t730 won't do SRIOV the "usual" way (so it doesn't work on ESXi). Is "Configure SR-IOV" enabled when you select the the Intel i350-T4, or is it greyed out/disabled (like below)?

 
C

csp-guy

Active Member
Jun 26, 2019
372
153
43
Hungary, Budapest
OK, i upgraded my I350 card to latest firmware (it is dell branded).

I tried to Enable sr-iov on my card, it was unsuccessful, but the option is visible.

I checked VMWARE HCL, the card is supported, but there is no mention about sr-iov.

I do not understand.

In intel's site the card is sr-iov capable.

I checked my x540-t2/t1 cards, they are sr-iov supported in vmware-s HCL page.
 
W

WANg

Well-Known Member
Jun 10, 2018
1,282
946
113
45
New York, NY
csp-guy said:
It seems, Dell has been removed the SR-IOV feature from their card:

Forums

In the forum above they suggest to buy i350 retail card.

Sorry for the bad news.
Click to expand...
...
Look through the Dell forums first. There is no way in hell Dell/EMC customers will let their vendor get away with this. Those cards aren't cheap back in the days.

In fact, look at the release notes for the 15 series of Dell i350/Ethernet firmware released in December 2013 (11 months after the posting on the Intel forums) - Dell did enable SRIOV on their cards, so the problem might not be on the card side...it could be on the thin client side. That's why I wanted you to boot into a Linux live environment and run lspci -vv to see how the PCIe setup is done on the thin client itself as what is claimed in ESXi is often not what the hardware is capable of doing, or even if it has, it might not be exposed on the BIOS. Also, look through the t740's BIOS and see if there is an IOMMU option present.
 
Last edited:
C

csp-guy

Active Member
Jun 26, 2019
372
153
43
Hungary, Budapest
[godmode@sexi:~] lspci -vv
0000:00:00.0 Host bridge Bridge:
Class 0600: 1022:15d0

0000:00:00.2 IOMMU Generic system peripheral:
Class 0806: 1022:15d1

0000:00:01.0 Host bridge Bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
Class 0600: 1022:1452

0000:00:01.1 PCI bridge Bridge: [PCIe RP[0000:00:01.1]]
Class 0604: 1022:15d3

0000:00:01.2 PCI bridge Bridge: [PCIe RP[0000:00:01.2]]
Class 0604: 1022:15d3

0000:00:01.3 PCI bridge Bridge: [PCIe RP[0000:00:01.3]]
Class 0604: 1022:15d3

0000:00:01.7 PCI bridge Bridge: [PCIe RP[0000:00:01.7]]
Class 0604: 1022:15d3

0000:00:08.0 Host bridge Bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
Class 0600: 1022:1452

0000:00:08.1 PCI bridge Bridge: [PCIe RP[0000:00:08.1]]
Class 0604: 1022:15db

0000:00:08.2 PCI bridge Bridge: [PCIe RP[0000:00:08.2]]
Class 0604: 1022:15dc

0000:00:14.0 SMBus Serial bus controller: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
Class 0c05: 1022:790b

0000:00:14.3 ISA bridge Bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
Class 0601: 1022:790e

0000:00:18.0 Host bridge Bridge:
Class 0600: 1022:15e8

0000:00:18.1 Host bridge Bridge:
Class 0600: 1022:15e9

0000:00:18.2 Host bridge Bridge:
Class 0600: 1022:15ea

0000:00:18.3 Host bridge Bridge:
Class 0600: 1022:15eb

0000:00:18.4 Host bridge Bridge:
Class 0600: 1022:15ec

0000:00:18.5 Host bridge Bridge:
Class 0600: 1022:15ed

0000:00:18.6 Host bridge Bridge:
Class 0600: 1022:15ee

0000:00:18.7 Host bridge Bridge:
Class 0600: 1022:15ef

0000:01:00.0 Ethernet controller Network controller: Intel Corporation Ethernet Server Adapter I350-T4 [vmnic0]
Class 0200: 8086:1521

0000:01:00.1 Ethernet controller Network controller: Intel Corporation Ethernet Server Adapter I350-T4 [vmnic1]
Class 0200: 8086:1521

0000:01:00.2 Ethernet controller Network controller: Intel Corporation Ethernet Server Adapter I350-T4 [vmnic2]
Class 0200: 8086:1521

0000:01:00.3 Ethernet controller Network controller: Intel Corporation Ethernet Server Adapter I350-T4 [vmnic3]
Class 0200: 8086:1521

0000:02:00.0 Ethernet controller Network controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [vmnic4]
Class 0200: 10ec:8168

0000:02:00.1 Serial controller Communication controller:
Class 0700: 10ec:816a

0000:02:00.2 Serial controller Communication controller:
Class 0700: 10ec:816b

0000:02:00.3 IPMI SMIC interface Serial bus controller:
Class 0c07: 10ec:816c

0000:02:00.4 USB controller Serial bus controller:
Class 0c03: 10ec:816d

0000:03:00.0 Network controller Network controller: Intel Corporation Wireless-AC 9260
Class 0280: 8086:2526

0000:04:00.0 Non-Volatile memory controller Mass storage controller: [vmhba0]
Class 0108: 1179:0116

0000:05:00.0 VGA compatible controller Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega [Radeon Vega 8 Mobile]
Class 0300: 1002:15dd

0000:05:00.1 Audio device Multimedia controller:
Class 0403: 1002:15de

0000:05:00.2 Encryption controller Encryption controller:
Class 1080: 1022:15df

0000:05:00.3 USB controller Serial bus controller:
Class 0c03: 1022:15e0

0000:05:00.4 USB controller Serial bus controller: [vmhba32]
Class 0c03: 1022:15e1

0000:05:00.5 Multimedia controller Multimedia controller:
Class 0480: 1022:15e2

0000:05:00.6 Audio device Multimedia controller:
Class 0403: 1022:15e3

0000:06:00.0 SATA controller Mass storage controller: Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] [vmhba1]
Class 0106: 1022:7901
 
Last edited:
W

WANg

Well-Known Member
Jun 10, 2018
1,282
946
113
45
New York, NY
csp-guy said:
[godmode@sexi:~] lspci -vv
0000:00:00.0 Host bridge Bridge:
Class 0600: 1022:15d0

0000:00:00.2 IOMMU Generic system peripheral:
Class 0806: 1022:15d1

0000:00:01.0 Host bridge Bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
Class 0600: 1022:1452

0000:00:01.1 PCI bridge Bridge: [PCIe RP[0000:00:01.1]]
Class 0604: 1022:15d3

0000:00:01.2 PCI bridge Bridge: [PCIe RP[0000:00:01.2]]
Class 0604: 1022:15d3

0000:00:01.3 PCI bridge Bridge: [PCIe RP[0000:00:01.3]]
Class 0604: 1022:15d3

0000:00:01.7 PCI bridge Bridge: [PCIe RP[0000:00:01.7]]
Class 0604: 1022:15d3

0000:00:08.0 Host bridge Bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
Class 0600: 1022:1452

0000:00:08.1 PCI bridge Bridge: [PCIe RP[0000:00:08.1]]
Class 0604: 1022:15db

0000:00:08.2 PCI bridge Bridge: [PCIe RP[0000:00:08.2]]
Class 0604: 1022:15dc

0000:00:14.0 SMBus Serial bus controller: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
Class 0c05: 1022:790b

0000:00:14.3 ISA bridge Bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
Class 0601: 1022:790e

0000:00:18.0 Host bridge Bridge:
Class 0600: 1022:15e8

0000:00:18.1 Host bridge Bridge:
Class 0600: 1022:15e9

0000:00:18.2 Host bridge Bridge:
Class 0600: 1022:15ea

0000:00:18.3 Host bridge Bridge:
Class 0600: 1022:15eb

0000:00:18.4 Host bridge Bridge:
Class 0600: 1022:15ec

0000:00:18.5 Host bridge Bridge:
Class 0600: 1022:15ed

0000:00:18.6 Host bridge Bridge:
Class 0600: 1022:15ee

0000:00:18.7 Host bridge Bridge:
Class 0600: 1022:15ef

0000:01:00.0 Ethernet controller Network controller: Intel Corporation Ethernet Server Adapter I350-T4 [vmnic0]
Class 0200: 8086:1521

0000:01:00.1 Ethernet controller Network controller: Intel Corporation Ethernet Server Adapter I350-T4 [vmnic1]
Class 0200: 8086:1521

0000:01:00.2 Ethernet controller Network controller: Intel Corporation Ethernet Server Adapter I350-T4 [vmnic2]
Class 0200: 8086:1521

0000:01:00.3 Ethernet controller Network controller: Intel Corporation Ethernet Server Adapter I350-T4 [vmnic3]
Class 0200: 8086:1521

0000:02:00.0 Ethernet controller Network controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [vmnic4]
Class 0200: 10ec:8168

0000:02:00.1 Serial controller Communication controller:
Class 0700: 10ec:816a

0000:02:00.2 Serial controller Communication controller:
Class 0700: 10ec:816b

0000:02:00.3 IPMI SMIC interface Serial bus controller:
Class 0c07: 10ec:816c

0000:02:00.4 USB controller Serial bus controller:
Class 0c03: 10ec:816d

0000:03:00.0 Network controller Network controller: Intel Corporation Wireless-AC 9260
Class 0280: 8086:2526

0000:04:00.0 Non-Volatile memory controller Mass storage controller: [vmhba0]
Class 0108: 1179:0116

0000:05:00.0 VGA compatible controller Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega [Radeon Vega 8 Mobile]
Class 0300: 1002:15dd

0000:05:00.1 Audio device Multimedia controller:
Class 0403: 1002:15de

0000:05:00.2 Encryption controller Encryption controller:
Class 1080: 1022:15df

0000:05:00.3 USB controller Serial bus controller:
Class 0c03: 1022:15e0

0000:05:00.4 USB controller Serial bus controller: [vmhba32]
Class 0c03: 1022:15e1

0000:05:00.5 Multimedia controller Multimedia controller:
Class 0480: 1022:15e2

0000:05:00.6 Audio device Multimedia controller:
Class 0403: 1022:15e3

0000:06:00.0 SATA controller Mass storage controller: Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] [vmhba1]
Class 0106: 1022:7901
Click to expand...
...which Linux distribution are you running this from, and are you root (or running as sudo)? If not, it won't tell you much - It's missing quite a bit of info (like the device capability flags). It should look like this when executed under root or with root privileges (example from my t730).
 
C

csp-guy

Active Member
Jun 26, 2019
372
153
43
Hungary, Budapest
It is inside esxi shell. This machine is used me as virtualization platform, and firewall.
I have v16 firmware installed on card.
 
Last edited:
W

WANg

Well-Known Member
Jun 10, 2018
1,282
946
113
45
New York, NY
*Ugh*. Alright, well, I cannot troubleshoot the hardware inside ESXi, since it's rather limiting - and there are also 2 assumptions made.

a) The hardware abilities (working IOMMU support, support for ACS/ARI, interrupt remapping (all needed for SRIOV on a hardware level) is reported to be present by the t740 - which it seem to be.
b) The hardware actually work as expected when you try to use it with a compatible card - which it isn't.

I did however just take delivery of a t740 myself, and my troubleshooting is done within Proxmox 6/Debian 10 (Buster). So far a) looks good on that side, while b)...not so much.

Until I work all the issues out of Linux I am not going to jump into ESXi, since my t730 is working and the t740 is more of a long term project.
 
You must log in or register to reply here.
Top Bottom