Looks like the drivers are finally coming!
https://tinkertry.com/intel-x552-x557-10gbe-vmware-vib-expected-nov-2015
https://tinkertry.com/intel-x552-x557-10gbe-vmware-vib-expected-nov-2015
Intel Xeon D-1540 10GbE X552/X557 driver/VIB for VMware ESXi 6 coming SOON!
- Thread starter JimPhreak
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Yea I am not a huge fan of these with ESXi. I have been working with the driver team at Supermicro though.
Uhhh you all DID see that the OFFICIAL ESXi 5.5/6.0 vib for the Intel X552/X557-AT 10Gbe dusal-port adapter are out now right? I sure hope so, I think I glossed over a thread that had some Xeon-D content and he showed this maybe...maybe it was just sr-iov.
IF I had a 1540/1520 board I'd do the review for ya Patrick :-D sniff...
https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI55-INTEL-IXGBE-4111&productId=353
Original VMTN thread so you can see how it unfolded:
When will Intel(R) Ethernet Connection X552/X55... | VMware Communities
Have at it, I am both insanely jealous/happy for those of you w/ this board...sometime soon maybe for me. If you could all throw me a bone and get some cacti/freenas/rrdtool goodness out of hypervisor 10G nics, NAS/SAN interfaces showing what these can push that would be appreciated.
Similar to this:
IF I had a 1540/1520 board I'd do the review for ya Patrick :-D sniff...
https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI55-INTEL-IXGBE-4111&productId=353
Original VMTN thread so you can see how it unfolded:
When will Intel(R) Ethernet Connection X552/X55... | VMware Communities
Have at it, I am both insanely jealous/happy for those of you w/ this board...sometime soon maybe for me. If you could all throw me a bone and get some cacti/freenas/rrdtool goodness out of hypervisor 10G nics, NAS/SAN interfaces showing what these can push that would be appreciated.
Similar to this:
Attachments
Last edited:
Moot point now I suppose w/ the official VMware vib for this out right? Dave, was that you beasting out over in that other thread w/ ESXi 6.0 native vib nic support as well as sr-iov goodness? Very cool stuff good sir!
FYI I'm seeing Ethernet negotiation issues with these ports and a cisco 10GE switch (SG500X) where 1000/Full is negotiated, or the link doesn't come up at all. This may be similar to problems reported elsewhere. Is anyone else seeing similar with X557s and ESXi?
I may post to VMTN soon and see what folks say. I haven't troubleshot this a whole lot, but here's what I have.
The flaky config:
Very often (approximately 40% of the time) one of the X557 ports either negotiate to 1000/Full...
[root@server03-esxi:~] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- ---------------------------------------------------
vmnic0 0000:05:00.0 igb Down Down 0 Half 00:25:90:5d:0e:08 1500 Intel Corporation I350 Gigabit Network Connection
vmnic1 0000:05:00.1 igb Down Down 0 Half 00:25:90:5d:0e:09 1500 Intel Corporation I350 Gigabit Network Connection
vmnic2 0000:03:00.0 ixgbe Up Up 10000 Full 00:25:90:5d:0f:2a 1500 Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
vmnic3 0000:03:00.1 ixgbe Up Up 1000 Full 00:25:90:5d:0f:2b 1500 Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
...or simply didn't come up at all...
[root@server03-esxi:~] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- ---------------------------------------------------
vmnic0 0000:05:00.0 igb Down Down 0 Half 00:25:90:5d:0e:08 1500 Intel Corporation I350 Gigabit Network Connection
vmnic1 0000:05:00.1 igb Down Down 0 Half 00:25:90:5d:0e:09 1500 Intel Corporation I350 Gigabit Network Connection
vmnic2 0000:03:00.0 ixgbe Up Down 0 Full 00:25:90:5d:0f:2a 1500 Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
vmnic3 0000:03:00.1 ixgbe Up Up 10000 Full 00:25:90:5d:0f:2b 1500 Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
Bouncing the link eventually brings the link up, though sometimes it would take 2-3 tries
Configuring the switch to advertise 10000/Full eliminated the lower speed outcome, but not the occurrences of links not coming up.
I couldn't find anything obvious in the logs around this issue. I've since fallen back to the following which works fine.
Luckily the sole PCIe slot was available due to my other problem (AOC-S2308L-L8i LSI 2308 based controllers not recognizing my SSDs) so silver lining I guess
I'm not seeing any conclusive patches, but with the number of X552/X557 related commits since 4.1.1, I think it best to wait until a later ixgbe release before depending on these ports at all.
I may post to VMTN soon and see what folks say. I haven't troubleshot this a whole lot, but here's what I have.
The flaky config:
- Dual Intel X552/X557-AT (integrated on SuperMicro X10SDV-8C-TLN4F)
- VIB net-ixgbe_4.1.1.1-1OEM.550.0.0.1331820
- ESXi 6.0 U1 (build 3073146)
Very often (approximately 40% of the time) one of the X557 ports either negotiate to 1000/Full...
[root@server03-esxi:~] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- ---------------------------------------------------
vmnic0 0000:05:00.0 igb Down Down 0 Half 00:25:90:5d:0e:08 1500 Intel Corporation I350 Gigabit Network Connection
vmnic1 0000:05:00.1 igb Down Down 0 Half 00:25:90:5d:0e:09 1500 Intel Corporation I350 Gigabit Network Connection
vmnic2 0000:03:00.0 ixgbe Up Up 10000 Full 00:25:90:5d:0f:2a 1500 Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
vmnic3 0000:03:00.1 ixgbe Up Up 1000 Full 00:25:90:5d:0f:2b 1500 Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
...or simply didn't come up at all...
[root@server03-esxi:~] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- ---------------------------------------------------
vmnic0 0000:05:00.0 igb Down Down 0 Half 00:25:90:5d:0e:08 1500 Intel Corporation I350 Gigabit Network Connection
vmnic1 0000:05:00.1 igb Down Down 0 Half 00:25:90:5d:0e:09 1500 Intel Corporation I350 Gigabit Network Connection
vmnic2 0000:03:00.0 ixgbe Up Down 0 Full 00:25:90:5d:0f:2a 1500 Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
vmnic3 0000:03:00.1 ixgbe Up Up 10000 Full 00:25:90:5d:0f:2b 1500 Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
Bouncing the link eventually brings the link up, though sometimes it would take 2-3 tries
- esxcli network nic down -n vmnic3
- esxcli network nic up -n vmnic3
Configuring the switch to advertise 10000/Full eliminated the lower speed outcome, but not the occurrences of links not coming up.
I couldn't find anything obvious in the logs around this issue. I've since fallen back to the following which works fine.
- Dual Intel X540-AT2 (add-in PCIe card)
- VIB net-ixgbe_3.21.6-1OEM.600.0.0.2159203
Luckily the sole PCIe slot was available due to my other problem (AOC-S2308L-L8i LSI 2308 based controllers not recognizing my SSDs) so silver lining I guess
I'm not seeing any conclusive patches, but with the number of X552/X557 related commits since 4.1.1, I think it best to wait until a later ixgbe release before depending on these ports at all.
I'm still seeing occasional negotiation problems along with strange performance issues on X557 and 4.1.1, so have been sticking with my trusty X540-T2 NICs. Looks like ixgbe 4.2.2 came out several days ago which I'm about to give a whirl. Crossing fingers!
VMware vSphere 5: Private Cloud Computing, Server and Data Center Virtualization
VMware vSphere 5: Private Cloud Computing, Server and Data Center Virtualization
That's odd. Even my Proxmox host that had the driver loaded by default hasn't had that kind of problem.
FYI in case anyone runs into a similar issue. I've been running 4.2.2 for awhile now while rebooting hosts and the switch, bouncing links, futzing with flow control, etc. and have not seen the issue resurface. I want to proceed cautiously but I should have some X540-T2s for sale in a few days.
On to performance tuning!
4.1.1 was already old when vmware made it available back in November. Seems like a delay due to the pipeline involved (vendor develops and QA tests before releasing to vmware, vmware does its own QA tests and finally releases the driver). I think we saw similar delays when the X540s were first released... I definitely saw it with multiple FC HBA vendors and with Cisco enic/fnic VIC drivers.
On to performance tuning!
What ixgbe driver version did you start with when running Proxmox and X557 NICs?
4.1.1 was already old when vmware made it available back in November. Seems like a delay due to the pipeline involved (vendor develops and QA tests before releasing to vmware, vmware does its own QA tests and finally releases the driver). I think we saw similar delays when the X540s were first released... I definitely saw it with multiple FC HBA vendors and with Cisco enic/fnic VIC drivers.
Last edited:
Why is that? Networking? Drivers? Performance? Just trying to figure which aspect is not well covered with these.
Speaking for Patrick here but fairly certain that it was related to the fact that the initial revision 1.0 of the board using the 1540 chip did not support SR-IOV. The new version 2.0 which utilizes the 1541 chip does support SR-IOV so I don't think he would say the same thing.
My opinion is that the board is awesome and so is SM support. Been running mine for a while now and love it. I had a few hiccups with bios 1.0a but they were fixed with bios 1.0b.