Intel Xeon D-1500 Series Discussion

@jonaspaulo

I suppose if you really have to have it you could try something like this :

Gmane -- Re: I40E: UTS UBUNTU RELEASE ABI is too large on Ubuntu 14.04 with 3.14.43 031443 generic Kernel

 

@ggg thanks for pointing that out, in fact I also have found that thread but I believed more on the release notes from the ixgbe drivers which mention:

"This is the most current release of the ixgbe driver for Linux, which supports kernel versions 2.6.18 up through 4.2.1."

In regards to features/bugs/fixes the most useful for me in each ixgbe release are (Intel Ethernet Drivers and Utilities - Browse /ixgbe stable at SourceForge.net):

-4.0.3:
* Support for new X550 hardware (new features not enabled yet)
* VxLan RX checksump offload support

-4.1.1
* Support for new X550 hardware

-4.1.5
the support for kernel versions 2.6.18 up through 4.2.1. (at least from what they mention)


So to be clear no one has successfully updated the ixgbe driver version to other than the default on linux kernels 4.x ?

@ggg can you tell me your current versions for the kernel and ixgbe? Thanks

Thanks again

 

@jonaspaulo

So it doesn't seem you need 4.1.5, just what is upstream in kernel 4.1+.

I think most have upgraded their kernel instead of compiling ixgbe for their kernel.

If you do build your own kernel on Ubuntu 14.04 from git rather then the Ubuntu sources then you might be more successful building ixgbe.

If you get a more recent kernel from Ubuntu that would work as well. Perhaps you can find a PPA or try this: medigeek/kmp-downloader · GitHub

One thing is clear, ixgbe 4.1.5 is not going to compile against your current Ubuntu kernel without modifications to the module's source.

 

But not the Ubuntu distributions of those versions

 

Yes that could be the reason. They also mention the distributions under which it is supported:
- RHEL 6.7 - RHEL 7.1 - SLES 11SP4 - SLES 12 - SLES 12SP1 Beta 3


Thanks a lot for all the help!

@EluRex can you just comment on my issue with your current setup?
You have Ubuntu with kernel 4.2.3 and a Xeon d-1540 chip with everything working right?
Did you upgrade the ixgbe driver or just let the default for 4.2.3?

 

I don't think @EluRex has been able to get a successful ping from his SR-IOV interfaces yet. So this may or may not work.

 

Sorry all, its been a terrible week for me so far.... some older pbx failing... and some pdc RAID1 failing... I just do not have to time to do the test.... I am actually more eager to do the testing then anyone else... lets someone can carry on the test https://forums.servethehome.com/index.php?threads/sm-5028d-tn4t-x10sdv-tln4f-x557-sr-iov-issue.7309/ where I left off..

 

Yes he just answered me in a PM.

So to recap and based on the experiments I made and all the people here in the thread:

- (@EluRex test) "Ubuntu 14.04.3 with Kernel 4.2.3 with success X557 VF and pci-passthrough but not yet test for a successful ping test. The other NIC Intel i350 confirmed with SR-IOV/pci passthrough/successful ping (also on Proxmox 4)" - with original ixgbe 4.0.1.k

- Ubuntu 14.0.4.3 with Kernel 4.2.3 doesn't allow to install ixgbe 4.1.5 so we stay at the default ixgbe 4.0.1.k - missing ping test on X557 SR-IOV
- Ubuntu 14.0.4.3 with Kernel 3.19.0-31-generic (default) allowed me to install ixgbe 4.1.5 - but still hadn't the time to search for how to properly activate SR-IOV and do a ping test on the X557 VF

So we either stay at the default kernel 3.19.0-31 with ixgbe 4.0.1.k or upgrade to kernel 4.2.3 with ixgbe 4.0.1.k. One must test which has the best results. For now it seems that the second option is on the lead.

Thanks all

 

if you can't successfully ping what do you have?

 

Right now not much. I have been busy.

a@ubuntukvm:~$ uname -r

3.19.0-31-generic



a@ubuntukvm:~$ lspci | grep Ethernet

03:00.0 Ethernet controller: Intel Corporation Device 15ad

03:00.1 Ethernet controller: Intel Corporation Device 15ad

05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)



a@ubuntukvm:~$ uname -r

3.19.0-31-generic



a@ubuntukvm:~$ dmesg | grep Gigabit

[ 3.965229] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k

[ 4.166681] igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection

[ 4.334709] igb 0000:05:00.1: Intel(R) Gigabit Ethernet Network Connection

[ 7.784969] Intel(R) 10 Gigabit PCI Express Network Driver - version 4.1.5

[ 9.348577] ixgbe 0000:03:00.0 eth0: Intel(R) 10 Gigabit Network Connection

[ 11.291505] ixgbe 0000:03:00.1 eth1: Intel(R) 10 Gigabit Network Connection





a@ubuntukvm:~$ echo '7' > sudo cat /sys/class/net/em1/device/sriov_numvfs

a@ubuntukvm:~$ cat /sys/class/net/em1/device/sriov_numvfs

0


It doesn't seem to allow me to write the sriov_numvfs value :s


[edit] @EluRex which bios version do you have on your supermicro board?

Thanks

 

Kind of off-topic but ran into major issue. Received all the hardware last week (see hardware spec beneath). System is hanging on "System Initializing w/ post code 00".

I do suspect the memory modules. Question: the LRDIMM modules are not supported on the Supermicro X10SDV-TLN4F ?

Glad to see the fork regarding SR-IOV!

Hardware specifications
- Supermicro X10SDV-TLN4F
- 4x Samsung Dram 32gb ddr4-2133 cl15 lrdimm qr (m386a4g40dm0-cpb)
- ChassisSupermicro SC-505-203b w/ additional fans
- Intel 750 1.2 TB

 

Thanks @EluRex

So a short update on this SR-IOV topic, and no I still didn't have the time to test it fully.

Requirements for Enablement of SR-IOV on both i350 and X520 eths:
-Linux Kernel 4.2.+ (as EluRex already proved)
-DO NOT UPGRADE BIOS to 1.0b because there is some kind of bug/error where in this version the SR-IOV option (or ASPM as Supermicro references it on their newest board BIOS) vanishes. I have been in talks with SM about this and already received a custom 1.0b BIOS which has the option reenabled. I am waiting for their response on the GA for this.

Meanwhile I have also reported the warning some of you already noticed on the PCIe bandwidth an waiting for their response.

[ 14.825404] ixgbe 0000:03:00.1: PCI Express bandwidth of 2GT/s available
[ 14.869386] ixgbe 0000:03:00.1: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[ 14.912905] ixgbe 0000:03:00.1: This is not sufficient for optimal performance of this card.
[ 14.998712] ixgbe 0000:03:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[ 15.087831] ixgbe 0000:03:00.1: A slot with more lanes and/or higher speed is suggested.


I will try soon to attach VFs (from different PFs, since there seems to be a bug even on the latest ixgbe/ixgbevf drivers that doesn't allow communication between VFs on the same PF) to different VMs and perform a simple ping test.

Thanks
BR

 

so you have it enabled, just haven't tested a ping and got warnings about bandwidth?

 

It was too late!
https://forums.servethehome.com/ind...0sdv-tln4f-x557-sr-iov-issue.7309/#post-65781

I updated one of the server which is ready to send to production rack! I am soooooo mad now! is there the original bios 1.0a available for downgrade?!

 

Please let us know how things go with SM ripping out SR-IOV from bios. 1.0a was not stable for me.

FYI, 10G is now enabled on ESXi.

 

How quiet are these 1540-boards?
I'm contemplating buying one for a new home-lab build, but it will run in a Node 304 which sits in my office.

If they are noisy I'd rather wait for the E3 v5 CPU's, and put on a liquid cooler or something.

 

I have the 5028 case. Quiet to me is basically hear nothing. That is how my office PC is. This thing definitely makes some noise but is not crazy loud. I keep mine upstairs behind the TV and the dogs still manage to sleep at night

 

Sounds good. Shouldn't be a problem swapping out the fan with a larger one either, which would be more quiet.
Or maybe do some home-modding to get a liquid cooler on it.

 

For those who have a xeon D (and linux with kernel > 3.9), does it support intel_pstate ?
It's a new power scaling driver for modern Intel CPUs (normally since Intel SandyBridge processors)

You can verify this with the command (it must return "intel_pstate" if support is ok):

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driver​