So a short update on this SR-IOV topic, and no I still didn't have the time to test it fully.
Requirements for Enablement of SR-IOV on both i350 and X520 eths:
-Linux Kernel 4.2.+ (as EluRex already proved)
-DO NOT UPGRADE BIOS to 1.0b because there is some kind of bug/error where in this version the SR-IOV option (or ASPM as Supermicro references it on their newest board BIOS) vanishes. I have been in talks with SM about this and already received a custom 1.0b BIOS which has the option reenabled. I am waiting for their response on the GA for this.
Meanwhile I have also reported the warning some of you already noticed on the PCIe bandwidth an waiting for their response.
[ 14.825404] ixgbe 0000:03:00.1: PCI Express bandwidth of 2GT/s available
[ 14.869386] ixgbe 0000:03:00.1: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[ 14.912905] ixgbe 0000:03:00.1: This is not sufficient for optimal performance of this card.
[ 14.998712] ixgbe 0000:03:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[ 15.087831] ixgbe 0000:03:00.1: A slot with more lanes and/or higher speed is suggested.
I will try soon to attach VFs (from different PFs, since there seems to be a bug even on the latest ixgbe/ixgbevf drivers that doesn't allow communication between VFs on the same PF) to different VMs and perform a simple ping test.