Supermicro X10SDV-TP8F

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
Moving this to its own thread, per @Patrick's suggestion.

A welcome present arrived in the mail today...



X10SDV-TP8F, D-1518, 2x 10G SFP+, 6x 1Gbe. Don't need the extra 1gbe ports but this was the board I could get the soonest.
 
  • Like
Reactions: Patriot and Patrick

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
Evan said:
@PigLover what case will you be using ?
Rack cases are plenty, but small desktop cases (needed as in apartment and no chance for a rack) for flex-atx seem few and far between.
SC113MTQ. 1U short depth with 8 2.5" hotswaps + an LSI9300 IT mode that I have on the shelf. For now. This will likely be the first of 5 CEPH OSD nodes in a small cluster. I plan to do some simple performance testing and may go down-scale to the 2C/4T model for the rest of the cluster (OSD really isn't an intensive use case).
 

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
As it turns out...it's a standard depth 113 that I had ready with cables staged, etc. The little baby board looks rather ridiculous in the big case ;) And the 563w PSU is probably overkill (duh!).

But this is what let's me fire it up tonight...

 
  • Like
Reactions: T_Minus

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
A couple of notes on the X10SDV-TP8F:

I loaded Ubuntu 16.04 daily last night. Came up perfectly and recognized all 8 NICs (2x i210s, 4x i350s and 2x 10Gbe). There was an annoyingly long pause by the installer at "detecting devices" - I estimated 90 seconds. Don't really understand that but I'm speculating that it was trying to figure out which NIC had a live internet connection.

Since the manual isn't published yet i had to guess on the NIC order. Appears that the leftmost two (facing from the rear) are the i210s with port 0 on the bottom (typical for SM). The next four are the i350s. The 10Gbe are sorta obvious as they are SFP+.

I did notice an oddity with the new Linux NIC renaming scheme. Firstly, it only seems to permit 4 "on MB" NICs, named "eno1", eno2", "eno3" and "en04". The other for get named "rename1", "rename2", etc. It was designed to "ensure" stable naming across reboots - but it doesn't. The names shift around across reboots, probably because of probe-response time randomness, which makes it impossible to get a clean startup across boots. For now I disabled the "new" naming and went back to MAC-based binding to eth[0-7].

I also successfully set up booting to a ZFS mirror rpool. This is still not supported directly in the installer (boo) and I had to follow the method published for 15.10. But it does appear to work. We'll see if it will still be stable after a kernel update.
 

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
Paul Bommel said:
@PigLover

Are these Supermicros 65L4-Fans?

If the case is closed and in the rack, does it sound like a jet engine?
Yes they are. And no it doesn't. They are not silent - or really even close to silent - but the MB holds them at about 5k RPM and they give a nice dull hum at that speed.

Not bedroom quiet, might not want it right next to you, but from across a small room it's good enough. For me it doesn't matter because they all live in the garage.[/quote]
 

Yamabushi

Member
Feb 19, 2016
33
5
8
53
A couple of notes on the X10SDV-TP8F:

I loaded Ubuntu 16.04 daily last night. Came up perfectly and recognized all 8 NICs (2x i210s, 4x i350s and 2x 10Gbe). There was an annoyingly long pause by the installer at "detecting devices" - I estimated 90 seconds. Don't really understand that but I'm speculating that it was trying to figure out which NIC had a live internet connection.

Since the manual isn't published yet i had to guess on the NIC order. Appears that the leftmost two (facing from the rear) are the i210s with port 0 on the bottom (typical for SM). The next four are the i350s. The 10Gbe are sorta obvious as they are SFP+.

I did notice an oddity with the new Linux NIC renaming scheme. Firstly, it only seems to permit 4 "on MB" NICs, named "eno1", eno2", "eno3" and "en04". The other for get named "rename1", "rename2", etc. It was designed to "ensure" stable naming across reboots - but it doesn't. The names shift around across reboots, probably because of probe-response time randomness, which makes it impossible to get a clean startup across boots. For now I disabled the "new" naming and went back to MAC-based binding to eth[0-7].

I also successfully set up booting to a ZFS mirror rpool. This is still not supported directly in the installer (boo) and I had to follow the method published for 15.10. But it does appear to work. We'll see if it will still be stable after a kernel update.
Last night I just received and built up an X10SDV-8C-TLNF4 with a Samsung 950 Pro M.2, and 64GB ECC RDIMM's and I think I may be having the same or at least a related issue. When starting up it takes a long time, maybe about 45-60 seconds or so to get get to the screen where I can enter the BIOS, from there on to Kubuntu 15.10 desktop takes almost another 120 seconds. Once in Kubuntu it is giving me "eno1", "eno2", "rename4", and "rename5" for my NIC's. Just FYI I did a EFI install and also have an nVidia GT640 video card installed. Please, keep us posted on your progress on this as these slow boots are a bit demoralizing. Any help you can provide would be greatly appreciated! Thank you in advance!!
 
  • Like
Reactions: Patrick

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
Idle or under (heavy) load?
They never seem to spin up no matter what I do. Did a prime59 burn on it and the fans just sat at ~5k rpm the whole time. At 35W I just don't think it every generates enough heat to need more than a slow breeze across the heatsink.
 
  • Like
Reactions: Patrick

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
Last night I just received and built up an X10SDV-8C-TLNF4 with a Samsung 950 Pro M.2, and 64GB ECC RDIMM's and I think I may be having the same or at least a related issue. When starting up it takes a long time, maybe about 45-60 seconds or so to get get to the screen where I can enter the BIOS, from there on to Kubuntu 15.10 desktop takes almost another 120 seconds. Once in Kubuntu it is giving me "eno1", "eno2", "rename4", and "rename5" for my NIC's. Just FYI I did a EFI install and also have an nVidia GT640 video card installed. Please, keep us posted on your progress on this as these slow boots are a bit demoralizing. Any help you can provide would be greatly appreciated! Thank you in advance!!
You can go back to legacy device naming pretty easily:

Edit /etc/default/grub and change the boot arguments to this:

GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"
Then run "update-grub" and reboot. You'll be back to NICs named "eth0", "eth1". With your board they should be stable across boot because the 1Gbe NICs always seem to init faster than their 10Gbe brothers. On my board I still get an ugly race between the i210 and i350 1Gbe NICs.

You fix that by writing a udev rule /etc/udev/rules.d/70-persistent-net.rules and force the names by MAC address.
 

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
BTW - I also have a trouble ticket open with SM asking them to consider fixing their BIOS so that it returns unique SMBIOS type-41 records for each NIC. This would be the correct fix.
 
  • Like
Reactions: BennyE_HH

BennyE_HH

New Member
Mar 30, 2016
6
2
3
37
BTW - I also have a trouble ticket open with SM asking them to consider fixing their BIOS so that it returns unique SMBIOS type-41 records for each NIC. This would be the correct fix.
I was just about to ask you ... Thank you and have a great weekend!

I truly hope that those boards become available in Germany one day ...

Benny
 

Yamabushi

Member
Feb 19, 2016
33
5
8
53
You can go back to legacy device naming pretty easily:

Edit /etc/default/grub and change the boot arguments to this:

GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"
Then run "update-grub" and reboot. You'll be back to NICs named "eth0", "eth1". With your board they should be stable across boot because the 1Gbe NICs always seem to init faster than their 10Gbe brothers. On my board I still get an ugly race between the i210 and i350 1Gbe NICs.

You fix that by writing a udev rule /etc/udev/rules.d/70-persistent-net.rules and force the names by MAC address.
Thank you for that! Pardon my ignorance, won't this only affect things after posting, between the Supermicro Logo and reaching the desktop. My underlying issue seems to between hitting the power button and reaching the Supermicro Logo. That is taking 45-60 seconds. My current bios is "1.0c", but I just saw that they have released "1.1". That may address the problem?
 

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
Slow startup on SM server boards is 'normal'. They do a lot of hardware checks and characterization on init. It's not a workstation or laptop board and your not really expected to be rebooting it very often anyway.
 
  • Like
Reactions: Yamabushi

Yamabushi

Member
Feb 19, 2016
33
5
8
53
Thank you again! That was something I wasn't expecting. I have a Supermicro A1SRi-2758F that is my pfSense firewall and I don't remember it taking so long to POST, although you are definitely correct, I very rarely reboot it.
 
  • Like
Reactions: solaris12

Yamabushi

Member
Feb 19, 2016
33
5
8
53
Quick update... I updated my BIOS, same issues. Then I made the GRUB change that you suggested and still the problem persists. I continue getting the slow boot and the NIC naming scheme referenced above. Does that mean I need to mod my "/etc/udev/rules.d/70-persistent-net.rules"? If yes, how?
 

Paul Bommel

Member
Oct 30, 2015
44
6
8
They never seem to spin up no matter what I do. Did a prime59 burn on it and the fans just sat at ~5k rpm the whole time. At 35W I just don't think it every generates enough heat to need more than a slow breeze across the heatsink.
hmm, "5k rpm" is still to much.
The 1541 is listed with 45W and I'd like to throw 100L4 in, but I fear they would always run at full speed.
 

Yamabushi

Member
Feb 19, 2016
33
5
8
53
OK, I moved the boot parameters you gave me from GRUB_CMDLINE_LINUX= to GRUB_CMDLINE_LINUX_DEFAULT= and that sorted me right out! I now have eth0, eth1, eth2, and eth3, and now from POST to my desktop is very fast. THANK YOU!!
 
  • Like
Reactions: PigLover

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
Got the engineering build BIOS loaded and tested. Seems to fix the Ethernet identification issue.

Using "dmidecode" confirmed that the on-board ethernet ports are enumerated corrected (on-die I210s #1 & #2, on-die 10Gbe #3 & #4, and off-die i350s #5, #6, #7 & #8).

Removed the boot arguments and rebooted Xenial - NICs named eno1..eno8 just like you'd expect.

I've asked SM when they expect to publish the 1.0a BIOS. No answer yet.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
Got the engineering build BIOS loaded and tested. Seems to fix the Ethernet identification issue.

Using "dmidecode" confirmed that the on-board ethernet ports are enumerated corrected (on-die I210s #1 & #2, on-die 10Gbe #3 & #4, and off-die i350s #5, #6, #7 & #8).

Removed the boot arguments and rebooted Xenial - NICs named eno1..eno8 just like you'd expect.

I've asked SM when they expect to publish the 1.0a BIOS. No answer yet.
Hoping it's soon along with the IPMI fix.
 

Yamabushi

Member
Feb 19, 2016
33
5
8
53
Got the engineering build BIOS loaded and tested. Seems to fix the Ethernet identification issue.

Using "dmidecode" confirmed that the on-board ethernet ports are enumerated corrected (on-die I210s #1 & #2, on-die 10Gbe #3 & #4, and off-die i350s #5, #6, #7 & #8).

Removed the boot arguments and rebooted Xenial - NICs named eno1..eno8 just like you'd expect.

I've asked SM when they expect to publish the 1.0a BIOS. No answer yet.
Good to hear! While not mission critical for me, it'd nonetheless be nice to have this fix for my board as well.