Weird eBay find: any clue what a Supermicro X10SDV-IBQ-AM04 is?

[FrankTL]

Did you order ffc0530P 120S and F05XXPD254SA 30P from Aitendo.com or did you find equivalent parts?

Once I find some time in some weeks I'll give booting from USB one more go, and if that doesn't work then soldering the VGA break-out seems like the only way forward.

 

[adaptivesystems]

no, just bought equivalent parts off ebay.

you can clearly see the numbers to search on the wire, and the 30 pin connector is just a 0.5 pitch 30 pin. again, waiting on things to arrive. cost me less than £10 for 2 wires, and 2 boards. - better err on the safe side in case my soldering is just as shit as it always is.

 

[FrankTL]

ok just ordered ; the cable and the 30-pin connector breakout will come from China/Hong Kong respectively, so it'll probably take a bit of time before I get them.

 

[adaptivesystems]

success. first boot, with picture. may have mixed up the colours on the vga, but thats an easy fix later...

 

[FrankTL]

success. first boot, with picture. may have mixed up the colours on the vga, but thats an easy fix later...

Excellent!

Just got back from holiday, got a nice surprise as all required components have arrived in the mail.
Will get busy soldering tomorrow, but won't have access to a VGA screen until next week.

 

[adaptivesystems]

colour issue was just a crap connection, reseated the pins properly on the flat flex to pins board and the text went white.

usb isnt working, so thats an issue to troubleshoot.


probably another bad connection...... those tiny pitch pin headers are irritating.

 

[FrankTL]

I soldered the VGA connector last night; the soldering job looks terrible but against all expectations it worked on the first attempt.
The system boots, bios isn't protected and it reports as built by ZTSYSTEM and the model some variation on SNOWBALLxxxxxx

After changing some BIOS settings, I could boot from my UEFI USB stick with Ubuntu live image and used that to reset the IPMI admin password. However, I don't see an easy way to restore access to the IPMI web console, and the IPMI shell is pretty useless (mostly seems like an interface to allow read access to sensor data; maybe a serial console).

next I'll probably check if I can boot a hypervisor from USB and/or through PXE boot, and if I can bring down the idle power consumption to more manageable levels (e.g. disable the Mellanox infiniband controller and anything else that doesn't serve much purpose).

Edit: SMCIPMITOOL can control service ports, but it says http and https services are running on the default ports. IP access control is not available due to absense of a BMC license.
Might try setting a license just to see if there's anything blocking in there.
Btw I remarked that the BIOS is fully enabled for the SATA controller, which is not wired to a connector on the motherboard. I wonder if the same is true for the integrated Intel 10GBase ethernet on the CIBF board variant.

 

[FrankTL]

Some more testing and tinkering results:
There are three Mellanox ConnectX-4lx on this motherboard, two that can run at 25 or 10 Gbps, one that does 40 or 10 Gbps.

Setting the IB jumper to disabled deactivates one of the 25Gbps and the 40Gbps interface, and shaves off 5-6 Watt in idle.

BIOS got stuck at '99' when I tried setting the BMC jumper to disabled. Too bad, as the BMC seems rather limited without working web access.

Idle draw is 35-ish Watt with CPU C6 state active, 85-ish Watt with stress-ng matrix 0 (only 1 32GB ECC DIMM, CPU fan, gbit ethernet connected to Intel 10Gbase-T interface, USB3 stick with Ubuntu).

Will test the Mellanox interfaces with a 10Gbit DAC and a 25Gbit DAC ; my intention is to use this as a hypervisor hosting a 10Gbit internet router (eg Mikrotik CHR) and some other VMs.

 

[adaptivesystems]

cant solder the 2.0mm pins, every time i do they break. more waiting while a new order of pre-done stuff comes from aliexpress....groan.


i have been thinking about these snowball motherboards, especially their weird form factor - perhaps both mobos are in one case, and the storage one is unlocked via bmc, and just serves as an iscsi target?

 

[FrankTL]

cant solder the 2.0mm pins, every time i do they break. more waiting while a new order of pre-done stuff comes from aliexpress....groan.


i have been thinking about these snowball motherboards, especially their weird form factor - perhaps both mobos are in one case, and the storage one is unlocked via bmc, and just serves as an iscsi target?

Yup that'd be my guess, interconnected through a DAC on the SFP28 port that's pointing to the 'wrong' direction.
Stil not sure why there would be a PCIe slot on mine, they could have used the lanes for a storage controller for the lower capacity snowballs.

 

[sam55todd]

IMV these are just compute nodes for a cluster, doesn't need much storage or other peripherals, only needs high bandwidth for comms with hive bus.

 

[FrankTL]

IMV these are just compute nodes for a cluster, doesn't need much storage or other peripherals, only needs high bandwidth for comms with hive bus.

I'd think that clustering Snowball units would be using externally exposed network ports.

 

[sam55todd]

Snowball units aren't clustered afaik; they're for edge computing to facilitate data transfer to cloud.

You're probably right (unless those are "snowball edge" {devices physically having orange handles} which have superseded "snowball" {grey handles} and can be used in a cluster from 3 to 16 nodes, therefore potentially can have differently designed worker nodes, control node and load balancer units, then those units would logically fit more towards worker node function). But I'm more specializing on MS/Azure (and even not on infra), so could get it all wrong about AWS side.

 

[FrankTL]

Yes I realized after my initial reaction that snowball edge could be clustered. But the clustering wouldn't entail opening up the unit to connect with the internal ports; i really think the sfp28 bracket that faces the opposite direction from the rear-exposed IO ports is for an internal connection only.

 

[FrankTL]

update: setting a license key for the BMC does not magically unlock the web interface. My guess is that the BMC firmware is a customized image and that the web server is not part of it.

It possibly would work with a standard BMC firmware for the X10SDV series, but I'm very reluctant to try as my board doesn't want to boot with BMC disabled - it gets stuck at bios code 99, which seems to be something related to the super I/O. I have yet to try plugging in >800rpm PWM fans in both the fan1 and fan2 headers, which is mentioned somewhere as being one of the possible causes for the BIOS stalling.

 

[adaptivesystems]

update: bought some pre-done usb 2.0mm header to 2.54mm header and they work, got into bios and can no actually use it as a computer....wooo
bought a mellanox branded dac as thats listed as compatable, loaded up a sata hdd with linux on it, but the nic doesnt show.

which jumper was the one that disabled it? mine may be lablelled differently? as its not showing up, annoyingly.


get nothing from BMC, prob going to flash that generic one and see if it works. not losing anything if it doesnt, tbh

 

[sam55todd]

...and can no actually use it as a computer...

USB network adapter?

 

[FrankTL]

update: bought some pre-done usb 2.0mm header to 2.54mm header and they work, got into bios and can no actually use it as a computer....wooo
bought a mellanox branded dac as thats listed as compatable, loaded up a sata hdd with linux on it, but the nic doesnt show.

which jumper was the one that disabled it? mine may be lablelled differently? as its not showing up, annoyingly.


get nothing from BMC, prob going to flash that generic one and see if it works. not losing anything if it doesnt, tbh

The ports that can be disabled by jumper are the 'extra' Mellanox ones on the -IBQ- model not present on the storage node.

I have a generic intel-compatible dac from fs.com and it worked. Did you bring up the interface with 'ip link set dev xxxx up' ? Ubuntu didn't see anything connected until I did that - after that step ethtool showed a dac connected to the interface.

So far I only tested the 2nd SFP28 port, but this weekend I can test the first port (that is present on your model) as well.

Edit: if lspci / dmesg doesn't show the mellanox device, I'd have another look in the BIOS. Maybe there's an option to disable it somewhere. I'll also look for jumpers when I'm back home again, but I don't recall seeing one for that port.

Edit2: another suggestion is to try booting with the BMC jumper set to disabled - if you get stuck at bios check 99 like me, you probably don't want to risk bricking the BMC by flashing different firmware. Another thing to check is how big a firmware can be flashed - is there enough room for a 'fully featured' bmc firmware?

Edit3: reminder to self, i need to check ip access control for ports 80 and 443 on the bmc that the license should have unlocked.

 

[adaptivesystems]

USB network adapter?

the one i hve apparently doesnt like it, is a shit asix one, - it shows up, but never connects.


yeah, the mellanox never shows up in dmesg, and no options for it in bios either.


haven't tried booting to bmc yet though, so i'll give that a go.

and yeah, i thought about the intel management engine locking me out if i do stuff to the BMC, so might not be worth it after all,

 

[FrankTL]

update: bought some pre-done usb 2.0mm header to 2.54mm header and they work, got into bios and can no actually use it as a computer....wooo
bought a mellanox branded dac as thats listed as compatable, loaded up a sata hdd with linux on it, but the nic doesnt show.

which jumper was the one that disabled it? mine may be lablelled differently? as its not showing up, annoyingly.


get nothing from BMC, prob going to flash that generic one and see if it works. not losing anything if it doesnt, tbh

sorry for the extended wait - here's the jumper info for the -IBQ- variant:

List of all jumpers on x10sdv-ibq-am041, in the format:
LABEL - current setting on my board - description

positioned right of IPMI RJ45 port:
JPTG1 - 1-2 - 10Gbit LAN enable (1-2)/disable (2-3)

positioned 20mm below left-hand side of DIMM slots
JIB1 - 1-2 Infiniband controller enable (1-2)/disable (2-3)

positioned 10mm above JKVM1 header:
JIB2 - 1-2 Infiniband controller enable (1-2)/disable (2-3)

block of jumpers positioned between 'SAS code' sticket location and bottom centre screw hole
JPME1 - 1-2 - normal (1-2)/ME recovery (2-3)
JSMB1 - open - SMBus header
JI2C1 - 2-3 - SMB to PCI-Exp. Slots Enable(1-2)/Disable (2-3)
JI2C2 - 2-3 - SMB to PCI-Exp. Slots Enable(1-2)/Disable (2-3)
JBR1 - 1-2 - normal (1-2)/Bios recovery (2-3)
JPME2 - 1-2 - normal (1-2)/ME manufacturing mode (2-3)

block of jumpers positioned at top left of board, 40mm right of IPMI RJ45 port:
JWD1 - 1-2 - Watchdog RST (1-2)/NMI (2-3)
JPB1 - 1-2 - BMC enable (1-2)/disable (2-3)
JPG1 - 1-2 - VGA enable (1-2)/disable (2-3)

JIB1/2 behaviour observations:
both set to 2-3 (disabled): both 40Gbps and 25 Gbps ethernet ports disabled
both set to 1-2 (enabled): 40Gbps port and both 25 Gbps ports enabled
JIB1 disabled, JIB2 enabled: 40 Gbps port enabled, both 25 Gbps ports disabled
JIB1 enabled, JIB2 disabled: 40 Gbps port disabled, both 25 Gbps ports enabled