lga3647 esxi build *Nov 2024 --> Began Switching from vSphere to Proxmox

[BennyT]

Hello

I'm not having good day. I'm going to buy an internal speaker/buzzer for the JD1 header. I can't figure out why it won't post or if I'm doing something wrong. The USB keyboard backlight I connected to the server lights up. But no signal on the VGA display . I'm open to suggestions or tips on troubleshooting this. I think I need that beeper speaker to at least determine if there is an error code.

But I was able to get into IPMI console and here is what I found:

I turn on the power to the server... no signal via VGA... the motherboard has a "BMC Heartbeat" LED which pulses green, that is good. The power LED near 24 pin is green, that is good too. All of the fans are on, not high speed... that is good. but still no signal on VGA.

I turn off power to the server by pressing it's power button on front, but keeping the PSU and therefore the LAN ports and IPMI ports powered
.
Then I connect both NIC1 and the IPMI to my router (DHCP server) and a single MAC address from appears in the router's client listing and it is given an IP of 192.168.1.3. At this point I'm suspecting the MAC address is from BMC for access to IPMI, not the NIC1. The other MAC and IP shown below is my laptop



Knowing the ip of the Super Micro MAC address I can then login to IPMI console as ADMIN/ADMIN by going to that ip address in chrome browser on my laptop.

Chrome doesn't like Java when launching console via java kvm. So I logged in again using Internet Explorer browser.

I launch a Java iKVM console window but it shows "No Signal". And I'm unable to turn the power off to the server via the console (because it doesn't see the server).


I log out of IPMI in browser window. Then walk to the server and turn off it's power by hand. Then I return to my laptop and login to IPMI while the server is off. I go to the SYSTEM tab again and click Power On button. It turns on the server. I bring up the Remote Control Console via Java again, but again No Signal.

I try the console again but this time using HTML5 console in a chrome browser:

It powers on but no signal. It's not POSTing.

I then look at the Maintenance logs. I'm seeing some things dated in October showing that IPMI was returned to factory default and reset. Is that suspicious? I will assume that is normal due to testing it at Supermicro, but seems kindof suspicious. The logs for those event from Oct 2018 show source of "Localhost" via KCS instead of an IP.



The dates from Jan 05 2019 via 192.168.1.134 is me, that's my laptop address. It says on a few logs that I was successfully able to shutdown the server via iKVM. But the iKVM console had "no signal" entire time the console was open, so that log saying it was succesful is not really true. I had to power off the machine manually walking to it.

I also see I'm not licensed. I thought I would be since I purchased the board, but maybe that was incorrect assumption on my part and this is normal.



I'm not sure what's going on at this point. I'll close her up for now and wait until I can get a piezo speaker/buzzer to see if the board gives any POST beep error codes. But if you have any ideas please share.

Thanks!

Benny

*edit/update: I removed processor from CPU1 socket. It was oriented correctly in the socket, so no problem there. I reseated it while making sure it was oriented correctly. Was careful to "try" keep it to 12 ft lbs tq, but I've no idea how close I am to that spec. Basically when it feels snug I go a tad more. I'll try various tensions to see if that fixes anything until I can find a small tq wrench and allen/hex bit tomorrow at hardware store.

I removed all but one stick of RAM in DIMMA1. I tried again but still no post.

 

[BennyT]

I looked at IPMI once more after resetting CMOS but still no change. While I was in IPMI I took a look at the hardware information screen. It should be empty since it cannot see my system.

Why do I see the hardware shown below in IPMI? Is this just dummy filler data because it cannot see my system? Or is this left over traces of a prior system? It shows different CPU than what I have and different RAM. It even shows various sticks of RAM from Samsung and Hynix and even shows their serial #s. Really weird unless that was the testing hardware, but different mfr sticks? Doesn't make sense. I've even returned IPMI to factory default and the bogus hardware info still is seen.

I'm wondering if I was sold an RMA'd board. Checking now about Supermicro parts warranty/replacment etc.

Let me know what you think. Thanks and have a great weekend.

Benny

 

[Rand__]

-The default activity would be to run CMOS reset - have you done that?

-Do you ever see "Server on" in IPMI or always off despite it being on? Are you sure you have connected the correct power cables?

-It should not display dummy system data in ipmi, you can also reset that to defaults; but might have been used before

-Not being 'licensed' is ok, since thats not automatically included in a board, you can buy a license for $30 or find alternative ways using a search which would enable licensed features (bios update via ipmi).

-Are you sure vga is enabled (JPG1)?

 

[BennyT]

Hi. Thanks @Rand__ , I really appreciate your instruction steps. I'm posting my answers to your questions below:

-The default activity would be to run CMOS reset - have you done that?

yes. I've reset cmos by following this but I didn't see a different result. No POST or VGA signal.

-Do you ever see "Server on" in IPMI or always off despite it being on? Are you sure you have connected the correct power cables?

No, IPMI has never shown "Server on" even when it is on and fans are spinninng. It always shows Host is not powered on.

When the server really is off I can power it on via IPMI "Power On". But IPMI never recognizes that it powered it on. After about 30 seconds it displays message "Performing power action failed. Please check." even though it did power it on.


The PSU 24pin and two 8pin power connectors are wired as such:


I disconnected and then reconnected the two 8pin and the 24pin connector at both ends to verify I was using correct cables They were correct and they are seated properly but with same no POST result.

-It should not display dummy system data in ipmi, you can also reset that to defaults; but might have been used before

I've restored to Factory Defaults. But the bogus hardware information still exists in the hardware info tree. Really strange. It is one of the clues that indicated to me this was at one time in a previous system. The Maintenance System Event logs also showed that someone had previously Restored to Factory once before back in October 2018 which may be when somone tried to erase previous systems configuration.

I can see that on Oct 5 at 12:30AM someone Restored to Default. Then on the following day Oct 6 at 2:40PM they erased the event logs a couple times. The logs have since been erased again by my own restoring to factory. I'm glad I took a snapshot image before I erased the logs. But the hardware information still shows bogus old system info from possible previous owner even after restoring to factory.

The logs and the weird hardware information makes me to believe this is a used motherboard. But I may be paranoid.

-Not being 'licensed' is ok, since thats not automatically included in a board, you can buy a license for $30 or find alternative ways using a search which would enable licensed features (bios update via ipmi).

Ok, that is good to know. I was beginning to think a previous owner kept a license while the vendor cleared out the key. I've been acting very paranoid after this ordeal.

-Are you sure vga is enabled (JPG1)?

Yes. Jumper is on pins 1&2 (VGA enabled)


Thanks again Rand___

I'm going to hardware store today to purchase a nm or inch lb 1/4" torque wrench and a T30 socket with extension. The internal speaker/buzzers should arrive on Monday. Maybe this isn't a previously used board and maybe I've simply not torqued the processor correctly to correct specs

Regardless, I've requested an RMA from the seller and they've sent me an RMA label. I listed reason for return is "Defective" Dead on Arrival. I did not alude that the board may have been previously owned. They say 20% restocking on the RMA label document. So I'm going to contact Supermicro Monday morning (Tech Support and RMA are not open weekends unless you have SuperServer support) and see about parts warranty and replacing the board via them instead of the distributor/seller.

But first I'm going back to bed to catch a few hours sleep. Enjoy your weekend

 

[BennyT]

There are a few more recent IPMI versions. A 1.55 and a 1.61.

The IPMI version I have is at 1.51 with a build time of August 21, 2018.

Do you think an IPMI firmware update might fix this? I don't think it would help since IPMI doesn't seem to be the problem. The problem is the board does not POST, therefore IPMI thinks it is shutdown. But let me know what you think.

Okay, now I'm going to bed for real.

 

[itronin]

When you wake up and since you may be returning for RMA or getting supermicro to repair the board you might consider running it outside of the chassis (AKA bench build). Do use a non conductive surface to hold the motherboard, cardboard sheet, carboard box etc. Leave the power supply in the chassis if the cables will reach or pull it out if they won't. If the power on cable doesn't reach make one with your spare breadboard wire (previous picture). Use extra care with ESD precautions outside of the box. touch the power supply before touching the motherboard, cpu, or memory or wear an A/S wristband connected to a grounded source. You've only got one CPU so that's good. Use only one stick of memory, and no add-in cards (can't remember if you have any) and verify in the manual the memory slot configuration for 1 cpu, 1 stick of memory. don't connect any drive cables, do connect your cpu fan cables. You might also try backing off tension on the cpu heat sink *before* you try tightening it up. I'd be really careful torquing the heat sink into the cpu. The Noctua's usually have a long driver and I've not had a situation where I had to use extra torque to tighten them down. I'd be really curious to see what the BMC reports (if it does) with no cpu, and no memory in the motherboard. fwiw, a used computer store may have a salvaged speaker from an old AT or early vintage ATX case (if you happen to go by one on your way to the H/W store).

 

[itronin]

You've been really thorough with the build (and great pictures btw) but when you pull the motherboard you may want to check the case to insure there are no "extra" standoffs are in there and there's no chassis flange or chunk of metal coming up from the motherboard tray and making unwanted contact with the bottom of the motherboard.

 

[BennyT]

Thank you @itronin. I will definately try testing outside of the chassis. I'm stepping away from it for the rest of the weekend though. Monday I should get the little internal speaker buzzer and check for beep codes. Then I can decide how to proceed.

But I did try one more time today after I returned from the hardware store.

By the way, the stores I visited didn't have the torque wrench I needed down to 12lbf (12 pound force - inch).

I removed the CPU from socket again. This time I very lightly snugged bolts 3&4 on the heatsink. I'm speaking of the middle bolts, not the two corner bolts. Corner heatsink bolts are not under spring tension. Only the middle bolts 3&4 are under spring tension and it's those bolts that are spec'd to 12 pound force - inch of torque.

Turned power on and none of the fans turned on (excpet PSU fan) and none of the front panel LEDs lit up. The only LEDs that lit up were the BMC heartbeat LED and the green power LED near the 24pin. No VGA output.

I powered it off at the PSU. I tighted the CPU heatsink onto socket another 3/4 turn of the hex wrench which I feel is approx 10- 12 inch lbs. tq. Powered it on again and this time the FANs spun and front panel LEDS lit up. But still no VGA output or POST.

 

[Rand__]

Never had that much trouble with mine (for a change, usually have a lot of trouble with my stuff)

But initial boot takes very very long with this board while it does RAM training, maybe its not displaying properly at that time. Have you let it just run for a few minutes?

 

[BennyT]

Hi @Rand__ I’ve waited at least 30 minutes and longer with it powered on while Im in IPMI looking over configuration and logs etc, and during this time it doesn’t POST.

I think the only things left to do is wait for beeper speaker to arrive tomorrow and check the error beep codes. Also I’ll try diagnosing the board outside of chassis. Then I plan to call Supermicro regarding my parts warranty. Maybe I’ll be able to refer SM to this thread since it is pretty well documenting my issues in detail.

 

[Rand__]

I am curious what the speaker will "say".
And I never realized that board did not have a built in speaker anymore, good thing I have one around.
And maybe I can rig one from the flight controllers to just sit on top of the pins... hmm

 

[BennyT]

Yeah, I'm curious to see what error codes I get too.

Can anyone recommend a good Supermicro motherboard reseller?

The distributer/seller says I would not have to pay 20% restocking if I RMA it back to them.

Granted I still want to diagnose the error codes before I seek to replace the board, but I do expect to be replacing this board. I'm trying to determine if I should RMA to seller or seek manufacturer warranty support.

If I seek warranty then Supermicro will either repair or replace the board. But if I RMA back to the seller I'll need to order another board. Can anyone recommend a good reliable distributor/seller of SuperMicro motherboards?

Thanks,

Benny

 

[Rand__]

Usually (in EMEA) you need to send back to your reseller as they have to return it to SM themselves and since that incurs cost usually only your reseller will do it.
This might be different in the states though; but unless it was very cheap (and you can't re-procure at that price) I'd send it back to the seller.

 

[BennyT]

Thanks Rand___ that is good advice.

My seller has been very aggreable to work with and they've not been contentious at all. They also recommend I return the board to them for a refund and then I can place an order for a new board. They say that would be the fastest turn around to get me a new board. I think I will go with that recomendation and simply return the board and order another one from them.

Unless anyone in the USA can recommend another good SuperMicro reseller I plan to order another from my current seller once my refund goes through.

Thanks!

 

[BennyT]

I finally connected speaker/buzzer to header JD1. No error codes at all. I removed the RAM sticks completely, no RAM. No error codes buzzed.

I removed the CPU and still no error codes. Its basically a dead board with exception of BMC IPMI, fan headers. Really weird.

Supermicro Troubleshooting says that in such cases of no error code when all RAM is removed then replace the board. So that's what I'm doing tonight. I'll package it all up in the original retail box and ship it out.



I just visited the supermicro website for this board to view the user manual PDF... but supermicro has taken that board off it's website. Im not sure what to make of that. It was there a couple days ago. I'll assume it's just a fluke with website maintenance and not to take it as a bad sign. Why am I so paranoid now.

 

[BennyT]

You've been really thorough with the build (and great pictures btw) but when you pull the motherboard you may want to check the case to insure there are no "extra" standoffs are in there and there's no chassis flange or chunk of metal coming up from the motherboard tray and making unwanted contact with the bottom of the motherboard.

Hi itronin,

I promised I would try diagnosing this outside the chassis, but I think I'm going to just ship it back at this point without trying outside the chassis. It's basically dead. The standoffs looked good though. They were all where they were supposed to be and there were no extra standoffs were they were not supposed to be. I didn't see anything that could've been trapped under the board or around edges that would short it out. I think the less time I spend with this board now, the better I'll feel.

I will do a pre-chassis check though on the next board, the replacement board.

 

[Rand__]

I just visited the supermicro website for this board to view the user manual PDF... but supermicro has taken that board off it's website. Im not sure what to make of that. It was there a couple days ago. I'll assume it's just a fluke with website maintenance and not to take it as a bad sign. Why am I so paranoid now.

Working fine for me

 

[BennyT]

I originally planned to boot from a USB flash. But I recieved the 80GB Intel enterprise SSD in the mail ($20 used from ebay). I plan to use it to boot ESXi and keep persistent scratch on it.

And if anyone else was curious like I was why the motherboard didn't have onboard buzzer / speaker, I found this FAQ explaining why

The X11 Supermicro EATX boards didn't have space. That is one crowded motherboard if can't find space for a buzzer. They still put them on other boards though.

 

[BennyT]

I'm now rreconsidering which motherboard to use as replacement for the X11-DPI-NT which I've returned for refund.

After discussing with another user here, the other board I'm considering is the Supermicro X11-DPH-T or perhaps the X11-DPH-TQ. The "Q" in the later means it has Intel's Quickassist which accelerates intensive CPU utilization for things like big data (HADOOP for example), realtime compression, encryuption... none of which I really do, but sounds cool and maybe I can use it later on.

The Xeon 6130 which I have has 3 UPI links. The DPH-T (with or without the Q) over the DPI-NT is that the DPH has 3 UPI links instead of 2 on the DPI. I'm won't pretend I understand UPI links, but I do know 3 is greater than 2.

The single m.2 slot on the DPI-NT is further downstream than the m.2 on the DPH. The DPI m.2 is via PCH. The two m.2 slots on the DPH are directly connected to CPU by 2x 4-lane.

The DPI-NT does have two addition occulink NVMe connectors though, and those are by 2x 4-lane.

It's almost a toss up to me... basically the DPH-T has 3 UPI links vs the DPI having 2 UPI links. The DPH-TQ also has quickassist as mentioned.

Let me know your thoughts. I'll probably order one or the other next week. Thanks!

Here is the x11DPI-NT system diagram:

and here is the x11DPH-TQ diagram:

 

[Rand__]

I thought UPI links were primarily for CPU to CPU communication (I.e. max #cpus in multi CPU systems). Not sure this will help at all on a dual cpu board?
I'd have a closer look at that before deciding...