The Dell PowerEdge C6320/C6300 Mega Thread

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

CA_Tallguy

Member
May 19, 2020
30
5
8
Throwing this against the wall to see if it sticks.... anyone else out there on the C6320? There is a great C6220 thread that has been informative....

... but thought it might be good to start a new one for the C6320. There are probably a lot of similarities between the previous model and this one, and perhaps a lot more in common than with the C6420 that is next in line. But there are certainly differences.

I think a lot of the chassis and parts may be similar between C6320 and C6220 i and ii. I am having a hell of a time with the fans on the C6300 chassis I picked up and I read that fans were also an issue on previous. Were any of you guys successful in updating the firmware or settings? Any tips?

A Fan Control Board firmware update refers to some fan control issues, I believe that occur when a C6320P sled is put into a chassis configured for C6320 or vice versa.... For C6320 - PowerEdge C6300 ENCLOSURE FCB FW Update | Driver Details | Dell US

The C6320P sled owners manual has a whole section about the fan control board and that is missing from the C6320 manual. It says to check the sticky bits: Owners Manual

That didn't work for me ... I got: "Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0xc8 rsp=0xd5): Command not supported in present state"

Another thing that is different from previous model is the SFP+ ports. I have tried THREE different base-t transceivers so far and can't get them to work pre-boot for iDRAC and LifeCycle Controller. Anyone have any tips? Can they be configured and enabled anywhere? I see some settings and ability to view them in iDRAC and such but the only thing I believe I am seeing is the embedded LOM, not the transceiver. The transceivers I have put in there just seem dead. I got a different C6320 sled delivered today and same thing.... just dead. The last transceiver I purchased was listed as compatible on Dell website so I'm starting to think there is a setting I may be missing to get them to light up. (posted about this on Dell community forums here ===> SFP's for C6320 Sleds - can't get them to light up so there may be a solution there at some point.)

Anyway, please join in on this thread if you have an interest in the C6320 or the poweredge C platform. There is a big lack of documentation on this particular model so I think anyone jumping on this platform will benefit from some discussion and sharing of advice. I have some specific issues that I'd like to discuss but I'll also jump in and start to post relevant info and documentation I come across to help others going through the same frustration later with docs. As I mentioned above, there is even more documentation in the C6320P manual than in the C6320 manual!
 
Last edited:
  • Like
Reactions: Patrick

CA_Tallguy

Member
May 19, 2020
30
5
8
Here is a post I made over on Dell community forums ( here ==> Half of C6300/C6320 Fans Don't Work ) about a problem specific to my system but since it has some links to outside resources I will post it in this thread too. A couple Dell employees have replied but have not offered a lot more info yet.

Half of C6300/C6320 Fans Don't Work
Hi folks -- having a heck of a time trying to understand what is going on with the fans in this C6300 chassis I'm trying to turn up.

SYMPTOM: The fans for the left two sleds are going full blast (assuming) and the fans on the right two slots are either not working or are barely turning. (I disconnected the left side and these right side to try to discern and the right side do seem like they are turning but barely. I could be wrong and they are just dead or turned off.

OBSERVATIONS:
  • There is nothing plugged into "fan speed control" header on power distribution board.
  • Using only 1 sled out of 4, moved sled to the right side but fans remain only engaged on left.
  • Swapped known working fan from left side to right side to eliminate chance of a defective fan.
  • It appears there have been some known bugs around fans and sled mismatching (C6320 in chassis configured for C6320P and vice-versa?) so a firmware update may rectify some problems: For C6320 - PowerEdge C6300 ENCLOSURE FCB FW Update | Driver Details | Dell US
TROUBLESHOOTING:
  • Within fan control of iDRAC, get message "RAC0709: Unable to retrieve the fan information. Power on the server. If the server is already powered on, wait for a few minutes and refresh the page. If the problem persists, contact your service provider." (NOTE: OpenManage manual says that RAC0709 error is actually "Unknown Server Inserted Into Chassis")
  • Using IPMI documentation for C6220 and C6320P (unable to find docs for C6320), attempted a few commands that failed:
NEXT STEPS (please provide feedback, guidance, tips and suggestions... thanks!)
  • Continue trying to verify/update Fan Control Board (FCB) firmware version. Here are some relevant links found on Dell site and web....
  • Continue attempts to check "sticky bits" to ensure chassis is configured for C6320 sleds not C6320P
  • Try additional sleds that are en route to note any different behavior or ability to better see/probe the chassis through those sleds.
  • Exchange this chassis for another from the seller, however their inventory may be similarly configured chassis enclosures from the same environment.
 
Last edited:
  • Like
Reactions: Patrick

CA_Tallguy

Member
May 19, 2020
30
5
8
NETWORK/SFP RESOLUTION -- problem was the switch!

Still working on the FCB but I figured out the issue I was having with the SFP's (I may try to swap that chassis for another tomorrow if I don't fix this today) -- turned out the switch was not properly negotiating the link with the SFP's.

To troubleshoot, I had a DAC cable and plugged that in between sleds. Lit right up. So then I plugged the sleds into each other and all the SFP's for base-T, even non-Dell, worked fine talking to each other. Then tried another switch and the SFP's lit right up.

In my research on this problem, I had heard about some of these SFP's needing to be manually set to a speed to be backwards compatible with some switches, but in places such as iDRAC configuration, the link speed configuration options are dimmed…. probably because iDRAC is sharing those NIC's so it can't/shoudn't override the system configuration. I don't see another place to configure this, however.

It sure would be helpful if Dell software would (1) better show the presence of SFP's and (2) the status of a present/inserted LOM/SFP could also be shown (such as no cable/network connection). To me, I couldn't discern the presence of an SFP in any information that I retrieved about the LOM's.

More information about my troubleshooting of the LOM/SFP issue in this thread on Dell.com... SFP's for C6320 Sleds - can't get them to light up
 
  • Like
Reactions: Patrick

CA_Tallguy

Member
May 19, 2020
30
5
8
Here's my latest post on the thread about fan speed over on Dell's forums.....

ATTEMPTS TO PROBE FCB (FAN CONTROLLER BOARD) VIA IPMI RAW COMMANDS
(TRYING TO COMMUNICATE BEFORE ATTEMPTING FIRMWARE UPDATE)

cut/paste:

The "raw" command is not just the fans and has to be in a format:

ipmitool –I lanplus -H ipaddress -U username -P password raw netfn cmd data
In this format, the netfn element is the network function, which identifies the functional message class and clusters IPMI commands into different sets. The cmd element represents a unique one-byte command value within a given network function. Finally, the data element provides additional parameters for a request or response, if any. (source: https://www.dell.com/downloads/global/power/ps4q07-20070387-Babu.pdf )

NOTE: I am color coding everything as: netfn cmd data and response

In my listed commands below and in original post, I appear to be issuing commands 0x12 and 0xc8 to "netfn" = 0x30 ( and to repeat above documentation, "network function ... identifies the functional message class and clusters IPMI commands into different sets" ). IPMI is doing a nice job of repeating back and parsing out how it interpreted original command and then what the response code is for the operation…. (0x80 = "Unknown" and 0xd5 = "Command not supported in present state").

$ ipmitool -I lanplus -H 192.168.50.16 -U root -P ***** raw 0x30 0x12
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x12 rsp=0x80 Unknown (0x80)

$ ipmitool -I lanplus -H 192.168.50.16 -U root -P ***** raw 0x30 0xc8 0x01 0x0A 0x05 0x00 0x00 0x00
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0xc8 rsp=0xd5 Command not supported in present state (0xd5)

I have not been able to see fan speeds when they are in operation anywhere at all, not just missing in IPMI. Above commands and as shown in my original post on this thread do not seem to do anything useful. I will try again today to elicit some useful info, even if not on the fans.

I am not sure if 0x30 is the correct "netfn" for this system. Any idea where I can verify this? The commands I am issuing are not giving me anything useful or expected, as shown. Or is there a way to probe for the system to tell me what the valid/active "netfn" codes are for my configuration?

I suspect it could be different for PowerEdge C platform vs regular PowerEdge as the fan control/power distribution board is shared between the 4 different compute sleds/motherboards whereas on a normal PowerEdge server, there is only one chassis and fan/cooling setup supporting a single "compute sled" or motherboard.

On the C6320P manual, they do have a section on checking "sticky bits" and that is where I am getting the 0x30 netfn…… https://www.dell.com/support/manuals/us/en/04/poweredge-c6320p/pec6320p_om_pub/checking-the-chassis-...

I assume that the "netfn" is the same for the C6320 (no p) and C6320p sleds as both are using the C6300 chassis. The S C6320p is just one processor not two.

The really confusing part of this setup is knowing if these IPMI commands issues to the various sleds are making it back to the actual chassis that holds four sleds. As per my previous post 05-26-2020 10:33 PM — I have seen indications in places that on previous generations the commands had to be issued from the fourth sled ONLY and that if issued from sleds 1 to 3 they may not work. There is nothing mentioned about this that I can see in the C6320P manual. And as I've said there is NOTHING about any of this in the C6320 manual. So I feel like I"m playing with minimally documented feature. As I've said, if I have to use 4th sled that is going to be tricky because the cooling seems almost non-existent (though the fans are turning). I likely will have to rig up some temporary cooling for the CPU's as the whole point of this thread is to get the fans to work properly on both sides.

If anyone has more info on all this I would be very grateful. Especially if anyone can verify that I should be issuing commands to 0x30 and if I have to use ONLY the fourth sled.
 

CA_Tallguy

Member
May 19, 2020
30
5
8
Amazing news.... I swapped for a different chassis today and fans seem like they are working great now. Still need to mess with the system to see if they are adjusting speed properly.

There are SO many settings and data points available in iDRAC that I was not seeing before. As I'm new to Dell enterprise hardware I had no idea what I should be seeing. But a BIG clue to anyone else facing these types of symptoms -- I was not seeing power supply info, fan speeds and the message "RAC0709: Unable to retrieve the fan information" as I mentioned in original post were conditions suggesting that the sled was not properly communicating with the chassis.

Once I powered up this chassis with only a single power supply inserted, I immediately got flashing power button and indicator lights that I hadn't seen on the previous chassis.

Among the theories I have been mulling over that may have been the issue with the original chassis I had....
  1. missing cable connection somewhere (this seems most straightforward)
  2. bad fan controller/power distribution board (I think they are integrated together?)
  3. Other chassis was configured for C6320P and I was using C6320 sleds. In my research I found some information that said this could cause fan problems (perhaps in early firmware). Also note: RAC0709 error is listed in some Dell docs as "Unknown Server Inserted Into Chassis" instead of "Unable to retrieve fan information" as displayed under fan information when my sleds were in the other chassis.
  4. Corrupted firmware
  5. Incompatible firmware (I've heard that if some firmware gets too far ahead of other firmware sometimes it is necessary to regress firmware versions in some devices in order to update the lagging one)
  6. Fan settings hard coded or corrupted by a previous user (although that doesn't explain why probing the board via IPMI failed)
 

CA_Tallguy

Member
May 19, 2020
30
5
8
Let's talk about MEZZANINE CARDS for a moment.... there are a bunch of interposer/bridge cards out there that LOOK like they may be interchangable out there but they are NOT compatible!

JKM5M Printed Wiring Assy.Daughterboard.Serial Attached Scsi.Bridge.X3.Data Center Solutions
HH4P1 Card.Interface.Mezzanine.Bridge.C6220.CLE
2NYRW Card.Expansion. Expand.Bridge.C8000
4NWN5 Printed Wiring Assy.Daughterboard.Mezzanine.Bridge.V2.V1.6300
MW5RY Printed Wiring Assy.Daughterboard.Mezzanine.Bridge.V2.V2.6300

The 4NWN5 card seems most often listed as compatible with C6320 and I can CONFIRM that it works with mellanox cards on C6320 sleds.

HH4P1 and 2NYRW will prevent booting up or even powering on! Fortunately I don't think they caued any damage but different cards appear to short out some of the pins in different ways so pay attention to what you are putting into your machine. Be careful when buying equipment to see what kind of card you are looking at because some ebay listings may say a card is compatible with C6320 but then ship it with a bridge card that may NOT be compatible.

I'm still trying to figure out if any other bridge cards can or must be used for C6320 but AFAIK the 4NWN5 is the one you want (at least for mellanox cards)

Further discussion and pictures:
 
  • Like
Reactions: gabor

josh

Active Member
Oct 21, 2013
615
190
43
There isn't much supply of these of fleabay. What did you pay for yours?
 

latezx

New Member
Apr 23, 2020
8
5
3
Amazing news.... I swapped for a different chassis today and fans seem like they are working great now. Still need to mess with the system to see if they are adjusting speed properly.

There are SO many settings and data points available in iDRAC that I was not seeing before. As I'm new to Dell enterprise hardware I had no idea what I should be seeing. But a BIG clue to anyone else facing these types of symptoms -- I was not seeing power supply info, fan speeds and the message "RAC0709: Unable to retrieve the fan information" as I mentioned in original post were conditions suggesting that the sled was not properly communicating with the chassis.

Once I powered up this chassis with only a single power supply inserted, I immediately got flashing power button and indicator lights that I hadn't seen on the previous chassis.

Among the theories I have been mulling over that may have been the issue with the original chassis I had....
  1. missing cable connection somewhere (this seems most straightforward)
  2. bad fan controller/power distribution board (I think they are integrated together?)
  3. Other chassis was configured for C6320P and I was using C6320 sleds. In my research I found some information that said this could cause fan problems (perhaps in early firmware). Also note: RAC0709 error is listed in some Dell docs as "Unknown Server Inserted Into Chassis" instead of "Unable to retrieve fan information" as displayed under fan information when my sleds were in the other chassis.
  4. Corrupted firmware
  5. Incompatible firmware (I've heard that if some firmware gets too far ahead of other firmware sometimes it is necessary to regress firmware versions in some devices in order to update the lagging one)
  6. Fan settings hard coded or corrupted by a previous user (although that doesn't explain why probing the board via IPMI failed)

Question for you sir, I know its an amazingly old thread but did you ever get fan control working on this guy?

I too have a C6300 I am enamored with, I've gone through some lengths to quite it down including making the custom fan connectors and swapping with some coolmaster 6k rpm ones, but I found they were quite frankly not cooling adequately enough and I ended up throwing the original ones back in - would love to be able to use raw commands to get this resolved *if* it works.


thanks in advance!
 

aarcane

Member
Feb 16, 2016
35
1
8
39
Be me. Get one of these for your birthday with no CPUs or RAM, and want to kit it out gradually over time. Decide to put one CPU and one DIMM per system, just to avoid having to buy more right now.
Have a panic attack when you boot up something that was working before to test it, but no network card is found

Onboard NIC just VANISHED! I have no idea why. I spend 30 minutes in BIOS and diagnostics and even try the other nodes, all of them show no NIC and a warning that the NIC has been removed. The ONBOARD nic. The literally soldered to the motherboard Intel 82599ES 10GBE ports. They're just missing, so here's an update to throw to the thread:

The Onboard NIC is attached to the 2nd CPU in Socket 2
If you ever boot one of these up with only one CPU and the onboard NIC has vanished, don't panic and start changing all your BIOS and iDRAC settings trying to get it back. Despite having no Motherboard Block Diagram available online, I can fairly conclusively say that the onboard NIC is hardwired to CPU 2. If you have no CPU 2, it will not appear in your OS or be accessible to any applications, including the iDRAC. Restoring a proper second CPU to these things will resolve the missing NICs.
 

aarcane

Member
Feb 16, 2016
35
1
8
39
I'm in the process of upgrading my system to support Mellanox QSFP NICs, and I can't get the right Mezzanine Bracket. I got one kit which included the MW5RY riser, and that also exhibits the temporary short behaviour you described. Removing the blade and the card and the system seems to be working fine again. ALL 4 NODES shut off and 3 came back. The Kit I ordered has a KIT part number of VG4T2 and a Bracket part number of DP32Y. The Bracket does NOT align properly with the QSFP ports on the Y5WKX Mellanox card. I'm going to return this bracket and riser kit, as none of it is usable.
Do you know which Bracket or Bracket & Riser kit to select to be able to connect the ConnectX-3 Card?