Dell C6100 XS23-TY3 2U 4-Node (8 CPU) Cloud Server

[dba]

I'm running out of "good news: there's a simple solution" ideas. If you don't have a firewall blocking that port, then maybe you do have a c6100 problem, though it's odd that it would happen on all four boards.

I did try from different computers. I get the same error. The java console is trying to connect to port 7578. There is nothing listening on that port hence the error. You can open the jnlp file in textpad to see the connection details that is failing.

 

[Smalldog]

No. Dual-rank is not bad. Its probably good. Its also why you maintained DDR1066 speeds.

Looking at the original memory I was using which when fully populated in a node dropped it to 800MHz, it arrears that it was quad ranked.

It was Elpida EBJ42HE8BAFA-AE-E (Dell P/N H959F) which is also sold in these machines when they were only being populated with 24GB per node.

Jeff

 

[MACscr]

Looking at the original memory I was using which when fully populated in a node dropped it to 800MHz, it arrears that it was quad ranked.

It was Elpida EBJ42HE8BAFA-AE-E (Dell P/N H959F) which is also sold in these machines when they were only being populated with 24GB per node.

Jeff

Ah, that makes sense then. Quad rank is definitely slower memory as well. Stinks thats what we have. Id move all the quad rank memory to one cpu and or even node and then move the dual rank to the other cpu/nodes.

 

[avnbs]

I'm running out of "good news: there's a simple solution" ideas. If you don't have a firewall blocking that port, then maybe you do have a c6100 problem, though it's odd that it would happen on all four boards.

I fixed the issue.

I was setting BMC LAN IP to be dynamic via DHCP. Changing this to static seems to work. I now have iKVM working. I played with all settings except IP being static, which did not make lot of sense.
BTW, I did brick one of the server while setting the LAN configuration to static. Thanks to the thread http://forums.servethehome.com/processors-motherboards/1448-dell-c6100-anyone-brick-board-yet.html I was able to quickly restore the BMC from a DOS USB.

Thanks for all the help. I post more tips and updates as I find them.

 

[hakalugi]

Some c6100 I/O testing:

I have a c6100 node with dual L5520s and 96GB RAM. The node has the LSISAS2008 SAS/SATA mezzanine card and is in a 24-bay chassis. I hooked up five old OCZ Vertex3 drives in addition to a non-SSD boot drive and ran some IOMeter tests to look for IO problems if any.

The good news is that the setup performs as expected. Maximum throughput for 1MB random reads over a 20GB data set was 2,601MB/S - right in line with the best that the LSISAS2008 can do in an x8 slot on any platform. Maximum 4kb IOPS was 304,804, likely held back just slightly by the five old drives. CPU utilization was very low - 1.3% in the throughput test and 11.2% in the IOPS test.

If anyone has the LSI1068e version of the card, I'd love to know how it performs compared to the newer 2008 version that I tested - the 1068e cards are far more available than the 2008 cards

my c6100 is 4 nodes @ dual six core 2.80Ghz - X5660, between 24 and 48GB of ram each. I have the base LSI raid controller offered by dell, the LSI1068e. I had 3 spare vertex4 drives, so i made a 3 drive raid0 array (no hacking up my backplane on this one, other than this node, the other 3 nodes are in production).

running 2008r2, 12 cores, 24GB ram, and iometer, using the original test posted here: Benchmarking your Disk I/O | Technodrone i'm seeing much more modest numbers (depends on the test but: 550 - 1050MB/s and about 45K-60k iops), granted this is the OS disk and i'm running windows update applying 92 updates on this node while i run the tests...

what tool did you use for your testing and if iometer, if you can send me the test file, i can do a proper apples to apples.

-hak

 

[dba]

my c6100 is 4 nodes @ dual six core 2.80Ghz - X5660, between 24 and 48GB of ram each. I have the base LSI raid controller offered by dell, the LSI1068e. I had 3 spare vertex4 drives, so i made a 3 drive raid0 array (no hacking up my backplane on this one, other than this node, the other 3 nodes are in production).

running 2008r2, 12 cores, 24GB ram, and iometer, using the original test posted here: Benchmarking your Disk I/O | Technodrone i'm seeing much more modest numbers (depends on the test but: 550 - 1050MB/s and about 45K-60k iops), granted this is the OS disk and i'm running windows update applying 92 updates on this node while i run the tests...

what tool did you use for your testing and if iometer, if you can send me the test file, i can do a proper apples to apples.

-hak

I'm not surprised that you are seeing lower throughput - the LSI1068e is an older chip. It's good to see that it's faster than the motherboard SATA ports, which max out at around 650-700MB/S combined.

For my throughput and max IOPS testing I use IOMeter. I don't have a test file saved - I just enter my standard setup each time. Start with all default values in IOMeter and then:

Test Setup tab. All default values except:

Run time: 1 minute if testing the card, 10 minutes if testing the disks​

Ramp-up time: 1 minute​

Number of workers: 1 except when testing very large systems​

Results Display tab:

All default values is OK, or view every 5-10 seconds.​

Access Specification tab:

Create a new specification with exactly 1MB size, 100% random, 100% reads​

Otherwise all default values​

Network Targets tab:

All default values (aka empty)​

Disk Targets tab:

Select the sever, not the individual worker.​

Maximum Disk Size: 12,000,000 sectors (which is arund 6GB​

# of outstanding OIs: 32 (this is queue depth)​

Pattern: pseudo random​

For IOPS, create a new access specification with 100% reads, 100% random, and 4kb (instead of the 1MB above)

Also, if you are testing the maximum throughput of some disks and/or a card (as opposed to complete formatted array), just leave each disk as JBOD. When you are in IOMeter, use control-click on the Disk Targets tab to select all of the disks at once. IOMeter will queue up transfers to each disk separately, driving the absolute maximum possible throughput.
Lastly: If you can, initialize the drives but do not create filesystems on them. In Windows, the drives should not appear with drive letters and should show up as "unallocated" when viewed using Disk Management. By using raw drives, the OS has no opportunity to mess with your results by doing its own caching. My HP MSA array, for example, will test at 4GB/S against formatted drives (because of OS caching) and 1GB/S with raw drives.

 

[dw5304]

Im still trying to find out if server 2012 does indeed work w/ the c6100's i know someone said something about hv 2012 but did not see anything saying it was working.

 

[dba]

I have four c6100 nodes running Windows server 2012 with Hyper-V. No OS issues at all.

I did have to hand-roll a customized firmware for the Mezzanine Infiniband cards, and I recommend taking the time to bring all of the various firmware bits up to date before installing your OS, but other than that the Windows install and configure was a non-event.

Im still trying to find out if server 2012 does indeed work w/ the c6100's i know someone said something about hv 2012 but did not see anything saying it was working.

 

[dw5304]

thanks for the information dba

 

[seang86s]

I have four c6100 nodes running Windows server 2012 with Hyper-V. No OS issues at all.

I did have to hand-roll a customized firmware for the Mezzanine Infiniband cards, and I recommend taking the time to bring all of the various firmware bits up to date before installing your OS, but other than that the Windows install and configure was a non-event.

Curious. What did you customize in the Infiniband firmware? A solution to the RDMA issue perhaps? I'm just getting ready to start my Hyper-V build on the C6100.

 

[dba]

I could not find a pre-made installer that had the latest version of the firmware, so I followed the following instructions:

Mellanox Technologies: Support

Occasionally the order of things matters. Here is the sequence that I followed (thanks to MS OneNote notes):
1) Update c6100 BIOS
2) Update ESM firmware
3) Update FCB firmware
4) Install and then update Windows 2012
5) Install IB card
6) Install WinMFT
7) Usse WinMFT to update IB card firmware
8) Install IB driver
...

Curious. What did you customize in the Infiniband firmware? A solution to the RDMA issue perhaps? I'm just getting ready to start my Hyper-V build on the C6100.

 

[markpower28]

DBA:

Could you please share the command to use WinMFT to update IB card firmware in Windows?

Thanks a lot!

 

[markpower28]

I have Dell Open Manage 6.5 running for C6100. It can do everything except storage.
https://docs.google.com/file/d/0BynPQojkIX8rNkxhT3RGMkRFUG8/edit?usp=sharing
https://docs.google.com/file/d/0BynPQojkIX8rQl9KZXBUZXlQV2s/edit?usp=sharing
https://docs.google.com/file/d/0BynPQojkIX8rcnpYcldYV2ZNdDA/edit?usp=sharing
https://docs.google.com/file/d/0BynPQojkIX8rN21Pbk1JaHpJTk0/edit?usp=sharing

 

[dba]

Yes - probably tomorrow (Tuesday). I'm testing a new firmware for Windows 2012 that may fix the RDMA issue; I'll share both that info and the commands in one post.

DBA:

Could you please share the command to use WinMFT to update IB card firmware in Windows?

Thanks a lot!

 

[shindo]

Hello, everyone. I'm new to STH but have done quite a bit of reading on the forum since I ordered a C6100 earlier this week. I'm interested in utilizing one of the nodes for storage and connecting two others via Infiniband (JR3P1 cards). I'm certainly no storage expert and have been a little confused by the cable types that would work for me. I believe that I'm supposed to be looking for QSFP QDR cables, but would QSFP+ cables also work in the JR3P1?

Would the following cable work?

NEW Brocade 58-0000033-01 4x10GE QSFP Cable 1M 40Gb Communications Twinax Cable

It is QSFP and says it is 4x10GE...so to me this seems right but I'd sure appreciate a second opinion.

Thanks for all the knowledge I've gained so far with the forum, I look forward to learning a lot more!

 

[dba]

For our purposes, QSFP and QSFP+ are the same thing. The eBay cable that you found will definitely work.

Hello, everyone. I'm new to STH but have done quite a bit of reading on the forum since I ordered a C6100 earlier this week. I'm interested in utilizing one of the nodes for storage and connecting two others via Infiniband (JR3P1 cards). I'm certainly no storage expert and have been a little confused by the cable types that would work for me. I believe that I'm supposed to be looking for QSFP QDR cables, but would QSFP+ cables also work in the JR3P1?

Would the following cable work?

NEW Brocade 58-0000033-01 4x10GE QSFP Cable 1M 40Gb Communications Twinax Cable

It is QSFP and says it is 4x10GE...so to me this seems right but I'd sure appreciate a second opinion.

Thanks for all the knowledge I've gained so far with the forum, I look forward to learning a lot more!

 

[shindo]

For our purposes, QSFP and QSFP+ are the same thing. The eBay cable that you found will definitely work.

Thanks for the quick reply, dba! Has anyone noticed any difference copper vs optical with the JR3P1? If prices are close, would it be better to get a longer optical cable instead of a 1m copper or should I just try to get as short as possible since I don't need the length at this time?

 

[dba]

For short cables, up to several meters, there is a theoretical difference between copper and optical Infiniband cables, but no practical difference. For 1 Meter or 2 Meter cables, buy based on price. For longer cables, buy optical.

Thanks for the quick reply, dba! Has anyone noticed any difference copper vs optical with the JR3P1? If prices are close, would it be better to get a longer optical cable instead of a 1m copper or should I just try to get as short as possible since I don't need the length at this time?

 

[Smalldog]

Thanks for the quick reply, dba! Has anyone noticed any difference copper vs optical with the JR3P1? If prices are close, would it be better to get a longer optical cable instead of a 1m copper or should I just try to get as short as possible since I don't need the length at this time?

I picked up a bunch of copper cable off ebay for a good price, but didn't need the long lengths that I bought (but they were far cheaper than buying 2m copper cables), so for me, the only difference is how much cable is laying all over the place. I have a 5m fiber cable (QSFP to CX4) which takes very little space room because it can be wound up fairly small. I cannot say the same about copper

Unfortunately I do not have a rack yet, so I have some cables running across the floor.

 

[markpower28]

Patrick:

Mine Dell 3.5" to 2.5" Caddy - 9W8C4 just come but it does not work well with C6100 since I can only put the screw on one side. We may want to remove that part number from the list. It does work with R710/R720 etc.