I've been watching this switch but I guess you must got a very good deal. Voltaire Grid Director TM 4036 | eBay
I got the switch on eBay and got a fantastic deal, but the seller I bought from doesn't have it listed anymore.
This was my backup plan if something fell through with the one I got
QLogic Silverstorm Infiniband Edge 9024 FC24 ST1 DDR 24 Port Rack Mount Switch | eBay
The server has the punch outs. Isn't that all I need?Alan,
It looks like the mezzanine cards that you bought don't have the mounting brackets. Any chance that you can return them?
Here is a card that does have the rear mounting bracket that surrounds the ports and extends to attach to the sled:
Yes, I found it for $499. I didn't realize just how good of a deal that was until later or I could have bought both they had and flipped one to pretty much pay for my whole setup.I've been watching this switch but I guess you must got a very good deal. Voltaire Grid Director TM 4036 | eBay
This is my first time with infiniband, the setup I'm replacing has 4 separate servers that communicate over gigabit ethernet. So even DDR would have been a major upgrade. I realize one port of QDR saturates the PCIe bus, I had wondered if hooking up both ports to a 20 gbit infiniband switch would give 40 gbit of bandwidth.QDR Infiniband has twice the bandwidth of DDR, and lower latency, but in practice you get a smaller throughput improvement. QDR PCIe2 x8 Infiniband cards are good for around 3.2GB/s, limited by the PCIe bus, while DDR cards are good for around 1.9GB/s. So DDR is 60% as fast as QDR in the c6100, but costs less than half... which makes the far less expensive DDR equipment a pretty good deal.
None the less, it seems that everyone here with QDR dreams of FDR.
Nope. The server sleds need to have the punch-outs and the cards need the brackets.The server has the punch outs. Isn't that all I need?
If you have a number of threads working simultaneously, then yes - dual DDR can approach or equal QDR speeds. The one thing that I haven't tested is whether Ethernet RDMA works when aggregating links, for example using SMB3.This is my first time with infiniband, the setup I'm replacing has 4 separate servers that communicate over gigabit ethernet. So even DDR would have been a major upgrade. I realize one port of QDR saturates the PCIe bus, I had wondered if hooking up both ports to a 20 gbit infiniband switch would give 40 gbit of bandwidth.
Insanely good price for a 32 port QDR switch - or did you get the 18 port version, which would still be a good deal at $500. Let us know when you get it working; we know that the used Mellanox switches are easy enough to get running but I think that you are the first to try a QLogic. It would be great to have two options.Yes, I found it for $499. I didn't realize just how good of a deal that was until later or I could have bought both they had and flipped one to pretty much pay for my whole setup.
I checked today, they ones I ordered didn't show the mounting brackets in the ebay photo, but did come with them. Everything has arrived now except for the C6100 itself, which is on it's way and should be here monday.Alan,
It looks like the mezzanine cards that you bought don't have the mounting brackets. Any chance that you can return them?
Here is a card that does have the rear mounting bracket that surrounds the ports and extends to attach to the sled:
Just FYI, I put in 4 x 16 GB DIMMS today and the BIOS sees it just fine. I have a L5520 processor. In other words, this server does certainly support 16 GB DIMMs!#1. No, you cannot use 16GB dimms with any L55xx or X55xx CPU. You must be using L56xx or X56xx CPUs in order to use 16GB dimms. With the L5520 you are limited to using 8GB or smaller dimms.
Thanks Joel very good info thereNever mind, figured it out. Well I haven't actually figured it out but I got past it. It was something to do with the drives in node 3 and 4. I swapped the drive from node 3 to node 1 and it too wouldn't see the drive on install. I grabbed another drive, put it in node 3 and it sees it. No idea what's up with these two drives because I literally just ran SMART tests on them and they passed with no problems. I'll run more tests on them later.
My c6100 is a L5520 (2 CPU per node) model with 48GB per node (192 total). I did some power usage tests for my own benefit and thought I'd share them with you all here.
Chassis powered, no nodes online: 22W, 44VA, 0.38A
Node 1 booting: 276W peak. In BIOS: 156W, 161VA, 1.36A
Node 2 booting: 298W peak. In BIOS: 263W, 268VA, 2.30A
Node 3 booting: 407W peak. In BIOS: 373W, 378VA, 3.22A
Node 4 booting: 531W peak. In BIOS: 483W, 487VA, 4.19A
Initial power-on, all nodes booting simultaneously: 680W peak.
All four nodes booted and logged into Windows, idle: 375W, 377VA, 3.20A
Amperage values are on 120V. I do not have any readings with only a single CPU, sorry.
That's great news.I checked today, they ones I ordered didn't show the mounting brackets in the ebay photo, but did come with them. Everything has arrived now except for the C6100 itself, which is on it's way and should be here monday.
Upgrade the BMC firmware first. Also which browser and Java version are you using?I just received my server and was able to rack it up last night. Now when I tried to use the iKVM, the iKVM connection disconnects every minute or so (the fps at the top goes to 0fps every now and then). It seems like the node is rebooting itself because there is no OS.
Now my real question is, can I configure everything headless? Or do I need to bring in a monitor and a keyboard to do the initial setup (upgrade firmwares, set BMC IP etc.). I appreciate somebody who has done this remotely from the start to give some insight to where I can look. Worse case is.. for me to haul a 17" monitor home from work... on the bus...
Thanks!
Can I upgrade BMC headless, remotely? I am using Chrome and Java 7. I can get into the iKVM no problem.. it's just it won't stay connected. Is the Firmware Revision on System Information --> Summary the BMC firmware version? If so.. it is on 01.32.28014Upgrade the BMC firmware first. Also which browser and Java version are you using?
You can. Try using IE and/or Firefox to see if that works. Usually pretty simple step.Can I upgrade BMC headless, remotely? I am using Chrome and Java 7. I can get into the iKVM no problem.. it's just it won't stay connected. Is the Firmware Revision on System Information --> Summary the BMC firmware version? If so.. it is on 01.32.28014