1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HP SL6500 4U 8-Node (16 CPU) Cloud Servers | 256 Threads | 2,048 GB RAM ~$2000

Discussion in 'Processors and Motherboards' started by Zankza, Jun 30, 2016.

  1. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    Inspired by this topic ( Dell C6100 XS23-TY3 2U 4-Node (8 CPU) Cloud Server )

    [​IMG]

    HP SL6500 seem to be the next way to getting cheap LGA2011 platform high density servers. With it's recently spotted bare SL230s. I know the SL6500 can house many other nodes but today we'll be focusing on SL230s.

    S6500 (QuickSpecs PDF)
    • 4U Chassis (Highly Efficient)
    • 8x Half-width or 4x full-width node
    • 8x Fan (Optionally hot-plug)
    • 4x PSU (Optionally hot-plug) (750W,1200W,1500W)
    • No Shared/integrated networking/management

    SL230 (QuickSpecs PDF)
    [​IMG]
    • 2x Intel Xeon 2670 (8C/16T 2.6Ghz) 115W @ $55 Per Chip
    • 128GB (8x8GB DDR3) @ $8 Per Module
    • 2x Hot Swap 2.5 (#4 By using FlexibleLOM-Compartment) *
    • 4x Cold Swap 2.5 (#1 By on-board Compartment)
    • 2x Gigabit ( HP Ethernet 1Gb 2-port 361i Adapter )
    • 1x iLOM (Integrated IPMI)
    • 1x FlexibleLOM Capabilities which provides access to additional NIC or options (Proprietary design)
    • 1x HHHL PCI-e Slot which provides basic access to vast hardware expansions (Half-Height/Half-Length)
    [​IMG]

    With rough estimations it is very much possible to secure a all-in-one chassis consisting of 128/256 Core/Threads With 1,024GB RAM for only $1,392 and doubled RAM capacity 2TB instead of 1,024GB for $1,904. This is just rough number based on my experience just to give you a idea.

    You may also install 10G or Infiniband adapter into the FlexLOM for high-speed connectivity. You may also additionally install an accelerator (PCIe SSD) such as Intel 750 into the PCIe for extra storage. Combining Infiniband and Accelerator can create serious performance hyper-converged setup all within single chassis. With PCIe SSD as cache layer and SATA SSD as capacity layer.

    [​IMG] [​IMG]

    The goal is to ultimately summarize the specs and obtain/release all critical files relating to support/drivers/firmware/etc. All contributions are welcome and appreciated.

    *(You lose PCIe access if you decide to install hot swap 2.5's)
     
    #1
    Last edited: Jun 30, 2016
  2. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    Reserved.

    Parts / Spare List
     
    #2
    Last edited: Jun 30, 2016
  3. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    Quick Crash:

    (This guide assumes you bought yours from the bectechglobal otherwise use those information at your own discretion)

    Those chassis apparently were bought and once-used by "hightail.com". They seem to be manufactured somewhere during 2012. They appear to be loaded with advanced ILO (which is loaded IPMI) and isn't usually free or cheap. The nodes are most likely to contain the following data:

    • iLO FWs are 1.22 and 1.05
    • System ROM at P75 04/04/2012
    • Static IP @ 172.24.0.0/24 subnet
    To reset and/or change ILO settings:

    1. set /map1/enetport1/lanendpt1/ipendpt1 IPv4Address=<your new ilo static ip> SubnetMask=<subnet mask of your network>
      Code:
      set /map1/enetport1/lanendpt1/ipendpt1 IPv4Address=10.0.99.1 SubnetMask=255.0.0.0
    2. set /map1/gateway1 AccessInfo=<your gateway ip>
      Code:
      set /map1/gateway1 AccessInfo=10.0.0.1
    Please note that after executing each of the command, The system will freeze as it attempts to reboot, Expect the fans to be running full during the duration of reboot.

    Should the ethernet routine doesn't go well for any reason, You may use SUV (Serial+USB+Video) to connect and start a terminal, And press "ESC+Shift+9" to get ILo login prompt and from there you may control (and execute same command set).

    Thanks to the following members for sharing their contributions: @raileon
     
    #3
    Last edited: Jul 6, 2016
  4. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    Gallery:

    When they shipped, it appears they just only had to use some very simple basic of methods that did not require cardboard at all expect for very super-durable L-Shaped at every corner, and wrapped in plastic shrink.

    Granted it frightens me at first seeing how UPS was carrying it into the house using the dolly, almost as if it looks like it's bound to take some damages but I was amazed it did not sustain any damages or let alone single scratch.

    [​IMG]

    And So I began to unload one of node and start inspecting the nodes.
    [​IMG]

    Removed all fans and they look brand new.. not single dust.
    [​IMG]
    [​IMG]

    The chassis has really good condition, Everything looks like it's been in some regular usage.
    [​IMG]

    I went ahead to identify the all PCIe accesses
    [​IMG]
    This one I couldn't fit into some regular PCIe adapter, as I was already expecting due to the fact it's called "FlexLOM" not PCIe Compartment. It's PCIe with different physical measurements, you may be able to force fit the PCIe into the seating but the layout wont work with the chassis, it's too far from the end of side.

    [​IMG]

    And this one is the true PCIe slot, Fully compatible with HH/HL (Half-Height Half-Length) however it seems to have quite a lot of bracket supporting. Additionally there is another seating right next to the PCIe that is also PCIe but in different format, it's 3.0 8x and you cannot use both of them at once.

    Here's a picture of both bracket removed revealing the PCIe and FlexLOM.
    [​IMG]
    As you can see there is really three PCIe bus, but you can only use up to two at once and two of them are proprietary as well meaning they are meant to be only used with HP designed-manufactured parts.

    And here's some adapter being installed and mounted on the PCIe interface.
    [​IMG]

    Updated 7/3/2016

    Next We have some proprietary PCIe hardware that is designed and manufactured by HP also known as Flexible-LOM or (Flex-LOM)
    [​IMG]

    This particular hardware is known as HP FLOM 10/40GB Dual Port QSFP 544FLR 656090-001 It's an essentially VPI, Which means 40G ethernet or 56G infiniband protocol.

    Here's a picture comparing the PCIe riser with some adapter and the HP FLOM
    [​IMG]

    And installed in it's FlexLOM slot:

    [​IMG]

    And both of the optional bays installed, The FlexLOM with the PCIe.
    [​IMG]
    [​IMG]
    [​IMG]

    Moving to HDD/SSD compartment, First of all coming from C6100, HP has truly luxury of space and the concept of being truly node-independent has made me realize why HP's design was better for most of scenarios. Without further ado, Here's close up pictures of the area and the air cooling shrouds.
    [​IMG]

    And here's some basic idea how truly big that cage is:

    [​IMG]
    [​IMG]
    [​IMG]
    As you can see I already can fit three SSD, and there is even room for fourth, fifth?! I hope those pictures show you why I think 16 SSD is very much possible reality, 20 would be pushing it.

    And That is all I have to show for now, I'm out of money to purchase 1 or 2 2670s', Once I obtain the chips I'll perform various experiments.
     
    #4
    Last edited: Jul 3, 2016
    linuxmanbg, raileon and J Hart like this.
  5. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    Very nice. It will be interesting to see how the builds evolve.

    My servers don't get here till Tuesday so all I have are 30 lbs worth of heatsinks to play with =)
     
    #5
  6. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    1,608
    Likes Received:
    225
    Very very nice indeed.
     
    #6
    Zankza likes this.
  7. dinskuns

    dinskuns New Member

    Joined:
    Nov 16, 2013
    Messages:
    2
    Likes Received:
    0
    Ddr3 at 8$ per 8gb - does any brand fit or just hp proprietary?
    Can you point at ebay listings?
    Thanks alot
    Also, which 10gb card have you put into the pci-e ?
     
    #7
  8. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    Thanks. Heatsinks wont be 30 pounds, They are 15 pounds at most, And probably less. They used Copper as well.
     
    #8
  9. themelon

    themelon Member

    Joined:
    Jan 26, 2016
    Messages:
    37
    Likes Received:
    6
    I would also like to know where $8/8GB dimm's can be acquired.
     
    #9
  10. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    I have just updated the gallery to formally introduce the FlexLOM compartment and exemplary of hardware installed. You can find the pictures at bottom of the gallery post. (which is the fourth post of this topic)

    1. There is nothing like "HP proprietary" Only "HP supported". Generally almost everything "DDR3" will work, Just that if you run into problems, Tough luck.
    2. I can't point ebay listing because they are so volatile. You need to be extremely aggressive with best offer too.
    3. It's some brocade network adapter. I forget the model name sorry.
    Please see the answer at above.
     
    #10
  11. themelon

    themelon Member

    Joined:
    Jan 26, 2016
    Messages:
    37
    Likes Received:
    6
    #11
  12. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    Yes, It is Brocade 1020. I only installed to ensure the PCIe is up to standard, and there wouldn't be any problem with physical dimension as in future it would be a PCIe NVMe SSD installed. Also as mentioned I ran out of money to purchase 1 or 2 2670s, So sadly I won't be able to test or benchmark for other month. But I hope meanwhile we can start gathering and contributing files such as BIOS/ILO firmware.
     
    #12
  13. themelon

    themelon Member

    Joined:
    Jan 26, 2016
    Messages:
    37
    Likes Received:
    6
    I have 2 E5-2670's and a pile of 4GB modules so I can test the sleds one at a time. Well, 2 at a time if I only test one socket but that would not be very through, would it? That is as long as the serial port is redirecting properly. I'm waiting on the USB/VGA breakout cable.
     
    #13
  14. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    I am more rather interested in how the chassis cooperates, IE treat the scenarios as real world usage, what happens if you take out one node unexpectedly? What happens if you have dead storage device, How would you manage to unplug all the cables so you can safely sled out a node, etc.

    And the adapters too of course, Gotta see how the 40G/NVMe SSD performs.

    IE I already found one caveat, the SAS onboard should require 90d but it appears the previous owner had some RAID card, so if you try to plug in the SAS cable, It's too tall and you would need to really bend it so hard to get 180 degree to able keep it inside within node.
    [​IMG]
     
    #14
    Last edited: Jul 3, 2016
  15. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    4,633
    Likes Received:
    821
    Lots of great info in here!! Thanks @Zankza
     
    #15
  16. gbeirn

    gbeirn Member

    Joined:
    Jun 23, 2016
    Messages:
    64
    Likes Received:
    12
    I've found in eBay that often times the best price for RAM comes from searching for uncommon terms. For example I recently purchased 6x8Gb DIMMS for $55 so slightly above $9/DIMM by searching for 16Gb kits composed of two DIMMS. Bought 3 sets.

    Sometimes searching for 2Rx4 or 4Rx4 or just PC3L registered comes up with good results. Get creative. Find a particular modual you like? Search by that part number. Find a seller that has 100s available for sale and low ball them for a small quantity like 6-8 and they might do it.
     
    #16
  17. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    My heatsinks came in on Friday. 32 in total (2x s6500) very well packaged, individually sealed antistatic bags for each and double boxed. Pretty sure the ups tag said 30lbs.

    I've been trying to pick up all the E5-2670s I need but the best offer I've gotten so far is $57 each. I'm kind of surprised. I figured I'd get below $55 since I'm buying almost 3 dozen.

    Does anyone know a better source for these?
     
    #17
    Last edited: Jul 3, 2016
  18. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    They too arrived, They were straight from HP (double boxed lmao). And it does state on the label that it's 15 pounds or under.

    Honestly, 3x dozen is nice but you need to purchase at 500's to get bulk pricing. 55$ is awfully cheap for the performance they load with.
     
    #18
    Last edited: Jul 3, 2016
  19. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    And BTW It's my goal to get some idea about those hardware, IE what manf date, what version bios/fw, and the ILO as well. If anyone has ram/cpu and PSU already, I would think that a lot of people would be happy to see if you shared your details. As it does appear the ILO tags are on them, I wasn't sure if that was normal or not, but sadly it doesnt mention version.
     
    #19
  20. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    That's too bad. I'll shop around a bit and if nothing else pops up I'll pull the trigger @57 on Tuesday morning.
     
    #20

Share This Page