1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HP SL6500 4U 8-Node (16 CPU) Cloud Servers | 256 Threads | 2,048 GB RAM ~$2000

Discussion in 'Processors and Motherboards' started by Zankza, Jun 30, 2016.

  1. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    I've begun the "Quick Crash" post consisting of information to get you quickly online with those nodes. They are located at original post. Big thanks to those who have already shared information regarding this product.
     
    #41
  2. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    For this part I actually used a regular RJ45 -> DB9 serial cable. I don't have the SUV cable yet but I'm sure that works as well.

    I ordered that cable yesterday when I went over to the server with a USB stick and realized... there is no external USB port lol
     
    #42
  3. Fredrik

    Fredrik New Member

    Joined:
    Mar 28, 2016
    Messages:
    5
    Likes Received:
    1
    Fortunately, a workaround is available. If you have a USB extension/hub, there's an internal USB port which can be temporarily routed out for OS install.
     
    #43
    raileon likes this.
  4. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    I played around with them some more today.

    The integrated LSI 2308s are at the highest version available from HP which is 15.10.09 so SPP must've updated them. Avago/LSI have firmware for their 2308s at P20. No clue if you can even compare the version numbers directly and since these are so integrated into other parts of iLO and HP stuff trying to cross-flash might be a terrible idea. I'm going to leave them alone unless I run into any compatibility problems.

    SPP definitely didn't update the chassis firmware which was 5.2 0r 5.3 (I forget which). So I updated that to the current 6.2 from within linux. Not sure what I gained here, sure as hell didn't help with sound, but it didn't break anything either.

    The FlexLOM 544FLR QSFP cards are working well. I had to install the Mellanox OFED driver package first from the Mellanox site. This package will usually update your firmware too if you're using a regular card. In this case you have to grab the HP firmware and then burn it with mstflint. After a cold reset, my switch could connect to the card at full 56G on both IB and ETH so these seem to be identical to regular ConnectX-3 VPI cards.

    I need to pick-up a bunch of 3.5" drive screws for all the PCIe/FlexLOM cards. This was posted in the deals forum: Supermicro HDD Screws (4.98$ for 100 screws w/ shipping)

    PC Connection has both the 3.5 and 2.5 drive screws for $1.91 per bag but the shipping is terrible so it only works better than ebay if you order a lot.

    Racks come in tomorrow. Hopefully I'll only need a screwdriver to get them mounted and not a dremel/hacksaw/hammer.
     
    #44
  5. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    Would you mind help me understand where are you getting SPP? And if it's possible you can share them too?
     
    #45
  6. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    PM.
     
    #46
  7. Fredrik

    Fredrik New Member

    Joined:
    Mar 28, 2016
    Messages:
    5
    Likes Received:
    1
    JFYI, cross flash to P20 DOES work - use 9205-8e FW, as these cards do not support PCIe 3.0. However, be warned that the fan speed will increase to 70%. This happens with the latest HP P15 firmware as well, when using non-HP drives. I couldn't find a way around this, so I reverted back to the P11 firmware shipped with the system. Fan speeds back to normal. Oh well...
     
    #47
  8. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    Hmm interesting. Thanks for the info. What is "normal" fan speed on your chassis?

    I don't remember the fans running lower before I did the updates though. I was also running with no drives installed in the beginning and the sound was about the same. Going to give P11 a shot and see if it's less horrible.
     
    #48
  9. Fredrik

    Fredrik New Member

    Joined:
    Mar 28, 2016
    Messages:
    5
    Likes Received:
    1
    I'll need to check again when I get home, but IIRC, it was somewhere between 10-30%? It is now a jet-engine at idle, as opposed to a jet-engine at full blast. Try popping in a non-HP drive in one of the nodes to see if that really triggers it. I was doing my LSI firmware comparison with drives in through the whole process, so I cannot accurately comment on how it would behave with no drives.
     
    #49
  10. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    I took some pics of these mounted in the rack before I start filling it up with other equipment. The rack is a Dell 4220 Standard which is 1070mm in length. That little thing on top of the two s6500s is a Dell 1U to give you an idea of scale, especially in the back. I'm using the 3rd Party Rail Kit for these minus the front brackets.

    In the front shots you can see two QSFP cables plugged into the FlexLOM slot of the right node. Those cables should be the biggest thing coming out of the front that adds any length to the unit.

    [​IMG]

    There are about 3 inches of space between the door and the front of the nodes. The blue Mellanox pull tab just touches the grill.

    [​IMG]
    [​IMG]

    The dell is tiny in comparison. I'm probably going to end up putting these at the very top or bottom of the case so I don't have to jump over them with cables for other boxes.

    [​IMG]

    There are about 2 inches between the back door and the fan tabs. I could've pushed these further back into the rack if I had needed the space up front but I'm not seeing the need.

    [​IMG]

    Getting these in there wasn't anything special really. They're the size of a boogie board but they fit a normal rack just fine.
     
    #50
    T_Minus, cookiesowns and Zankza like this.
  11. cookiesowns

    cookiesowns Member

    Joined:
    Feb 12, 2016
    Messages:
    85
    Likes Received:
    21
    dang it raileon, really made me butthurt for missing the deal. Would fit perfect in a 1200mm deep rack.
     
    #51
  12. markarr

    markarr Active Member

    Joined:
    Oct 31, 2013
    Messages:
    290
    Likes Received:
    76
    Has anyone tried the sata port on the board?
     
    #52
  13. GuybrushThreepwood

    Joined:
    Aug 2, 2015
    Messages:
    51
    Likes Received:
    19
    raileon,

    You mean it could fit a 1000mm depth rack and still able to close the doors?! Sigh, I guess I have to rip apart my wallet now.
     
    #53
  14. cellcreator

    cellcreator New Member

    Joined:
    May 16, 2016
    Messages:
    3
    Likes Received:
    0
    Hello,

    I also ordered one unit of this machine and it arrived today. Unfortunately, the packaging materials were completely damaged, and some stuff might be missing.

    Could anyone provide a list of items delivered with the chassis so I can contact with the seller about missing items? Besides, the top right node is a little bit difficult to remove and install. I'm worried if any of the nodes may fail running. Could anyone give me some advice on dealing with such situation?

    Here are some photos of this terrible shipping:
    Dropbox - HP_Proliant

    Thanks!
     
    #54
  15. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    That firs pic is how it showed up at your door? I think everyone got the same crappy packaging but the sticky pinkish shrink wrap stuff was on tight.

    As far as parts, you seem to have everything. All nodes, the rail kit is the 2 front parts, the skirt for the back and the bag of screws. I'm assuming all your fans are there, you even got a custom wood handle on one of them. Did you get your heat sinks in a separate delivery?

    A couple of my nodes are a little hard to put in and slide out, but I doubt it was from shipping since mine were locked on tight to a pallet. The chassis itself just isn't completely rigid.

    You should definitely contact the seller with pics of how it arrived in case they want to file a claim with the freight company. It also gives them plenty of notice that some of your nodes might not work and why that's the case.
     
    #55
    cellcreator likes this.
  16. Layla

    Layla New Member

    Joined:
    Jun 21, 2016
    Messages:
    12
    Likes Received:
    1
    Anybody figure out a better way yet?

    Are the 2 SATA ports Intel 6G ports from the chipset? I'm tempted to just use those for the moment.

    I'm really shocked that there don't seem to be any 90 degree SAS couplers, or any HP brand cables with a 90 degree SAS connector..
     
    #56
  17. cellcreator

    cellcreator New Member

    Joined:
    May 16, 2016
    Messages:
    3
    Likes Received:
    0
    @raileon Thanks for the reply. Yes, the first picture shows exactly how it arrived at my door...

    I just called the seller today. I was told the heatsinks were missing and will be sent right away, arriving next day. The seller asked me to test the server but he said it should be functionally fine. If there are some not working, some amont of refund can be issued. He also told me so far for this batch of machines, only one user reported that one fan was not working; and in another case, one of the nodes was difficult to remove and install. My top right node is a little bit difficult to remove but seems not a big deal. The seller was very nice and patient to answer my questions.

    I'll need to test the server in the next a few weeks.

     
    #57
  18. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    As time passed, The SAS isn't tricky anymore, Bending has stayed in shape and I no longer worry or have issues taking out/in the nodes anymore.

    You'll just have to cringe as you bend them real hard for first time but afterwards it's all good.
     
    #58
  19. Zankza

    Zankza Active Member

    Joined:
    Feb 1, 2014
    Messages:
    66
    Likes Received:
    13
    To everyone who bought this chassis/server:

    How has the servers been working out for you?
     
    #59
  20. raileon

    raileon Active Member

    Joined:
    Jun 22, 2016
    Messages:
    134
    Likes Received:
    33
    The servers have been working well so far. I still need to put a lot of work into them to get them running as distributed storage and/or distributed DB. I'm not sure what the final form will be as far as the software, I plan to experiment a lot there.

    I've had the hardware setup for a while now. All blades are running dual cpus and are fully loaded with dimms. They all have those great mlnx network adapters in the flexlom slot that a few of us picked up around the time of the original deal.

    I did end up with one bad board where adding a second cpu caused post to fail every time. The seller was great though and refunded me $80 for it. I picked up 2 replacement motherboards for $100 from alan tech and it was no problem switching the bad one out. I just added the previous serial and the ilo advanced license and everything worked perfectly. Right now I'm thinking I might have a second bad board since one of the blades keeps dropping 2 dimms during boot. It seems random and switching the dimms hasn't fixed it so I'm not sure what's going on there.

    I've been amassing an unhealthy amount of ssds for this project. Now I need to work out modifications to the cages where I plan to run 8 ssds so I'll be busting out the dremel there soon. I also have some blades where I want to run sas3 drives hooked up to LSI 9340-8i cards but I've yet to find a break out cable for those that works. I wasn't expected that to be so difficult to find...

    Also on my todo list is a complete reworking of the fans to get the sound under control. I already have a game plan there but I keep pushing that project back in favor of other stuff I'm working on.

    Overall though I'm very happy with the blades. They are really hard to beat from a price vs feature set point of view especially when you throw in those dirt cheap connect-x3 cards into the mix.

    Forgot to mention: Two of my blades have sl250s stickers in the front. Thought that was odd since the sl250s chassis looks like it's one piece. Did someone really flip those stickers around? lol
     
    #60
    Last edited: Sep 16, 2016

Share This Page