Refreshing servers, consolidating with new high end boxes, less power consumption

mephisto

Member
Nov 6, 2013
36
3
8
London
Hi guys,

I'm planning now to replace all my Dell R710 boxes which are 5-6 years old with dual socket quad core cpus, lagging a lot these days and consuming a lot of power. I'm considering supermicro kit, and actually I've noticed that are a few more manufacturers around but I'm still not sure about their reliability, so perhaps my options for something that is tested and widespread would be supermicro. I used some HP kit along the way, but to be honest I got a bit annoyed how they deal with support and how everything is propriety and closed from all angles.

My idea is to get some 4 or 5 R630 boxes with 2x 18core (E5-2695) cpus which seem to be a good balance between price and performance, this will replace about 11x R710s that are consuming a hell lot of electricity and not offering that much CPU performance. I considered the R830 to consolidade more with 4 sockets, but the prices are 6 times what a R630 costs, not worth it.

I'm considering some Supermicro kit as well, but still undecided. I would welcome suggestions, I'm open minded to anything these days.

I'm using 10gbit network and a couple of boxes running ZFS on debian which so far has worked very well, perhaps later on I'll migrate to infiniband 56gbit using RDMA as I expand and get the storage boxes filled with NVME SSDs.

What are your thoughts? I'm using VMware at the moment, but also I'm planning to start migrating things to another hypervisor, perhaps KVM as costs with VMware are going to far up, and Hyper-V even though has more advantages now, may become what VMware is now in a few years, so perhaps time to move to something more open.
 

MiniKnight

Well-Known Member
Mar 30, 2012
2,999
909
113
NYC
Sounds like you're buying for work. Lenovo is getting more proprietary so I'm kinda meh on that. Dell and HPE are proprietary but have some value add for it. SM is good since it is like the anti proprietary. SM has support options but you're prob going to get through a reseller.

Now... if you want to do this on the cheap....

I'd add Intel to your list. If you just wanted super cheap there's some options like this Intel R2208GZ4GC Server w/ DA0S6GMB8C1 Rev:C Motherboard no Xeon E5-2600 | eBay

Add a few hundred for E5-2670's and you're at sub $600 per node. Add external disk shelves as you want they're cheap. Those servers have many expansion slots since they're 2U and have full DIMM slots. With 16 cores each you'd get down to 5 nodes easy.

Supermicro I like these (saw this one cheap) NEW Supermicro SuperServer SYS-1028R-WTRT Dual LGA2011 W/ X10DRW-IT 1U Server | eBay

It'd be going away from your original plan but you could consolidate to quieter servers get newer tech and come close to even money on the deal.
 
  • Like
Reactions: mephisto

mephisto

Member
Nov 6, 2013
36
3
8
London
I'm trying to stay as far away as possible from any propriety solution. I had a nightmare with Dell storage boxes and the floods in thailand several years ago, remember? We could not get Dell to supply us disks, and the damn storage box only accepted disks with Dell firmware so we could not get the disks anywhere else. Never again! On another node, the normal Dell server are okay, not excellent, but comparing to HP I can say one thing, HP stinks pretty bad!

I'm going for a SM reseller in the UK, I've checked a couple of them so far, still specing servers but so far prices are very much in line with Dell, if not more expensive. I was expecting to be the opposite. Is SM hardware any cheaper or it is just a myth?

I want to go for latest CPUs, getting the xeon V4s as we are limited on electricity and heat produced by the servers. These servers need to be running for the next 5 years. Getting second hand hardware is out of question, too much at stake. I need warranty and agreed SLAs, with Dell I can get 4 hour support on site, that has saved my arse once.

I've been purchasing second hand Dell R710 servers from ebay for a while, but now we need to refresh completely and get the best and the latest.

Is there anything else than SM out there that is actually good these days?

Thanks!
 

cheezehead

Active Member
Sep 23, 2012
718
174
43
WI
You could always buy 2nd hand SM chassis and then buy new mobo/cpu/ram to pop in them for cheaper, but you'll need to do some research on what exact configs you want to use. Another thing to consider is the Brexit effect on exchange rates, SM gear isn't made in the UK so it's all going through the exchange hell and might hurt a bit more there than usual.
 

mephisto

Member
Nov 6, 2013
36
3
8
London
yep, that is a valid idea but considering the risks involved as I have all production stuff in there, I would still go the route of all new kit this time.

Dell will increase around 5% their prices this Friday here in the UK due to the currency exchange :/
 

Blinky 42

Active Member
Aug 6, 2015
568
203
43
44
PA, USA
If you are doing production stuff and want to pay for support, I would go with Supermicro from a system integrator that can offer you a service plan / warranty on the whole setup (anything from return for repair to 4hr on-site I have seen).

Pricing should be on par or better than Dell/HP/Lenovo, but I would not worry about the prices their online configurators will give you and go directly to the vendor to get a good quote on the whole solution.

We have used Dell/HP/IBM/Lenovo servers over the years, and migrated to SM across the board as it is cheaper in the long run and you get more value for your $ over the lifetime of the hardware. Ask your VAR who they use to do the support work - it is often the same companies that Lenovo, Dell or HPE would send out depending on your location.

We have deployed hundreds of SM and IBM/Lenovo servers throughout the USA and internationally and they have been on par in terms of build quality and longevity when deployed in a mix of environments.
 

mephisto

Member
Nov 6, 2013
36
3
8
London
What I like most about supermicro is that is the customisation, it feels like you are building your on PC, you know every single component, unlike Dell rebranding LSI controller with their firmware, and then always giving warning if a disk has no Dell firmware. I'm considering supermicro more seriously now as I've been reading a bit more, and I loved the HTML5 IPMI, nice to not need Java. I'm going to ask who are the companies behind the support onsite, that is a very relevant question, thanks for the tip!

I'm quite interested in a few models from Supermicro that have NVME. I'm using at the moment zetavault which is ubuntu with ZoL on top, and so far on the 2 boxes I'm testing using Dell R510 with 128GB of RAM and 12x 600GB 15k SAS disks and 1TB Samsung 850 PRo for L2ARC seems to be working very well, I've replaced the H700 RAID controller for H200 flashed in IT mode, so far I found one 600GB disks having CRC error and I replaced it just fine with several VMs running from the storage box.

NVME seems to be wired with no way of using a RAID controller, right? So the only way to have redundancy with NVME is to use ZFS or some other filesystem which has disk management and software raid.

I've been looking at these 2 servers, they seems very in line with my thoughts for storage nodes:

SYS-2028U-TN24R4T+ - 2U 24x 2.5 NVME, XEON V4, 24 DIMM
SSG-2028R-NR48N - 2U, 48x 2.5 NVME, Xeon E5 V4, 24 DIMM

I may get one of those 2U 24x 3.5 inches as well for long term storage and backups, not sure yet.

I'm not sure if I'll run then over 10GB ethernet as I have already the infra-structure in place, or go for infiniband with 56GB and RDMA as it seems to lower considerably the latency.

What do you think? I'm seeing NVME ssd going down in price, and comparing performance with SATA has a massive diference
 

J--

Active Member
Aug 13, 2016
179
40
28
37
I think you're about a year too early on NVMe, from a cost perspective. As the volume increases, it should achieve price parity with SATA rather quickly.

Someone posted a little bit about it earlier, but the general feeling is that since there are few players in the NVMe space, by driving costs down heavily and quickly, they're essentially barring entry of the smaller players in the business and holding onto market share. From a bill of materials perspective, the silicon is no more expensive.
 

mstone

Active Member
Mar 11, 2015
505
117
43
43
In some cases nvme is getting cheaper than SATA already. Component count on an aic is lower than a 2.5in SATA.
 

mephisto

Member
Nov 6, 2013
36
3
8
London
I think you're about a year too early on NVMe, from a cost perspective. As the volume increases, it should achieve price parity with SATA rather quickly.

Someone posted a little bit about it earlier, but the general feeling is that since there are few players in the NVMe space, by driving costs down heavily and quickly, they're essentially barring entry of the smaller players in the business and holding onto market share. From a bill of materials perspective, the silicon is no more expensive.
I believe you have not see the latest models?

Prices here in the UK:

  • SATA S3520 1 DWPD 480GB £214 - 800GB £385 - 1.2TB £500 1.6TB £675
  • NVMe P3520 3 DWPD 450GB £253 1.2TB £530 2TB £850
Considering the performance difference, I think it is a no brainer to go for NVMe if we don't consider you need a new server for it. As I'm going to refresh all my storage, I think it makes more sense to go for infiniband, use RDMA and NVMe with ZoL.

What do you think in this case?
 

mephisto

Member
Nov 6, 2013
36
3
8
London
by the way, I'm checking specs and endurance, the P3520 with 3 DWPD seems a very good candidate. the P3600 has also a 3 DWPD and costs nearly twice. I think the P3520 is a very good option for my workload.

I'm going to check Samsung prices now on the SATA models to compare and NVMe if they have any model already out.
 

mephisto

Member
Nov 6, 2013
36
3
8
London
Suggested reading NVMe Falling Below SATA Pricing? Get ready it is happening

And Going fast (inexpensively) 48TB of "near SATA" pricing NVMe SSDs

The parity point is already here in many cases. @mephisto that second link is a 24 bay.
Very good article Patrick, very well written! Indeed, SATA seems to be on the way out.

So, is the only way to have RAID on a system with NVMe drives would be with SDS like vSAN, ZFS, EMC scaleIO?

I wonder if there will be any equivalent to a hardware RAID controller for PCI-E in the future.

I wonder how the hotswap works at the moment with this supermivro server, does it work well?
 

mephisto

Member
Nov 6, 2013
36
3
8
London
by the way I can't seem to find any official Intel documentation about the endurance of the P3520, some places are saying 0.3 some others 1, is there any official place you found this information?
 

i386

Well-Known Member
Mar 18, 2016
2,109
556
113
31
Germany

Patrick

Administrator
Staff member
Dec 21, 2010
12,003
4,988
113
Adding a card will add latency and can also limit bandwidth. HighPoint needs a big OEM win to make the cards profitably.