Wanted: expertise on server setup

[mmmann]

I'd like to solicit expert opinions on a server(s) setup. Although a software engineer, I'm finding the hardware aspects to be incredibly complex, much more so than I would have imagined, even after weeks of reading. I'm an old hand at building hardware (desktops), but designing, selecting and configuring a rack of hardware is proving beyond me.

I'm building a server closet to hold (1) a BlueIris security camera system, (2) an internal file server, (3) an internal web site, and (4) later, perhaps, a public-facing, low volume web site if I decide against renting and using remote administration. I have a Cisco SG 200-26P gigabit PoE switch, a packed Leviton wiring box, and an empty rack that needs filling!

Requirements:

6-8 cameras recording to 4-6-8GB (?) of hard drive space.
12 - 20TB of file server space.
Processes: BlueIris, SQL Server, IIS, Emby, uTorrent, QuickBooks, Outlook, et al.
Windows: most of what I run is Windows-based.
All data directories are currently segmented to fit on individual 4-6TB drives and are robocopy'd to backup drives. RAID is not req'd for a huge partition, but I'd like the file server to have 8+ hot-swaps if I decide to upgrade to hardware RAID.

Questions:

Build or buy? Mr. Rackables or Mr. do-it-yourself?
Best practices to separate internal files and cameras from outside access?
Setup of VLAN's on the Cisco?
Dual Ethernet motherboard(s) to bond for high traffic?
Overall best-practices config for web server <-> file server <-> security cams?
Can (should) security cams or file server or SQL Server coexist?
Virtualization on one massive server or use individual servers?
SAS and status LEDs for eventual RAID hardware?
One large UPS (what capacity?) or multiple, smaller UPSes?
BTW, how do you backup a huge RAID…w/another external RAID? $$$.
Is RAID req'd for FASTEST file server speed, or are single, 7200RPM drives equivalent?
Windows 10? Windows Server 2012 R2?
Redundant power supplies? I've never had one fail, before.
So many other questions I don't even know enough to ask.

Experts, please chime in. I'm looking for best practices in separation of concerns for security, configuration of machines and networking, hardware and software requirements, building vs. buying, specific components if building or specific model #'s if buying, future-proofing, etc.

Thanks!

 

[j_h_o]

I'd do 1 big server, running Server 2012 R2 Hyper-V.

• Chassis
  • 1U with direct attached storage with SA120, SE3016, or any other number of options...
  • or 4U SC846 or something, with room for 24 hotswap drives
• RAID
  • 3ware 9750 for hardware RAID
  • or an M1015 or something to connect your drives to the chassis
• VMs
  • I'd run a separate VM for each of your roles, or at least segment by relative risk/role.
• Backup
  • Dump snapshots of your VHDs to individual disks, or ideally, into another chassis entirely.

How loud can the server be? Any noise constraints/limits?

Your needs seem fairly modest, so CPU-wise, you're probably OK with C2750-based system, or maybe an Xeon-D if you want more room to grow.

What hardware are you currently running on?

 

[mmmann]

Currently, I'm running the "file server" on a 6-core LGA 2011 workstation. The impetus for this pedal-to-the-metal upgrade is (a) BlueIris security cam software and (b) finally getting around to implementing all business practices in code. This is a small-becoming-medium-sized property management company, BTW, so we're talking an internal database, property management system, accounting functions, automated messaging, internal functions, external access by lessees, remote access and administration (from a remote site), access from Android, etc. Then, admittedly, there's Emby (MediaBrowser3) since I love watching movies...

Note that BlueIris is VERY CPU intensive. See their web site for more information, but suffice it to say that ten 1080p web cams could bring a Skylake to its knees. It seems to be very poorly threaded; per-core Ghz appears to be important. For that reason, I was thinking that it may need to be its own machine, perhaps writing to local drives since it's very disk-intensive. They recommend 7200 RPM drives or even an SSD or RAID.

But if BlueIris (and et al.) write to a separate file server, that would necessitate RAID for its faster throughput, I imagine? All those layers of software and hardware and Cat5 cables would probably destroy single-drive performance. Perhaps that's the reason to keep it all on one big box? If one big box, then I suppose Xeons with high per-core performance might be necessary so that BlueIris can tap out a core, SQL Server and Emby can run quickly, etc. In other words, the workloads I intend probably need fewer, faster cores rather than a slower, wider architecture. Plus, a single machine would provide local SATA / SAS drives for FAST access.

The server can be loud(ish); I'm building a fully-enclosed (but well-ventilated) server closet that will have sound insulation in the stud cavities and a security door. The closet will be located in a cool basement.

How many machines do I need? Can one, beefy machine do all this AND be secure? Perhaps a fast server for SQL Server, Emby, BlueIris, file serving and everything else, with local drives (and someday, RAID), and then, later, a small web front end, carefully fire-walled?

There are so many permutations, I can't even make a start at best practices.

I'd do 1 big server, running Server 2012 R2 Hyper-V.

• Chassis
  • 1U with direct attached storage with SA120, SE3016, or any other number of options...
  • or 4U SC846 or something, with room for 24 hotswap drives
• RAID
  • 3ware 9750 for hardware RAID
  • or an M1015 or something to connect your drives to the chassis
• VMs
  • I'd run a separate VM for each of your roles, or at least segment by relative risk/role.
• Backup
  • Dump snapshots of your VHDs to individual disks, or ideally, into another chassis entirely.

How loud can the server be? Any noise constraints/limits?

Your needs seem fairly modest, so CPU-wise, you're probably OK with C2750-based system, or maybe an Xeon-D if you want more room to grow.

What hardware are you currently running on?

 

[j_h_o]

Virtualization (like Hyper-V) should be secure enough for what you're doing.

If all 10 webcams write at the same time, what's the throughput you'd need? How many MB/s? How much recording would you be doing (on motion? always?) and how much would you expect to write a day, for the security cameras?

 

[T_Minus]

Do you need support, and a warranty? That should be #1 question.
If yes, then don't DIY.

 

[Naeblis]

Why not get a 8 bay NAS, even as a stop gap solution, then you have the flexibility to find the shortcoming and likes and design your system better

 

[mmmann]

Do you need support, and a warranty? That should be #1 question.
If yes, then don't DIY.

I'm fully capable of handling any software and hardware issues. I just don't have enough IT experience to envision a best-practices design and to select components. I used to tell people that no software engineer should be hired until he's been working for at least 10 years. After weeks of reading I've concluded the same about designing an IT infrastructure: better not solo it until you have a decade of hard-knocks experience under your belt. That's why I'm throwing up my hands and asking for help, here in the forum.

 

[mmmann]

Why not get a 8 bay NAS, even as a stop gap solution, then you have the flexibility to find the shortcoming and likes and design your system better

 

[mmmann]

In answer to the "How many MB/s" question: the information is hard to come by. BlueIris will saturate a 4-core, high end Core i7 with 10 cams. It appears to be okay with one, 7200RPM drive. I'll probably write only on motion detect, but I could change my mind, and if so, throwing $ at an 8TB He6 HGST drive would probably fix the problem...or moving to RAID would do it.

I'm going to work with j_h_o's post re: "I'd do 1 big server, running Server 2012 R2 Hyper-V..."

I did a search for "2x E5- or E6-" servers on eBay. Since I need fast cores, the first that matched 3.?Ghz was this: Dell Poweredge R630 Server 2x E5-2637 V3 QC 3.5GHz 32GB RAM 2x 300GB 10K SAS. Almost certainly overkill but, on the other hand, I've always preferred to "do it once, do it Right(tm)" and leave room for future-proofing. This is a business I'm running, after all.

Can I build something like the above? I've always been able to build $4000 Falcon-NW-type workstations for 1/2 the price; I imagine it's possible in the server realm, too.

What case, motherboard, CPUs, etc. seem to be current, best practices to build out a dual-Xeon server like this?

Thanks

 

[j_h_o]

If you want to build it yourself, then I'd use a Supermicro board.

But you haven't really described your workload enough:

• Since you have BlueIris running already, go generate some recordings. How large is 1 minute of video recording?
• During the recording, open Task Manager and watch CPU usage. Grab a screenshot -- let's see how much CPU is in use, across how many cores.
• I'm assuming your SQL utilization is basically nothing, and that your load will be largely dominated by BlueIris. Is that fair?
• Noise is OK. Do you care about power usage?

 

[JeffroMart]

Note that BlueIris is VERY CPU intensive. See their web site for more information, but suffice it to say that ten 1080p web cams could bring a Skylake to its knees. It seems to be very poorly threaded; per-core Ghz appears to be important. For that reason, I was thinking that it may need to be its own machine, perhaps writing to local drives since it's very disk-intensive. They recommend 7200 RPM drives or even an SSD or RAID.

Just a small note that I found in running BlueIris on our system. As far as CPU usage goes it all depends on the configuration of the output of the streams, and if you put overlays etc on the recorded output as it has to be encoded. We have 8 Hikvision cameras all running 1080p streams using the overlays built into the cameras rather than using the ones that are encoded from BlueIris itself. We also output it to direct-to-disc with no transcoding and no overlay under the video output options.

This obviously will create a lot larger output files, but will have better overall video quality, and lower CPU usage.

In this configuration it runs 15-20% cpu usage inside a VM on 8 cores of a Dual E5-2430 system.

 

[mmmann]

Hi j_h_o and everyone.

I want to express my thanks for allowing such a general (read: noob) server question. I'm hoovering up IT knowledge daily (for weeks!) but it's an insanely steep learning curve. Thanks JeffroMart for the input, as well.

I think I've come to one conclusion: I need to break up my system into three component parts rather than one, huge server. The reason is that the workloads are very different and would be happier on separate hardware. BlueIris requires an extremely fast CPU but merely a single HD. The file server (the next component, time-wise) will server files, Emby, SQL Server, etc., and will require a very broad architecture and very likely, RAID, at some point. I believe it may run VM's. Then finally, a web server that can encapsulate its software very strictly with a connection to SQL Server. Advantages: fewer single points of failure; I can get BlueIris going 1st before tackling the file server; aggregated (doubled-up) ethernet from the switch to both B.I. and the file server; VLANs enforced in the switch to firewall the internal data; etc.

Build #1: the BlueIris server. It needs to be a VERY beefy Xeon system (for the ECC memory) but can write to a single, large HD. I'm not worried about redundancy since a failure is not catastrophic.
Build #2: some bada$$ Supermicro-like system. I'm nowhere NEAR being able to understand how to put that together or even make a build-vs-buy decision.
Build #3: a web server that connects to the file server. That's a 2016 or 2017 addition (after the software is mid-stream in development).

Concentrating on the first, BlueIris server, let me attempt to answer j_h_o's questions, above.

#1: using the StarDot "Bandwidth and Storage Calculator" I come up with 3.6TB for 8 cameras @ 1080p 15FPS, 7 days of storage.
#2: my testing on a LGA2011 (6-core) running at 4.2Ghz shows 5% CPU use for one camera at 720p. BUT that's not direct-to-disk; the demo mode intercedes in order to spray "demo mode" on the video.
#3: SQL load will be negligible; File serving will be measurable (1-2 streams of 1080p+ video and various file serving activity).
#4: Noise is no problem; power usage is no problem. They should be considered, of course, at least to be Green(tm) and to lower the bills, but they're of secondary considerations.

JeffroMart's reply is YAA: Yet Another Anecdote (and much appreciated!) I've concluded that nobody has a formula for BlueIris CPU consumption. But there are several threads that are informative: BI CPU Usage, Which is better for BI: Intel 5820 or 6700, 36 IP Cams Blue Iris Setup Notes.

The conclusion from the last thread is:
"1. Frame Rate Matters. At night, many cams auto reduce frame rates to 15FPS. CPU (I7-5930K) drops from 60% to 30%.
2. Screen Refresh Matters. It accounts for about 10% of CPU (I7-5930K) usage on my setup.
3. Megapixel matters. A 3MP cam triggering recording and transcoding will tax CPU signficantly. VGA cams, not so much.
4. PC Spec. Really Matters. In order to drive that many megapixels, it's important to get a beefy system. As Fenderman repeatedly reminds us, not all i7 are created equal. With this setup, inbound network traffic is about 12MB/S (about 100Mb/s) making gigabit ethernet a must. But SSD for CURRENT CLIP may be an overkill as writing 12MB/S (continuous recording, if needed) to HDD is within the comfort zone of normal magnetic HDDs.
5. DTD (Direct to Disk) Really Matters. Even if you have a stable setup with CPU at 50-60% usage WHEN NOT recording, triggering multiple HD cam transcoding will kill your high end CPU and start to cause dropped frames, pixelations, and a generally unresponsive system. Enabling DTD on as many CAMs as one can (if not on all) allows the video stream to be directly saved to disk without transcoding (CPU intensive work). One can research this site on the many good suggestions and caveats of using DTD (no time/weather video overlay, rotation of images, etc)."

My belief is that I need 1x 3.{4,5,6,7+}Ghz Xeon (6-8 cores) and at least 16GB of memory to keep up with recording AND allow two simultaneous live-views (the main view and a monitor at the front door). I can run Server 2012 R2 express on this machine.

Question: Build or buy? Dell or HP or ? Who has the better mgmt interface? Better bang-for-buck? I'm currently surfing eBay and thinking that something like a ProLiant DL320e might do nicely (eBay). Or, with help here, I could build something similar at lower cost (?).

Cheers.

 

[PigLover]

Note that BlueIris is VERY CPU intensive. See their web site for more information, but suffice it to say that ten 1080p web cams could bring a Skylake to its knees. It seems to be very poorly threaded; per-core Ghz appears to be important. For that reason, I was thinking that it may need to be its own machine, perhaps writing to local drives since it's very disk-intensive. They recommend 7200 RPM drives or even an SSD or RAID.

Just a small note that I found in running BlueIris on our system. As far as CPU usage goes it all depends on the configuration of the output of the streams

+1 to this - exactly the same as my experience.

Running 9 cameras right now, all 1080p 30fps. Running inside VM, but this is the only VM on the server. CPU is 8 core Avoton (C2750), all 8 cores allocated in VM.

With BI "transcoding" on all 9 cameras: 90%+ CPU, dropped frames, missed motion detects, unusable UI and stutters on playback.
With 8 of 9 cameras using Direct-to-Disk: 24% CPU, perfect recordings, perfect motion detect, perfect playback. The 9th camera has some "special needs" and I have to overlay in BI.

Its not that BI is "poorly threaded". Its that video processing is CPU intensive (like - duh!).