ArcGIS for Server cache farm

[smccloud]

I've been playing around with building a caching cluster at work for our big customers, the ones who's GIS data takes well over a week to cache (one takes 65+ hours just for two levels). This is the start of the build log for the cluster I've speced out. I chose NAS4Free instead of FreeNAS or OmniOS + napp-it due to the fact that I have the most experience with it.
Storage Server

• Supermicro 6047R-E1R24L
• 2x Intel Xeon E5-2603 Sandy Bridge-EP
• 4x Crucial 16GB (2x8GB kit)
• 2x WB Blue 250GB HDD (mirrored for boot, probably OmniOS + napp-it now)
• 10x WD Black 2TB (5 stripped mirrors)
• Intel X540-T2
• 4x Icy Dock EZConvert MB882SP-1S-1B
• 4x Intel DC3500 300GB SSD (2 mirrored for ZIL & 2 stripped for L2ARC)

Caching Servers (3 or 4)

• Supermicro 5018D-MF
• Intel Xeon E3-1275V3 Haswell
• Crucial 16GB (2x8GB kit)
• WB Blue 250GB HDD (fast enough since not much data will need to be local, will be backed up to another storage array)
• Intel X540-T1
• Server 2012 R2 Standard
• ArcGIS for Server 10.2.2

Netgear 12-port ProSafe 10Gbe Switch XS712T
Coboc 7' CAT7 cables, orange because it isn't used in our server room yet

Current testing shows a 76% speed improvement going from a single server to four servers in my VM environment, and I expect it to be even larger with physical machines (storage for the VMs is a Lenovo Haswell i5 desktop w/ 8GB RAM, single 1Gbe NIC & 2 320GB Hitachi SATA drives in RAID 1 on Ubuntu).

I expect to be able to start ordering parts by the end of the month at the latest and will definitely enjoy posting build pictures.

 

[spazoid]

I have to ask. Why the VelociRaptors? The 600 GB drives are over 3 years old, and slower in most regards than new 2-4 TB drives. I'm sure you've thought about it, I'm just curious.

 

[smccloud]

The random access on them is slightly better than a 7200 RPM drive, and since we will be reading from and writing to them at the same time.....Plus the smaller size means IT won't take part of it over for a VM datastore They do also have a better warranty.

 

[smccloud]

Minor specification update, going to OmniOS + napp-it for the OS and updated the drives accordingly.

 

[BackWoodsTech]

Hey there. Not trying to hijack your thread, but I do quite a bit with AGS myself...usage and implementation, not so much on architecting and large deployments like what it sounds you're up to.

Curious about what your caching cycles are and if your just recaching everything in bulk or if there's some change detection going on? Sounds like fun. I'm trying to design an interactive tile creation download tool for our ArcPad users (we just acquired over 30 licenses and tile caching is going to be key to this roll out). Again, sorry to hijack.

 

[smccloud]

We actually do both full recaching & AOI recaching. Unfortunately the AOI stuff is super buggy, hence the big deployment. It will be used for full recaches/initial caches.

 

[capn_pineapple]

Just out of curiosity, is your only reason for not going SSD array price? Or does the workload just not benefit from that style of storage i.e. need large storage size over speed?

 

[smccloud]

Just out of curiosity, is your only reason for not going SSD array price? Or does the workload just not benefit from that style of storage i.e. need large storage size over speed?

Mostly price. No SSDs to test caching using them. I think the would probably be great to store the base GIS data on, but I'm not going to build two arrays so I can have the GIS on SSDs and the cache on spinning disks. Plus some of our customers have a final cache size of over 1TB and the aerial imagery can be close to that too.