workstation for huge graphics files

[nk215]

If I were you, I would get my hand on a motherboard with the most pci3 x8 slots such as x9drx+-f which has 10 pci3x8 slots and a pci2 slot. Load it up with pci ssd cards and RAID10 them. You'll get 8+ GB/sec in transfer speed in no time.

Enterprise SSD is nice especially since your budget is so high but I would start out with a few 970 2TB EVO drives such as 8 of them for 8TB storage in RAID10 as starting out.

Load it up with RAM and use primocache to cache the SSD.

Use one PCI for a i8 or i16 controller to get more SATA SSDs to serve as warm storage in the same PC.

You are going after read speed above all else.

 

[aag]

Rand_ thank you for your detailed thoughts.

Workflow: its definition is in flux, and it will probably evolve repeatedly over time. In general, we are trying to automate as much as we can. Because the scanning is so slow, my most important goal is to operate the scanning automatically day-and-night hands-off on a 24/7 basis. However, for the time being we are nowhere near that.

Until last month a technician needed to readjust things in the scanner every 30 min. Now we can acquire a full stack (ca. 7 hours of scanning) hands-off, but our self-made scheduler (python) has bugs, and sometimes the same slice is repeated many times. Hence the necessity to inspect the stacks after acquisition. The ultimate goal is to automate sample handling with a comau e.do robotic arm.

The "analysis server" will be used to perform complex transforms on the data (segmentation, registration to a brain atlas, morphometry etc.).

The RAID box was purchased already, and is now attached to a single PC via Thunderbolt (I think). I certainly would like to repurpose it for the planned new setup. I assume that it may also have an ethernet port, hence its topology might be changed.

As for the tape unit, nothing is decided yet - except that we need one. Currently all data are only on a local drive + on the RAID. This is no good, also because the RAID will be soon filled. I'd rather have a fast tape unit (depending on price), because it's not just backup, it's also an archive. We will probably have to go back and reanalyze scan collections several times, and I don't want to go crazy waiting for the tape restores.

256 GB RAM for the clients would be very nice, because the first rough analysis has to be done on them, and the stacks are huge. Less RAM will mean more technician time, which would ultimately become more expensive (probably already within few months).

I didn't mention ECC (memory) at all because I am fundamentally ignorant! Might you take the time to explain pros and cons?

I would rather not go for old/used HW, if only for warranty reasons. I want to be responsible in my purchases, of course, but any unused rest of my budget will expire at the end of 2018, hence I have no incentive to buy cheap while compromising warranty.

 

[Rand__]

How many images/TB of data is a full stack? i.e. Is that one image or a number of them; whats the scanning speed (input data flow) and review/analysis speed (output data flow) from the server.

I'd set up a tape server with a cache device (for a single or 2 images). That can receive the done images from the analysis server, store them to the Raid box and then run tape backup from the raid box (when its speedy enough or while no new pic comes in, depending on analysis speed).

Do you really need visual inspection of the acquired image? If the same image is scanned twice is it then identical? Or only similar? Maybe a change in workflow would also help

ECC is a error correcting method (in this case on memory) to ensure data is not modified while in memory, see wikipedia. Your need depends on whether single bits are important to you or not. If you are looking for highest image quality then I'd say yes. Just something that you need to follow end to end (written data verification runs etc) which costs time.

 

[mstone]

thank you for your viewpoint. Might you elaborate? I am now considering deploying a small optical fiber network, so that we could upgrade to faster network speeds in the future

If your primary concern is read/write speeds at the edge of the network, running entirely off network storage is the least viable and most expensive approach. A consumer grade NVMe SSD can easily read sequentially at 2GByte/s (16Gbit/s). If you put in another one and stripe you're looking at 4GByte/s (32Gbit/s). If you have two clients reading at the same time, your server needs to be able to push 64Gbit/s to do the same. That's more than half the capacity of a 100Gbps link, and more than 6 times the capacity of a 10Gbit link. Pushing that much over a network is much, much harder than doing so locally. You also generally need much more expensive/reliable storage if everything is centralized because if it goes down you're completely down instead of losing 1 out of N workstations.

So go ahead and have centralized storage for archiving or whatever, but don't lose the local storage and expect to be able to hit the same performance for any reasonable amount of money.

I also agree with everyone who says that attaching a big raid array with a small number of disks via thunderbolt is a meh idea. I'd suggest a SAS RAID HBA (e.g., areca arc1883-x, lsi 9480-818e, etc.) and a SAS-expander JBOD shelf with at at least 10 or 12 drives (including a hot spare; the drives can all be relatively low-cost SATA) in RAID6, with a battery on the RAID controller so you can use a write cache. Then you can keep the size of the individual drives relatively small (like 4-6TB). The problem with using really big drives in a RAID array is that rebuilds take forever, and the window of vulnerability when you're rebuilding lasts a long time (and the array will be almost unusably slow during the rebuild). If you try to get the same capacity as the RAID shelf I described above in a 4 drive RAID5 with 10-12TB drives, the odds are better than even that the loss of one drive will mean the loss of the array.

 

[aag]

I like the idea of striping two consumer grade NVMe SSD, 2 TB each. May I ask you to elaborate? Which mainboards would support that? Or would we need a PCIe adapter for these SSDs? And which software would be required/recommended for the striping? Many thanks in advance!

I forgot to mention that most data will have to be shipped to our collaborators in Stanford for additional processing (we are in Europe). That is unrelated to the present set of question, but is going to be a lot of fun </sarcasm>. I suspect that we will end up fedex-ing HDDs rather that coercing humongous files through the internet...

 

[kapone]

I have a humble suggestion.

This *seems* like critical work. While saving money is a good thing...in this case, contact HP/Dell/etc etc directly or through a reseller. Get a system that is supported, and supported well. I'm talking things like onsite support, overnight/4 hr replacement parts etc etc.

You do not want to be in the position of debugging things because you DIYed your hardware and are not sure if it's the hardware or the software or the network or... There's a popular saying, I'm sure you've heard it. Penny wise...Pound foolish...

 

[aag]

Dear kapone, thank you so much for your "humble" suggestion. I am considering it, and I am certainly not dismissing it. However, it is not immediately obvious that your suggestion will produce a better cost/benefit ratio than others. At least in my country, service contracts are hideously expensive. In almost 30 years of life-science research, I have never encountered a situation where a service contract would be advantageous to anybody but the company offering it. Also, our team includes two colleagues with a PhD in computer science (which however does not mean that they would be hardware/network experts!) and we are not bad at DIY. We have constructed two state-of-the-art selective-plane illumination microscopes, and that endeavor is infinitely more complex than building a small computer network (and we had no choice, since no such instruments exist commercially). Nevertheless, you are right that this needs to be researched, and not dismissed a priori. I appreciate your input.

 

[aag]

ECC is a error correcting method (in this case on memory) to ensure data is not modified while in memory, see wikipedia. Your need depends on whether single bits are important to you or not. If you are looking for highest image quality then I'd say yes. Just something that you need to follow end to end (written data verification runs etc) which costs time.

May I ask for clarification? Does this mean that non-ECC RAM is prone to single-bit errors? That would be extremely worrisome, and I would expect it to produce CRC errors and all kind of crashes, the accuracy of scanned images being the least of all problems. Or am I completely off?

In more general terms, might you (and the community following this thread) give me specific advice on the server to purchased? I used Supermicro servers in the past, and I like their price/performance ratio. However, the Supermicro web site is rather chaotic and lists an enormous number of products, without any usable comparison of features. Now that you know the details of what I intend to do, I gather that this community might give me the best advice on this purchase.

 

[Rand__]

Have you read the wikipedia article? That explains pretty thoroughly.

And honestly at the moment without actually having established the actual workflow (or nitty gritty details) would feel hard presses to specify exact specifications for hw to buy. For example the number of PCIe cards you will be using decided the type of mainboard you need (slots vs bifurcation vs #of nvme bays on chassis [0-many])

 

[aag]

Have you read the wikipedia article? That explains pretty thoroughly.

Thank you for pointing me to that article. The maximal expected error rate is 10e-10 (but this is the worst-case, probably 10e-15 is more realistic). One 3D scan is approx. 4e+12 bits. Hence we might expect one bit error every few hundred scans. Of course this follows Poisson's law, hence even the first three scans may all suffer from errors. I therefore believe that you are right; we should go ECC.

 

[Rand__]

As I said its important to decide on that since it has significant impact on your boxes.

Only Xeon (or i3 CPUs iirc) actually support ECC memory (in intel world). Only Xeon E5's (+) support Registered memory allowing more than 64 GB (i3/e3). I think Xeon W also supports more than 64 GB (256/512) on FCLGA2066., so that might your you platform of choice for the clients (or xeon e5/SP o/c)

The next item you need to choose is
1. Required number of (nvme) drives (1 big one, multiple smaller ones, what form factor (pcie|aic vs u2 = pcie slots vs drive bays). Not sure you actually can HW raid nvme drives but o/c you can software raid them in windows if thats your thing
2. Network connectivity - Is 10G enough or not (can you use onboard 10g nic or do you need add-in nic (10|40|100G) which needs another x8|x16 slot). Can you afford the necessary switch (educational/company discount). Do you need RDMA/RoCE for max transfer speed
3. How many nodes total? 3 (network capable tape or attached to server) or 4 (dedicated backup host)
...

 

[aag]

Thanks. I feel that I am getting closer to a reasonable setup. Might a plausible choice for the clients be:

• X11SRA | Motherboards | Products - Super Micro Computer, Inc.
• Win10
• Xeon W-2195
• 256 GB DDR4 ECC RDIMMHighPoint rSSD7101B 4TB NVMe SSD (data)
• Samsung 970 EVO 500GB, M.2) (software)
• 10GBASE adapter card (which?)
• 10GBASE-SR transceiver (which?)

As for the network, I would go with 10GB-SR (optical). If need be, we can upgrade later. At the moment, the cost of 40-100G hardware is prohibitive, but at least the optical cabling will allow for it in the future. My next question, however, is: which network switch would be most appropriate for this LAN?

 

[kapone]

Dear kapone, thank you so much for your "humble" suggestion. I am considering it, and I am certainly not dismissing it. However, it is not immediately obvious that your suggestion will produce a better cost/benefit ratio than others. At least in my country, service contracts are hideously expensive. In almost 30 years of life-science research, I have never encountered a situation where a service contract would be advantageous to anybody but the company offering it. Also, our team includes two colleagues with a PhD in computer science (which however does not mean that they would be hardware/network experts!) and we are not bad at DIY. We have constructed two state-of-the-art selective-plane illumination microscopes, and that endeavor is infinitely more complex than building a small computer network (and we had no choice, since no such instruments exist commercially). Nevertheless, you are right that this needs to be researched, and not dismissed a priori. I appreciate your input.

There IS no cost/benefit ratio. You're buying a premade system with a service contract for support and peace of mind, not cost.

And that system is recommended to you by the manufacturer/integrator based on the requirements you've given them. If it doesn't perform to spec, it's their fault.

P.S. You're not taking into account your (or your PhDs) time value.

Your time, as an e.g. is worth ~$100/hr, because that's what you can get paid when you work for a client. A day's worth of work is ~$800 in that case, so if you have to spend some money to not have to account for that time, it's worth it.

When you're diddling around a homelab that's different, because there is no expectation to get paid for those hours. In your case this sounds like actual business work, which should be accounted for differently.

 

[Rand__]

There are three ways to do this kind of thing
-"Full Service" as @kapone recommends - Get a company to design the spec/setup of the solution. You will pay a premium for their services (and like the hw) but you will (should) benefit from their experience with these things. The better defined the workflow is the better is the result. Alternative is they design the workflow too based on your preliminary process idea (more expensive, easy to get wrong).
This is the well funded/uneducated/no-risk company way that often works well but also often fails miserably due to overestimation of knowledge, a missing understanding of the process or it intricacies or simply ego (all of those on either side).

- "Full HW service" - Buy a premade box based on your design/specs. (significantly) more expensive but verified functionality and interoperability. You define the process and its requirements, you simply don't bother with building the blocks. Can be significantly more expensive than no service, but depends on your contracts/haggling skills/need to get the newest hardware (i.e. yesterdays hw gets thrown out pretty cheaply yet with full warranty quite often).

-No Service - build your own. Cheapest, most time consuming, least amount of money.

Totally your choice
Given that your process is still in flux (depended on the new HW apparently) you'd either need a capable contractor helping you properly design it or you work on it iteratively and hope that you make the right call (or the hw is flexible enough to cope with changing design which it usually is if you retain expandability and don't spend the whole budget on the first run).

 

[Rand__]

Thanks. I feel that I am getting closer to a reasonable setup. Might a plausible choice for the clients be:

• X11SRA | Motherboards | Products - Super Micro Computer, Inc.
• Win10
• Xeon W-2195
• 256 GB DDR4 ECC RDIMMHighPoint rSSD7101B 4TB NVMe SSD (data)
• Samsung 970 EVO 500GB, M.2) (software)
• 10GBASE adapter card (which?)
• 10GBASE-SR transceiver (which?)

As for the network, I would go with 10GB-SR (optical). If need be, we can upgrade later. At the moment, the cost of 40-100G hardware is prohibitive, but at least the optical cabling will allow for it in the future. My next question, however, is: which network switch would be most appropriate for this LAN?

Why on earth would you want to add a 2k€ 18 core CPU to this built?

Your primary goal is memory support followed by copying speed. You are limiting to 10G here so more or less any medium clocked current CPU should be able to do that, so not even fastest single core is needed.

Now you still have not defined the client side workflow (scan, review, copy in parallel or serially) appropriately, so the actual number of cores is yet unknown, but I am fairly sure it does not have to be 18.
-Also unless the client software is huge you don't need a nvme drive for that.
-Also you dont have your originially stated local backup option - intentionally or oversight?
- NIC, no idea regarding recent 10G models but I usually go with Intel nowadays.
-Also, I'd still use 2 data nvme drives separating in- and outflow.
I see the drive you picked is a raid10, so it might suffice or not. Which model do you take?`You will need to consider flash lifetime. P modell recommended. O/C its m.2 based so limited capabilities (plp, actual size, thermal output etc)

Lastly - spec out the box, eval the price and then compare it to yesterdays E5 based setups (E5-1660 or so). Way better choice of boards available there, with included 10G options. Maybe also Xeon-D (not sure on high clock freq's). Lots of choices.

Also we have not spoken about cooling/power draw/noise requirements. Rack or Desktop?

 

[aag]

Thanks. At this stage my flowchart is:

• try to educate myself (I am a medical doctor and a professor of pathology; HW is far from my expertise) as far as I can. This forum is excellent for the purpose. I am getting a lot of detailed, competent and up-to-date advice. Much better than StackExchange where people tend to be condescending and never get into details.
• Define a possible system with the help of forum members
  
• Propose the system to the HW professionals at my institution, and see what they say. From past experience, I can say that they are competent but very slow. Polling the "wisdom of the crowd" here seems a much more efficient way to come up with preliminary specs.
• Have in-house technical service install fiber network. The media channels are all available and preinstalled, and the runs are short; hence this is not going to be very expensive.
  
• Buy "Full HW service" per your definition. The idea is definitely not to buy motherboards, cases etc. and assembling them by myself (unless that turns out to be the only possible path, but this is unlikely). Installing specific adapter cards etc. is a tolerable task though.

 

[aag]

Also we have not spoken about cooling/power draw/noise requirements. Rack or Desktop?

The SPIM room has strong cooling. Hence room thermal will not be limiting, but space is very tight. The two acquisition machines are next to each other (in fact I would like to run one of them "headless", controlled from the other by RDP. Hence rack mount might be the better choice.

I'd still use 2 data nvme drives separating in- and outflow.

Not sure I understand. Do you mean, having 2 identical nvme drives, and joining them as software RAID? That may be indeed a good idea. Since most boards have max 2 M.2 connectors, the "software drive" would then have to be a SATA3 SDD.

 

[Rand__]

You could also do 2/4 nodes in 2 U if you really need to save rack space, but thats limiting add on cards.

You already have software raid with your highpoint controller as I dont image they actually do HW raid.
My point would be to have them parallel (basically one per workflow step)
Scan to disk 1. When done, review on disk 1. Scan to Disk 2. Move to server from disk 1. When done, review on disk 2. Scan to Disk 1. Move to server from disk 2 ...
Not needed if the drive in question can cope with scan, review and copying out at the same time.

 

[aag]

I would rather not copy these huge files from an incoming to an outgoing disk within the same box. The workflow will be eventually carried out by biologists with zero computer experience, hence it should be kept as simple as possible. Of course it could be automated, but this always comes at the expense of flexibility, which in a research environment is paramount. Also, the data acquisition is so slow that I do not envisage major bottlenecks with concomitant access.

So Rand_, based on all you know and all I told you, which motherboard and CPU would you recommend? My main problem is that I am confused by the myriad of options and flavors of Xeons etc.

As I said I like supermicro, but I am not obsessing on them. For the two clients, all I need is a bunch of PCIe slots, at least two NVMe connectors, 256 GB RAM, and a really speedy processor. If the board offers a slot for an fibre GBIC, so the better - but an adapter would also do the job.

For the server, requirements are not really much different, except that more RAM (e.g. 1 TB) would come handy.

 

[Rand__]

I didnt say copy between drives.
I would have switched (ideally programatically) the scan/review destination.
But thats workflow and if you say as easy as possible then stay with one drive, perfectly fine.

You might have different requirements for server and clients as server has computational requirements has it not? What program, how many concurrent users, how many concurrent images etc