Can't get more than 20Gbps out of a 40GbE network - Suggestions?

[lzvolensky]

couple of points :

- PCI Express 2.0 8x is capped at about 25.6Gbit/s = 3.2GB/s. It's 4Gbit a lane, 8 lanes = 32Gbit/s but it uses inefficient 8/10 encoding due to which you only have 8/10 real throughput so 32*8/10 = 25.6Gbit/s. This exactly is the reason why you didn't see more than 3.2GB/s = 25.6Gbit/s in any test of RAID controller or NIC hooked to PCI-Express 2.0 8x.

- PCI Express 3.0 4x is pretty much in the same position. The maximum has shifted from 3.2GB/s to about 3.6GB/s.

- PCI Express 3.0 8x has maximum at about 6.4GB/s = 51.2Gbit/s. Once again, you will hardly see any RAID controller faster than 6.4GB/s with any amount of SSDs hooked to it. Seagate has the fastest SSD in the world (nobody seen yet) with claimed 10GB/s throughput just because they use PCI Express 3.0 16x connection.

- beware where you place your cards. Correctly said already.

- beware how your motherboards are set in BIOS ! It's easily possible that 8x slots are 4x in reality and there's your limit. Out of 25.6Gbit/s, you are able to squeeze out ~21Gbit in tests because you have to generate so much data each second and every tiny microlatency (despite now showing in CPU utilization) adds up. Million times a second, it's a significant factor then.

- beware of any M2, U2 usage. It's seen quite often that some PCI slots have degraded performance when non-SATA storage is used in motherboard. Switch it off, if you can.

- you will NOT achieve 1200MB/s with EIGHT spinning drives you have. Those disks attack almost 200MB/s on the most outter tracks (longest) = when they are empty, but they fall to ~120MB/s on the inner shortest tracks when filled. You have 8 of those, so in the worst case you will be getting 8x 120MB/s = 960MB/s which is far cry away from your required target. Not talking about parallel operations from 2 workstations to which we get in a second.

- ZFS. How good are you in optimizing ZFS ? That is a science, believe me. Especially with writes which you will have a ton.

- RAID overhead. Don't even think about having the performance of all disks available in logical RAID. If you create RAID0 = striped vpool, well we might be getting close to simple linear scaling. With Raid5 (Raid-Z) or Raid6 (Raid-Z2), forget it instantly. Apologies for using Raid5/Raid6 naming convention, we all know it's incorrect with ZFS.

- you require TWO PARALLEL operations with those spinning disks. No way in the world you will achieve 50/50 performance when you split it. Let's say you have disk subsystem able to provide 3000MB/s either for reads or for writes. At the moment you will try to read AND write AT THE SAME TIME from the same subsystem, you will not have 1500MB/s for reading and 1500MB/s for writing no matter what RAID controllers, operating system, cache, RAM or networking adapters you have. I realistically ESTIMATE in such case you will have close to 200MB/s reads and 150MB/s writes, yes that much performance will be hit. Spinning drives are EXTREMELY bad in handling anything that is non-linear non-sequential. Even ZFS fragmentation due to copy-on-write mechanism WILL play EXTREME role after you use that filesystem a bit, after it will fill up a bit. Please note this is not american lawyer style speech "may have", it guaranteed will have performance impact.


The reason ? Seek times. Spinning drives have to move their heads back and forth. Forget linear scaling or linear split in performance. Even SSD drives do not scale that good - any model with 100.000 IOPS read, 50.000 IOPS write will not achieve 75.000 IOPS read/write combined together (see any Intel DC SSD specification, it's simply not mathematically averaged due to introduced latencies).


- introducing two or more separated pools : extremely clever idea. You will not achieve your goals without them. Eight disks in pool seem to be on the poor side to me, given the performance characteristics I would not go with less than 12 or probably 16, but even with dozen you will hit cruel laws of scalability and failure tolerance design. Split those disks evenly across controllers in order to load their CPUs and PCI bus equally. Don't put "read array" on controller1 and "write array" on controller2, mix 'em - half of "read array" disks on controller1, the other half on controller2. The same with "write array" disks.

- carefully choose your disks. I wouldn't go with Seagate and especially not with DM001 models you have (this is DESKTOP MODEL), but that's my personal choice only. Reasons ? Performance, longevity, no designation for 24x7 operation, heat/shock resistance (yes, snapping 12 disks into enclosure is not the same as having three in the desktop) and some others.



Executive summary : your project CAN NOT achieve requested 1200MB/s read and any amount of write throughput at the same time, as it is configured now. No matter how you try - due to disk subsystem and amount of physical disks you have. You will have hard times achieving that long-term with two or more separated pools, but it's doable with careful planning (and lots of low-level horsepower = physical disks).



I'm very sorry to say this bad news.

 

[lzvolensky]

forgot to say - operating systems and their drivers play huge role. It's not exactly the OS difference (Windows X versus Linux Y), it's The Driver. Especially with FreeBSD [!!!!] it plays huge role. I would recommend to try to swap adapters : you have Chelsio in FreeNAS now, pull it out and test with Mellanox X-3 instead. If it works = if there is driver for it, test throughput. If there is no driver, try to compile one, I believe ConnectX-3 has support for FreeBSD from Mellanox.

Both adapters you have are definitely from the better end of spectrum. The catch might be in FreeBSD drivers and their optimization. If possible, I would also CAREFULLY suggest to test any CentOS or Ubuntu storage repository instead of FreeNAS. You can use ZFS if totally required, but MDADM and the "classical" arrays are no slouch either. Pay attention to resilver = rebuild times which are the main differentiator between those filesystems : ZFS ONLY will rebuild as much data as needed whereas old-fashioned Raid5/6 implementation will have to go through ALL disk capacity.

This easily means two or three days rebuild times with 6TB disks you have. Such solution is extremely risky in 8+ drive environment. What happens when your rendering data are lost ? Is it "only" two days of calculations done or is it far more expensive ? Performance will be also CATASTROPHIC during any rebuilds, you easily will not get even 1200MB/s reads you request - once again this is due to seeks and physical head movements. If you hammer the storage during rebuild, you easily might get into 5 or 6 days total rebuild time. No kidding. Seen that.


Decisions, decisions...

Swap the adapters in FreeNAS, test with Mellanox. Swap FreeNAS for CentOS if you have the time.

And absolutely, absolutely ditch Windows7. Windows8 or Windows10 are so much better with these super-modern high-end technologies and this is said by a chap (me) who uses Win7 EVERYWHERE (where I don't have Windows Server 2012 ... I don't have single Win8/10.

And last but not least, because you talk about 6TB movie size total, I would absolutely go SSD route. Samsung SM863, 1TB each, priced around $400, you need eight of them considering some capacity will be lost due to RAID/ZFS redundancy. More reliable than HDDs. Intel P/DC3x00. Yes, this adds some financial cost to your solution, but the performance is orders-of-magnitude better than spinning drives, especially with parallel access from two or three workstations. Each SSD can easily provide around 450MB/s read performance for sequential transfers all-disk-long no matter how much capacity is used, no internal or external tracks to deal with. Eight of them and you are definitely on the safe side when 1200MB/s is requested. Scaling will be your major problem then - how to get all the performance from disks through enclosure (pay attention to SAS/SATA expanders !) through Raid/HBA controllers down to PCI Express bus to operating system.

You are on The Great Forum here, don't even think about opening GREAL DEALS forum section as you will spend as much money as we all do listen you can find 800GB / 960GB ENTERPRISE SSD offers there for $200, yes slightly used but... but... but... dozen of those and you have 10TB to 12TB of raw capacity, about 8TB to 10TB usable after RAID/ZFS penalty. That all for circa $2500... Rebuild times in case of failure ? Just couple of MINUTES, compared to DAY(S) with HDDs. You definitely want to have one, two or probably three available on your shelf just for spare purposes. Hey they only cost $200 a piece !

Bada bada boom boom !! Feeling so sorry for spending your money

 

[jerrytsao]

couple of points :

- PCI Express 2.0 8x is capped at about 25.6Gbit/s = 3.2GB/s. It's 4Gbit a lane, 8 lanes = 32Gbit/s but it uses inefficient 8/10 encoding due to which you only have 8/10 real throughput so 32*8/10 = 25.6Gbit/s. This exactly is the reason why you didn't see more than 3.2GB/s = 25.6Gbit/s in any test of RAID controller or NIC hooked to PCI-Express 2.0 8x.

- PCI Express 3.0 4x is pretty much in the same position. The maximum has shifted from 3.2GB/s to about 3.6GB/s.

- PCI Express 3.0 8x has maximum at about 6.4GB/s = 51.2Gbit/s. Once again, you will hardly see any RAID controller faster than 6.4GB/s with any amount of SSDs hooked to it. Seagate has the fastest SSD in the world (nobody seen yet) with claimed 10GB/s throughput just because they use PCI Express 3.0 16x connection.

Just wanna point out the real life PCIe bandwidth isn't calculated by that formula, 8/10b and 128b/130b are encoding overheads on the transmission line only.

Gen 2 x1: 5.0 GT/s*8b/10b = 4 Gbit/s (500 MB/s)
Gen 2 x8: 40 GT/s*8b/10b = 32 Gbit/s (4 GB/s)

Gen 3 x1: 8.0 GT/s*128b/130b = 7.88 Gbit/s (985 MB/s)
Gen 3 x4: 32 GT/s*128b/130b = 31.51 Gbit/s (3.94 GB/s)
Gen 3 x8: 64 GT/s*128b/130b = 63 Gbit/s (7.88 GB/s)

Keep in mind the above GB is Gigabyte rather than Gibibyte that is usually measured (4 GB/s = 3.73 GiB/s, 3.94 GB/s = 3.67 GiB/s, 7.88 GB/s = 7.34 GiB/s), also there are many other overheads contributing to the actual 3.2 GB/s, 3.6 GB/s and 6.4 GB/s limit you referred to, but not through the line code formula again, it's a pretty complicated process overall.

http://www.xilinx.com/support/documentation/white_papers/wp350.pdf

 

[Patrick]

@lzvolensky the Chelsio adapters are very well supported in FreeBSD and are usually my top recommendation for pfSense, FreeNAS and etc. I have had more issues with ConnectX-3 and FreeBSD systems so I have standardized on Chelsio to save headaches.

I do completely agree that OS/ drivers make a huge difference.

 

[lzvolensky]

I messed it up with that PCI badly oh I learn many new things every day, thanks for the Xilinx PDF. At least those "final throughput numbers" were pretty spot on, 3.2GB/s and 6.4GB/s are what I see almost everywhere.

The rest I wrote should hold pretty strong unless someone proves me blatantly wrong again in a minute

yes, Chelsio has been top favorite for FreeBSD for very long time. Stability-wise... that doesn't necessarily mean it will have the best performance. Careful checks on hardware side have to be performed to rule out any misconfiguration, heck how many times I inserted fast adapters into slots being 4x electrically and wondered then...

 

[Perry]

Perry, just a few points as regards your system apart from the networking.

Er, thanks, but almost none of that is relevant to our setup.

We are working in 4K, not uhd. Uhd is the consumer 4K format and is a derivative from the kind of higher resolution files were working with.

H264 and H265 are delivery formats and are pretty lightweight in terms of bandwidth requirements (20-30 times less bandwidth than the uncompressed files we deal with day to day). But the system we're talking about has absolutely nothing to do with those formats. We're not encoding on the freeNAS machine, it's merely a file server for a film scanner, a color correction system and a film restoration system.

I've been doing this for 25 years and am well aware of the standards.

 

[Patrick]

@Perry I have wanted to use some of the higher-end footage for our tests/ reviews. The test footage I have from a friend is only around 3.7GB/ minute (ProRes 422). I know that goes WAY up from there.

 

[lzvolensky]

comment to bcache and company : they are linux based. Perry doesn't use Linux in his environment - he has Win7 on clients and FreeBSD on fileserver. EnhanceIO has been declared as dangerous and instable with potential of damaging filesystem, as far as I recall [not for my house, definitely].

Even if he had Linux, you can't get too much benefits in his environment : one local SSD is much slower than Perry's requirement is (1200MB/s, SATA SSD can provide 500MB/s on 6Gbit interface, about 1100MB/s on 12Gbit interface, so he would have to use PCI-Express flash on each client, which has relatively little capacity (no, Perry is not going to buy 4TB PCI-Express drives to each client as they are terribly expensive and for split of that price he can install proper infrastructure), provides next to no benefit for write caching as this is superlarge sequential thingy, probably has absolutely no usage for read caching as these video files are not reused - you don't render one file seven times in a row) etc, etc. And we didn't even start with write endurance yet.

When proper background infrastructure is available with 2000 - 3000MB/s throughput, I see not too much benefits for local cache of any form.

Non-repetitive processing. This is not database. This is not web server with main page articles being accessed by thousands of people each minute.


I feel we somehow deviated from networking to disk subsystem 40Gbit adapters are able to spit out 38Gbit/s of data, that's tested and proven.

 

[Perry]

@Perry I have wanted to use some of the higher-end footage for our tests/ reviews. The test footage I have from a friend is only around 3.7GB/ minute (ProRes 422). I know that goes WAY up from there.

It does. On our web site we have a chart for clients to estimate data rates and file sizes.

There are two different types of files we typically deal with: Containerized files (like Avid, MXF, AVI or Quicktime where the entire movie is in a single file), and image sequences such as 10bit or 16bit DPX. With Image Sequence sets you have a single folder with all the files in it, one for each frame, sequentially numbered.

For testing, you could easily simulate an image sequence by making a file that's about the same size as each frame, then duplicating it and sequentially numbering them. For a Quicktime file, if you have Quicktime Pro ($25 I think) on Mac or Windows, you only need a small movie, and you can copy/paste it over and over then save as self contained to the codec of your choosing (though you can't save to ProRes from Windows, only Mac). You can also use ffmpeg to make most file formats for free.

The problem with these files is that they're too big to be practically transferred over most internet connections. 1 minute of 4k 10bit DPX sequence is nearly 70GB. Also, most of our footage is restricted use because it belongs to our clients, so I can't share it.

 

[Perry]

I feel we somehow deviated from networking to disk subsystem 40Gbit adapters are able to spit out 38Gbit/s of data, that's tested and proven.

That's fine - it's important stuff!

The fact is, I'm able to get 40Gb throughput by running iperf in parallel so I know the network is running at full speed. What I find curious is that a single stream can't saturate the connection - It's largely academic, because the 20Gb we're getting is more bandwidth than we're going to need for quite some time. My approach to this kind of thing is to make sure the kinks are worked out of the network so that when I start testing disk speed in earnest, I know that any slowdowns I'm seeing aren't caused by some sort of underlying network issue. At this stage, I'm pretty confident I can start doing that, but I do wonder why a single iperf stream doesn't saturate. Can it just not do that?

Regarding disk speed - we have multiple 8-drive direct attach RAIDs that can hit our targets. Each brand and capacity of disk that we've tried, however, is different - some are faster than others. I haven't built one yet with 6TB drives - this will be the first. Most of our existing direct-attach RAIDs are either 2TB or 3TB disks. The primary reason for using such large drives is the dropoff in disk speed as the drives fill up. So with a larger drive, we can pretty comfortably work on a couple of projects at the same time and keep the usage under 50%, which is the point where we typically start to see performance issues. Yeah, it's a lot of wasted space, but speed is more important than efficiency here.

SSDs just aren't there yet in terms of price. Once they've dropped, we'll consider them for sure. Our current FreeNAS system can hold 36 disks. That's 216TB if we use all 6TB drives, with 108TB being usable if we keep below our 50% usage goal. That's a decent amount of space to work with, and the cost to get that with spinning drives is only about $7700. If we were to try to get that much space out of SSDs, we'd need way more of them -- which means more HBAs, which means a different motherboard with more slots, and we'd have to get a different enclosure to hold more drives. Not to mention the cost/GB for SSDs being several orders of magnitude more expensive in the first place...

When SSD is near $.25/GB for drives over 3TB, then it'll be worth considering. But for now, it's just way too expensive when a single project will need upwards of 16TB of storage for all the files.

 

[lzvolensky]

>> The primary reason for using such large drives is the dropoff in disk speed as the drives fill up.

rightfully expected - those are the "tracks" I was talking about. Throughput differs tremendously, every single test shows that.
With eight disks and their prices, this is not about wasted space or money as we are talking about ~800$ trashed (8x 3TB disks compared to 8x 6TB disks is about $800 difference). Every single day, more money would be lost due to performance issues and loss of productivity. No brainer.


I get your calculations for SSDs. Been there, done that. How does 960GB enterprise Samsung SSD for $199 sound ?
You can't replace ALL your storage instantly. No. You don't need to.


16TB for a single project is stretching it a bit, but in your current config you seem to have 16 slots free. You might find it surprisingly easy to convince your boss to spend $3200 for sixteen 960GB Samsung disks, offer to be found in Great Offers section here on forum... just to test it out. That would provide you enough capacity to test ONE PROJECT. If they don't work as expected, hell yeah you will sell them five days later for the same money. There is no "fill-factor" to count with. You can fill SSD to 98% and still get 520MB/s sequential read performance out of it. You know that.



Iperf : I believe it might be the architecture itself. Not all testing software is created identically. I'm too lazy to search for iperf 40Gbe performance testing articles on internet

 

[Perry]

Well, I *am* the boss, so I'll need more convincing...

I get the advantages of SSD. But don't forget that for us, we're writing tens of thousands of files to each drive. We use SSDs as system drives in most of our machines, and they've been pretty good. But the failure rate for SSDs has been a lot higher than it has for our spinning disks. We probably have 60-70 spinning disks in use right now in various machines, and we lose maybe one or two per year to failure. In the past few years, we've had half a dozen SSDs (different brands), and have lost 2 to failures. Granted, that's a small sample size, but it's a much higher percentage. At $200/drive I'm not really willing yet to experiment like that. When it's half the price, probably we'll go for it.

The other thing to consider is the sheer amount of space we need - a 1TB SSD takes up the same slot as a 6TB disk. Our current setup would be maxed out at 36TB if we did all SSDs. That's barely enough for two jobs. We usually have 3-4 projects going at once, some of which can drag out for months.

Show me a reliable $200 2TB SSD, and you'll definitely get my attention!

 

[keybored]

... In the past few years, we've had half a dozen SSDs (different brands), and have lost 2 to failures. Granted, that's a small sample size, but it's a much higher percentage. At $200/drive I'm not really willing yet to experiment like that. When it's half the price, probably we'll go for it. ...

That's an interesting data point on SSDs. Can you provide a bit more detail? Are they all consumer level SSDs? Enthusiast/Pro consumer? Enterprise?

 

[Perry]

I don't know if we have the exact model numbers, since I wasn't really paying close attention - they're relatively inexpensive consumer 2.5" SSDs. We've installed OCZ, PNY, SanDisk, Samsung and I think one IBM drive in various machines. One of the failed drives was an OCZ and one a PNY, and one was 128GB the other 256, but I'm not sure which was which off the top of my head. We have other OCZ and PNY drives in different machines, and they're chugging along just fine so far.

SSDs may be different, but I don't buy into the notion that Enterprise drives are better. There's too much data to support the opposite finding, and the extra cost isn't worth it (at least when you're talking about commodity spinning drives). We've had so few drive failures for the number of disks we go through, that the few hundred bucks spent on replacements is vastly less than if we had put much more expensive enterprise drives in those machines. I would imagine, though, that with SSD there may be a difference, simply because it's a newer, less proven technology than traditional drives and you're probably more likely to get a better disk for more money.

I would think the $200 1TB Samsubg SSD being suggested would be classed in with consumer drives, purely based on price, no?

 

[keybored]

... One of the failed drives was an OCZ and one a PNY, and one was 128GB the other 256, but I'm not sure which was which off the top of my head. ...

PNY and OCZ along with Patriot and SiliconPower don't really have a good reputation in the SSD domain. They usually crank out bottom of the barrel, cheapo consumer SSDs. I personally wouldn't put them into anything but the most basic desktops and certainly not into video editing workstations.

SSDs may be different, but I don't buy into the notion that Enterprise drives are better. There's too much data to support the opposite finding, and the extra cost isn't worth it (at least when you're talking about commodity spinning drives). ...

In the case of SSDs, Enterprise drives are definitely better. You get better quality high-endurance NAND, better controllers, and power loss protection. Many consumer SSDs will choke under sustained load but enterprise SSDs will keep chugging along because that is what they're built for.
Ultimately, everything depends on your load profile. If your typical workload continuously stresses your SSDs with continuous writes then you'll likely be better off with enterprise SSDs in the long run.
Regarding pricing -- yes, enterprise SKUs have higher price premiums vs. their consumer counterparts, but if you're willing to self-warranty and if your demand for drives isn't very high then you can actually get them cheaper than the consumer drives of the same capacity. How? You get them used or refurbished. Good quality enterprise SSDs have insane endurance ratings and they have plenty of life left in them even if you buy them used/refurbed.

I would think the $200 1TB Samsubg SSD being suggested would be classed in with consumer drives, purely based on price, no?

If we're talking about the SM863 mentioned earlier in this thread then at $200 a pop for 960GB they wouldn't even be in the same universe quality-wise compared to the consumer drives in that price range. If you look at the current consumer SSD offerings, you're going to be paying $200-$230 for 1TB of storage which will use inferior, planar TLC NAND.
Only problem is, I don't think I've ever seen SM863s go that low. They're relatively new. But there are other drives in the same price range that go on sale relatively frequently; e.g., Samsung PM863, Samsung SV843, Intel S3700.

Bottom line: examine your typical client SSD workloads. If you see heavy sustained disk activity then enterprise SSDs should be a good fit. You can always start with one or two and gather some feedback from users. Those used ones won't cost you any more than the bottom of the barrel consumer drives.

 

[lzvolensky]

>> Well, I *am* the boss, so I'll need more convincing...

ouch it's getting better every minute



Something like this : SAMSUNG Data Center Series SV843 2.5" 960GB SATA III V1 MLC VNAND Enterprise Sol

is worlds apart from your experience with OCZ and PNY [WTH is that]. Failure rate of SSDs is statistically lower than failure rate of HDDs. That's manufacturers stats, I didn't say that - they do. Also, manufacturing process and reliability have been refined from the old era of OCZ. Samsung and Intel are world class manufacturers with their own NAND supply so rest assured they keep the best quality chips for themselves and only sell what they don't need/want to competitors.


Samsung's V-NAND or 3D NAND is extremely more reliable than any "planar" 2D consumer SSDs. First and foremost, customer models typically use TLC chips instead of MLC as pointed out already. Next, Samsung 3D Enterprise SSDs are manufactured with 4x nm technology compared to 1x consumer class - it's exactly the same as comparing the most luxurious Mercedes/Maybach to cheapest car you can find on market. I will add that 3D arrangement of cells in Samsung drives contributes to extremely low cross-interference, heavily prolonging their useful life. Plus the difference between 4x and 1x NAND process can't be described in words. 3D is the only way forward, it's the future.

Samsung said (Samsung SSD 850 Pro (128GB, 256GB & 1TB) Review: Enter the 3D Era) they have consumer-class 850Pro model in 128GB capacity that kicked 8000TB of writes and still going strong - to illustrate, those drives are warranted for 150TB only. Enterprise class is warrantied for much much more.

Let me confirm what has been said above : there are many factors why consumer class and enterprise class SSDs are worlds apart. Well it's directly Patrick, The Guru The Owner here, who has extremely good experience with used/refurbished SSDs. Enterprise class SSDs.


We fully understand your density requirements. Nobody says you should dump all your storage. Ehm no. We are just trying to let you think about using SSDs because you said you have slots free. Maybe you could think about building additional storage box which could cost you around 1000$ - that box will be SSD TEST to split your load. Some workstations will be using existing storage, some less important/deadline relaxed/less risky project could test SSDs.

Such solution would provide you, among other benefits, failover possibilities. What happens with your data when motherboard of current storage breaks down ? How long it will take to restore service and be fully operational again ? We don't know, you didn't mention that. What is the cost of your outage, per hour ? Having additional box directly on site - yes test box with SSDs - would mean just reconnecting disks from failed storage (sure, and having to stop the less important less critical project rendering on SSDs) and kicking back to action in 10 minutes.


As you can see on linked auction, hundreds of SSDs were sold essentially every day so there is no single question you would be able to sell yours, if you hit any wall with them. As they say in Jamaica "I buya $180, dat sht no worky worky, I sell $180 next day". Moreover, depreciation on those SSDs will be low during time, too. Buy them for $200 today, sell them for $99 in year. Heh.


For me, THE MAJOR factor to test The Strange Storage would be essentially no financial loss arising from being brave (=testing them out a couple of days) AND IMPOSSIBILITY OF 36 HDDs TO PROVIDE ENOUGH PERFORMANCE FOR TWO OR THREE CONCURRENT PROJECTS DEMANDING 1200MB/s EACH.


What we didn't touch yet is how you access the data in storage box. iSCSI, SMB, NFS ? They differ in capabilities, resiliency and of course performance. Windows7 is not exactly what we call champion in offloading techniques and efficiency - this means your network adapters probably will load CPUs more than they could with Windows8/10, effectively leaving less CPU performance for rendering applications. It's not so easy to transfer 20Gbit/s over network...

 

[azev]

but I do wonder why a single iperf stream doesn't saturate. Can it just not do that?

The maximum throughput of a single tcp flow is determined based on the window size and latency between the 2 endpoints.
You can you the this online calculator to calculate how big your window size have to be to reach the kind of throughput you are looking for.
There's also other factor such as tcp buffers, link reliability etc etc.
I think I read somewhere although in modern OS you can increase your tcp window size to 1Gb there's still laws of diminishing return.

I cannot remember just how many time I have to answer why our 10gb circuit between 2 site with very low utilization and 70ms latency cannot allow an application which only use single flow to use every drop of bandwidth available.

 

[Perry]

Sorry - had a ton of work to do for the past few days, and didn't get a chance to respond here.

I'm willing to try the Samsung SSDs, but only if they're in the $200-$225 range. The ones on ebay are gone, and there is but one used drive on Amazon right now. I'll poke around and see what I find, but my thinking is that these won't go in the FreeNAS box at all. Rather, they're going to go into the individual workstations, software striped to RAID 0, to be used for caching drives. That leaves bays open in the FreeNAS server for higher capacity drives.

The density problem remains though - 1TB per disk simply isn't enough space for us. When these drives hit 2TB at that price point, then it'll be something worth considering in the server. But for now, it's not cost effective to buy 6-8 of these (roughly what we'd need to hold a typical job in progress).

Thanks,

-perry