raidix eraraid on 4 nvme - request your perf test

[lihp]

*REMARK* - Update follows (Friday eve or Saturday noon) - below numbers included caching.

Just putting up some benchmarks without additional comments. Request your own - discuss freely.

System environment:
Hardware

• EPYC Milan 7443P
• 256GB RAM 3200 MHz
• PCIe 4.0 (SuMi H122SSL-NT)
• 4x KIOXIA CM6-V Enterprise - 3DWPD Mixed Use SSD 3.2TB, U.3 (KCM61VUL3T20)

OS

• Ubuntu 20.04 HWE
• Raidix eraraid 3.4
• RAID5, standard values

Performance test inside Windows 10 VM (KVM):

 

[Rand__]

Not sure whats your goal here (information, advertisement, discussion), but I'd wonder

- how a single drive compares against this
- how regular OS software raid [mdadm, windows software raid] compares against this
- how other raid solutions (intel vroc [not on thios board o/c]) compare

-What the use case is? Secure storage for a single VM Host?
-How networked performance looks like (i.e. how much do you loose in a distributed setup)

-Whats the caching strategy of Raidix? Memory? What happens on powerloss?

 

[lihp]

Not sure whats your goal here (information, advertisement, discussion), but I'd wonder

Plain discussion only and ofc curiosity for additional input to improve. What buffled me on today's test run are the 40µs latency, which made me like: ooook, just post on sth and see for the opinion of others.

Apart from that, eraraid is free for up to 4 drives - so its an option for any home server.

- how a single drive compares against this

Good question - gotta see, once I can test it.

- how regular OS software raid [mdadm, windows software raid] compares against this

Way worse, but I didnt screenshot those several days ago. mdadm improved in read speed in this years update. Write speed sucks still with mdadm.

- how other raid solutions (intel vroc [not on thios board o/c]) compare

Sequential large file reads perform quite similar over solutions - up to -30%, but at 30GB/s, which is more theoretical, it doesnt matter. In real world applications, quite close to the 3rd screenshot - that is where solutions break usually.

-What the use case is? Secure storage for a single VM Host?

In my case it's twofold:
- Ahigh end block I/O workstation for graphics in general, DTP and video editing (burst high IO usage).
- High end network shares for directSMB.

-How networked performance looks like (i.e. how much do you loose in a distributed setup)

Thats the next part - I already have tests and setups, but not yet satisfied to show. The server is connected at 100G to the network - my goal (hope) is to saturate 100G with single-threaded, synchronous loads of large files (10GB - 2TB size).

-Whats the caching strategy of Raidix? Memory? What happens on powerloss?

Caching can be configured to some extent for eraraid. Generally I see transaction oriented continous reads, while writes happen in bursts. In these test cases its direct (raw) virtio access of the VM on eraraid. If you like, I can see for a more complete answer.

 

[ectoplasmosis]

I’m waiting for an 8-drive RAIDIX license before testing 8x Samsung PM9A3 1.9TB U.2 drives in a RAID0 array.

Could you possibly provide mdadm figures for your array? Would be interesting to know how much extra performance the $800 RAIDIX license buys...

 

[amalurk]

Here is a single Kioxia CM6-R in my Windows which is a lesser drive than your drives as it is only rated at 1DWPD. While your SEQ1M is impressive the rest isn't for a 4x drive array compared to this single drive.

 

[ectoplasmosis]

@lihp Your SEQ1M results also appear to be tainted by caching, as they are greater than the rated and tested combined speeds of the 4x drives, let alone in RAID5, and are therefore meaningless.

Can you test using fio?

 

[lihp]

@lihp Your SEQ1M results also appear to be tainted by caching, as they are greater than the rated and tested combined speeds of the 4x drives, let alone in RAID5, and are therefore meaningless.

This. You are 100% correct. (Sitting here blushing that I didnt cross-check once more after 1 month of testing).

Can you test using fio?

Will do, need some time.

PS - @ectoplasmosis : Except one value (SEQ1M Q8T1) the values are actually physically possible. Thats exactly where I should have seen it for a RAID5 array.
PPS: Will post a complete update ground up from raw host OS numbers to VM.

 

[Rand__]

@lihp Your SEQ1M results also appear to be tainted by caching, ...

The reason why i basically never run 1GiB CDM tests

 

[lihp]

The reason why i basically never run 1GiB CDM tests

Tbh: I dont know, why I didnt look twice - figure the low µ-latency plain got me really early in the morning. Then again nice reason to do it right - thats why I dont delete the thread, but instead update with new numbers, latest Sat noon.

 

[shoosh]

@lihp Hi there, I have been trying unsuccessfully to get a satisfactory (better than a single drive) software RAID10 (or 5/6) speed with 6 NVMe's.
I wanted to try RAIDIX Era 4 but the free license directed me to a google drive with just documentation. Still waiting on a response from them, but since it's been almost a year, what are your thoughts on running it in production?

 

[ectoplasmosis]

@lihp Hi there, I have been trying unsuccessfully to get a satisfactory (better than a single drive) software RAID10 (or 5/6) speed with 6 NVMe's.
I wanted to try RAIDIX Era 4 but the free license directed me to a google drive with just documentation. Still waiting on a response from them, but since it's been almost a year, what are your thoughts on running it in production?

RAIDIX are a russian company; their taxes directly and indirectly fund putin’s war machine. Hence, I ditched all instances of their software months ago.

 

[shoosh]

ooph. not good. have you found any good alternatives? I've tried Dell's S150, Windows server 2022 Storage Spaces, StarWind's SAN software. So far the best is SS (about 50-65% of single disk), worse is StarWind's by far (about 10-20% of single disk), but admittedly that could be a settings issue.

 

[lihp]

RAIDIX are a russian company; their taxes directly and indirectly fund putin’s war machine. Hence, I ditched all instances of their software months ago.

Nope, no longer. Raidix is now Xinnor, an Israel company.

Actually they decided on that move around March 2022, meaning I see their move as protest and not opportunism.

 

[lihp]

@lihp Hi there, I have been trying unsuccessfully to get a satisfactory (better than a single drive) software RAID10 (or 5/6

Works like a charm. Personally I had to learn much more about storage caching than I anticipated. Some findings:

1. RTFM - it's all in the manuals. Those are exceedingly good and precise.
2. Tuning: you must tune the RAID to your workload. You can tune it for low latency (eg. iSCSI, SRP, iNVME), high sudden sequential throughput, eg. Video editing), high steady low-threaded low latency throughput (database storage),... This actually takes some expertise and time: understanding your workload.
3. Kernel version: while Xinnor plays nice, you still need to make sure you dont upgrade kernel versions too early.
4. Implementation: you need to think closely how your storage interacts with your server.
   1. When to load the kernel module? (usually DKMS)
   2. When do you need to start eraraid?
   3. Use LVM or not? (gonna be tricky)
   4. Startup/Shutdown management - your dependancy increased: what if you use it as Samba-share, but smbd.service hangs on shutdown
5. Oftentimes your CPU can be limiting to seek times or throughput. So monitor closely. For testing I defined the CPUs to use - spread over chiplets (trying to avoid thermal throttling) and set those to performance and boost, while setting C1 on each.
6. ...

Bottom line: Know what you do. The difference in settings in raidix/Xinnor is HUGE for the workload you configure it for.

Some examples - 4 drives ( 6900MB/s read and 4200MB/s write, Kioxia CM6-V) :

• RAID 0: achieved like 24-26GB/s reads and 14-16GB/s writes. Steady. Without "Caching" (difficult word with a SW-solution).
• RAID 5: achieved total r/w of app. 3x single drive: 18GB/s read and 11-12 GB/s write.
• RAID 5: High multi-thread async load - handled 20.000+ parallel connections, error-free. In overload situations, eraraid doesnt write errors to disk, but prefers to lock down. In light load situations it recovers easily over time, in heavy ol it is lock down to prevent data loss. While uncool, data corruption doesnt happen.
• ...

@lihpspeed with 6 NVMe's.

4 drive license is free, 6 needs a test key. If you need one: holler me, I can get you one (no, I am not from Xinnor, former Raidix).

I wanted to try RAIDIX Era 4 but the free license directed me to a google drive with just documentation. Still waiting on a response from them, but since it's been almost a year, what are your thoughts on running it in production?

They moved from Russia to Israel, which is imho a good thing. In the Goggle documentation on eraraid 4.0 you will find the repository to connect to. Just follow step by step the installation guide.

In HPC environments eraraid is SICK. Fast as hell and reaches the physical maximum. Using it at that edge, needs tight administration: not due to eraraid problems but because such a high load is difficult for any system. Once you step beyond 20GB/s sustained in 24/7 critical workload (eg. dbs) you plain need to know what you do.

Depending on configuration, performance can range from 2GB/s to 24+GB/s (80K IOPS to 1.5M+ IOPS) on the same workload.

----------------------
!Production use: IMPORTANT!

1. eraraid is very stable - if you know what you do.
2. plan ahead and get the right server(s) and SSDs (stable performance like Kioxia CM6-V, Optane-drives etc)
3. Test and tune for your workload, esspecially: CPU (governor, C-state, boost and cores) and eraraid settings.
4. Implementation: think twice how you implement it. Many options.
5. Plan your workflows around your requirements.
6. Edit: choose your memory, CPU cores according NUMA-nodes wisely

!Home use: IMPORTANT!

1. RTFM
2. Test your needs
3. If you plan to use VMs, you need to think twice, so you dont burn the performance due to VM inefficiences.
4. If you plan to use it as hot storage or samba shares or SRP: understand eraraid first - RTFM Samba, SRP and eraraid again.

 

[lihp]

ooph. not good. have you found any good alternatives?

Xinnor, former raidix is now a pure Israel company.

I've tried Dell's S150, Windows server 2022 Storage Spaces, StarWind's SAN software. So far the best is SS (about 50-65% of single disk), worse is StarWind's by far (about 10-20% of single disk), but admittedly that could be a settings issue.

Tbh I didnt find any getting close to eraraid performance. Imho it is very mature and stable.

Yet cope with it: its a scissor, a razor, not a hammer. Meaning RTFM.

 

[lihp]

Hints on server sizing:

• Single threaded, sync loads are 1 core only loads. At 6-8+ current Enterprise SSDs you can put a core on an Epyc 7003 system at 100% load easily. In production for this you want a 7003+ Epyc (7443(P) for low budget imho minimum) or current TR Pro, optionally Ryzen 9 7950 (high single core performance, many cores, consistent performance over cores - no good workstation/server boards resp Enterprise servers yet for 7950+). For the 7950 your PCIe lanes are (too) limited at 6 SSDs max directly connected to CPU. Depending on drives (PCIe 3), up to 16(32) drives are imho max for consumer AMD-CPUs (JBOD SSD-controller).
• IF AMD creates a 7950X3D, I would love to test it with a 12+ drives optane P5800 array. I am quite sure the extra cache works wonders.
• SSDs: for production you need consistent performance. Samsung 1725/1735 are lacking here. Prefer WD, Intel, Kioxia.
• Plan your network: 10G Ethernet wont cut it, when serving over network. You want 100G, 200G or 400G RDMA links. Preferably Infiniband.

 

[shoosh]

WOW @lihp , oh my goodness thank you so much for taking the time to share all this.
I will certainly go over this.

The hardware environment I get to play with is:
7525 Dell server with dual AMD EPYC 7252 8c/16t. and the storage in question is 6 (soon to be 8) 8TB Samsung PM9A3 NVMe U.2 drives
with a dual 25Gbps nic
The whole thing is completely overkill compared to the environment needs, but no sense in leaving processing power on the table, so to speak.
Plus I'm learning as well.
I was initially thinking Win 2022 /Hyper-V with a CentOS 9 VM with xinnor eraid (then feeding storage to) Windows VM for fileshare/AD/DC/etc. to play nicely with the rest of the resources here.

I just sent in a request to Xinnor, but if you've got a test key I can mess around with Monday that would be awesome.

ok time to go RTFM on those you suggested! Thanks again.

 

[NISMO1968]

That's smart move! We work with federal contracts and have been “strongly advised” by the government to avoid dealing with Russian companies even if they are not legally bound by sanctions law. It's better to be safe than sorry!

RAIDIX are a russian company; their taxes directly and indirectly fund putin’s war machine. Hence, I ditched all instances of their software months ago.

 

[NISMO1968]

Nope, no longer. Raidix is now Xinnor, an Israel company.

Actually they decided on that move around March 2022, meaning I see their move as protest and not opportunism.

I've heard different story. RAIDIX sold their Russian business to some third parties, just to avoid being sanctioned. Xinnor, their new company based out of Haifa, Israel has launched worldwide operations with their original team and codebase. You don't expect anyone to build such a complex product from scratch just in two months, do you? IP. Customers. Partners. There was no non compete signed and no non solicitation agreement of any kind, wasn't it? This leads to a dilemma: either we deal with someone who's trying to avoid secondary (?) sanctions, or we run the risk of being sued for IP infringement.

Xinnor is old RAID wine in a fresh new bottle – Blocks and Files

A company called Xinnor has apparently sprung up out of nowhere with mature xiRAID software RAID technology that seemingly goes faster than the high-speed GPU-powered SupremeRAID card developed by GRAID. Xinnor, with the company itself based in Israel, is actually RAIDIX reinvented. The story...

blocksandfiles.com

 

[lihp]

I've heard different story.

Xinnor is old RAID wine in a fresh new bottle – Blocks and Files

A company called Xinnor has apparently sprung up out of nowhere with mature xiRAID software RAID technology that seemingly goes faster than the high-speed GPU-powered SupremeRAID card developed by GRAID. Xinnor, with the company itself based in Israel, is actually RAIDIX reinvented. The story...

blocksandfiles.com

Seriously spreading false rumours is exactly what Russia does. Interestingly you do the same. Coincedence?

So no, xinnor is not funding Russia or Russia's war at all, actually more on the funding side for #Ukraine. And I lack words for people like you.