have you done any testing with raidix yet?
Yes, but not on our hardware. Bottom line: I saw those Ultrastar NVMEs fly. Impressive enough to plan with it for myself architecture-wise. So far - from what I saw - I can confirm that read performance is around 90% of the added value of all drives in a RAID5 NVME array. And write depends on the size of the RAID5 array, so basically 90% of the added value of n-1 drives write performance in an array. Kinda impressive.
I am curious as to how the raidix folks got to 55.8GB/s.
You can actually read it on their website. Basically they sat together with a bunch of mathematicians and rewrote RAID logic optimized for NVME drives.
Have you thought about using the mellanox 516-cdat card instead of the IB card?
I am a complete newbie to Mellanox cards - well not 100% but close to. I dont even have those cards up for now (but inserted and connected). So I dont know on the benefits with the 516 cdat. The 455A were plain a bargain once I saw pricing. Like: miss diner with a business partner twice and buy both cards
.
Are you using an IB switch?
No. So direct link for now. Mellanox and IB are for now my hobby. Then again I have some great ideas for business scenarios.
Exactly what I think. I want to push it as far as possible. So far I tested a whole lot on architecture and bottlenecks beforehand. Main issue in the end was single core performance. Thats why I wanted the 7443P - cost-effective, high single core performance, new chip design, PCIe 4.0, well and ofc AMD for the PCIe lanes should I upgrade to more NVME drives. Because once this really works, its time to think about scaling it up... And then 55 GB/s total performance is actually not the end of the story in a multi-user environment.
Single user, single threaded maybe even 15GB/s is possible (I wouldn't bet on it but consider it possible as a close call). But multi-user, asynchronous loads... by then only PCIe lanes are the limit.
I like the added twist of involving your son
It was a blast - he is telling in preschool that he built a server... he still does. Kinda funny.