If you remember, i did ask you if you can run sequential write speed test?I have not. I can disable the cache any time, but a benchmark wouldn't be very accurate with the number of VMs running on it. Once the build is complete, I can give it a try, since I plan on moving most of the high activity VMs to the NVMe SSDs.
If it's comfortable for you.
At my experience with HW RAID controllers - more accurate result i got then copy huge file from NVMe to RAID. But it was with my 64Gb RAM server and 90Gb file
I have no ideas, if it's possible get pure and "cacheless" result with your 1Tb RAM server