The pny xlr8 are consumer ssds without powerloss protection (consumer ssd performance numbers drop in steady state)....
truenas with new intel p5800x optane
- Thread starter jpmomo
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
thanks for reading through my admittedly wandering post 
. You did seem to get to my main questions and confirm my assumptions that the ram/arc might help proxy the responses but in the end, the m.2 ssds will take a beating with any writes. Workload in this case would be copying files to the dataset via smb. I was sustaining around 16Gbps for the writes to the NAS. As you mentioned, this was probably hitting the arc so not really useful at this point. I am still working on setting up our testing sw which will allow for simulating a more demanding workload. I will also try and disable the arc if possible to get a clearer picture of the actual performance of the drives/config.
thanks again for taking the time to help.
. You did seem to get to my main questions and confirm my assumptions that the ram/arc might help proxy the responses but in the end, the m.2 ssds will take a beating with any writes. Workload in this case would be copying files to the dataset via smb. I was sustaining around 16Gbps for the writes to the NAS. As you mentioned, this was probably hitting the arc so not really useful at this point. I am still working on setting up our testing sw which will allow for simulating a more demanding workload. I will also try and disable the arc if possible to get a clearer picture of the actual performance of the drives/config.thanks again for taking the time to help.
good point and thanks for the heads up. I was aware of the lack of plp but didn't think about the drop off from steady state. this will probably come into play when trying to transfer really large data sets which will be part of these tests.
thanks.
I have some details here:
I think the 53G were with 4 or 6 Optane 900p's in a Mirror config, else same setup as in the last post of the thread i linked
I got it from provantage. I was trying to get a couple of the 400G versions but couldn't find any in stock. the 800G versions showed up for a couple of days at a few different sites. They have now seemed to disappear along with the intel 5510 nvme "normal" drives. Might be a chia thing!
Hello Rand,
Thanks for the link. that is what I have been searching for. It looks like you striped 12 sas3 drives for your pool/vdev and got around 40Gbps as measured by both windows perf mon and the fio output. what type of dataset did you use? ex. samba for smb shares.
It looks like you tested this setup with the following fio cmd:
fio --direct=1 --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=windowsaio --size=100G --bs=128k --iodepth=1 --numjobs=16 --rw=write --filename=fio.test -name win1
Can you help me understand some of the details of the cmd above? Where did you run this command from? I am assuming that you would run the command from some client machine on the 100G network and point it at the vdev that you want to test.
How do you ensure that you are testing the actual vdev and not just the ram?
You also mention that you use one xeon gold 5122 cpu. Isn't that just a 4c cpu?
Can you also elaborate on your comments above regarding getting around 53Gbps with 4 or 6 optane 900Ps in a mirror? Did you use the optanes in a separate vdev and test just those instead of the 12 striped sas drives? or was it some combination of all of them?
Thanks for taking the time again to help get me further down the road!
jp
Hi,
yes it was SMB, running on a windows Server as client, i.e. mount share via Samba, run on client box.
I see I said 5122, can't remember the exact setup, maybe this was after switching to the other board. Will need to see if I find some old notes.
Its likely that some of the performance was from memory, but I used a 100G test file size, so hopefully that excluded most of it; but in the end it shows what FreeNas can do with good hardware (and rdma), how many drives of what type/setup will depend on specific requirements.
Re 53Gbps - thats what I think (remember) was the maximum I hit in various experiments, and it either was with the same 12 drives or a mirror'ed optane setup, didnt have anything else at that time.
Combining both would not speed up things.
yes it was SMB, running on a windows Server as client, i.e. mount share via Samba, run on client box.
I see I said 5122, can't remember the exact setup, maybe this was after switching to the other board. Will need to see if I find some old notes.
Its likely that some of the performance was from memory, but I used a 100G test file size, so hopefully that excluded most of it; but in the end it shows what FreeNas can do with good hardware (and rdma), how many drives of what type/setup will depend on specific requirements.
Re 53Gbps - thats what I think (remember) was the maximum I hit in various experiments, and it either was with the same 12 drives or a mirror'ed optane setup, didnt have anything else at that time.
Combining both would not speed up things.
Thanks again for your help. I did some more research and found out the answers to some of my questions. the direct=1 param should force fio to bypass the ram. I tried your exact command first on a local drive on windows 10 with a single pm1733 ssd:
I then mapped a smb share from the TN server. I tried to run the same command and the test started but then the TN server would reboot. This happened repeatedly. I was using the latest build of TN ver 12.0U3.1. It has been very inconsistent with trying to map network shares created by TN. the permissions don't seem to work the same twice! I have read a few other posts regarding similar issues. I was able to map a share but it took several tries and I am still not sure how I was able to get it to connect. Using your fio script without any changes doesn't seem to work with my TN setup. I will try and tweak it when I can find some time (difficult with my current day job!) to see where it breaks in my setup. At least you have shown me a basic way to test the performance with some opensource tools.
If the server starts to reboot under load then there is either an OS issue or a HW issue.
Can you run the same command successfully locally on the box (substitute windowsaio through posixaio) ?
If so that would point to a network issue, if not a local/hw issue
You aren't per chance fluid in perl? I have an (very rough) perl script to automatically test different pool layouts with different log devices which used to run on FN11.3. Have not touched since so not sure it still does, but I wrote it exactly for situations like this where you have lots of drives and options and no idea what the best setup is. Its just dumping output to huge csv files, but you do get all commands it used so can copy & paste for trying individual parts...
I really should put it on Github one day...
Can you run the same command successfully locally on the box (substitute windowsaio through posixaio) ?
If so that would point to a network issue, if not a local/hw issue
You aren't per chance fluid in perl? I have an (very rough) perl script to automatically test different pool layouts with different log devices which used to run on FN11.3. Have not touched since so not sure it still does, but I wrote it exactly for situations like this where you have lots of drives and options and no idea what the best setup is. Its just dumping output to huge csv files, but you do get all commands it used so can copy & paste for trying individual parts...
I really should put it on Github one day...
I don't give up that easily! Got almost 90Gbps with your fio cmd at least on local storage.
