Home Setup - Design changes

[marcoi]

freenas as client out to ubuntu as server

 

[Rand__]

well there is some of your theoretical speed that's getting lost...

With "iperf3 server setup on freenas / client on ubuntu 18.4", your bandwith 12,8 GB/s , thats going to be ~1300 MB/s (o/c the bandwith is no static value and may change every few seconds for unknown reasons, especially on esxi).

And just realised that your read perf was significantly lower than write which is exactly a scenario I have seen with CDM/FreeNas/SSD Pools before. I never found an a solution but @gea had provided a possible explanation at some point. I think I was performing tests with S3700's back then, might be this thread: https://forums.servethehome.com/index.php?posts/164693/ , unfortunately no time to actually look it up, sorry

 

[marcoi]

ok so i been messing with two ubuntu images using iperf3. both vms on the same host. I enabled tso and lro and jumbo frames. Tesitng using vmnet3 nics and storage switch where freenas is setup to use.
Info from: VMware Knowledge Base





So far this is the best test VM to VM.

 

[marcoi]

This is same VMs as above but doing it local so only one VM is involved.





Seems like there should be room for improvement on the VM to VM side still?

 

[Rand__]

Have you looked at cpu load at that time?

Dont think that this is it, but just to be sure

 

[marcoi]

Have you looked at cpu load at that time?

I looked at briefly. The host has 2 e5-2680 v2 CPU and only had the few test VMs running at the time. So I dont think im even causing the CPUs any real load.

 

[Rand__]

whats the commands you are running?

 

[marcoi]

server side iperf3 -s -f M -i 60
client side iperf3 -c IP address

 

[Rand__]

you could run multiple instances?
iperf3 at 40Gbps and above

 

[marcoi]

At three threads it failed. with two I got the below results.
I did disabled the mtu = 9k on the vm and esxi hosts and enabled tso and lro.

 

[Rand__]

So its not really scaling well.

Still you should be able to reach higher values with that network bandwith...

 

[marcoi]

I agree - I dont know if I should create a new Vswitch and move the vms onto it or reinstall ESXI.

 

[Rand__]

well vswitch is a quick test, but I'd be surprised if reinstall would help. Not sure we discussed that, but all bios settings/energy saving options are off?

 

[marcoi]

well vswitch is a quick test, but I'd be surprised if reinstall would help. Not sure we discussed that, but all bios settings/energy saving options are off?

i have the bios setting for optimized RAM and Dell Power to watt profile. Ill go back in and re-check.

 

[Rand__]

Both bios and esxi should be on max performance

 

[marcoi]

Both bios and esxi should be on max performance

worth a shot- so far Watts increased by 50-60 ish when i changed the bios to performance.

 

[marcoi]

slightly better performance with 2 threads with performance set in bios and esxi. I also removed the nic from the vswitch.






single performance



Not sure its worth the extra 5-60watts of power since it isnt that much more over prior setting.

 

[Rand__]

and cdm?

 

[marcoi]

i deleted the test storage but ill try again.

I did some other testing on my 2nd host. vm to vm on 2nd host






esxi host to esxi host seems to hit the 10GB limits

 

[marcoi]

i recreated my stripe pool 8x800 gb no slog and sync=standard
datastore to ESXI local host as ISCSI with 200GB drive added to w7