Oracle Solaris 11.4

Gremlin190

New Member
Dec 14, 2018
4
0
1
Pass through of an LSI 2308 with ESXI 6.7U1 and 11.4 doesn't seem to work. System just freezes
Tried copying the mpt files from omnios 15028 and get a bunch of 'invalid kernel relocation type' messages and the driver does not load.
Any suggestions or am I out of luck with this controller?
If I am out of luck, can someone recommend one that would work?

Snapshot of the freeze by using the “-m verbose -v” after $kern on file /rpool/boot/grub/grub.cfg
 

Attachments

Last edited:

Gremlin190

New Member
Dec 14, 2018
4
0
1
This has been an issue before with possibly *some* LSI 9207 cards[URL="https://forums.servethehome.com/index.php?threads/esxi-6u1-passthrough-with-solaris-11-3.8595/#post-78498"]ESXI 6U1 passthrough with Solaris 11.3[/URL]

I bought the IBM 1015 and cross flashed it to the LSI 92111 and all is fine. The card had worked for months in OmniOS

Everything is working better than expected. One thing I'd like to know is how I can get rid of the picket fence when writing. I tried adding write cache, but it still happens. I understand that it's a flush from memory, but I have over 550MB/s bandwidth and I am only writing at gigabit speeds

upload_2018-12-22_13-12-44.png
 

gea

Well-Known Member
Dec 31, 2010
2,513
851
113
DE
This is the effect of the writecache flush in Solaris every 5s.
Solaris collects all (small) writes in RAM for 5s and writes it then as a single large sequential write to pool. This is why performance goes to upper limit (write to RAM) and zero duting a flush to pool.

Open-ZFS (OmniOS) behaves different. There the cache is size determined, ex 4GB/ 10% of RAM. A flush to disk is initiated there when the cache is full not after a given time.

The result is that on short write peaks Solaris ZFS is much faster (ok, Solaris is mostly faster) while Open-ZFS has a more even write behaviour on a steady write load without reaching the upper limit but without the drops. The effective performance is the area covered by the graph over say your 60s.
 
Last edited:
  • Like
Reactions: sth

DedoBOT

Member
Dec 24, 2018
38
5
8
Is this real, i want to believe but its suspicious, zpool iostat 2 output:IMG_20190526_161104.jpg

10k +write iops?! 16 hdds in raid10, no dedicated cache drives. There is something fishy.
 

gea

Well-Known Member
Dec 31, 2010
2,513
851
113
DE
10k write operations in 2s means around 5k per second.
You are correct that this is far above of the pool's physical capability.

A single disk can give around 100 physical io iops. A pool from 8 mirror vdevs can give around 800 write iops and 1600 read iops. Your remark about a cache drive points into the right direction. But a cache drive (L2Arc) is only for reads.

Here you see the effect of the rambased write cache. On Solaris it has a size of around 5s of writes until it is flushed as a single large write operation to disk. Your 5k write operations are mainly writes to ram.

If you enable sync without an Slog where every write commit must land on disk your write operations per second should be more in the region of the physical capability of the pool.
 

DedoBOT

Member
Dec 24, 2018
38
5
8
Thank Gea for detailed explanation.
Default pool's setup,atime=off datasets. Ram 128 gb .
 
Last edited:

Evan

Well-Known Member
Jan 6, 2016
3,122
522
113
I wound up @gea estimate of 100 write IOP’s to maybe 150 though on the newer disk would be possible.
But a safe rule of thumb as pointed out is 100 iops per spinning disk.
 

DedoBOT

Member
Dec 24, 2018
38
5
8
Not bad for predominately second hand eBay's setup , except hdds and more ram .
Filled mostly with random files, multiplied. The goal was 80 percent, but exceeded it with little bit - 18 percents free space left:
aja-d63-64tb-16GBtest.jpg
aja's disk cache disabled , zfs sync disabled ,atime off,
Wife got new camera. I'm ready :)
 

Attachments

gea

Well-Known Member
Dec 31, 2010
2,513
851
113
DE
A little tuning may even improve performance

On Solaris:
enable Jumbo
increase tcp buffers

On Windows
enable Jumbo
With Intel nics: use newest Intel drivers and disable int throttelling in driver settings

On Switches
enable Jumbo

But even with current values, you are able to edit 4k videos from server.
Only care about pool fillrate with disks (due increased fragmentation).
ZFS becomes slower with pool fillrate and slow > 80%,
 
Last edited:

DedoBOT

Member
Dec 24, 2018
38
5
8
Yeh Gea thanks,
Played with the NICs drivers setting on both ends. At the Windows side the major difference made interrupt moderation from default "adaptive" to "low" and disabling the flow control . Thats the cost of under 5 percent CPU load . With other types of calculations offloaded to the CPU the load go above 10 percent and I skip them . Same with the Jumbos - the wifi and the wired LAN are in bridge - I want other devices to be able to connect to the storage and to the windows host too. Iperf3 gives me 9 Gb/s in both directions , its fine for me, I'm not jealous :)
About the pool filtrate - mmy next free day I will torture it with the test in conditions of 15, 10 and finally 5 percent free space.
Just wonder about the outcome . Then I will destroy the pool and create it again.