ZFS native on Ubuntu 16.04 LTS

[rubylaser]

That's an old bug and doesn't seem to be applicable as you can get mpt2sas to load reliably with the 3.19 kernel. I am also able to use my m1015s and H310 without issue in baremetal all the way up to a 4.4 kernel (haven't tried anything newer).

I believe this is a module conflict between The hypervisor and some that was introduced in the newer kernels.

 

[whitey]

Wonder if I should update my ESXi hypervisors to 6.0 U2 now that it's out on the fleeting chance it helps. Was thinking it's time for a vSphere upgrade up from 6.0 U1 just have not had the few spare free hrs to do it yet.

Just bothers me that none of the other stg appliance running in vt-D in AIO mode give me this hell...never even had freenas barf on me like that other gents was recommending in the other thread abt a advanced config option for msi disabled...ONLY happens on newer Linuxs'

 

[rubylaser]

Wonder if I should update my ESXi hypervisors to 6.0 U2 now that it's out on the fleeting chance it helps. Was thinking it's time for a vSphere upgrade up from 6.0 U1 just have not had the few spare free hrs to do it yet.

Just bothers me that none of the other stg appliance running in vt-D in AIO mode give me this hell...never even had freenas barf on me like that other gents was recommending in the other thread abt a advanced config option for msi disabled...ONLY happens on newer Linuxs'

I wouldn't think that would help unless they mention something in the release notes about new Linux support, but if you are willing to give it a try, give it a go

 

[canta]

That's an old bug and doesn't seem to be applicable as you can get mpt2sas to load reliably with the 3.19 kernel. I am also able to use my m1015s and H310 without issue in baremetal all the way up to a 4.4 kernel (haven't tried anything newer).

I believe this is a module conflict between The hypervisor and some that was introduced in the newer kernels.

I do not believe hypervisor problem..

ubuntu move fron upstart (init script model) to rhel systemd
systemd is a daemon, and the sequence of loading system is not predictable,
old init script is predictable aka sequential

as I said
basically, mfg page id, got corrupted in PCI memory..

we are talking mpt3sas(merged with mpt2sas).

the interesting part, megaraid_sas is not affected.

 

[rubylaser]

I do not believe hypervisor problem..

ubuntu move fron upstart (init script model) to rhel systemd
systemd is a daemon, and the sequence of loading system is not predictable,
old init script is predictable aka sequential

as I said
basically, mfg page id, got corrupted in PCI memory..

we are talking mpt3sas(merged with mpt2sas).

the interesting part, megaraid_sas is not affected.

But he said that Rockstor (Centos) has the same issue, so it's not just Ubuntu. I run baremetal Ubuntu 14.04.4 with a 4.4 kernel and 2 of these cards in IT mode for my bulk media fileserver with no issue. Yesterday, I tried baremetal installs on three different hardware setups (MSI X99 SLI Plus, Supermicro X9SRL-F, and Intel S2600CP) across different versions of Ubuntu (14.04.4 and 16.04) and they all consistently initialized and all disks were available.

Hence, my belief this may be a hypervisor issue with newer Linux kernel where vt-d is used.

 

[Patrick]

I guess, aside from the ZFS issue, if you are installing Ubuntu server today the question is whether to use 14.04 LTS or 16.04 nightly. As bare metal, I have been fairly happy with 16.04 thus far.

 

[rubylaser]

I guess, aside from the ZFS issue, if you are installing Ubuntu server today the question is whether to use 14.04 LTS or 16.04 nightly. As bare metal, I have been fairly happy with 16.04 thus far.

I'd love to use the 16.04 nightly on my home fileserver, but I'm confused why the ZFS root install option appears to not be included in the Server versions (unless I completely missed it).

 

[Patrick]

I'd love to use the 16.04 nightly on my home fileserver, but I'm confused why the ZFS root install option appears to not be included in the Server versions (unless I completely missed it).

Yea I have not seen that either. Strange since the Proxmox guys have had this working on Debian for some time now.

 

[canta]

But he said that Rockstor (Centos) has the same issue, so it's not just Ubuntu. I run baremetal Ubuntu 14.04.4 with a 4.4 kernel and 2 of these cards in IT mode for my bulk media fileserver with no issue. Yesterday, I tried baremetal installs on three different hardware setups (MSI X99 SLI Plus, Supermicro X9SRL-F, and Intel S2600CP) across different versions of Ubuntu (14.04.4 and 16.04) and they all consistently initialized and all disks were available.

Hence, my belief this may be a hypervisor issue with newer Linux kernel where vt-d is used.

speaking on centos, does rockstar using vanilla centos? is using 7.X (systemd).
if rockstar using vanilla centos 7.X, I could answer that.
linux kernel is 3.1o on centos 7
just check rockstor blog -> we’ve also updated the default kernel to 4.3.3
if they are based on centos. they are out of scope on centos, I believe they did modification heavily.

as I know, out of sync systemd can cause memory pci corruption and other issues.

if you want to know the detail on mpt3sas(merge from mpt2sas) source code, grab 4.4.6 vanilla Kernel source code, and compared with 3.X mpt2sas source kernel.
mostly are identical, they just add support 2.5 model that 12G speed..

I do not believe this is the hypervisor with 4.4.X kernel. linux kernel 4.X is not new actually..
and still believe out of sync sequence in systemd

 

[whitey]

I do not believe hypervisor problem..

ubuntu move fron upstart (init script model) to rhel systemd
systemd is a daemon, and the sequence of loading system is not predictable,
old init script is predictable aka sequential

as I said
basically, mfg page id, got corrupted in PCI memory..

we are talking mpt3sas(merged with mpt2sas).

the interesting part, megaraid_sas is not affected.

POS systemd...grrr, I KNEW that beast was garbage :-D

 

[whitey]

But he said that Rockstor (Centos) has the same issue, so it's not just Ubuntu. I run baremetal Ubuntu 14.04.4 with a 4.4 kernel and 2 of these cards in IT mode for my bulk media fileserver with no issue. Yesterday, I tried baremetal installs on three different hardware setups (MSI X99 SLI Plus, Supermicro X9SRL-F, and Intel S2600CP) across different versions of Ubuntu (14.04.4 and 16.04) and they all consistently initialized and all disks were available.

Hence, my belief this may be a hypervisor issue with newer Linux kernel where vt-d is used.

Yep, happens on Rockstor as well (based on CentOS7, doesnt that use systemd as well...may be onto something)? Some gent over in rockstor forums even saw same oddity of backporting an older kernel, detected disks, rebooted into newer mis-behaving kernel and it worked for him, I can't deal w/ the unpredictable nature of that though.

 

[rubylaser]

Yep, happens on Rockstor as well (based on CentOS7, doesnt that use systemd as well...may be onto something)? Some gent over in rockstor forums even saw same oddity of backporting an older kernel, detected disks, rebooted into newer mis-behaving kernel and it worked for him, I can't deal w/ the unpredictable nature of that though.

But, it doesn't happen here for me in Ubuntu (didn't try Centos) even with both 14.04.4 or 16.04 baremetal? Where these Rockstor users baremetal or virtualized as well (if you recall)?

 

[rubylaser]

speaking on centos, does rockstar using vanilla centos? is using 7.X (systemd).
if rockstar using vanilla centos 7.X, I could answer that.
linux kernel is 3.1o on centos 7
just check rockstor blog -> we’ve also updated the default kernel to 4.3.3
if they are based on centos. they are out of scope on centos, I believe they did modification heavily.

as I know, out of sync systemd can cause memory pci corruption and other issues.

if you want to know the detail on mpt3sas(merge from mpt2sas) source code, grab 4.4.6 vanilla Kernel source code, and compared with 3.X mpt2sas source kernel.
mostly are identical, they just add support 2.5 model that 12G speed..

I do not believe this is the hypervisor with 4.4.X kernel. linux kernel 4.X is not new actually..
and still believe out of sync sequence in systemd

Good info I just don't understand why I don't see this in baremetal installs. Also, what would be the fix for this as I don't see any systemd related mpt2sas bug reports or fixes yet?

 

[whitey]

But, it doesn't happen here for me in Ubuntu (didn't try Centos) even with both 14.04.4 or 16.04 baremetal? Where these Rockstor users baremetal or virtualized as well (if you recall)?

Virtualized/vt-D, the rockstor dev's seemed less than enthusiastic in the long run getting to the bottom of it although they were quite helpful and dug in initially.

 

[chilipepperz]

More of an OP reply but I did my first Xenial install today from "current". No ZFS but a server that was KILLING me due to random odd bugs (no VGA output! Freeze on sudo apt-get update) is now working 100%

 

[whitey]

And the fix for me was:

mpt2sas.msix_disable=1 (for 4.3 or older kernels)
or
mpt3sas.msix_disable=1 (for 4.4 or newer kernels)

Added to kernel boot line parameters, adding here as well for prosperity's sake.

 

[unwind-protect]

I just switched to ZFS as the on-disk format for my USB drives (travel and off-site backup). I used to use HFS which is so-so supported by everything but I got annoyed at all the warnings in Linux.

OpenZFS for OSX seems to do well, and I didn't see any problems in Linux. Of course now the problem is that I have machines down to 8 GB RAM try to grog ZFS. So dedup is out. But I didn't have dedup before either.

In any case, I compared ZFS on FreeBSD and Linux before and it just obviously is the same code and runs as advertised. So I fully support these pushed for ZFS.

I just learned that ZFS now has mutable snapshots so that you can branch from a snapshot. I always wanted that for my PXE boot environment to quickly clone machine installs. So far only btrfs offered it to me. That's good now, and it also solves the previous problem that the snapshots were hard readonly and if you backed up a bogus file the only way to remove that was to copy out the entire snapshots, modify it and copy it back in. All the while that ZFS was out of normal operations.

 

[gea]

I just learned that ZFS now has mutable snapshots so that you can branch from a snapshot. I always wanted that for my PXE boot environment to quickly clone machine installs. So far only btrfs offered it to me. That's good now, and it also solves the previous problem that the snapshots were hard readonly and if you backed up a bogus file the only way to remove that was to copy out the entire snapshots, modify it and copy it back in. All the while that ZFS was out of normal operations.

Yes, Sun did a good job with ZFS with CopyOnWrite and Snapshots.
The exact procedure:

A ZFS snapshot is a freeze of a filesystem that can be done without a delay
and is always readonly.

You can create a clone from a snapshot.
A clone is a writable volume or file system whose initial contents are the same as the snapshot from which it was created. Creating a clone is nearly instantaneous and initially consumes no additional disk space. In addition, you can snapshot a clone.

You can use the zfs promote command to replace an active ZFS file system with a clone of that file system. This feature enables you to clone and replace file systems so that the original file system becomes the clone of the specified file system. In addition, this feature makes it possible to destroy the file system from which the clone was originally created. Without clone promotion, you cannot destroy an original file system of active clones.

 

[MicheleM79]

I'm talking about the kernel version. On a fresh 16.04 install modinfo on mine (previous page) was only showing 12.100.00.00 vs. 20.100.00.00 on yours. Just weird...

Hello everybody guys, I followed the thread coming from a search about mpt3sas and mpt2sas
driver. I know that in recent linux kernel (for sure in 4.4 of Ubuntu 16.04) there is a mpt3sas only
(no more mpt2sas) because there was a merge (Single driver module which supports both SAS 2.0 &
SAS 3.0 HBAs).
I've done the same modinfo test as rubylaser, and the result is 12.100.00.00, that is the last mpt3sas driver revision.
I'm using a SAS 2.0 HBA and so Ubuntu 16.04 is using the merged "mpt2sas part" of the mpt3sas module.
My question is: how can I know the exact version of the "mpt2sas" driver used in the 12.100.00.00 "mpt3sas" driver module?
I assume it's the 20.100.00.00, but how can we assure of this?
I would know because of the suggested match beetween firmware and driver used (recommended by LSI/Avago).
Many thanks in advance, sorry for english (not my native lang)

 

[whitey]

Update/more info

mpt2sas.msix_disable=1 (for 4.3 or older kernels)
or
mpt3sas.msix_disable=1 (for 4.4 or newer kernels)

Added to kernel boot line parameters, adding here as well for prosperity's sake.

Under ubuntu or CentOS using grub2 to make it stick edit the /etc/default/grub to the following:

GRUB_CMDLINE_LINUX_DEFAULT="mpt2sas.msix_disable=1"
(mpt3sas.msix_disable=1 for 4.4 or newer kernels)

Save file:

Then update grub2 files:

Ubuntu - update-grub
CentOS - grub2-mkconfig -o /boot/grub2/grub.cfg (BIOS based machines)
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg (UEFI based machines)