ZFS native on Ubuntu 16.04 LTS

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Correct, just vt-D passedthru HBA's from vSphere 6.0 U1 to recent kernel Linux VM's. I guess i could try baremetal but if you have already done that no point really...I have always and need to use these as AIO stg appliances and if all the other flavors of Illumos, FreeNAS, Solaris, and older ubuntu ZoL 14.04.3 work w/out ANY tweaks to .vmx or pci=realloc=off settings then i just cannot wrap my head arnd what is going on here. The paranoid man in me says 'they broke my sh|t cause they don't want me using it in vt-D mode' haha
 

rubylaser

Active Member
Jan 4, 2013
846
236
43
Michigan, USA
Correct, just vt-D passedthru HBA's from vSphere 6.0 U1 to recent kernel Linux VM's. I guess i could try baremetal but if you have already done that no point really...I have always and need to use these as AIO stg appliances and if all the other flavors of Illumos, FreeNAS, Solaris, and older ubuntu ZoL 14.04.3 work w/out ANY tweaks to .vmx or pci=realloc=off settings then i just cannot wrap my head arnd what is going on here. The paranoid man in me says 'they broke my sh|t cause they don't want me using it in vt-D mode' haha
I feel relieved slightly that it doesn't bother baremetal installs, but disappointed that it's not working virtually. Have you determined if it's all 4.x kernels that cause it to fail (sorry if you already covered this, I just don't remember)? If it's just the kernel, you could try installing an older 3.19 kernel in 14.04.4 and see if that corrects the issue.

Index of /~kernel-ppa/mainline/v3.19-vivid
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
I feel relieved slightly that it doesn't bother baremetal installs, but disappointed that it's not working virtually. Have you determined if it's all 4.x kernels that cause it to fail (sorry if you already covered this, I just don't remember)? If it's just the kernel, you could try installing an older 3.19 kernel in 14.04.4 and see if that corrects the issue.

Index of /~kernel-ppa/mainline/v3.19-vivid
I believe I have tried that but I will surely give it another whirl now. So you looking for 3.19 kernel AND pci=realloc=off in kernel boot line or leave it default?
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
WOWZERS, that worked, now the perplexing ?, why not in newer/recent kernels/distro releases. Seen this in rockstor and now latest ubuntu ZoL setups that I attempt.

ubuntu-lts-14.04.4-kernel-3.19.0-031900-generic-WORKING.png
 
  • Like
Reactions: rubylaser

rubylaser

Active Member
Jan 4, 2013
846
236
43
Michigan, USA
WOWZERS, that worked, now the perplexing ?, why not in newer/recent kernels/distro releases. Seen this in rockstor and now latest ubuntu ZoL setups that I attempt.

View attachment 1930
That's awesome. Is this without the pci=realloc=off option passed to the kernel? If this is the default kernel, it appears you have discovered a bug in the newer kernels and should probably report it, so it can get fixed for future kernel releases.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
That's awesome. Is this without the pci=realloc=off option passed to the kernel? If this is the default kernel, it appears you have discovered a bug in the newer kernels and should probably report it, so it can get fixed for future kernel releases.
Yep, no 'pci=realloc=off' option just advanced boot, select 3.19, done deal. Is there a lkml/bugzilla/git bug-tracking system, what's the best way to report?
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Welcome to strangeland...now if I reboot into 4.2 kernel it's working there, only reason I know that is I installed ubuntu-zfs and the modprobe was unhappy in 3.19 but a reboot into 4.2 and disks and zfs is working properly...but for how long.

Goodness gracious. I think some fellow in the rockstor community and even here on another thread mentioned this oddity as well.

EDIT: rebooted and it is hosed up again, quick reboot into 3.19 and immediate reboot back into 4.2 and all is well again...WTF? haha

I'm gonna snag a quick video to show you guy the calamity.
 
Last edited:

rubylaser

Active Member
Jan 4, 2013
846
236
43
Michigan, USA
  • Like
Reactions: whitey

rubylaser

Active Member
Jan 4, 2013
846
236
43
Michigan, USA
Welcome to strangeland...now if I reboot into 4.2 kernel it's working there, only reason I know that is I installed ubuntu-zfs and the modprobe was unhappy in 3.19 but a reboot into 4.2 and disks and zfs is working properly...but for how long.

Goodness gracious. I think some fellow in the rockstor community and even here on another thread mentioned this oddity as well.
Maybe a supporting module wasn't being loaded until you installed the 3.19 kernel? What version of mpt2sas does modinfo show?
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Link to video.

https://71.196.231.126/

servethehome/S3rv3th3h0m3 (zero in h0me, that doesnt format real well on here)

mpt2sas-bug-newer-linux-kernels.mp4
 
Last edited:

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Maybe a supporting module wasn't being loaded until you installed the 3.19 kernel? What version of mpt2sas does modinfo show?
Code:
root@ubuntu-lts-14:~# modinfo mpt2sas
filename:       /lib/modules/4.2.0-27-generic/kernel/drivers/scsi/mpt2sas/mpt2sas.ko
version:        20.100.00.00
license:        GPL
description:    LSI MPT Fusion SAS 2.0 Device Driver
author:         Avago Technologies <MPT-FusionLinux.pdl@avagotech.com>
srcversion:     F74E004728BCB0A8B19A944
alias:          pci:v00001000d0000007Esv*sd*bc*sc*i*
alias:          pci:v00001000d0000006Esv*sd*bc*sc*i*
alias:          pci:v00001000d00000087sv*sd*bc*sc*i*
alias:          pci:v00001000d00000086sv*sd*bc*sc*i*
alias:          pci:v00001000d00000085sv*sd*bc*sc*i*
alias:          pci:v00001000d00000084sv*sd*bc*sc*i*
alias:          pci:v00001000d00000083sv*sd*bc*sc*i*
alias:          pci:v00001000d00000082sv*sd*bc*sc*i*
alias:          pci:v00001000d00000081sv*sd*bc*sc*i*
alias:          pci:v00001000d00000080sv*sd*bc*sc*i*
alias:          pci:v00001000d00000065sv*sd*bc*sc*i*
alias:          pci:v00001000d00000064sv*sd*bc*sc*i*
alias:          pci:v00001000d00000077sv*sd*bc*sc*i*
alias:          pci:v00001000d00000076sv*sd*bc*sc*i*
alias:          pci:v00001000d00000074sv*sd*bc*sc*i*
alias:          pci:v00001000d00000072sv*sd*bc*sc*i*
alias:          pci:v00001000d00000070sv*sd*bc*sc*i*
depends:        scsi_transport_sas,raid_class
intree:         Y
vermagic:       4.2.0-27-generic SMP mod_unload modversions
signer:         Build time autogenerated kernel key
sig_key:        73:6E:2F:9E:A1:B4:72:8A:15:AC:16:9B:18:69:26:7E:11:28:D6:E8
sig_hashalgo:   sha512
parm:           logging_level: bits for enabling additional logging info (default=0)
parm:           max_sectors:max sectors, range 64 to 32767  default=32767 (ushort)
parm:           missing_delay: device missing delay , io missing delay (array of int)
parm:           max_lun: max lun, default=16895  (int)
parm:           diag_buffer_enable: post diag buffers (TRACE=1/SNAPSHOT=2/EXTENDED=4/default=0) (int)
parm:           prot_mask: host protection capabilities mask, def=7  (int)
parm:           max_queue_depth: max controller queue depth  (int)
parm:           max_sgl_entries: max sg entries  (int)
parm:           msix_disable: disable msix routed interrupts (default=0) (int)
parm:           max_msix_vectors: max msix vectors  (int)
parm:           mpt2sas_fwfault_debug: enable detection of firmware fault and halt firmware - (default=0)
parm:           disable_discovery: disable discovery  (int)
root@ubuntu-lts-14:~#
That's WITH it working btw after a quick reboot into 3.19 and back into 4.2, you're gonna laugh when you see the vid and scratch your head.
 

rubylaser

Active Member
Jan 4, 2013
846
236
43
Michigan, USA
Code:
root@ubuntu-lts-14:~# modinfo mpt2sas
filename:       /lib/modules/4.2.0-27-generic/kernel/drivers/scsi/mpt2sas/mpt2sas.ko
version:        20.100.00.00
license:        GPL
description:    LSI MPT Fusion SAS 2.0 Device Driver
author:         Avago Technologies <MPT-FusionLinux.pdl@avagotech.com>
srcversion:     F74E004728BCB0A8B19A944
alias:          pci:v00001000d0000007Esv*sd*bc*sc*i*
alias:          pci:v00001000d0000006Esv*sd*bc*sc*i*
alias:          pci:v00001000d00000087sv*sd*bc*sc*i*
alias:          pci:v00001000d00000086sv*sd*bc*sc*i*
alias:          pci:v00001000d00000085sv*sd*bc*sc*i*
alias:          pci:v00001000d00000084sv*sd*bc*sc*i*
alias:          pci:v00001000d00000083sv*sd*bc*sc*i*
alias:          pci:v00001000d00000082sv*sd*bc*sc*i*
alias:          pci:v00001000d00000081sv*sd*bc*sc*i*
alias:          pci:v00001000d00000080sv*sd*bc*sc*i*
alias:          pci:v00001000d00000065sv*sd*bc*sc*i*
alias:          pci:v00001000d00000064sv*sd*bc*sc*i*
alias:          pci:v00001000d00000077sv*sd*bc*sc*i*
alias:          pci:v00001000d00000076sv*sd*bc*sc*i*
alias:          pci:v00001000d00000074sv*sd*bc*sc*i*
alias:          pci:v00001000d00000072sv*sd*bc*sc*i*
alias:          pci:v00001000d00000070sv*sd*bc*sc*i*
depends:        scsi_transport_sas,raid_class
intree:         Y
vermagic:       4.2.0-27-generic SMP mod_unload modversions
signer:         Build time autogenerated kernel key
sig_key:        73:6E:2F:9E:A1:B4:72:8A:15:AC:16:9B:18:69:26:7E:11:28:D6:E8
sig_hashalgo:   sha512
parm:           logging_level: bits for enabling additional logging info (default=0)
parm:           max_sectors:max sectors, range 64 to 32767  default=32767 (ushort)
parm:           missing_delay: device missing delay , io missing delay (array of int)
parm:           max_lun: max lun, default=16895  (int)
parm:           diag_buffer_enable: post diag buffers (TRACE=1/SNAPSHOT=2/EXTENDED=4/default=0) (int)
parm:           prot_mask: host protection capabilities mask, def=7  (int)
parm:           max_queue_depth: max controller queue depth  (int)
parm:           max_sgl_entries: max sg entries  (int)
parm:           msix_disable: disable msix routed interrupts (default=0) (int)
parm:           max_msix_vectors: max msix vectors  (int)
parm:           mpt2sas_fwfault_debug: enable detection of firmware fault and halt firmware - (default=0)
parm:           disable_discovery: disable discovery  (int)
root@ubuntu-lts-14:~#
That's WITH it working btw after a quick reboot into 3.19 and back into 4.2, you're gonna laugh when you see the vid and scratch your head.
I just watched the video... That makes no sense at all unless there is some communication that is happening between the vm and the hypervisor that is fixed after starting up with 3.19. What happens if you try to reboot the 4.2 kernel again and pick 4.2? Does it continue to work, or does that break it again?

The other thing that's weird to me is that even on a fresh install of 16.04 Server, my mpt2sas version number was showing 12.100.00.00 not 20.100.00.00 like yours is (not that it matters, but I'm not sure why my version is so much older even in the latest daily build of 16.04).

The other weird thing is that the server install didn't provide me the option to install to ZFS root. I wonder if only the Desktop version supports this feature?
 
Last edited:

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
That breaks it, that's where I started the vid from I believe, a working 4.2 config, I reboot try to let it go back into 4.2, you notice the LONG hang on scatter elements for mpt2sas...something seriously unhappy, then it boots, ya see the failures from the dmesg output, no pools/filesystems, a quick reboot into 3.19 (very fast reboot) then a quick reboot immediately into 4.2 again and what do you know it boots FAST/no hang on gather scatter elements mpt2sas BS, and becomes fully functional again and everything looks golden in dmesg output again and zpool/zfs filesystems are again accessible.

Certainly does NOT leave a warm fuzzy feeling to run anything off of it.
 

rubylaser

Active Member
Jan 4, 2013
846
236
43
Michigan, USA
That breaks it, that's where I started the vid from I believe, a working 4.2 config, I reboot try to let it go back into 4.2, you notice the LONG hang on scatter elements for mpt2sas...something seriously unhappy, then it boots, ya see the failures from the dmesg output, no pools/filesystems, a quick reboot into 3.19 (very fast reboot) then a quick reboot immediately into 4.2 again and what do you know it boots FAST/no hang on gather scatter elements mpt2sas BS, and becomes fully functional again and everything looks golden in dmesg output again and zpool/zfs filesystems are again accessible.

Certainly does NOT leave a warm fuzzy feeling to run anything off of it.
No, that would make me feel very uneasy trying to run anything important off of it when it's that buggy.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
We talkin firmware on HBA or driver w/in kernel/OS? I have my HBA flashed to IT mode v20, looks like driver I thought was v20 as well. My eyes hurt :-(
 

rubylaser

Active Member
Jan 4, 2013
846
236
43
Michigan, USA
We talkin firmware on HBA or driver w/in kernel/OS? I have my HBA flashed to IT mode v20, looks like driver I thought was v20 as well. My eyes hurt :-(
I'm talking about the kernel version. On a fresh 16.04 install modinfo on mine (previous page) was only showing 12.100.00.00 vs. 20.100.00.00 on yours. Just weird...
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
I'm talking about the kernel version. On a fresh 16.04 install modinfo on mine (previous page) was only showing 12.100.00.00 vs. 20.100.00.00 on yours. Just weird...
You sure that's not a diff between mpt2sas and mpt3sas versions? I seem to have seen this at some point I could have sworn. You'd figure they just sunset mp2sas in favor of mpt3sas but I don't know the history or if one addresses older/newer generations devices.
 

rubylaser

Active Member
Jan 4, 2013
846
236
43
Michigan, USA
You sure that's not a diff between mpt2sas and mpt3sas versions? I seem to have seen this at some point I could have sworn. You'd figure they just sunset mp2sas in favor of mpt3sas but I don't know the history or if one addresses older/newer generations devices.
I queried them both, and they are both the same version.
Code:
root@xenial:~# modinfo mpt2sas | grep version
version:        12.100.00.00
srcversion:     3946C07EF122A6D7F0CF884
vermagic:       4.4.0-15-generic SMP mod_unload modversions

root@xenial:~# modinfo mpt3sas | grep version
version:        12.100.00.00
srcversion:     3946C07EF122A6D7F0CF884
vermagic:       4.4.0-15-generic SMP mod_unload modversions
 

canta

Well-Known Member
Nov 26, 2014
1,012
216
63
43
I posted on other thread -> Bug 92351 – mpt2sas fails to load for a LSISAS2008 card Bug 92351 - mpt2sas fails to load for a LSISAS2008 card ( on ubuntu) :p.
<= 3.4.0 ubuntu kernel works!!!!

definetly some issue on the sequence of low layer PCI in kernel that cause picky LSI2008 got corrupted firmware config info

starting 3.4 higher kernel, Ubuntu enabled :
CONFIG_PCI_REALLOC_ENABLE_AUTO=Y

this is the problem, this must be turned off on ubuntu...he.

post to ubuntu bugtracking, I has my instinct that this is ubuntu issue during building the kernel

btw. linux kernel megre mpt2sas to mpt3sas that would use identical kernel module :)

that could be out of sync when ubuntu start using systemd :D.....
 
Last edited: