Need help: Docker, ZFS & SAP HANA

[nthu9280]

zfs experts -
I have a Ubuntu 18.04.1 LTS host with ZFS hosting /var/lib/docker and it generally works fine. I did experience some quirks but that can be another topic.

I'm trying to setup SAP HANA express edition docker and ran into an issue. It's not a supported config by SAP. But someone has posted a work around but that is not quite working for me.

Installing SAP HANA, express edition with Docker - persistence on zfs · Issue #2234 · SAPDocuments/Tutorials

I created a 200G image as described above and mounted but still getting the error.

dd if=/dev/zero of=HANA.img bs=128k seek=1638400 count=0
mkfs.ext4 HANA.img
mount HANA.img /mnt/hxe1data
chown hxeadm.sapsys /mnt/hxe1data
chmod 666 /mnt/hxe1data

I'm using the latest release if that makes a difference
store/saplabs/hanaexpressxsa:2.00.033.00.20180925.2
The base OS is Ubuntu 18.04.1 LTS
Docker -
Server Version: 18.06.1-ce

Snippet of error log:

Change hardware key...
nameserver hxehost:39001 not responding.
Opening persistence ...
assign failed with persistence startup error. exception 1: no.2000004 (Basis/IO/FileAccess/impl/FileDescriptors.cpp:334)
Cannot open file ""/datavolume_0000.dat" ((closed, mode= RW, access= rw-rw-r--, flags= DIRECT|MUST_EXIST|LOCK), factory= (root= "/hana/shared/data/HXE/mnt00001/hdb00001/" (access= rw-rw-r--, flags= AUTOCREATE_DIRECTORY, usage= DATA, fs= UNKNOWN (zfs), config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=64,min_submit_batch_size=16,max_submit_batch_size=64))", rc=22: Invalid argument
exception throw location:

 stopping service...
error: Unknown persistence layer error
failed. trace file nameserver_hxehost.00000.000.trc may contain more error details.
~~~

*****

 

[Evan]

I had no idea such a toy was available for HANA, cool. Sorry can’t help to fix that error though.

 

[nthu9280]

Yes, HANA Express & HANA Express XSA are both free upto 32GB license even for production use.

Developer license has higher limit 64gb memory instance but the hassle of renewing every 90 days and can't be used for production purpose.

I think I'm close to get the docker version working. HANA doesn't like ZFS. Just need to present the volume for persistence as described above in a supported fs such as ext4. Don't know what I'm doing wrong.

 

[gigatexal]

For testing maybe try chmod 777 on the file for now. Perhaps the docker user doesn’t have perms to the file you created?

 

[nthu9280]

I've already tried that.

 

[gigatexal]

I've already tried that.

Hmm... maybe the mailing list if there is one ?

 

[marcoi]

did you do the basic troubleshooting steps? Sometimes the obvious is easily overlooked.

Check the storage area? IE can you write a file, read it, edit it, etc.?
Can you temporary unmount the zfs storage and use local disk to confirm if its a ZFS issue vs container issue?

 

[nthu9280]

For now, I'll shelve the docker approach and use a VM instead.

I'm presenting ext4 FS (/mnt/hxe1data) for volume but the install inside the docker still sees it as ZFS and don't know how to work with it.
-v /mnt/hxe1data:/hana/mounts \
~~
Cannot open file ""<root>/datavolume_0000.dat" ((closed, mode= RW, access= rw-rw-r--, flags= DIRECT|MUST_EXIST|LOCK), factory= (root= "/hana/shared/data/HXE/mnt00001/hdb00001/" (access= rw-rw-r--, flags= AUTOCREATE_DIRECTORY, usage= DATA, fs= UNKNOWN (zfs)
~~

 

[marcoi]

Maybe your missing something at the host level that tells dockers what kind of mount it suppose to be?
IDK if related but maybe how you configure the mount and using the /etc/fstab?

Maybe you need to configure the mount in fstab with different settings.

Currently im using freeNas and NFS mount to my ubuntu images using the below line in fstab
dev-freenas:/mnt/Storage/ /Software/ nfs auto,noatime,nolock,bg,nfsvers=3,intr,tcp,actimeo=1800 0 0

I'm not an expert on fstab and not sure if you are using that or not is your issue but I figured i mention it.

 

[nthu9280]

did you do the basic troubleshooting steps? Sometimes the obvious is easily overlooked.

Check the storage area? IE can you write a file, read it, edit it, etc.?
Can you temporary unmount the zfs storage and use local disk to confirm if its a ZFS issue vs container issue?

Guess I can mount another drive and format ext4 and present that for volume for testing. However, /var/lib/docker is on ZFS. Someone has done this as described in the link I posted in my OP. So like you said I'm probably overlooking something simple

I've done basic testing of creating a test file and deleting it etc.

Spoiler: Permission test

root@ubnt-x10sra1:/home/nthu9280# su hxeadm
hxeadm@ubnt-x10sra1:/home/nthu9280$ cd /mnt/hxe1data/
hxeadm@ubnt-x10sra1:/mnt/hxe1data$ ls
lost+found
hxeadm@ubnt-x10sra1:/mnt/hxe1data$ touch test.txt
hxeadm@ubnt-x10sra1:/mnt/hxe1data$ ls
lost+found  test.txt
hxeadm@ubnt-x10sra1:/mnt/hxe1data$ rm touch.txt
rm: cannot remove 'touch.txt': No such file or directory
hxeadm@ubnt-x10sra1:/mnt/hxe1data$ rm test.txt
hxeadm@ubnt-x10sra1:/mnt/hxe1data$ ls
lost+found
hxeadm@ubnt-x10sra1:/mnt/hxe1data$ cd ..
hxeadm@ubnt-x10sra1:/mnt$ ls -l
total 12
drwxr-xr-x 3 root   root   4096 Oct 29 19:45 disk1
drwxr-xr-x 3 root   root   4096 Oct 30 09:45 hana1
drwxrwxrwx 3 hxeadm sapsys 4096 Oct 31 11:48 hxe1data
hxeadm@ubnt-x10sra1:/mnt$

 

[marcoi]

does an application like gparted show zfs mounts devices? If so maybe you can check to confirm it formated correctly as well?

 

[nthu9280]

I followed this to configure Docker to use ZFS.

Use the ZFS storage driver

It works ok for other test containers. I'm playing with Hortonworks HDP sandbox etc.

I do have a issue but that is documented.

~~
Storage Driver: zfs
Zpool: error while getting pool information strconv.ParseUint: parsing "": invalid syntax
Zpool Health: not available
~~

Spoiler: docker info

# docker info
Containers: 4
 Running: 0
 Paused: 0
 Stopped: 4
Images: 85
Server Version: 18.06.1-ce
Storage Driver: zfs
 Zpool: error while getting pool information strconv.ParseUint: parsing "": invalid syntax
 Zpool Health: not available
 Parent Dataset: zpl2/docker
 Space Used By Parent: 256438519920
 Space Available: 5491961909104
 Parent Quota: no
 Compression: off
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc nvidia
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-38-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 220.3GiB
Name: ubnt-x10sra1
ID: VLTZ:ETKS:DO4D:6IZP:MHRW:3SQZ:FY72:U5JC:YZOJ:WAZV:YF2D:EVRF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: nthu9280
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support