Anybody seen this container/ceph-fuse bug?

Discussion in 'Linux Admins, Storage and Virtualization' started by zdude, Apr 8, 2018.

  1. zdude

    zdude What is a Computer?

    Joined:
    Aug 29, 2017
    Messages:
    49
    Likes Received:
    7
    I am trying to mount a ceph-fuse client in a ubuntu 16.04 lxc container within Proxmox v5.1-46. I have gotten far enough that it begins to complain about kernel dentries before failing.

    Code:
        -9> 2018-04-08 23:23:36.597140 7ff01595f700  1 client.2979582 using remount_cb
        -8> 2018-04-08 23:23:36.598849 7ff01495d700  3 client.2979582 ll_getattr 0x1.head
        -7> 2018-04-08 23:23:36.598863 7ff01495d700  3 client.2979582 ll_getattr 0x1.head = 0
        -6> 2018-04-08 23:23:36.598876 7ff01495d700  3 client.2979582 ll_forget 0x1 1
        -5> 2018-04-08 23:23:36.598947 7ff01515e700  3 client.2979582 ll_getattr 0x1.head
        -4> 2018-04-08 23:23:36.598957 7ff01515e700  3 client.2979582 ll_getattr 0x1.head = 0
        -3> 2018-04-08 23:23:36.598965 7ff01515e700  3 client.2979582 ll_forget 0x1 1
        -2> 2018-04-08 23:23:36.599479 7ff01595f700 -1 client.2979582 failed to remount (to trim kernel dentries): return code = 32
        -1> 2018-04-08 23:23:36.599497 7ff01595f700 -1 client.2979582 failed to remount for kernel dentry trimming; quitting!
         0> 2018-04-08 23:23:36.600052 7ff01595f700 -1 *** Caught signal (Aborted) **
     in thread 7ff01595f700 thread_name:tester
    
     ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
     1: (()+0x6d4784) [0x55a824ac4784]
     2: (()+0x11390) [0x7ff021e2e390]
     3: (gsignal()+0x38) [0x7ff020bba428]
     4: (abort()+0x16a) [0x7ff020bbc02a]
     5: (Client::_do_remount()+0x303) [0x55a824680b83]
     6: (Client::test_dentry_handling(bool)+0x18f) [0x55a824680ebf]
     7: (()+0x20ebe3) [0x55a8245febe3]
     8: (()+0x76ba) [0x7ff021e246ba]
     9: (clone()+0x6d) [0x7ff020c8c41d]
     NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
    I wasn't able to find anything on google, any ideas here? The host has 128GB of ram, 60GB of which is free and the container has 8GB of ram, of which all but 70MB is free. (specified because alot of these errors appear to come form out of memory issues)

    UPDATE: I can verify it is not an issue with the fuse inside the container. I was able to use sshfs to fuse mount a directory without issue.
     
    #1
    Last edited: Apr 8, 2018
  2. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    10,966
    Likes Received:
    3,921
    I have one for ZFS rpool and Docker (not supported by Proxmox) that I documented the fix for, but I have not seen this one.
     
    #2
  3. zdude

    zdude What is a Computer?

    Joined:
    Aug 29, 2017
    Messages:
    49
    Likes Received:
    7
    Do you have a link to the fix you have documented? At this point I am reading into stuff that is only tangentially related to try finding a fix...
     
    #3
  4. zdude

    zdude What is a Computer?

    Joined:
    Aug 29, 2017
    Messages:
    49
    Likes Received:
    7
    Small update, I am diving into the ceph source code and compiling with debug messages in the troublesome methods. This is what I have gotten so far before the fuse client crashes.

    Code:
    2018-04-19 22:45:25.544 7fecf9371c00  0 ceph version 13.0.2-1552-ga4778e9 (a4778e9d62fd38eb1e4eba1cfb3bc27a2ae5e101) mimic (dev), process (unknown), pid 581164
    2018-04-19 22:45:25.544 7fecf9371c00 -1 init, newargv = 0x560d9424e760 newargc=7
    ceph-fuse[581164]: starting ceph client
    ceph-fuse[581164]: starting fuse
    mount: cannot remount /home/super/ceph/build/bin/ceph-fuse read-write, is write-protected
    I have narrowed it down to a call to libfuse that is failing at src/client/fuse_ll.cc:1203 Not sure what is causing the failure at this time, however I will continue plugging away at the issue when I have time.
     
    #4

Share This Page