1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Memory usage madness (oom-killer starts at 50% of ram)

Discussion in 'Linux Admins, Storage and Virtualization' started by ISRV, Jan 30, 2017.

  1. ISRV

    ISRV Member

    Joined:
    Jul 11, 2015
    Messages:
    60
    Likes Received:
    8
    i have latest debian x64 on 32gb server with swap turned off.
    i want to use ram as much as possible and the rest - leave for system cache.
    everytihng was fine until i noticed that oom-killer hit my mysql daemon from time to time.
    it looks like this:

    [​IMG]

    mysql growing and growing until all used memory come close to 50% and then oom-killer making a strike.
    i also noticed that i never ever actually had used any more than 50% of my 32gb.

    looks like it's because of the /proc/sys/vm/overcommit_ratio default to 50 and /proc/sys/vm/overcommit_memory default to 0 - meaning kernel is making own decisions about overcommit memory or not.
    i've tried to set overcommit_ratio to 99, but it only works together with overcommit_memory = 2.
    which means overcommit is disabled and that's bad, because i have lots of processes that seem to declare to a kernel they need a huge amount of ram (but actually using very low amount of it), and the kernel should say - sure, no problem, i'll give you 100 gigs of ram when you need it :]
    so memory overcommit should be always enabled as it is by default.

    what i don't understand, is why mysqld can't keep growing and growing, taking about 31gb of the memory and only then after it cross the line, it should be killed.
    why this happens at approx 16gb checkpoint (50% of total)?

    i feel that i need to add 32gb of swap, so the system grand total become 64gb and this 50% will mean 32gb (the ram i actually have).
    but i don't want to use swap at all, how can i move that 50% point to 99% for example?

    anyone had such experience?
    when you run "free -g", your "-/+ buffers/cache used" showing more than half of the ram you have?
    because i've never saw anything more than 15
    so my system is actually using only 16gb and never more.
    yes, i know the other 16gb is also in use by the cache, but i'd like to sacrifice system cache to the processes like mysqld.

    any ideas?
     
    #1
  2. Jb boin

    Jb boin Member

    Joined:
    May 26, 2016
    Messages:
    49
    Likes Received:
    12
    Might be a NUMA related issue if its a multi CPU system.
     
    #2
  3. ISRV

    ISRV Member

    Joined:
    Jul 11, 2015
    Messages:
    60
    Likes Received:
    8
    no, it's single cpu.

    i've added 16gb swap file and swappiness=0
    now feel better a bit, at least no more oom-killing since then
    but my used memory still stuck at 50%. here's 24 hour graph:
    [​IMG]
    swap usage was 0 for some time, but started to growing:
    [​IMG]
    [​IMG]
    what else is wrong i don't understand? why processes can't use more than half of my ram?

    here's current meminfo:
    [​IMG]
     
    #3
    Last edited: Jan 31, 2017
  4. tuxflux85

    tuxflux85 New Member

    Joined:
    Dec 8, 2015
    Messages:
    1
    Likes Received:
    0
    #4
  5. ISRV

    ISRV Member

    Joined:
    Jul 11, 2015
    Messages:
    60
    Likes Received:
    8
    my kernel - 3.16.0-4

    i wonder if i'm the only one with such problem?
    anyone else is using latest debian and able to see in "free" command "-/+ buffers/cache used" more than half of all available ram?

    i understand the importance of system cache, but still i'd like to be able to give my mysqld the most of all memory.
    but for some mysterious reason kernel not allowing that.
     
    #5
  6. EffrafaxOfWug

    EffrafaxOfWug Radioactive Member

    Joined:
    Feb 12, 2015
    Messages:
    398
    Likes Received:
    140
    No idea why it's being so kill happy, but if you "disable" oom-killer for mysqld does it allow it to allocate more than 50%?

    Code:
    echo "-17" > /prod/`pidof mysql`/oom_adj
     
    #6

Share This Page