1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Any bash text processing gurus?

Discussion in 'Linux Admins, Storage and Virtualization' started by Patrick, Jan 18, 2017.

  1. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    8,947
    Likes Received:
    2,622
    Working on my monero Docker container and ran into a roadblock.

    Using nproc was easy to get cores and use them as a base for threads to use.

    Now I need to get L3 cache and then take half the MB value and round.

    The math is easy. Being able to store a value for L3cacheMB is beyond me. I can see the number in lscpu, but parsing is another matter.

    Any ideas on how to do this?
     
    #1
  2. xhypno402

    xhypno402 New Member

    Joined:
    Dec 7, 2016
    Messages:
    10
    Likes Received:
    2
    you can use the following to get the cache amount

    Code:
    lscpu | grep L3 | tr -s " " | awk -F':' '{print $2}'
     
    #2
    Patrick and gigatexal like this.
  3. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    1,736
    Likes Received:
    242
    confirmed that works - newbie for the win. props.
     
    #3
  4. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    8,947
    Likes Received:
    2,622
    Thank you!
     
    #4
  5. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    8,947
    Likes Received:
    2,622
    Ok this is ugly but seems to be working:
    Code:
    $ lscpu | grep L3 | tr -s " " | awk -F':' '{print $2}' | cut -c 2- | rev | cut -c 2- | rev
    46080
    
    Thank you again @xhypno402 !
     
    #5
  6. TuxDude

    TuxDude Well-Known Member

    Joined:
    Sep 17, 2011
    Messages:
    611
    Likes Received:
    331
    If you're going to use awk anyways, which is a full on language of itself that can do math and such, why also involve so many other tools and pipes? @xhypno402 's solution can be simplified down to just:

    lscpu | awk '/^L3/ {print $NF}'

    And you can do further math or whatever within that awk statement too if required.
     
    #6
    gigatexal, T_Minus and Patrick like this.
  7. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    8,947
    Likes Received:
    2,622
    You should not underestimate how bad I am at text parsing!

    Sitting delayed on the plane at our gate at SFO but the above has it working on the 4 socket system so that is a good start.

    Thanks for the awk tip will see if I can edit en route.
     
    #7
  8. TuxDude

    TuxDude Well-Known Member

    Joined:
    Sep 17, 2011
    Messages:
    611
    Likes Received:
    331
    So you want L3 cache in MB as a final output? I just tested my line on a VM that thinks it has 2x dual-core chips (4 cores total), where the output is simply:

    20480K
     
    #8
    Patrick likes this.
  9. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    8,947
    Likes Received:
    2,622
    We pushed from the gate but I got it dividing by 1024 for MB, multiplying by number of sockets, dividing by 2 and rounding down.

    The program runs in 2MB L3 cache so the object was to get that number for the entire system to plug into the miner's thread parameter.

    Had to put away my laptop but will update what I have soon (even incorporates your suggestion @TuxDude ). It is working properly.

    Next step is creating the docker swarm and running the container as a service.

    Something to do after Whistler's slopes close.
     
    #9
  10. TuxDude

    TuxDude Well-Known Member

    Joined:
    Sep 17, 2011
    Messages:
    611
    Likes Received:
    331
    Sounds like this should do what you want then.

    lscpu | awk '/^L3/ {l3=sprintf("%u", $NF)/1024} /^Socket/ {sockets=sprintf("%u", $NF)} END {print l3*sockets/2}'

    Grabs both the L3 cache amount and number of sockets from lscpu output and stores them into variables, then does the bit of math you described at the end and prints out the result.

    Given this lscpu output:
    it outputs just the number 20:

     
    #10
    Patrick and tjk like this.
  11. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    405
    Likes Received:
    34
    nice, how are conditions ?
    have done some backcountry-hiking near german/austrian border today with bluebird and 80cm fresh from the weekend :)
     
    #11
  12. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    8,947
    Likes Received:
    2,622
    Rain at the bottom. Will see tomorrow how it is at higher elevations.

    Thanks @TuxDude
     
    #12
  13. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    1,736
    Likes Received:
    242
    Awk is well awkward -- i find this a bit easier to follow, but it probably won't work in really bare environment without python:

    check it out:

    repl.it

    Code:
    import os
    import pprint #optional
    
    lscpu = {}
    
    for line in list(os.popen('lscpu')):
      elem = line.splitlines()
      for e in elem:
        tmp = e.split(':')
        key = tmp[0]
        val = tmp[1].strip()
        lscpu.update({key:val})
    
    pprint.pprint(lscpu) #can comment out, for debug purposes
    print('Cache as an integer: ',int(lscpu['L2 cache'][0:-1])) #input any key here
    
    granted one could make this a lot better by allowing it to take input from the command-line and make it executable with a #/usr/python etc
     
    #13
    Last edited: Jan 19, 2017
    Patrick likes this.

Share This Page