AMD EPYC 7401P CentOS 7.4 benchmarks - cpu soft lockups?

Discussion in 'Processors and Motherboards' started by eva2000, Feb 26, 2018.

  1. eva2000

    eva2000 Active Member

    Joined:
    Apr 15, 2013
    Messages:
    242
    Likes Received:
    48
    Finally managed to do some testing with AMD EPYC 7401P on CentOS 7.4 with both default 3.10 Linux Kernel and 4.15 Linux Kernel. But I ran into some cpu soft lockups during UnixBench test and was wondering if anyone else have run into these before Benchmarks - Packet.net bare metal cloud provider review & benchmarks ?

    packet-amd-epyc-carbon-03.png

    Code:
    uname -r
    4.15.5-1.el7.elrepo.x86_64
    Code:
    48 x Dhrystone 2 using register variables  1 2 3 4 5 6 7 8 9 10
    48 x Double-Precision Whetstone  1 2 3 4 5 6 7 8 9 10
    48 x System Call Overhead  1 2 3 4 5 6 7 8 9 10
    48 x Pipe Throughput  1 2 3 4 5 6 7 8 9 10
    48 x Pipe-based Context Switching  1 2 3 4 5
    
    Message from syslogd@epyc at Feb 26 12:39:47 ...
     kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [sshd:72234]
    
    Message from syslogd@epyc at Feb 26 12:39:51 ...
     kernel:watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [context1:72109]
    
    Message from syslogd@epyc at Feb 26 12:40:03 ...
     kernel:watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kworker/8:0:4089]
    
    Message from syslogd@epyc at Feb 26 12:40:15 ...
     kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [sshd:72234]
    
    Message from syslogd@epyc at Feb 26 12:40:19 ...
     kernel:watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [context1:72109]
    
    Message from syslogd@epyc at Feb 26 12:40:31 ...
     kernel:watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kworker/8:0:4089]
    seems raising the kernel.watchdog_thresh value from 10 seconds to 20 seconds fixed it though. But haven't seem anyone mention this yet and whether it's unique to AMD EPYC + CentOS combo on ELrepo mainline 4.15 kernels ?

    Also seems there were definite performance improvements switching from CentOS 7.4 3.10 Kernels to 4.15 mainline Kernels i.e. building binutils 2.30 + GCC 8.0.1 profile guided optimization based RPMs for CentOS 7 were ~38% faster on newer Kernel :)
     
    #1
  2. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,952
    Likes Received:
    861
    I haven't seen the soft lockup. The 4.1x kernel is what you've gotta do.
     
    #2
  3. eva2000

    eva2000 Active Member

    Joined:
    Apr 15, 2013
    Messages:
    242
    Likes Received:
    48
    Yup using 4.15.6-1 by default for CentOS 7.4.

    @Patrick What's the best way to check AMD EPYC's cpu frequency and turbo scaling on CentOS 7 ? turbostat is showing around 2.35-2.55Ghz for all core tests while 7401P is rated 2.8Ghz for all cores and 3.0Ghz for single AFAIK. This is during GCC 8.0.1 compile tests with make -j utilising all cpu threads.
     
    #3
  4. eva2000

    eva2000 Active Member

    Joined:
    Apr 15, 2013
    Messages:
    242
    Likes Received:
    48
    Reinstalled CentOS 7.4 with default 3.10 kernel and I see at 24 cpu thread load that cpu jumps to 2700+Mhz and 48 cpu thread drops to 2560Mhz

    Code:
            Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
            -       -       1363    50.05   2724    1996    128036
            0       0       2725    100.00  2725    1996    5436
            0       24      0       0.00    2702    1996    12
            1       8       2724    100.00  2724    1996    5039
            1       32      0       0.00    2694    1996    7
            2       16      2723    100.00  2723    1996    5038
            2       40      0       0.01    2699    1996    12
            4       4       2724    100.00  2724    1996    5036
            4       28      9       0.33    2709    1996    1125
            5       12      0       0.00    2682    1996    10
            5       36      2725    100.00  2725    1996    5442
            6       20      5       0.17    2730    1996    20
            6       44      2724    100.00  2724    1996    5041
            8       1       2725    100.00  2725    1996    5036
            8       25      0       0.00    2720    1996    6
            9       9       0       0.00    2707    1996    6
            9       33      2725    100.00  2725    1996    5034
            10      17      2725    100.00  2725    1996    5036
            10      41      0       0.00    2713    1996    9
            12      5       41      1.54    2689    1996    4953
            12      29      2724    100.00  2724    1996    5036
            13      13      0       0.00    2689    1996    6
            13      37      2724    100.00  2724    1996    5036
            14      21      0       0.00    2711    1996    6
            14      45      2725    100.00  2725    1996    5036
            16      2       2724    100.00  2724    1996    5036
            16      26      0       0.00    2702    1996    6
            17      10      2725    100.00  2725    1996    5036
            17      34      0       0.00    2722    1996    6
            18      18      2722    100.00  2722    1996    5036
            18      42      0       0.00    2665    1996    8
            20      6       0       0.00    2721    1996    6
            20      30      2725    100.00  2725    1996    5036
            21      14      1       0.03    2705    1996    91
            21      38      2725    100.00  2725    1996    5036
            22      22      0       0.00    2723    1996    6
            22      46      2724    100.00  2724    1996    5036
            24      3       2724    100.00  2724    1996    5036
            24      27      0       0.00    2685    1996    6
            25      11      2725    100.00  2725    1996    5036
            25      35      0       0.00    2719    1996    6
            26      19      2724    100.00  2724    1996    5036
            26      43      0       0.00    2694    1996    6
            28      7       2725    100.00  2725    1996    5036
            28      31      0       0.00    2706    1996    6
            29      15      2725    100.00  2725    1996    5036
            29      39      0       0.00    2726    1996    6
            30      23      2725    100.00  2725    1996    5036
            30      47      6       0.21    2724    1996    33
    
    Code:
    turbostat
            Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
            -       -       2561    99.96   2562    1996    245435
            0       0       2560    99.99   2560    1994    5298
            0       24      2560    99.98   2560    1994    5089
            1       8       2557    99.93   2559    1994    5192
            1       32      2562    99.94   2564    1998    5170
            2       16      2563    99.99   2563    1998    5088
            2       40      2563    100.00  2563    1998    5090
            4       4       2563    99.97   2564    1998    5090
            4       28      2563    99.96   2564    1998    5091
            5       12      2563    99.93   2565    1998    5091
            5       36      2564    99.96   2565    1998    5405
            6       20      2560    99.96   2561    1995    5092
            6       44      2560    99.96   2561    1995    5096
            8       1       2561    99.97   2562    1995    5089
            8       25      2562    100.00  2562    1995    5069
            9       9       2560    99.96   2561    1995    5086
            9       33      2560    99.93   2561    1995    5090
            10      17      2560    99.95   2562    1995    5088
            10      41      2560    99.95   2562    1995    5090
            12      5       2558    99.94   2560    1995    5119
            12      29      2558    99.93   2560    1995    5129
            13      13      2559    99.96   2560    1995    5089
            13      37      2560    99.99   2560    1995    5088
            14      21      2560    99.94   2562    1996    5167
            14      45      2561    99.98   2562    1996    5208
            16      2       2560    99.94   2562    1996    5091
            16      26      2561    99.94   2562    1996    5089
            17      10      2561    99.97   2562    1996    5090
            17      34      2561    99.95   2562    1996    5088
            18      18      2558    99.97   2558    1996    5089
            18      42      2558    99.98   2558    1996    5091
            20      6       2561    99.95   2562    1996    5089
            20      30      2561    99.94   2562    1996    5085
            21      14      2561    99.93   2562    1996    5128
            21      38      2562    99.98   2562    1996    5160
            22      22      2561    99.97   2561    1996    5089
            22      46      2560    99.93   2561    1996    5087
            24      3       2560    99.98   2561    1996    5152
            24      27      2559    99.93   2561    1996    5121
            25      11      2561    99.97   2562    1996    5087
            25      35      2561    99.95   2562    1996    5089
            26      19      2560    99.98   2560    1996    5088
            26      43      2559    99.95   2560    1996    5089
            28      7       2563    99.99   2563    1996    5087
            28      31      2563    99.99   2563    1996    5089
            29      15      2563    99.99   2563    1996    5088
            29      39      2561    99.93   2563    1996    5088
            30      23      2563    100.00  2563    1996    5087
            30      47      2563    100.00  2563    1996    5090
    
    and 1 cpu thread load hits the max 3.0Ghz turbo speed
    Code:
            Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
            -       -       65      2.16    2994    1996    7555
            0       0       3       0.11    2893    1996    516
            0       24      0       0.00    2717    1996    7
            1       8       0       0.01    2653    1996    16
            1       32      0       0.00    2690    1996    7
            2       16      0       0.01    2802    1996    13
            2       40      0       0.00    2567    1996    12
            4       4       91      3.03    2993    1996    269
            4       28      0       0.00    2630    1996    7
            5       12      0       0.00    2648    1996    11
            5       36      2       0.07    2891    1996    506
            6       20      0       0.00    2602    1996    7
            6       44      3       0.11    2745    1996    812
            8       1       0       0.00    2746    1996    7
            8       25      0       0.00    2722    1996    7
            9       9       0       0.00    2724    1996    7
            9       33      0       0.01    2703    1996    16
            10      17      0       0.00    2712    1996    7
            10      41      0       0.00    2714    1996    7
            12      5       0       0.00    2540    1996    8
            12      29      0       0.00    2594    1996    7
            13      13      0       0.00    2523    1996    7
            13      37      0       0.00    2574    1996    7
            14      21      0       0.00    2528    1996    7
            14      45      0       0.00    2590    1996    7
            16      2       0       0.00    2752    1996    5
            16      26      0       0.00    2707    1996    7
            17      10      0       0.00    2687    1996    7
            17      34      0       0.00    2716    1996    7
            18      18      0       0.00    2667    1996    7
            18      42      4       0.14    2959    1996    21
            20      6       0       0.00    2579    1996    7
            20      30      0       0.00    2648    1996    7
            21      14      1       0.02    2590    1996    66
            21      38      0       0.01    2667    1996    31
            22      22      0       0.00    2576    1996    7
            22      46      0       0.00    2643    1996    7
            24      3       0       0.00    2782    1996    5
            24      27      0       0.00    2716    1996    7
            25      11      0       0.00    2783    1996    8
            25      35      0       0.00    2684    1996    8
            26      19      0       0.00    2586    1996    7
            26      43      0       0.00    2651    1996    7
            28      7       0       0.00    2593    1996    7
            28      31      0       0.00    2653    1996    7
            29      15      0       0.00    2886    1996    7
            29      39      2994    100.00  2994    1996    5012
            30      23      0       0.00    2581    1996    7
            30      47      3       0.11    2970    1996    17
    
    So looks like it's something to do with either my 4.15 Kernel update or additional tweaks I did manually.
     
    #4
    Last edited: Feb 27, 2018
    Patrick likes this.
  5. eva2000

    eva2000 Active Member

    Joined:
    Apr 15, 2013
    Messages:
    242
    Likes Received:
    48
    Looks like it was due to my own tweaking of kernel boot command parameters - the culprit was setting idle=poll ! But that only gets me to 2.55-2.65Ghz max frequency and not 2.80Ghz rated max for AMD 7401P.

    Removing that idle=poll and reboot server shows turbostat

    at idle
    Code:
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       0       0.01    2797    1996    978     374     588     1.65    98.34
    0       0       2       0.06    2763    1996    182     10      167     0.11    99.84
    0       24      0       0.00    2822    1996    12      7       4       0.09    99.91
    1       8       1       0.02    2595    1996    110     11      96      3.15    96.83
    1       32      0       0.00    2735    1996    12      7       4       0.08    99.91
    2       16      0       0.00    2637    1996    14      10      5       0.10    99.89
    2       40      0       0.00    2722    1996    12      8       4       0.10    99.89
    4       4       0       0.00    2686    1996    14      8       4       0.12    99.88
    4       28      1       0.02    2906    1996    110     20      86      24.98   75.00
    5       12      0       0.00    2732    1996    12      6       5       0.08    99.92
    5       36      0       0.01    2758    1996    69      18      60      47.45   52.54
    6       20      0       0.00    2659    1996    17      13      4       0.11    99.88
    6       44      0       0.00    2743    1996    12      5       6       0.07    99.92
    8       1       0       0.00    2775    1996    12      8       4       0.11    99.89
    8       25      0       0.00    2874    1996    11      7       3       0.09    99.91
    9       9       0       0.00    2770    1996    11      7       4       0.09    99.91
    9       33      0       0.00    2880    1996    11      7       3       0.09    99.91
    10      17      0       0.00    2672    1996    13      5       7       0.07    99.93
    10      41      0       0.00    2907    1996    10      5       4       0.07    99.93
    12      5       0       0.00    2634    1996    12      9       3       0.12    99.88
    12      29      0       0.00    2662    1996    11      7       3       0.08    99.92
    13      13      0       0.00    2581    1996    11      8       4       0.10    99.90
    13      37      0       0.00    2737    1996    9       5       3       0.08    99.92
    14      21      0       0.00    2660    1996    12      8       4       0.09    99.90
    14      45      0       0.00    2743    1996    9       5       3       0.08    99.92
    16      2       0       0.01    2794    1996    15      10      5       0.11    99.89
    16      26      0       0.00    2790    1996    10      6       3       0.07    99.93
    17      10      0       0.00    2728    1996    11      5       4       0.07    99.93
    17      34      0       0.00    2851    1996    10      6       3       0.07    99.93
    18      18      0       0.00    2658    1996    12      5       5       0.07    99.93
    18      42      0       0.00    2899    1996    9       4       4       0.05    99.95
    20      6       0       0.00    2636    1996    12      5       4       0.08    99.92
    20      30      0       0.00    2728    1996    10      4       4       0.04    99.96
    21      14      0       0.01    2733    1996    15      6       9       0.08    99.92
    21      38      0       0.00    2794    1996    8       4       3       0.06    99.94
    22      22      0       0.00    2621    1996    10      7       3       0.07    99.92
    22      46      0       0.00    2735    1996    8       5       2       0.08    99.92
    24      3       0       0.00    2800    1996    10      8       2       0.09    99.91
    24      27      0       0.00    2714    1996    14      5       8       0.07    99.92
    25      11      0       0.00    2760    1996    9       6       3       0.07    99.93
    25      35      0       0.00    2841    1996    9       6       2       0.07    99.93
    26      19      0       0.00    2780    1996    11      6       4       0.07    99.93
    26      43      0       0.00    2860    1996    8       4       3       0.05    99.95
    28      7       0       0.00    2659    1996    10      7       3       0.06    99.94
    28      31      0       0.01    2729    1996    11      4       5       0.03    99.97
    29      15      0       0.00    2723    1996    13      10      4       0.14    99.86
    29      39      0       0.00    2810    1996    15      29      2       0.14    99.86
    30      23      0       0.01    2741    1996    14      15      4       0.13    99.86
    30      47      2       0.08    2964    1996    16      3       9       0.05    99.87
    
    1 cpu load @2994Mhz
    Code:
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       63      2.09    2993    1996    6182    454     681     2.26    95.64
    0       0       2       0.07    2675    1996    191     12      173     0.16    99.77
    0       24      0       0.00    2770    1996    18      9       8       0.12    99.87
    1       8       1       0.02    2592    1996    113     23      91      9.89    90.09
    1       32      0       0.00    2831    1996    16      9       7       0.12    99.88
    2       16      0       0.00    2552    1996    17      10      7       0.14    99.86
    2       40      0       0.00    2698    1996    16      10      5       0.14    99.86
    4       4       0       0.00    2662    1996    16      11      5       0.15    99.84
    4       28      1       0.02    2853    1996    123     26      85      47.88   52.10
    5       12      0       0.00    2764    1996    17      10      6       0.13    99.87
    5       36      0       0.01    2719    1996    74      19      64      46.07   53.92
    6       20      0       0.00    2596    1996    17      11      6       0.15    99.85
    6       44      0       0.00    2707    1996    16      8       7       0.11    99.89
    8       1       200     6.69    2994    1996    354     9       7       0.13    93.19
    8       25      4       0.12    2991    1996    25      10      7       0.13    99.75
    9       9       0       0.01    2725    1996    23      10      12      0.14    99.85
    9       33      0       0.00    2837    1996    15      8       6       0.10    99.90
    10      17      0       0.00    2662    1996    17      7       9       0.08    99.92
    10      41      0       0.00    2822    1996    15      7       7       0.10    99.90
    12      5       0       0.00    2658    1996    16      12      4       0.15    99.85
    12      29      0       0.00    2694    1996    14      8       5       0.12    99.89
    13      13      0       0.00    2654    1996    15      13      4       0.15    99.85
    13      37      0       0.00    2674    1996    14      7       6       0.09    99.91
    14      21      0       0.00    2544    1996    15      6       8       0.09    99.91
    14      45      0       0.00    2726    1996    14      6       7       0.07    99.93
    16      2       2789    93.17   2994    1996    4676    0       2       0.00    6.83
    16      26      0       0.00    2958    1996    15      7       6       0.10    99.89
    17      10      0       0.00    2701    1996    15      11      5       0.14    99.86
    17      34      0       0.00    2831    1996    13      7       5       0.10    99.90
    18      18      0       0.00    2648    1996    15      7       7       0.09    99.90
    18      42      0       0.00    2789    1996    13      7       5       0.09    99.90
    20      6       0       0.00    2626    1996    15      8       6       0.11    99.89
    20      30      0       0.00    2742    1996    13      7       5       0.09    99.91
    21      14      0       0.01    2661    1996    19      6       11      0.08    99.91
    21      38      0       0.00    2696    1996    13      7       5       0.09    99.91
    22      22      0       0.00    2614    1996    14      8       6       0.10    99.89
    22      46      0       0.00    2697    1996    13      7       5       0.08    99.91
    24      3       3       0.09    2958    1996    27      14      8       0.21    99.70
    24      27      0       0.00    2666    1996    17      6       9       0.10    99.90
    25      11      0       0.00    2744    1996    14      9       5       0.12    99.88
    25      35      0       0.00    2840    1996    12      6       5       0.08    99.92
    26      19      0       0.00    2714    1996    14      7       6       0.09    99.90
    26      43      0       0.00    2830    1996    12      6       5       0.07    99.92
    28      7       0       0.00    2667    1996    13      8       4       0.11    99.89
    28      31      0       0.00    2733    1996    12      6       5       0.07    99.93
    29      15      0       0.01    2713    1996    13      22      4       0.10    99.89
    29      39      0       0.01    2780    1996    12      9       3       0.10    99.89
    30      23      0       0.01    2684    1996    14      15      5       0.10    99.89
    30      47      3       0.10    2914    1996    17      3       8       0.03    99.87
    
    all 24 cpu threads load ~@ 2.63Ghz
    Code:
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       1327    50.48   2629    1996    126226  2260    2936    8.60    40.92
    0       0       34      1.28    2628    1996    4229    1686    2506    42.57   56.21
    0       24      2629    99.99   2629    1996    5052    10      0       0.01    0.00
    1       8       0       0.02    2624    1996    104     17      88      14.97   85.02
    1       32      2629    99.99   2629    1996    5053    10      0       0.01    0.00
    2       16      306     11.66   2628    1996    16      6       6       0.09    88.25
    2       40      2629    99.99   2629    1996    5051    14      0       0.01    0.00
    4       4       2629    99.99   2629    1996    5052    15      0       0.01    0.00
    4       28      1       0.03    2627    1996    150     38      102     58.41   41.56
    5       12      0       0.01    2629    1996    35      8       24      1.22    98.77
    5       36      2629    99.98   2629    1996    5053    10      0       0.02    0.00
    6       20      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    6       44      0       0.01    2627    1996    72      20      63      60.97   39.01
    8       1       2629    99.99   2629    1996    5053    14      0       0.01    0.00
    8       25      0       0.00    2622    1996    14      5       6       0.06    99.94
    9       9       2629    99.98   2629    1996    5053    10      0       0.02    0.00
    9       33      0       0.00    2632    1996    17      8       6       77.33   22.67
    10      17      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    10      41      0       0.00    2634    1996    16      6       7       0.08    99.92
    12      5       2629    99.99   2629    1996    5053    10      0       0.01    0.00
    12      29      0       0.00    2630    1996    15      6       6       0.08    99.92
    13      13      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    13      37      0       0.00    2629    1996    14      5       6       0.06    99.94
    14      21      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    14      45      0       0.00    2622    1996    17      7       7       77.78   22.21
    16      2       2629    99.98   2629    1996    5051    10      0       0.02    0.00
    16      26      0       0.00    2622    1996    16      8       5       0.12    99.88
    17      10      268     10.19   2629    1996    21      9       7       0.13    89.68
    17      34      2629    99.99   2629    1996    5054    15      0       0.01    0.00
    18      18      2629    99.99   2629    1996    5053    14      0       0.01    0.00
    18      42      0       0.00    2619    1996    14      5       6       0.06    99.94
    20      6       2629    99.99   2629    1996    5053    14      0       0.01    0.00
    20      30      0       0.01    2624    1996    22      6       12      0.10    99.90
    21      14      2629    99.99   2629    1996    5054    14      0       0.01    0.00
    21      38      0       0.00    2621    1996    16      6       7       0.08    99.92
    22      22      2629    99.99   2629    1996    5051    10      0       0.01    0.00
    22      46      0       0.00    2617    1996    15      6       6       0.08    99.92
    24      3       2629    99.99   2629    1996    5053    10      0       0.01    0.00
    24      27      0       0.00    2622    1996    18      6       9       0.10    99.90
    25      11      3       0.13    2620    1996    38      14      15      0.23    99.64
    25      35      2629    99.99   2629    1996    5054    14      0       0.01    0.00
    26      19      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    26      43      0       0.00    2631    1996    20      8       9       0.12    99.88
    28      7       2629    99.99   2629    1996    5053    10      0       0.01    0.00
    28      31      0       0.01    2624    1996    24      90      3       0.21    99.78
    29      15      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    29      39      0       0.02    2624    1996    39      19      17      77.45   22.53
    30      23      2629    100.00  2629    1996    5035    0       0       0.00    0.00
    30      47      3       0.12    2623    1996    29      7       13      0.11    99.78
    
    all 48 cpu threads load ~@2.55Ghz
    Code:
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       2536    99.98   2537    1996    242691  491     0       0.02    0.00
    0       0       2537    100.00  2537    1996    5109    0       0       0.00    0.00
    0       24      2536    99.97   2537    1996    5050    10      0       0.03    0.00
    1       8       2536    99.98   2537    1996    5064    13      0       0.02    0.00
    1       32      2536    99.97   2537    1996    5049    10      0       0.03    0.00
    2       16      2536    99.98   2537    1996    5055    11      0       0.02    0.00
    2       40      2536    99.97   2537    1996    5053    10      0       0.02    0.00
    4       4       2536    99.97   2537    1996    5054    10      0       0.03    0.00
    4       28      2536    99.98   2537    1996    5195    11      0       0.02    0.00
    5       12      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    5       36      2536    99.97   2537    1996    5049    10      0       0.03    0.00
    6       20      2536    99.98   2537    1996    5051    14      0       0.02    0.00
    6       44      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    8       1       2536    99.98   2537    1996    5052    15      0       0.02    0.00
    8       25      2536    99.98   2537    1996    5052    10      0       0.02    0.00
    9       9       2536    99.98   2537    1996    5052    10      0       0.02    0.00
    9       33      2536    99.98   2537    1996    5053    10      0       0.02    0.00
    10      17      2536    99.97   2537    1996    5049    10      0       0.03    0.00
    10      41      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    12      5       2536    99.97   2537    1996    5053    10      0       0.03    0.00
    12      29      2536    99.98   2537    1996    5052    10      0       0.02    0.00
    13      13      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    13      37      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    14      21      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    14      45      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    16      2       2536    99.97   2537    1996    5053    10      0       0.03    0.00
    16      26      2536    99.98   2537    1996    5051    11      0       0.02    0.00
    17      10      2536    99.97   2537    1996    5051    10      0       0.03    0.00
    17      34      2536    99.97   2537    1996    5053    10      0       0.03    0.00
    18      18      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    18      42      2536    99.97   2537    1996    5052    11      0       0.03    0.00
    20      6       2536    99.98   2537    1996    5053    10      0       0.02    0.00
    20      30      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    21      14      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    21      38      2536    99.97   2537    1996    5050    10      0       0.03    0.00
    22      22      2536    99.98   2537    1996    5054    10      0       0.02    0.00
    22      46      2536    99.98   2537    1996    5050    10      0       0.02    0.00
    24      3       2536    99.97   2537    1996    5050    10      0       0.03    0.00
    24      27      2536    99.97   2537    1996    5050    10      0       0.03    0.00
    25      11      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    25      35      2536    99.98   2537    1996    5053    10      0       0.02    0.00
    26      19      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    26      43      2536    99.98   2537    1996    5052    10      0       0.02    0.00
    28      7       2536    99.97   2537    1996    5053    11      0       0.03    0.00
    28      31      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    29      15      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    29      39      2536    99.98   2537    1996    5052    10      0       0.02    0.00
    30      23      2536    99.97   2537    1996    5052    11      0       0.03    0.00
    30      47      2536    99.98   2537    1996    5051    13      0       0.02    0.00
    
    Or am I reading the turbostat correctly ?
     
    #5
  6. eva2000

    eva2000 Active Member

    Joined:
    Apr 15, 2013
    Messages:
    242
    Likes Received:
    48
    So again it was my own manual tweaks I needed to remove for another kernel command boot parameter skew_tick=1 LOL
    24 cpu thread load now gives 2760Mhz clock speed
    Code:
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       1007    36.46   2761    2000    95200   5272    2920    26.23   37.44
    0       0       1978    71.50   2767    2004    3714    81      24      18.70   9.92
    0       24      0       0.01    2751    2004    92      83      17      20.88   79.52
    1       8       577     20.81   2774    2004    405     69      19      28.89   50.61
    1       32      1982    71.63   2766    2004    3712    83      26      21.03   7.45
    2       16      1987    71.84   2765    2004    3709    86      1       28.27   0.00
    2       40      1       0.03    2722    2004    89      44      45      15.83   84.51
    4       4       1987    71.85   2766    2003    3717    65      33      18.54   9.71
    4       28      1       0.05    2759    2003    192     80      100     72.80   27.50
    5       12      1986    71.82   2765    2003    3711    82      4       28.27   0.00
    5       36      1       0.04    2766    2003    271     70      205     10.00   90.29
    6       20      1985    71.81   2765    2003    3718    89      3       28.27   0.00
    6       44      1       0.03    2753    2003    205     97      105     45.95   54.34
    8       1       1985    71.80   2765    2003    3708    86      5       28.28   0.00
    8       25      0       0.01    2743    2002    106     86      16      20.98   79.30
    9       9       0       0.01    2739    2002    86      74      11      20.88   79.42
    9       33      1985    71.82   2764    2002    3726    88      15      20.94   7.33
    10      17      1984    71.78   2764    2002    3709    76      11      28.29   0.02
    10      41      0       0.02    2772    2002    117     86      23      21.08   79.18
    12      5       24      0.86    2738    2002    3381    1391    1969    55.93   43.51
    12      29      1983    71.78   2762    2001    3709    88      2       28.28   0.00
    13      13      1971    71.38   2762    2001    3691    84      1       28.68   0.00
    13      37      0       0.01    2765    2001    93      82      13      20.97   79.26
    14      21      285     10.19   2796    2000    602     72      7       41.08   48.93
    14      45      1981    71.75   2762    2000    3703    78      6       28.30   0.01
    16      2       1934    70.09   2760    2000    3712    128     50      29.02   0.95
    16      26      0       0.01    2751    2000    91      78      13      20.93   79.25
    17      10      1709    62.01   2755    2000    3248    99      12      28.57   9.49
    17      34      0       0.01    2694    2000    99      83      16      21.03   79.14
    18      18      1979    71.75   2758    1999    3711    90      1       28.30   0.00
    18      42      0       0.01    2720    1999    98      82      16      21.01   79.14
    20      6       1871    67.87   2757    1999    3518    82      2       32.17   0.00
    20      30      0       0.01    2776    1999    98      80      17      21.00   79.14
    21      14      1979    71.73   2759    1999    3709    83      2       28.30   0.00
    21      38      0       0.01    2769    1999    96      81      15      21.00   79.12
    22      22      1979    71.73   2759    1998    3710    84      1       28.30   0.00
    22      46      0       0.01    2776    1998    95      79      16      20.96   79.14
    24      3       1978    71.72   2758    1998    3709    82      2       28.31   0.00
    24      27      0       0.01    2733    1998    94      81      13      21.00   79.08
    25      11      1978    71.71   2758    1998    3710    87      1       28.31   0.00
    25      35      0       0.01    2787    1998    93      78      15      20.95   79.12
    26      19      2301    83.32   2762    1997    3712    20      4       16.69   0.00
    26      43      0       0.01    2752    1997    92      80      13      20.97   79.08
    28      7       1977    71.70   2757    1997    3709    82      2       28.31   0.00
    28      31      0       0.02    2791    1997    92      81      12      20.98   79.04
    29      15      1976    71.69   2756    1997    3710    87      3       21.89   6.42
    29      39      3       0.11    2790    1997    118     163     16      20.88   79.04
    30      23      1975    71.68   2756    1996    3710    85      1       28.32   0.00
    30      47      4       0.13    2787    1996    100     77      16      20.88   79.00
    
    48 cpu threads gives 2580-2603Mhz cpu clock speed - is this the max or should it be 2.8Ghz ?
    Code:
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       1773    68.42   2591    1988    167510  772     730     0.55    30.90
    0       0       1773    68.72   2580    1979    3506    26      27      0.18    30.84
    0       24      1722    66.84   2577    1980    3414    14      14      0.13    32.76
    1       8       1754    68.00   2579    1980    3473    18      12      0.11    31.63
    1       32      1710    66.36   2577    1980    3397    9       22      0.08    33.28
    2       16      1731    67.15   2578    1981    3421    12      7       0.09    32.50
    2       40      1698    65.90   2578    1981    3362    9       11      0.08    33.76
    4       4       1772    68.61   2583    1981    3539    16      52      5.42    25.74
    4       28      1809    69.87   2589    1982    3636    9       48      4.26    25.66
    5       12      1798    69.50   2587    1982    3545    8       15      0.07    30.21
    5       36      1794    69.38   2586    1983    3571    13      45      0.11    30.30
    6       20      1769    68.45   2584    1983    3489    16      6       0.09    31.25
    6       44      1808    69.81   2590    1983    3641    29      36      10.42   19.57
    8       1       1757    67.98   2584    1984    3476    17      19      0.11    31.70
    8       25      1793    69.28   2588    1984    3528    12      12      0.09    30.44
    9       9       1756    67.93   2585    1984    3461    9       12      0.07    31.81
    9       33      1791    69.18   2589    1985    3522    11      10      0.09    30.56
    10      17      1793    69.25   2589    1985    3537    18      16      0.22    30.36
    10      41      1770    68.42   2587    1985    3487    11      13      0.13    31.28
    12      5       1814    70.02   2591    1986    3568    16      15      0.16    29.67
    12      29      1783    68.91   2587    1986    3512    21      11      0.21    30.72
    13      13      1770    68.39   2589    1987    3478    19      6       0.16    31.30
    13      37      1782    68.80   2590    1987    3504    12      12      0.14    30.92
    14      21      1765    68.17   2589    1987    3473    18      9       0.18    31.52
    14      45      1780    68.70   2591    1988    3501    10      15      0.11    31.05
    16      2       1745    67.44   2588    1988    3448    25      20      0.25    32.18
    16      26      1740    67.24   2588    1988    3424    10      13      0.11    32.52
    17      10      1806    69.55   2597    1989    3533    20      5       0.15    30.19
    17      34      1716    66.30   2588    1989    3380    9       17      0.11    33.47
    18      18      1810    69.73   2596    1989    3543    20      7       0.12    30.04
    18      42      1707    65.98   2587    1990    3365    15      15      0.19    33.73
    20      6       1778    68.57   2592    1990    3484    20      6       0.15    31.18
    20      30      1747    67.46   2590    1990    3432    11      11      0.13    32.32
    21      14      1760    67.88   2592    1991    3453    21      9       0.19    31.84
    21      38      1786    68.82   2596    1991    3500    11      13      0.12    30.97
    22      22      1772    68.31   2594    1991    3474    15      10      0.15    31.47
    22      46      1807    69.49   2600    1992    3531    15      11      0.14    30.30
    24      3       1777    68.47   2596    1992    3482    14      13      0.12    31.34
    24      27      1749    67.44   2593    1993    3427    10      11      0.10    32.40
    25      11      1766    68.04   2596    1993    3462    17      13      0.19    31.72
    25      35      1713    66.05   2594    1993    3372    15      21      0.21    33.69
    26      19      1777    68.41   2597    1994    3477    20      10      0.19    31.36
    26      43      1819    69.88   2604    1994    3549    13      14      0.11    29.98
    28      7       1815    69.72   2604    1994    3551    21      14      0.24    30.01
    28      31      1803    69.31   2602    1995    3521    21      10      0.17    30.50
    29      15      1799    69.14   2602    1995    3509    20      6       0.16    30.68
    29      39      1801    69.20   2602    1995    3533    49      12      0.16    30.62
    30      23      1802    69.27   2602    1996    3516    19      8       0.14    30.58
    30      47      1793    68.87   2603    1996    3503    8       16      0.08    31.05
    
     
    #6
  7. Thias

    Thias New Member

    Joined:
    Mar 1, 2018
    Messages:
    1
    Likes Received:
    0
    Hi @eva2000 !

    I have also been testing some 7401P based servers on Packet.net for the last week or so.

    I have sent some real production traffic to them, and I can confirm that with the original 3.10.0 CentOS/RHEL kernel, the performance is decent, but that with a recent 4.15.x from elrepo the performance is MUCH better... BUT...

    ...I have also been seeing soft lockup messages with 4.15.x kernels, and in my case when they appear, the server has become completely unresponsive and requires a hard reboot.

    Code:
    Message from syslogd@epyc-web at Feb 28 18:18:42 ...
     kernel:watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [php-fpm:2080]
    
    Message from syslogd@epyc-web at Feb 28 18:18:42 ...
     kernel:watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [memcached:1935]
    
    Message from syslogd@epyc-web at Feb 28 18:18:42 ...
     kernel:watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [telegraf:5660]
    
    Message from syslogd@epyc-web at Feb 28 18:18:42 ...
     kernel:watchdog: BUG: soft lockup - CPU#31 stuck for 22s! [php-fpm:2136]
    
    Message from syslogd@epyc-web at Feb 28 18:18:42 ...
     kernel:watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [migration/43:271]
    Today I have come across https://developer.amd.com/resources/epyc-resources/epyc-white-papers/ and I'm poking around a few of the settings mentioned in the "Performance Tuning Guidelines for Low Latency Response on AMD EPYC™ Based Servers" document, in case it helps.

    Were you also "crashing" the server when the soft lockups messages were appearing, or was it recovering?

    Matthias
     
    #7
  8. eva2000

    eva2000 Active Member

    Joined:
    Apr 15, 2013
    Messages:
    242
    Likes Received:
    48
    Yeah 4.15 kernel and upstream had a few soft lockup bug reports listed here. Some are related to AMD Ryzen based systems and C6 states so not a far stretch to see AMD EPYC might have similar issues. I solved my soft lockup bug related to UnixBench tests by raising kernel.watchdog_thresh from 10 seconds default to 20 seconds Benchmarks - Packet.net bare metal cloud AMD EPYC 7401P review & benchmarks.

    Hehe that performance white paper was part of why my turbo frequencies were messed up following that guide for idle=poll / skew_tick=1 and other kernel boot parameters messed up AMD EPYC's ability to properly turbo boost ! So be aware !

    As to crash server needed hard reboot heh.
     
    #8
  9. wujj123456

    wujj123456 New Member

    Joined:
    Sep 20, 2017
    Messages:
    6
    Likes Received:
    0
    I am running into same problem with my Ubuntu 18.04 daily build. I haven't done any tuning with kernel parameters, but the few parameters you mentioned here isn't passed to my kernel cmdline. I am kinda surprised that 3.10 kernel works with EPYC. Or does CentOS 7.4's 3.10 have lots of backports to enable EPYC? With 48 threads, I also see 2.5GHz for a while except if I wait long enough (like half an hour or more, it eventually slowly drops down to 2.35GHz. So far I haven't found anyway of making 7401P reaching anywhere close to 2.8GHz.

    I described my system's behavior in a bit more detail in my own post because I failed to find this post before posting mine: https://forums.servethehome.com/index.php?threads/epyc-7401p-is-not-reaching-all-core-boost.19021.

    [0][21:33:24]wujj@S8026 ~ $ cat /proc/cmdline
    BOOT_IMAGE=/boot/vmlinuz-4.15.0-10-generic root=UUID=8f5f92d5-692c-4193-b26c-058809a2ca42 ro quiet splash vt.handoff=1
    [0][21:38:38]wujj@S8026 ~ $ uname -a
    Linux S8026 4.15.0-10-generic #11-Ubuntu SMP Tue Feb 13 18:23:35 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
     
    #9
  10. Hydrogen_Bombaklot

    Joined:
    Jul 16, 2019
    Messages:
    38
    Likes Received:
    3
    Any updates on this? 7401P's are pretty cheap these days, and it would be nice to know if this has been addressed somehow.
     
    #10
  11. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,569
    Likes Received:
    4,509
    We have something like 10x 7401P systems installed running Debian but have never seen this on any of them.
     
    #11
  12. Hydrogen_Bombaklot

    Joined:
    Jul 16, 2019
    Messages:
    38
    Likes Received:
    3
    Thanks, so you're hitting all-core boost just fine on them?
     
    #12
  13. eva2000

    eva2000 Active Member

    Joined:
    Apr 15, 2013
    Messages:
    242
    Likes Received:
    48
    haven't tried AMD EPYC since unfortunately - costly to test AMD EPYC in cloud on packet.net still heh
     
    #13
Similar Threads: EPYC 7401P
Forum Title Date
Processors and Motherboards AMD EPYC 7401P running temperatures Jul 18, 2018
Processors and Motherboards EPYC 7401P is not reaching all-core boost Mar 12, 2018
Processors and Motherboards AMD Epyc ROME Memory Population Performance - 4 or 8 RDIMMs? Dec 6, 2019
Processors and Motherboards Epyc 3000 platforms beyond mini-ITX Dec 1, 2019
Processors and Motherboards Possible to run EPYC gen1 in gen2 boards? Nov 20, 2019

Share This Page