IB - CPU Frequency Mismatches - Sun Falcon (Mellanox MHRH2A-XSR)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

gb00s

Well-Known Member
Jul 25, 2018
1,189
600
113
Poland
I have a little issue while testing InfiniBand connections. While system (Bios-wise) is set to performance and the CPU governance is set to performance, I still receive the message/alert/warning of CPU frequency mismatches. Both CPU's are E5-2673v3's. The cards are old 10EUR eBay findings of SUN Falcons (aka Mellanox MHRH2A-XSR's). Firmware is the last 2.11.2010 and I'm unable to get RDMA working.

Here is a test result with the alert/message/warning:
Code:
root@pve2:~# ib_send_lat 192.168.1.71
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF        Device         : mlx4_0
 Number of qps   : 1        Transport type : IB
 Connection type : RC        Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 236[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x01 QPN 0x0211 PSN 0x5fe420
 remote address: LID 0x02 QPN 0x0211 PSN 0x2d3b1c
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
Conflicting CPU frequency values detected: 3073.762000 != 2856.115000. CPU Frequency is not max.
Conflicting CPU frequency values detected: 3040.750000 != 2727.213000. CPU Frequency is not max.
 2       1000          0.74           9.07         0.77                0.81            0.31           1.70            9.07   
---------------------------------------------------------------------------------------
root@pve2:~# ib_send_bw 192.168.1.71
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF        Device         : mlx4_0
 Number of qps   : 1        Transport type : IB
 Connection type : RC        Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x01 QPN 0x0212 PSN 0x7dca80
 remote address: LID 0x02 QPN 0x0212 PSN 0xc1ba80
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 2967.297000 != 2432.842000. CPU Frequency is not max.
 65536      1000             3756.01            3755.94           0.052095
---------------------------------------------------------------------------------------
root@pve2:~#
Ok, I can drop the message with -F, but even aligning the CPU's to performance mode did not help or just lowered the gap of the CPU frequencies. Is there any more room for optimization in this case?

Thank you in advance.