Mellanox ConnectX-3 help

[Stephan]

Just found this on my live Arch Linux diag stick, example:

lspci -vv 2>/dev/null | grep -E "^.+:.+\.|Lnk(Cap2?|Sta|Ctl2):"
...
05:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller (prog-if 01 [AHCI 1.0])
    Subsystem: JMicron Technology Corp. Device 0000
        LnkCap:    Port #0, Speed 8GT/s, Width x2, ASPM not supported
        LnkSta:    Speed 8GT/s, Width x2
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
...

A bit easier on the eyes maybe. stress-ng needs a bit of swap, then:

nice -n19 stress-ng --vm $(nproc) --vm-bytes 86% --vm-keep --vm-populate --vm-madvise willneed --verify -v -t 4h --tz --perf

86% is about the most I could use, before kernel out-of-memory killer started attacking worker processes. Watch with rasdaemon for ECC errors. To test TDP a Linux kernel compilation from RAM will put a few extra watts of load on the system but not torture RAM as much. All completely free and open source.

Loopback-Test is a bit involved but still easy. Server:

#!/bin/sh

echo Setup
ip netns add ns_server
ip netns add ns_client

ip link set enp1s0 netns ns_server
ip netns exec ns_server ip addr add dev enp1s0 192.168.1.1/24
ip netns exec ns_server ip link set dev enp1s0 up
ip netns exec ns_server ethtool -s enp1s0 speed 56000 autoneg off

ip link set enp1s0d1 netns ns_client
ip netns exec ns_client ip addr add dev enp1s0d1 192.168.1.2/24
ip netns exec ns_client ip link set dev enp1s0d1 up
ip netns exec ns_client ethtool -s enp1s0d1 speed 56000 autoneg off

ip netns exec ns_server iperf -s -B 192.168.1.1 -w 16M

echo ""
echo Teardown

killall iperf
killall bwm-ng
sleep 0.5

ip netns del ns_server
ip netns del ns_client

echo Done

exit 0

Client:

#!/bin/sh

while :; do
    ip netns exec ns_client iperf -c 192.168.1.1 -B 192.168.1.2 -P 2 -w 16M -t 300
    sleep 0.1
done

For CX3 I only ever use FDR == 56 Gbps capable DACs and fiber modules. Haven't switched to 100/200 Gbps because that really needs PCIe 4 or 5 x16 and gear for this is too steep on the cost per performance curve. Also, my god, what's with all those post-Ice-Lake bugs and performance regressions. Intel really needs to get its act together. Fire 90% of all those useless suits and hire true engineers.

 

[tinfoil3d]

what's with all those post-Ice-Lake bugs

I'm lagging on intel side, my latest intel is i7 10th gen. For servers that i have direct access to it's first gen scalable. What bugs are you referring to?

Fire 90% of all those useless suits and hire true engineers.

They already work elsewhere like IBM.

 

[tinfoil3d]

@Stephan Don't you hate these weirdest random names enp1dfxasd03asd18ads41248970891?
I thought like this is a "guaranteed unique name", (sometimes for SOME but not all USB-CDC ethernet adaptors it is enx[mac-addr] which is unique. But I once swapped that USB NIC to a port right next to it and... had to pull out keyboard and display again to figure out what's wrong, TLDR its name changes if you plug same NIC to a different physical USB port! Such unique, so code, very systemd, WOW!
So sometimes i use some custom scripts, and for these tests also simply assign names with ip l set enp... name p1 and p2 respectively.
And most of times just use distros that said NO to systemd

 

[Stephan]

@tinfoil3d Can't name users, let's say HPC. x4xx gen slower than x3xx gen in some in-house benchmarks. Some as of yet inexplicable worker crashes. Analysis is ongoing but I do not follow that chat group too closely.

The "random" names are actually deterministic. Use kernel boot parameter net.ifnames=0 to disable. I, on the other hand, am a fan of systemd. It allows me to put an extra security layer around publicly reachable daemons. So even if you manage to execute shellcode in one such daemon, you can't write anything to disk for persistence, or use system calls which would endanger security more. Check out unbound/unbound.service.in at master · NLnetLabs/unbound for an example how to toughen up a daemon.

 

[tinfoil3d]

Sure I use ifnames, and isn't setuid for 99.9% of daemons already supposed to "fix" many(not all) security issues?

 

[Stephan]

Not running as root but a lower-privilege user is only one tiny step towards security mitigation. With systemd you can take away many more possibilities that a normal user still has. Like look around in /home, talk to devices, spam syslog, try to elevate privileges by exploiting vulnerable systems calls which the daemon never needed, force entire daemon into a private /tmp to prevent symlink bait-and-switch attacks, talk to the network at all, etc. etc. Or you can forbid in principle to write anywhere in the entire system, even if its the daemon's home or it would be otherwise allowed, with exception of /var/spool/mail to send alerts through mail for example. So even if your filesystem has a privilege hole, the daemon can't use it. Really good to nail that box tightly shut.

On the other hand some daemons like chrony (for timekeeping) NEED special, extra privileges, namely to correct a systems time. Normal users can't so traditionally this would mean the daemon runs as root. Or best case, run as privileged process and worker process, communicating to each other over some method, i.e. to initiate a time stepping to correct time. With systemd, you can have the daemon run as a simple user, and give it just the one extra privilege through Capabilities namely to correct system's time.

I am using alot of other features, like the coupling with udev. I have a Alix 4d4 with a 4G modem, which also can receive GPS for time. If you just start gpsd, it will complain there is no device (yet!), maybe because it is still booting or the USB stack hasn't fully come up yet. Whatever. So I made a rule to start the gpsd systemd-service only when the device has appeared. And I use BindsTo= in the systemd-unit-definition so the service is stopped, should the device disappear. And I give it CAP_SYS_TIME so it can write to chrony's shm. This way I can keep time precise at +/- 1ms in the house, without resorting to more esoteric PPS-capable solutions.