Search results

  1. N

    I want some QSFP+ switch, what should I look at?

    Routing In/Out of tunnels, you don't need this unless you're into large data center networks.
  2. N

    I want some QSFP+ switch, what should I look at?

    you're right, 7050QX-32S is still in the HCL. Just haven't seen any posts about running SONiC on these recently. High-end I meant something current/interesting that can do RIOT and better telemetry (Trident 3/Spectrum/Tomahawk2)
  3. N

    I want some QSFP+ switch, what should I look at?

    SX6012 (the most compact apart from being cheap), requires the most tinkering. Arista seems to be okay for your needs. High-end switches supporting SONiC are either expensive (SN2010/2100) or loud (Celestica).
  4. N

    Remove ONIE and install MLNX-OS

    Unfortunately, I can't find it anymore, and I haven't saved it. @boliniak1 has also been offline for 1.5 years...
  5. N

    Cross-flashed Mellanox CX354A-QCBT to FCBT only run at QDR (32Gb/s)

    Hey WadeW, What does iblinkinfo show? Have you ever seen FDR10 on the switch working? Connecting back-to-back (HCA to HCA), do you get a better rate?
  6. N

    Cross-flash HPE ConnectX-6 VPI

    Cross-flashed from HPE0000000035 to HPE0000000034 and then to MT_0000000223, card is alive @1G link. As expected, the differences between HDR100 and HDR are minimal. Card is really hot, reaching 100C in under 2 min without cooling in a big tower, had to mount 50x50x10mm fan on top. FS4 image is...
  7. N

    Patrick's 2021 Fiber Build-out

    https://forums.servethehome.com/index.php?threads/suggestion-of-quiet-10-gbe-switches.21801/post-207669 SN2100s seem to be okay for your use case, relatively quiet and small, start with 16 ports and expand if necessary. My guess, the smallest and the most quiet 100G product so far.
  8. N

    HPE 620QSFP28 4x 25GbE Single QSFP28 Port Ethernet Adapter

    https://www.servethehome.com/hpe-620qsfp28-4x-25gbe-single-qsfp28-port-ethernet-adapter-review/ this one, right? What's on the other side? Similar adapter or a switch? Are you using a known good cable? What do you have in ethtool output, do you get any signal levels (TX, RX), what do you have in...
  9. N

    Cross-flash HPE ConnectX-6 VPI

    QSFP56 @PCI 4.0 x16 is capable of 200G, would be interesting to see if it's binned chips that can't reach 200G, or a continuation of our favorite QCBT-FCBT reflash.
  10. N

    Connectx-7 Datasheet

    Thanks! The biggest question remains - how many Mpps and at what latency... Definitely no IPSec decryption at 400G.
  11. N

    Cross-flash HPE ConnectX-6 VPI

    Does --no_fw_ctrl do anything? Could you post a couple of pictures with jumpers and debug interfaces (i2c, jtag)? https://docs.mellanox.com/m/view-rendered-page.action?abstractPageId=12005445
  12. N

    Infiniband/opensm on Debian strange behaviour

    General rule of thumb is to have a running SM as daemon somewhere in the fabric. SRP, for example, won’t establish new connections if SM is down, iSER wants to open an RDMA session which requires SM as well. Do you run some weird SM, or it runs once and exits?
  13. N

    Infiniband/opensm on Debian strange behaviour

    This partitions.conf limitation is essentially a rate limiter to IPoIB multicast groups (which is declared, but it’s a challenge to go beyond 30Gb/s). If you use native RDMA for your SAN like SRP, you can leave that partition at default values, IPoIB is not used. If you want to experiment, I’d...
  14. N

    2x 56GbE Optimization, slow ceph recovery, and MLNX-OS S.O.S.

    I wouldn’t expect great performance on a NAT’ted interface. Did you have the same on IPoIB? Also, what are you buffers set to (Ethtool -g)? Bond config seems correct on both sides, but I’d start without it while debugging.
  15. N

    Infiniband noob with some routing issues

    This could indicate a firewall or a routing issue on pfsense, IPoIB doesn’t seem to be broken. ip r s, iptables dump would be great to see
  16. N

    Infiniband noob with some routing issues

    So you can ssh over IB from pfsense to some other servers? Can you ping pfsense over IB?
  17. N

    Juniper MX204

    These are good, a single line card strapped to a server. Complex MX chassis have more opportunities to break (fabric issues, ksync not working, hardware components compatibility, software upgrades). I had more insights on Juniper platforms in mailing lists and Juniper forum, not many here have...
  18. N

    SX6036 Unresponsive

    There are normally 2 partitions you can boot from. Is the boot loader available? Then you could download a test image, mount the main partition and read the dmesg/other logs. If you’re not sure this tinkering is worth it - return/sell for parts, debugging can be tedious and still end with the...
  19. N

    Any value in Mellanox InfiniHost III Ex MHGA28-XTC Rev 4A for IPoIB?

    IPoIB can get you around 30 Gbit/s on mlx4 without deep tuning(obviously less on DDR) but I just wouldn’t invest my time in these ancient cards unless I had some retro vibes.
  20. N

    Any value in Mellanox InfiniHost III Ex MHGA28-XTC Rev 4A for IPoIB?

    Woah, PCIe Gen.1, as you have Arch I believe you can make it work with rdma-core. SDR/DDR cables are damn ugly and cant be reused with modern cards. If something goes wrong I’d dump these and buy CX5s.