Search results

  1. N

    Time to switch to 200GbE ? (QSFP56 and/or OSFP vs QSFP-DD)

    From my experience with QSFP-DD: https://forums.servethehome.com/index.php?threads/go-on-admit-it-who-here-is-already-running-at-200gbe.30563/#post-320310 I'd recommend to stay on QSFP56, it's really backwards compatible and the DACs are cheap. Next one is QSFP112, can't say that OSPF dominates...
  2. N

    How to bridge Infiniband and Ethernet utilizing a Homebrew router

    Yes, it's easier than you think - just do it! Install opensm, configure IPoIB interfaces, your Linux will route packets between interfaces as it should.
  3. N

    RAM failing on new 6port router from AliExpress

    Doesn't seem like this basic UEFI will give you any advanced settings like timers. Anything you can change in CPU configuration that's related to memory? From the DDR5 standard, base frequency is 4000MHz, might be possible to set that in this UEFI. But also, just please take a look at the test...
  4. N

    How to bridge Infiniband and Ethernet utilizing a Homebrew router

    I'd recommend InfiniBand if you have primarily RDMA workloads (NVMeoF, SRP, SMB 3.0, AI/GPU Direct). Connecting your nodes to a switch makes sense, then you connect the switch to the "gateway" Linux box. If you have no RDMA, just go Ethernet. Sure, you can have a Linux machine as a gateway...
  5. N

    RAM failing on new 6port router from AliExpress

    ^from the MemTest manual. Look for test #13, there's some interesting info there. https://www.memtest86.com/downloads/MemTest86_User_Guide_UEFI.pdf Overall, from my experience, DDR5 is way more picky - have you tried relaxing timers? Does this tiny box do memory training during POST?
  6. N

    Mellanox/Nvidia Connectx-7 fw update

    No ideas as of now. The production images are fully encrypted, there are no visible sections. An upgrade is possible from a card which itself is encrypted and can do the decryption. I assume the worst, that HW_POINTERS sections could be used as additional security layer - full flash dump of a...
  7. N

    Very fun and different content today

    It's okay man, the burnout is real (last example ipspace.net), absolutely go for every "weird" opportunity, talk about powerline, take breaks and vacations, pick up the other hobbies. I knew the guy who connected cruise ships over BGP, if I get a hold of him will let you know. Re: video - lots...
  8. N

    Mellanox switch IB <-> ETH switching

    You can set the max MTU to 4092 in partitions.conf I actually meant Ethernet MTU, of course 9K is not available in datagram mode. Connected mode is perhaps only worth it on old CX2/CX3 cards, it's not great.
  9. N

    Mellanox switch IB <-> ETH switching

    Yes, you would need a VPI license (various ways to get it, you can search the forum). However, you would need to run OpenSM on one of the Proxmox nodes (maybe you are already) Yes. See the VPI configuration examples here on the forum. You're going to be running TCP/IP presumably, I'd...
  10. N

    Xyratex SBB enclosures and compatible controllers

    Do you have a copy still? The link is there, but it seems the file has been pulled. WinServer 2016,2019,2022 aren't booting on my controllers for some reason (graphics)? If anyone has access to HP ClusterStor downloads, old Cray Sonexion files, or AP-LS-1 Seagate files, would appreciate your help.
  11. N

    Problem installing onie seastone DX010, help

    tbh I'm behind on this one, at home still at 202012 which doesn't have auto negotiation. 202211 is a better candidate at this point.
  12. N

    Does a Mellanox SN2010 SFP28 port support 1GbE?

    1GbE SFP-T flashed for Cisco in a QSA adapter works on my SN2100 without issues. 10/100M however is not supported and won't work, that was the same with SX6012 (SwitchX-2 based).
  13. N

    Xyratex SBB enclosures and compatible controllers

    what's called SATIT2 here seems to be a Xyratex CS-6000-AB, a server with an LSI SAS adapter and a PCIe slot for extensions (IB/RoCE/iWARP/iSCSI with the correct HHHL NIC installed). Basically erase SSD from the StorSimple abandonware, install Linux or Windows Server, and start serving shares...
  14. N

    Xyratex SBB enclosures and compatible controllers

    Would you recommend SATIT? I'm choosing between it and IOM12 (NetApp or Oracle with Mini-SAS connectors), like Lot of 5 Sun Oracle IOM12 SAS3 I/O 7318548 489089M-1710CX0178 Controller Module | eBay. SATIT setup seems to be complex (guess you don't need to connect it to Azure anymore Deploy...
  15. N

    EMC Mellanox SX6005 and I²C for "external management"

    mst could be conflicting with mstflint, it has a closed-source alternative of mstflint. Can you try to do the same command with flint (ibswitches; flint -d lid-x q; )? "Can't find device id" looks like a error within your system - I could suggest stopping the current server, booting from a stock...
  16. N

    EMC Mellanox SX6005 and I²C for "external management"

    what about mstflint -d lid-6 q? which version of mstflint do you have? are you running OpenSM?
  17. N

    Picked up an RTX A5000 for £450, any fun ideas to use it for?

    It's a steal! But this blower man, it blows it scares me. What do you think of AIO?
  18. N

    EMC Mellanox SX6005 and I²C for "external management"

    the program that talks to switch to change firmware is flint or mstflint (open-source counterpart). It should be able to talk to the switch both in-band and out-of-band. For out-of-band, I've used MTUSB-1 as it's a supported device. Fan control is an advanced topic, the examples from other...
  19. N

    EU gone - Mellanox SN2100, "40G", 16 port €1500 (MSN2100-BB2F)

    Beware that this is not an officially supported SONiC platform - I had to do some modifications to make it work. With default Onyx it was working fine even at 100G. overall it’s my choice for a home lab switch - small, just enough ports, really quiet with stock fans, fans assembly can be easily...
  20. N

    Mellanox IB SB7800 Drive Failure

    if you're running 7800 in anything resembling a production environment, I'd rather buy support and RMA the switch. From the SX6012 experience and a few forum threads here, there are some manufacturing scripts that run on the device that populate the internal database which is located on a...