Search results

  1. 40gorbust

    Ring network with dual port network cards without a switch

    Hmmm. While I think still it's feasible to have a ring-network even in 2019 (I think we only had one cat5 cable that died on us in 4 years) I do agree that it's looking for trouble. I've found a Dell 4810-ON switch with 48 ports @ 10 gbit (SFP+) and 4 ports @ 40 Gbit (QSFP+). Before I only...
  2. 40gorbust

    Ring network with dual port network cards without a switch

    I never thought about L2 bridging. Not sure how to set that up in both Windows nor Linux clients on these cards. Any tips in that direction would be appreciated. It would definitely make things much easier, no static routing, no dynamic routing using demons or even BGP which I know but don't...
  3. 40gorbust

    Ring network with dual port network cards without a switch

    You mean this? I whipped out my best drawing skills! I added optional 1Gbit links between the T3 hosts. The next challenge would be how to setup fail-over with static routing without using scripts to detected failed links. Is there some gateway discovering protocol that would follow the links...
  4. 40gorbust

    Ring network with dual port network cards without a switch

    Interesting the 'just connect hosts and forget about a switch' is actually discussed by vendors. Interesting. I'm less crazy than I thought. I've yet to build the setup (cards are coming, spread between office and home). I test between 2 servers ; with 2 DACS between them, from port 1A via 2A...
  5. 40gorbust

    Ring network with dual port network cards without a switch

    Interesting switch. I see it's nearly silent until it isn't. That's good. So no noise at night in the home-lab (if I need to keep the network running anyway). Issues: 16 ports. Full is full. No (high bandwidth) uplink so connecting two switching requires 2 bonded ports (for at least 20 Gbit)...
  6. 40gorbust

    Ring network with dual port network cards without a switch

    I have a few servers that I do like to connect with 10Gbit and later 40Gbit (Mellanox ConnectX-3 cards in ETH mode) but without an 'expensive' and loud switch. I can get the cards with 2 ports for a little bit more than cards with 1 port so the initial investment is nearly the same ; 6 cards...
  7. 40gorbust

    32x40GbE $400 OBO if you are crazy

    Noice. We can do that. Have to dig up a serial cable and enable a port or get a USB to serial converter but that's no problem. That is in IB or ETH mode ? I assume ETH but not sure. 99% speed is more than fine. Even >90% would be fine as it's a big improvement over 10 GBit ethernet. Indeed...
  8. 40gorbust

    32x40GbE $400 OBO if you are crazy

    https://forums.servethehome.com/index.php?threads/enable-56gbe-on-hp-649281-b21-mellanox-354-with-direct-connect.21632/ Looks like that will work. Good find, that's a very cheap card.
  9. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    I found a great article about iSCSI, iSer and SRP performance over ETH, IB etc iSCSI vs iSER vs SRP on Ethernet & InfiniBand
  10. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    Would normal ipoib benefit from a 56G link vs a 40G link or would it max out at the regular "lower" speed e.g. 19/20/21 Gbit ?
  11. 40gorbust

    32x40GbE $400 OBO if you are crazy

    Hi, I found the same switch on Ebay from a different vendor but he mentioned he doesn't have the Ubuntu password. Is it easy to hack/crack/start Ubuntu on the switch in single user mode like on a normal Linux PC? I mean it won't be as easy as this How to Reset the Root Password in Linux - Make...
  12. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    To reply to you all regarding scp (the tool) ; we were happy to get 37 Gbit at the end of the day with iPerf, learned around 20:00 (on Friday) that the speed of SCP was just 50 MB/sec from server to server with just a DAC between them, so no switch, from ramdisk to ramdisk, so we called it a day...
  13. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    I think the newer CPUs are more efficient with I/O than the older e5-2450s v1 we use. On the other hand we paid less than $100 each second hand so I'm not complaining. I saw 37 Gbit so I'm far from complaining. Now we're trying to understand why we see 50 MB/sec from ramdisk to ramdisk SCP copy...
  14. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    But .. real world kicks in and then we see 50 MB/sec download and upload over the 40G link. :eek: Back to the drawing board.
  15. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    And finally we are close to the maximum. 37 Gbit with 16 parallel connections! Anything lower and the speed drops, quite a bit. Now we have to check if this is useful in reality when doing single file-transfers etc. We managed this change/improvements by changing the cards from their existing...
  16. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    various speed tests with -P 4,8 and more In full ETH mode we managed to squeeze 27 Gbit/s out of the card. :) We applied all sysctl changes to the tcp/ip stack. cpupower freq info Turbo boost is enabled and available up to a nice 3.1 Ghz but even though the single core speed is higher, with...
  17. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    Yeah in the new mainboards we have Gen.3 x8 speed so not hitting any limits (7.9 GB/sec raw), in practice I see GPUs transfer around 6 GB/sec over PCIE 3 x8. I'll apply those OS tuning tips. Are your cards direct connected with a DAC to another card, or via a switch? Are your cards in IB or ETH...
  18. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    Thanks I read that ipoib is slower, but I have no idea IF that's what I'm using :) We tried to set the cards to ETH mode on the bottom slots, but , after switching the cable , there was no connection. I'll try again Friday, to see if I can set the top slots to ETH. I left them alone because they...
  19. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    1. Ok so far we tried in 3 different mainboards ; 1 x Intel S5500 (X58) PCIe Gen.2 X8 max with dual Intel Xeon E5645 cpus (2.4 Ghz) 1 x Intel S5520HC PCIe Gen.2 X8 max with dual Intel Xeon E5645 cpus (2.4 Ghz) 2 x Intel S2400GP2 PCIe Gen.3 X8 max with dual Intel Xeon E5-2470 cpus (2.3 Ghz)...
  20. 40gorbust

    [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G

    Impressive speeds for that CPU (usage). This also helps me to not give up if we cannot reach (near) 40 Gbit on these 2 servers. Danke!