Actually that's 5 (five) fans blowing between the 2 power supplies, which each also has a fan.Considering that there are 3 small diameter fans blowing between the 2 power supplies, I would say that it is likely VERY noisy.
No, ONIE is the open boot loader for switches - think of it as like GRUB2 for the switch firmware.ONIE means SONiC support right?
Well, it's a fun little switch if you are trying to get into the open networking thing, but the obvious issue is the learning curve and applicability to the shops that you will end up in - it could be extremely good for learning purposes, or it could end up being one of those things that you'll have piling up in the corner. I bought an Arista switch at a very good price not too long ago (so I can give both SoNIC and EOS a try), but frankly, with the end-of-the-year projects @ work + home (need to get a MiniUSB power injection cable so I can run an external DVD burner on an old PowerBook G4, TV popped a fuse, wife's battery swap turned into a damaged loudspeaker, damn those Motorola Athene phones), I didn't even have a chance to take it home and really play with it yet.So... as a guy that's getting out of the IT field (temporarily) because of ... leadership issues (theirs, which became mine obviously), would this be a good 'investment' to screw around with? Not that I have free time. However I'm not going to be traveling a ton, and 40gbE is exotic enough that there's a ton of stuff out there for cheap IF you pounce when it comes up.
-btw- both of ya'll in the firmware/enterprise discussion camp do great work. I'm a guy that loves digging into binaries, but I also switch the hat when I'm discussing hardware for enterprise. It's a fun ability to go back and forth and lets me set the BS detector off at appropriate levels. Unfortunately my leadership has a bit of 'turtle' mentality. Likes to 'trust the experts' (which means, not me). Didn't like if I asked if he only gets one quote on major house improvements. But that's as it goes.
... and this switch supposedly will run well with Cumulus Linux ...
Is there an easy/cheap way to get Cumulus Linux for ... evaluation/educational purposes?...and Cumuls Linux is pretty easy to navigate around... last job used nothing but penguin switches + CL.
Arista 7050SX-128 is Intel basedmeh, EOS is just x86 fedora with a lot of custom shit on top making BRCM API calls to the ASIC. if it were a different ASIC I would say no, but both these switches are the exact same Trident II. Should just be a recompile of coreboot for the intel SoC board
x86 = intel 8086 + successors (& clones)Arista 7050SX-128 is Intel based
Try reaching out to cumulus, they're stuff is licensed based, so they might be able to give you a trial key.Is there an easy/cheap way to get Cumulus Linux for ... evaluation/educational purposes?
It is obvious. But most of Aristas have AMD processors, not Intel. 7050SX-128 is small exception - not sure if aboot is the same for AMD and Intel based boxes.x86 = intel 8086 + successors (& clones)
Hi,Hi there, crazy checking in here I suppose. I purchased one of these a couple weeks ago and went ahead and purchased a second when I saw you lot found them
...
It is trivial to reset the password, it boots as a standard Ubuntu system and from the serial console you can access BIOS, GRUB, and Linux command line. In order to reset the password you simply hit `e` at the GRUB screen to edit the default boot entry, throw an `init=/bin/bash` in there, wait a bit, and then `passwd`.Hi,
I found the same switch on Ebay from a different vendor but he mentioned he doesn't have the Ubuntu password. Is it easy to hack/crack/start Ubuntu on the switch in single user mode like on a normal Linux PC?
I mean it won't be as easy as this How to Reset the Root Password in Linux - Make Tech Easier right ?
I have been able to sustain 37.7Gbps TCP/IP iperf3 via a single port of the D4040 in the following configuration:Can you confirm that you can run near 40 Gbit ETH speed on the switch between two nodes/servers/clients connected to the switch ? What is your experience with this fine piece of Chinese manufacturing after a few months ?
The necessary and optional configuration for the lag member interfaces:interface lag 10
no port-channel static
port-channel load-balance 6
adminmode
The lag configuration sets the hash mode to L2+L3 and disables the static bonding mode, dynamic mode is LACP and that's what you more than likely want. The load balance mode is not strictly necessary, but if you are establishing a L3 adjacency traffic will of course only use one port of the bond in either direction without including some L3 header entropy in the hash. For lag interfaces `adminmode` is enable, `no adminmode` is disable. For everything else, it's `no shutdown` and `shutdown`. The port configuration enables what's commonly referred to as active mode, and also sets short timeout (duh, right?). The lag will take much longer to come up (30s vs 1s) and to stop forwarding to partner lag members that become inactive without a link down event (90s vs 3s). In practice I found it would not come up at all with a Linux bond-driver partner unless I set active mode and would not reliably come up after a Linux bond-driver partner reboots until I set short timeout.interface 0/1
port lacptimeout actor short
port lacptimeout partner short
no lacp actor admin state longtimeout
no lacp partner admin state longtimeout
It was for me. You'll need a console cable of course, but yes, it's just linux so resetting the root password isn't hard.Hi,
I found the same switch on Ebay from a different vendor but he mentioned he doesn't have the Ubuntu password. Is it easy to hack/crack/start Ubuntu on the switch in single user mode like on a normal Linux PC?
I mean it won't be as easy as this How to Reset the Root Password in Linux - Make Tech Easier right ?
https://forums.servethehome.com/ind...1-b21-mellanox-354-with-direct-connect.21632/It was for me. You'll need a console cable of course, but yes, it's just linux so resetting the root password isn't hard.
My question is, can I use hp infiniband qsfp+ cards with this? E.g.:
HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21 661685-001 Low Profile 4872503154148 | eBay
Can I use these like a regular network card or are they only good for storage networking?
Noice. We can do that. Have to dig up a serial cable and enable a port or get a USB to serial converter but that's no problem.It is trivial to reset the password, it boots as a standard Ubuntu system and from the serial console you can access BIOS, GRUB, and Linux command line. In order to reset the password you simply hit `e` at the GRUB screen to edit the default boot entry, throw an `init=/bin/bash` in there, wait a bit, and then `passwd`.
That is in IB or ETH mode ? I assume ETH but not sure.I have been able to sustain 37.7Gbps TCP/IP iperf3 via a single port of the D4040 in the following configuration:
Dual Xeon E5-2650V2 + Mellanox ConnectX3-EN -> D4040 -> D4040 -> Dual Xeon E5-2650V2 + Intel XL710-QDA2
99% speed is more than fine. Even >90% would be fine as it's a big improvement over 10 GBit ethernet.This is around ~99.4% theoretical line rate with 78 byte TCP/IP, VLAN, and Ethernet framing overhead per 1542 bytes on the glass. Until I have time to run some more thorough tests I am willing to give it a 0.6% benefit of the doubt considering the system under test in that configuration was a virtual machine binding a single SR-IOV virtual function of the XL710-QDA2.
Indeed, before I found out you can 'assign' IRQs to cards we just kept moving the NIC around in the (dual-CPU) server and in slot 1 (close to CPU) it jumped from 20 Gbit to over 36/37 Gbit.This result took a decent amount of tuning to achieve, but the ample documentation from both NIC vendors should point you in the right direction. In particular I found isolating the NIC's interrupts to specific CPUs in the virtual context and pinning those vCPUs to cores of the pCPU whose PCI bus the NICs were attached to be critical in sustaining >99% line rate, otherwise it dips by about 20-30% when cross-socket communication is involved.
Does the ICOS SSH daemon also have a password and if so how to reset that ?Now, back to the switch. Some aspects of the switches default configuration are not what you would expect if you are familiar with pretty much any other vendor's hardware of this class. The device comes with two SSH daemons that can be configured independently: one that connects to ICOS directly, and one that connects to Ubuntu directly. I disabled the ICOS SSH and enabled the Ubuntu SSH.
Is that also required for basic 'switch' operations, where the switch is the single switch in the whole network, just connecting a bunch of switches, no up- and down-links, no bonding? Just plain "server with 40Gbe card connected to switch using a QSFP+ (DAC) cable?You'll need to use Linux policy routing and multiple routing tables if you will learn/configure default routes for both the management and data plane interfaces (ip route .. table x, ip rule add ..., don't recall the exact config). Routes including the default learned from ICOS' routing protocols will be installed in the Linux main table, and so will routes configured for the management interface unless you specify an alternate table. There are separate interfaces for control plane to management interface, and control plane to data plane interfaces. They both must be configured if you want in band and out of band access to the control plane.
I had to google cut-through haha, I guess I've been spoiled with plug 'n play 1Gbit and 10Gbit switches. I see it helps with latency but if latency is less of a problem and throughput is more important (e.g. for iSCSI/iSER and other (file)transfers) ; does it matter to have cut-through enable ?You have to specifically enable cut-through mode. There is no reason not to that I'm aware of and it is a significant reduction in latency for jumbo frame traffic. `cut-through mode` from configuration mode and a `reload` should do it. I do not believe cut-through will function between 40GbE and broken out 10GbE interfaces.
just follow this https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015It was for me. You'll need a console cable of course, but yes, it's just linux so resetting the root password isn't hard.
My question is, can I use hp infiniband qsfp+ cards with this? E.g.:
HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21 661685-001 Low Profile 4872503154148 | eBay
Can I use these like a regular network card or are they only good for storage networking?