Here's a $1.3k switch for you: 18x 40Gig QSFP and 40x 10gig SFP+

[stormqloud]

Anyone have any more information or docs on these gnodal switches?

The "docdroid" like doesn;t seem to work for docs.

I have a pair of GS0018 arriving in the next few days.

Hopefully not a nightmare to work with!

 

[Scott Laird]

A minor followup on my GS0018: I was using a Mellanox QSFP-1xSFP+ cable to connect my GS0018 to a LB4M, without a ton of success. It looked okay at first--the link-layer showed up as connected on both ends, LLDP worked, the Gnodal's MAC table filled up with traffic from the LB4M, while the LB4M's MAC table saw the Gnodal itself. The only issue was that I couldn't ping the Gnodal over the SFP+ link, and when I connected both the 1000baseT admin port and the SFP+ port to the LB4M at the same time, I had intermittent connectivity to the Gnodal's IP address. Whenever the LB4M's MAC table showed the Gnodal's MAC heading out the SFP+ port, I couldn't ping the Gnodal. The the MAC would move to the 1000baseT port, and pinging/ssh/etc worked fine. And then it'd head back to the SFP+ port.

Things got more interesting when I plugged a 40G device into the Gnodal. It couldn't see out, and the LB4M never saw its MAC. In addition, it was unable to ping (or arp) the Gnodal's admin address from the 40G box.

After wasting way too much time on this, it looks like a cable issue--I swapped my Mellanox QSFP-1xSFP+ cable for a QSFP-4xSFP+ cable, and suddenly my test box can access the outside world via the Gnodal-LB4M link.

I still can't ping the Gnodal over the fabric, though. Does anyone know if Gnodals support this? Or am I going to have to create a little admin VLAN and plug it in there? No big deal, I guess.

 

[stormqloud]

I'm just setting up a GS0018 right now.

The CLI seems "industry standard".

I've got a Mellanox QFP to 4 SPF. It's the only 40GB cable I own right now ;-)

According to the docs I think you need to reconfigure the port to split out the channels if you want to use the QSFP-1xSFP+

Any idea what the small jack on the front is? Looks like an audio cable. I'm guess it's the RS232 interface?

I don;t think you can speak to the gnodal over the fabric. The mgmt is out of band.

My guess is if they added it in, it would slow the fabric down. They would have to put a dual speed switch inside and all the related buffers and queue etc.

I'll report back after trying it out with some Chelsio 10GB cards.

 

[stormqloud]

Any idea how to put a default route on the mgmt network?

 

[stormqloud]

after screwing around with the switch for a couple of hours it's all setup.

I even managed to get one single QSFP to hookup to 4 different 10GB servers with the octopus cable.

The Gnodal switch is a bit strange. It requires multiple reboots to flush the settings through and get working.

For example, changing the setup of a QSFP port from QSFP to 4 times 10GB individual connections requires rebooting the switch.

It's nice bit of hardware for the price. Lacks a bit of polish on the CLI but otherwise seems solid.

The only thing that doesn;t work is connecting to the mgmt interface from the switch fabric. I doubt that was meant to work tho.

 

[sergeant72]

Received 2 new qsfp cables today, both Mellanox brand, happy to report that both are working fine with the switch.
Here is the transceivers info:

Eth0/49 is present
name is Mellanox
part number is MC2210310-100
revision is A1
serial number is ES231201947
type is optical
length is 0
temperature is 0.00
lane Rx power is 0.0uW 0.0uW 0.0uW 0.0uW

and the second one:

Eth0/49 is present
name is Mellanox
part number is MC2207310-060
revision is A2
serial number is ES261200596
type is optical
length is 0
temperature is 0.00
lane Rx power is 0.0uW 0.0uW 0.0uW 0.0uW

Also, it seems I have some kind of issue with the 2 Windows 8.1 machines I'm using for testing - I can't break the barrier of ~4.5 - 5.0 Gbits/sec between them (with the switch or without, 10GBe or 40GBe, different cables). iperf with 10 threads with different parameters - doesn't matter, still hitting that barrier.
Any advice on configuration/different testing technique will be appreciated.

 

[britinpdx]

On the QSFP front, I've had success with a Mellanox MC2206130-003 connected to a Mellanox ConnectX-2 VPI (26428, configured as 10GBe not IPoIB).

On the SFP+ front, I've had success with the Finisar FTLX8571D3BCL transceiver and Monoprice 10Gb Fiber Optic Cable, LC/LC, Multi Mode

 

[Scott Laird]

Has anyone had any success with Mellanox QSA QSFP->SFP+ adapters plus SFP+ optics? I have a QSFP->4x SFP+ cable that works fine, but I need one more 10G port on my GS0018. I tried swapping in a the QSA + a Finistar FTLX8571D3BCLs, and all I get is a red light. Swapping the 4xSFP+ cable back in makes everything work again.

 

[Patrick]

Just wanted to check. Did anyone that purchased the QSFP+ and SFP+ version wish they instead got an 18x QSFP+ version? If so I would be happy to trade. I may have room for a high-speed switch in the colo.

 

[Scott Laird]

I'm fairly happy with my GS0018, but I can't figure out how to get SNMP working on it. Or, rather, I *think* I can get it working, but it's taking 30s to respond to each request, and my client is timing out before then. Is it just me, or is there something odd about SNMP on these things? I'm running firmware 2.6.4.

 

[Scott Laird]

FWIW, I figured out the SNMP problem. It comes with SNMP enabled, for community PUBLIC, in caps. I'm not quite sure why they did that vs. 'public' or leaving unauthenticated SNMP off entirely, it seems an odd choice.

 

[bacterialbag]

I am very interested in buying such a switch, but I need to know if it does RDMA with Mellanox Connectx-3 Pro VPI?

Any luck getting RDMA to work using mellanox cards? Does SMB direct function properly?

Would u have any reason to advice against buying such a switch?

Hope to hear from you soon!

Bacterialbag

 

[csmz]

Hi all,

I have a GS4008 in operation, it was purchased in unopened box and came with all accessories.
It is 2.6.1 version.

I have had the following success, if not specified I use LC-LC OM3 cables.

Finisar FTLX8571D3BCL 10 Gbit/s 850 nm SFP+ Transceiver

the Finisar Transciever is very cheap and works with the GNODAL and was tested with the following NIC:

HP NC522SFP 10GbE DP SFP PCIe (despite being HP branded Qlogic, they also work flawlessly with Dell etc servers)
MELLANOX MNPA19-XTR CONNECTX 2

Does NOT work with the Finisar Transciever:

Brocade 1020

using the DAC cable "Brocade 10G 5M Twinax Active FCoE 5m Cable 58-1000023-01" it works perfectly fine with the Gnodal.



Does anyone have the image for the latest firmware?

Also, to connect and enable the switch use the following procedure, assuming your network uses a 192.168.0.x subnet, if not change accordingly (and ensure .2 isnt used!):

Use a notebook with ip 192.168.1.2 subnet 255.255.255.0, direct connect it to the top right RJ45 port

download putty
PuTTY Download Page
connect to 192.168.1.1 TELNET with putty
a window should show up
enter user "root" password "admin123"

configure terminal
default ip address 192.168.0.2 subnet-mask 255.255.255.0
interface cpu0
ip address 192.168.0.2 255.255.255.0

close the now not responding putty, change your ip adress to 192.168.0. something or connect the management port to your network with rj45 and connect the notebook just to that, it should work. reopen putty
login using aforementione credentials

write startup-config

You have now set the new ip and saved that configuration. By default ports are off. Use the following to enable ports (if you use jumbo frames go to step 2):

Login like above

configure terminal
interface range ethernet 0/*
no shutdown
exit
exit
write startup-config

Now all ports are active. If your network has Jumbo Frames, I heavily emphasize to modify aforementioned step:
configure terminal
interface range ethernet 0/*
mtu 9216
no shutdown
exit
exit
write startup-config

This should be a quick get up and running and startup guide with some compatability advice.
Hope it helps!

 

[bds1904]

Finisar FTLX8571D3BCL 10 Gbit/s 850 nm SFP+ Transceiver

the Finisar Transciever is very cheap and works with the GNODAL and was tested with the following NIC:

MELLANOX MNPA19-XTR CONNECTX 2

Just to confirm the Finisar FTLX8571D3BCL works with the CONNECTX-2?

I guess I know what 10Gb NIC i need to buy considering I have a few of those transceivers laying around and a spare SFP+ on a LB4M!

 

[csmz]

Just to confirm the Finisar FTLX8571D3BCL works with the CONNECTX-2?

I guess I know what 10Gb NIC i need to buy considering I have a few of those transceivers laying around and a spare SFP+ on a LB4M!

I can confirm that this NIC and these Finisar not only work with the Gnodal but also the LB4M, as I have one of those next to the Gnodal to handle RJ45 traffic.

And yes, you can hook any port of the Gnodal to the SFP+ of the LB4M using Finisars and LC-LC Om3 to have those switches connected with 10G.

 

[Psycho_Robotico]

Just wanted to check. Did anyone that purchased the QSFP+ and SFP+ version wish they instead got an 18x QSFP+ version? If so I would be happy to trade. I may have room for a high-speed switch in the colo.

Can't offer a trade but if you're willing to part with a GS4008 I'd be interested Or does anyone know a place where a Gnodal is sold at a reasonable price?

 

[Chuckleb]

We bought 2 Gnodals on eBay and they work well. Plan on picking up another pair. The 72port SFP+ models.

 

[Patrick]

Can't offer a trade but if you're willing to part with a GS4008 I'd be interested Or does anyone know a place where a Gnodal is sold at a reasonable price?

Actually @PigLover saved me a few months ago.

 

[PigLover]

Was glad to help. I consider it a small token of thanks for you running the site/forum.

BTW, I don't really need the GS0018. It's a fun toy but running a 40gbe backbone between servers in my garage really is bit over the top even for folks that frequent here. If anybody needs or wants an 18x 40gbe Gnodal switch I'm open to offers

 

[Patrick]

18x 40gbe Gnodal switch

I believe that is also splittable to 4x 10GbE too using a QSFP+ to 4x SFP+ cable.