Quanta LB6M (10GbE) -- Discussion

[djflow195]

My server sees the internet, my workstation pc sees the internet, my personal pc sees the internet, all are connected through DAC cables to ports 24, 23 and 22, but it seems like the 10gb ports are not talking to each other?

(FASTPATH Routing) #show port all

...
0/22 Enable 10G Full 10G Full Up Enable Enable long
0/23 Enable 10G Full Down Enable Enable long
0/24 Enable 10G Full Down Enable Enable long
0/25 Enable Auto 1000 Full Up Enable Enable long
0/26 Enable Auto 1000 Full Up Enable Enable long
0/27 Enable Auto 1000 Full Up Enable Enable long
0/28 Enable Auto 1000 Full Up Enable Enable long

Ports 23 and 24 are Down. So whatever is plugged into them is not talking to the switch.

What is plugged into 26 - 28?

 

[Sleyk]

Port 25 is the internet/lan coming from the pfsense router. Then I have my wireless access point plugged into port 26, and two more 8 port 1gigabit switches plugged into 27 and 28

I will go back and make sure they are plugged in properly once more, also i will try to swap out my DAC to see if maybe they are bad.

 

[Sleyk]

No go. Swapped out direct attached cables for different ones. Still no contact from server to pc through sfp+ ports.

 

[Sleyk]

ok, swapped out DAC cables and made sure all connections are tightly plugged in and in the right ports. I had pc and server plugged into ports 21 and 22 instead of 23 and 24. i plugged them into ports 23 and 24 so that it is inline with what i previously wrote. Still unfortunately, no go. I did another show port all: Ports 23 and 24 now show up as being "up" instead of ports 21 and 22:


(FASTPATH Routing) #show port all

Admin Physical Physical Link Link LACP Actor
Intf Type Mode Mode Status Status Trap Mode Timeout
--------- ------ --------- ---------- ---------- ------ ------- ------ --------
0/1 Enable 10G Full Down Enable Enable long
0/2 Enable 10G Full Down Enable Enable long
0/3 Enable 10G Full Down Enable Enable long
0/4 Enable 10G Full Down Enable Enable long
0/5 Enable 10G Full Down Enable Enable long
0/6 Enable 10G Full Down Enable Enable long
0/7 Enable 10G Full Down Enable Enable long
0/8 Enable 10G Full Down Enable Enable long
0/9 Enable 10G Full Down Enable Enable long
0/10 Enable 10G Full Down Enable Enable long
0/11 Enable 10G Full Down Enable Enable long
0/12 Enable 10G Full Down Enable Enable long
0/13 Enable 10G Full Down Enable Enable long
0/14 Enable 10G Full Down Enable Enable long
0/15 Enable 10G Full Down Enable Enable long
0/16 Enable 10G Full Down Enable Enable long
0/17 Enable 10G Full Down Enable Enable long
0/18 Enable 10G Full Down Enable Enable long

--More-- or (q)uit


Admin Physical Physical Link Link LACP Actor
Intf Type Mode Mode Status Status Trap Mode Timeout
--------- ------ --------- ---------- ---------- ------ ------- ------ --------
0/19 Enable 10G Full Down Enable Enable long
0/20 Enable 10G Full Down Enable Enable long
0/21 Enable 10G Full Down Enable Enable long
0/22 Enable 10G Full Down Enable Enable long
0/23 Enable 10G Full 10G Full Up Enable Enable long (this is the server) > 10gigabit sfp+ port
0/24 Enable 10G Full 10G Full Up Enable Enable long (this is the PC) > 10gigabit sfp+ port
0/25 Enable Auto 1000 Full Up Enable Enable long (this is the Pfsense router) > 1gigabit port
0/26 Enable Auto 1000 Full Up Enable Enable long (this is the wireless access point) > 1gigabit port
0/27 Enable Auto 1000 Full Up Enable Enable long (this is an 8 port 1gigabit switch) > 1gigabit port
0/28 Enable Auto 1000 Full Up Enable Enable long (this is an 8 port 1gigabit switch) > 1gigabit port
1/1 Enable Down Disable N/A N/A
1/2 Enable Down Disable N/A N/A
1/3 Enable Down Disable N/A N/A
1/4 Enable Down Disable N/A N/A
1/5 Enable Down Disable N/A N/A
1/6 Enable Down Disable N/A N/A
1/7 Enable Down Disable N/A N/A
1/8 Enable Down Disable N/A N/A

--More-- or (q)uit


Admin Physical Physical Link Link LACP Actor
Intf Type Mode Mode Status Status Trap Mode Timeout
--------- ------ --------- ---------- ---------- ------ ------- ------ --------
1/9 Enable Down Disable N/A N/A
1/10 Enable Down Disable N/A N/A
1/11 Enable Down Disable N/A N/A
1/12 Enable Down Disable N/A N/A
1/13 Enable Down Disable N/A N/A
1/14 Enable Down Disable N/A N/A
1/15 Enable Down Disable N/A N/A
1/16 Enable Down Disable N/A N/A
1/17 Enable Down Disable N/A N/A
1/18 Enable Down Disable N/A N/A
1/19 Enable Down Disable N/A N/A
1/20 Enable Down Disable N/A N/A
1/21 Enable Down Disable N/A N/A
1/22 Enable Down Disable N/A N/A
1/23 Enable Down Disable N/A N/A
1/24 Enable Down Disable N/A N/A
1/25 Enable Down Disable N/A N/A
1/26 Enable Down Disable N/A N/A

--More-- or (q)uit


Admin Physical Physical Link Link LACP Actor
Intf Type Mode Mode Status Status Trap Mode Timeout
--------- ------ --------- ---------- ---------- ------ ------- ------ --------
1/27 Enable Down Disable N/A N/A
1/28 Enable Down Disable N/A N/A
1/29 Enable Down Disable N/A N/A
1/30 Enable Down Disable N/A N/A
1/31 Enable Down Disable N/A N/A
1/32 Enable Down Disable N/A N/A
1/33 Enable Down Disable N/A N/A
1/34 Enable Down Disable N/A N/A
1/35 Enable Down Disable N/A N/A
1/36 Enable Down Disable N/A N/A
1/37 Enable Down Disable N/A N/A
1/38 Enable Down Disable N/A N/A
1/39 Enable Down Disable N/A N/A
1/40 Enable Down Disable N/A N/A
1/41 Enable Down Disable N/A N/A
1/42 Enable Down Disable N/A N/A
1/43 Enable Down Disable N/A N/A
1/44 Enable Down Disable N/A N/A

--More-- or (q)uit


Admin Physical Physical Link Link LACP Actor
Intf Type Mode Mode Status Status Trap Mode Timeout
--------- ------ --------- ---------- ---------- ------ ------- ------ --------
1/45 Enable Down Disable N/A N/A
1/46 Enable Down Disable N/A N/A
1/47 Enable Down Disable N/A N/A
1/48 Enable Down Disable N/A N/A
1/49 Enable Down Disable N/A N/A
1/50 Enable Down Disable N/A N/A
1/51 Enable Down Disable N/A N/A
1/52 Enable Down Disable N/A N/A
1/53 Enable Down Disable N/A N/A
1/54 Enable Down Disable N/A N/A
1/55 Enable Down Disable N/A N/A
1/56 Enable Down Disable N/A N/A
1/57 Enable Down Disable N/A N/A
1/58 Enable Down Disable N/A N/A
1/59 Enable Down Disable N/A N/A
1/60 Enable Down Disable N/A N/A
1/61 Enable Down Disable N/A N/A
1/62 Enable Down Disable N/A N/A

--More-- or (q)uit


Admin Physical Physical Link Link LACP Actor
Intf Type Mode Mode Status Status Trap Mode Timeout
--------- ------ --------- ---------- ---------- ------ ------- ------ --------
1/63 Enable Down Disable N/A N/A
1/64 Enable Down Disable N/A N/A

(FASTPATH Routing) #

Also, here is a new show run:

(FASTPATH Routing) #show run
!Current Configuration:
!
!System Description "Quanta LB6M, 1.2.0.14, Linux 2.6.21.7"
!System Software Version "1.2.0.14"
!System Up Time "1 days 23 hrs 5 mins 9 secs"
!Additional Packages FASTPATH QOS
!Current SNTP Synchronized Time: Not Synchronized
!
network protocol none
network parms 192.168.0.5 255.255.255.0 192.168.0.1
vlan database
exit
ip ssh server enable
no ip telnet server enable
configure
ip routing
username "admin" password cda5d072920fca940b3710a175f0fe3a25fe83f3e9ad17ceca72ca27966aa7f012067fbfe8877e9e7df30b7b92725bede2969f8e93846d3787cf4c1f86c3aa04 level 15 encrypted
aaa authentication enable "enableList" enable
line console
exit
line telnet
exit
line ssh
exit

--More-- or (q)uit

spanning-tree configuration name "04-7D-7B-87-01-8A"
!
router rip
exit
router ospf
exit
exit

(FASTPATH Routing) #

Same problem persists. I cant log into the server from the sfp+ ports, but wireless i can login through my access point with my laptop. It is odd how the internet and lan ethernet signal passes through to the 10gigabit ports, but if i try to go from pc to server, whether through windows networking or webgui, the server doesn't show.

 

[djflow195]

Does "show port all" show the ports up or down? May be a bad DAC, wrong kind of DAC (is it passive or active), too long a DAC, etc. Validate all the cables with the known working device. If they are good, what are the other differences? What make cards? Try different ports on the switch, 23 and 24 may be bad. One at a time, use port 22 with your other devices and see if they come up.

 

[djflow195]

Okay, Layer 1 looks ok. Do "show interface 0/23" and "show interface 0/24" and post the output to verify Layer 1.

Then do "show mac-addr-table" and post the output to verify Layer 2. Make sure they match the devices MAC addresses.

If you are not using the routing features do a "no ip router" to simplify troubleshooting.

See if you can ping each device from the switch.

 

[Sleyk]

First, so sorry to everyone for completely taking over this thread with this issue. I appreciate all your kind efforts to help me.
I am not gonna give up. I was thinking about maybe seeing if the issue is with my server. I am going to install windows on another pc and put a 10gbit card in there, then connect it to port 1 and see if i can see another windows machine. prior to this, i have only been trying to go from windows pc to server. I can still ping my server and the server can still ping the windows pc. Will keep you posted. In the meantime, any and all help is truly appreciated.

djflow, I will post all info you asked for right away. Thanks

 

[Sleyk]

Ok djflow, here is the added info:

First, i swapped out the DAC for different ones, still no go on the ports.
The DAC are passive i believe. They are 3 meters (10feet) long.
The PC has a mellanox connectx-2 single port 10gbit card in a x8 2.0 slot, and the server has the same card. I also tried a chelsio dual port 10gbit card in the same slot in the server to see if maybe the mellanox card wasn't working, but the chelsio card is showing the same thing.
I will switch over to ports 1 and 2 on the switch, then try 6 and 7, all the way to port 20.

show interface 0/23:

(FASTPATH Routing) #show interface 0/23

Packets Received Without Error................. 25388
Packets Received With Error.................... 0
Broadcast Packets Received..................... 153
Packets Transmitted Without Errors............. 44198
Transmit Packet Errors......................... 0
Collision Frames............................... 0
Time Since Counters Last Cleared............... 1 day 23 hr 37 min 17 sec

show interface 0/24:

(FASTPATH Routing) #show interface 0/24

Packets Received Without Error................. 4600
Packets Received With Error.................... 0
Broadcast Packets Received..................... 3082
Packets Transmitted Without Errors............. 6052
Transmit Packet Errors......................... 0
Collision Frames............................... 0
Time Since Counters Last Cleared............... 1 day 23 hr 37 min 57 sec

ok, and here is show mac-addr-table:

(FASTPATH Routing) #show mac-addr-table


(FASTPATH Routing) #show mac-addr-table

VLAN ID MAC Address Interface IfIndex Status
------- ------------------ --------- ------- ------------
1 00:02:C9:53:AD:20 0/23 23 Learned
1 00:07:43:06:F2:1F 0/24 24 Learned
1 00:18:dd:31:2A:3B 0/27 27 Learned
1 00:18:dd:31:34:BF 0/27 27 Learned
1 00:1B:21:A7:BD:61 0/25 25 Learned
1 00:25:90:CC:59:07 0/28 28 Learned
1 00:25:90:CC:59:88 0/28 28 Learned
1 04:7D:7B:87:01:8A 3/1 29 Management
1 04:A1:51:9d:d0:95 0/26 26 Learned
1 28:9A:FA:55:48:B0 0/26 26 Learned
1 5C:FF:35:2D:F5:F8 0/26 26 Learned
1 7C:05:07:0F:02:88 0/27 27 Learned

(FASTPATH Routing) #

All mac addresses verified as different cards, access point, router and devices on lan and wireless network.
Finally i tried the no ip router command. it seems like the "no ip router" command is invalid, or not working on the lb6m.

 

[Sleyk]

ok, preparing to head off to work for tonight. as soon as i get back home, i'm gonna install windows on a separate pc and see if i can get the two windows pc's to talk to each other through the same 10gbit sfp+ ports. At least that will tell me if the ports are bad or if the problem lies with my server. Thanks for all the help so far, and any further suggestions are appreciated! Just tell me what to do and i will do so as soon as i get home in the am. Will keep you posted...

 

[djflow195]

Finally i tried the no ip router command. it seems like the "no ip router" command is invalid, or not working on the lb6m.

Woops, I meant "no ip routing". You have L3 active on the switch. That should disable it.

Since you have MACs on the switch. Layer 2 seems to be ok. And there is traffic on the ports.

On the PC/Server try to ping the other and see if the ARP table shows the correct MAC to IP mapping.

 

[derekvw91]

I'm having an issue with vlan configurations.

I have a router tagging vlans 5,10,15,20,25,30,35. I linked the switch to my router on port 27 and included all of those vlans in the participation with a pvid of 1.

I then went to configure port 26 and included vlan participation on vlan 5 and set the pvid to 5. When I go to plug in something to port 26, I don't get an IP from my router.

Can you guys help me figure out what I'm doing wrong?

Thanks!

 

[dwright1542]

I'm having an issue with vlan configurations.

I have a router tagging vlans 5,10,15,20,25,30,35. I linked the switch to my router on port 27 and included all of those vlans in the participation with a pvid of 1.

I then went to configure port 26 and included vlan participation on vlan 5 and set the pvid to 5. When I go to plug in something to port 26, I don't get an IP from my router.

Can you guys help me figure out what I'm doing wrong?

Thanks!

What you did is exactly right. I'll ask the obvious first:

1. Are you actually running a DHCP server on that VLAN?
2. If you set a static, can you ping anything on vlan 5?

 

[djflow195]

@derekvw91: Post your configs.

I assume that your router is running DHCP on that segment and VLAN interface.

Post the output of: "show vlan", "show vlan 5", "show vlan port all"

As with #2 from @dwright1542: give the device a static IP in that subnet and see if you can ping the ip address of VLAN 5. That will tell us if the VLAN setup is correct.

 

[Sleyk]

EUREKA!!! oh my Goodness! EURE-freakin-KA!!!!

Let me start off by saying that i appreciate the efforts of all who tried their best to help me. You guys are totally awesome, and I could not have done this, or arrived to this conclusion without your suggestions and help. Thank You, Thank You so MUCH.

I finally figured it out. I finally got this pony to start riding. Oh my goodness it was a struggle. And the answer was so damn simple, I am so mad at myself right now. So very mad.

The answer was JUMBO FRAMES.....Stay away. Stay far away.

So I was working on installing windows on a small business pc that had a x4 pcie 2.0 slot. I installed a low profile mellanox card and booted. I installed windows 8.1 and proceeded to test. After installation, my hopes were lifted as i was able to log into main pc through the network, but still not through to the server. This gave me hope, but i was still not able to see the server.

At this point I knew it was a setting on my server. It had to be, because now i could see windows to windows across the 10gb sfp+ ports. So i logged into the server wirelessly and started poking at my settings. Finally after spending all day and not sleeping at all after working all last night, I stumbled upon it.... JUMBO FRAMES.

I had previously set an MTU of 9000 for my mellanox and chelsio 10gb cards. This was my fatal mistake. Nothing, and I mean nothing passed through or left the server with the mtu set at 9000. I said what do I have to lose. I disabled the mtu and left it at default.

Oh my goodness. I was able to log in no problem and see the server and all is now freakin' good! I am so damn happy to get this fixed man!

I am abit curious though... Does this switch NOT support jumbo frames???? I mean, don't get me wrong. I am DONE with that old sinister technology. This had me stumped for days.

I went ahead and disabled jumbo on all my cards and devices on the network and everything is working peachy and bright. I have learned my lesson. Jumbo is a threat. Jumbo kills. Jumbo is now my arch enemy.

I was reading through a few articles and the impression I got anyway was that Jumbo is almost not needed in today's networking environment as the offloading capabilities of today's NIC's do most of the work anyways? Can someone confirm?

Anyways, I thank all for helping, and hope my posts can prove useful for some other poor schmuck who comes along. I am now a for life member of servethehome.

I hope I contributed to the knowledge.

P.S. dwright, djflow, and anyone else i might have forgotten....you totally ROCK.

 

[djflow195]

It will do Jumbo but not by default. I assume you had both endpoints set Jumbo but not the switch.

 

[PigLover]

Default does not support JumboFrames. Simple to configure across the whole switch:

enable
configure
interface 0/1-0/28
mtu 9216
exit
copy system:running-config nvram:startup-config
exit
logout

 

[dwright1542]

You need to set

mtu 9216

on each port to support jumbo frames. No need to not set it.

 

[Fritz]

How do you check the mtu?

 

[PigLover]

Show running-config.

If there is no MTU listed for an interface then it is default (1516). If a different MTU is set the interface will show it.

 

[djflow195]

Show running-config.

If there is no MTU listed for an interface then it is default (1516). If a different MTU is set the interface will show it.

Mine shows default MTU is 1518.

Do "show interface ethernet x/y" and look for "Max Frame Size................................. 1518" in the output.