Hi guys,
Hoping to find someone with some insights into 10Gbe networking.
I recently picked up a "cheap" G8124 in the hope I could replace my Lenovo/Intel X520-DA2 strait DAC connection between my DL160-G6 and QNAP TVS-1271U-RP with a switch and add the other Xen servers and backbone switches to the party.
The configuration I want to go for is a single switch with multiple servers and nas servers attached running Xenserver.
However, whatever I try, I can not get the 8124 to recognize my DAC cables. I have several sfp+ DAC cables, all either Lenovo 2m 00AY765 or HP 1m 487652-B21. Think the cables are OK as they report to the servers but the switch keeps telling me that the tranceivers are NOT APPROVED. The logs shows me:
Jul 5 11:09:00 phssw02 NOTICE system: Port 1 Installed Unknown
Jul 5 11:09:01 phssw02 NOTICE system: Port 1 Wrong Transceiver Type Unknown
I have tried all sorts of configurations but no luck so far. I have the feeling I need to turn on or turn off something..... or need other tranceivers/cables all together or I may have gotten a broken switch.... Anyone with an idea on how the configure the 8124 or at least confirm it's fubar.
Some info on the Switch and cards:
show version:
IBM Networking Operating System RackSwitch G8124
Switch has been up for 0 days, 1 hour, 31 minutes and 5 seconds.
Last boot: 13:20:40 Tue Jul 5, 2016 (reset by WEB-UI)
MAC address: 34:40:b5:6b:a5:00 IP (If 1) address: 192.168.49.50
MGMT-A Port MAC Address: 34:40:b5:6b:a5:fe
MGMT-A Port IP Address (if 127): 172.16.0.34
MGMT-B Port MAC Address: 34:40:b5:6b:a5:ef
MGMT-B Port IP Address (if 128): 192.168.51.50
Hardware Revision: 0
Board Revision: 2
Switch Serial No: Y350UN26S01B
Hardware Part No: BAC-00045-00 Spare Part No: BAC-00045-00
Manufacturing date: 12/27
MTM Value: 1455-24E
ESN: 23W0435
Software Version 7.11.7 (FLASH image1), factory default configuration.
Boot kernel version 7.11.7
------------------------------------------------------------------
NOTE: System dump exists in FLASH.
The dump was saved at 12:05:05 Tue Jul 5, 2016.
Use 'copy flash-dump' to
extract the dumps for analysis and 'clear flash-dump' to
clear the FLASH region.
------------------------------------------------------------------
Temperature Sensor 1: 41.5 C
Temperature Sensor 2: 37.5 C
Temperature Sensor 3: 42.75 C
Temperature Sensor 4: 46.50 C
Temperature Sensor 5: 44.75 C
Warning at 85C and Failure at 100C
Speed of Fan 1: 8307 RPM (75 PWM)
Speed of Fan 2: 8503 RPM (75 PWM)
Speed of Fan 3: 8896 RPM (75 PWM)
Speed of Fan 4: 8503 RPM (75 PWM)
Speed of Fan 5: 8640 RPM (75 PWM)
Speed of Fan 6: 8095 RPM (75 PWM)
System Fan Airflow: Front to Rear
State of Power Supply 1: On
State of Power Supply 2: On
RS G8124#show transceiver
Port Device TX TXuW RXuW TXFlt Volts DegsC Laser Approval
-------- ------- --- ----- ----- ----- ----- ----- ----- ------ ----------
Port 1 < WRONG Device Type (SFP vs SFP+) >
Detail -> Vendor: Error
Part: Error
Date: Error
S/N: Error
Port 2 < WRONG Device Type (SFP vs SFP+) >
Detail -> Vendor: Error
Part: Error
Date: Error
S/N: Error
end more like that...
X520-DA2 info
lspci -vvv -s 07:00
07:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 24
Region 0: Memory at fbc00000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at e880
Region 4: Memory at fbdf8000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at fbb80000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Vital Product Data
Product Name: Intel X520 Dual Port 10GbE SFP+ Adapter
Read-only fields:
[SN] Serial number: 11S00JY855Y050KK58N274
[PN] Part number: 49Y7960
[MN] Manufacture ID: 37 41 31 32
[V1] Vendor specific: Intel X520 Dual Port 10GbE SFP+ Adapter
[V2] Vendor specific: 00JY855
[VL] Vendor specific: 49Y7962
[VR] Vendor specific: 22
[VS] Vendor specific: 32773
[VT] Vendor specific: 9
[EC] Engineering changes: N34030
[VU] Vendor specific: 2C55B7D5DE7C435A885E592FFA246522
[RV] Reserved: checksum good, 0 byte(s) reserved
End
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number 90-e2-ba-ff-ff-ae-d4-d8
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000c0000000 (64-bit, prefetchable)
Region 3: Memory at 00000000c0100000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe
Server info (DAC between GL160 en QNAP installed):
ethtool eth2
Settings for eth2:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: external
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
and nas shows:
Interface :
Adapter 5 (10GbE)
DHCP : No
IP Address : 172.16.11.2
Submask : 255.255.255.0
Gateway : 172.16.11.1
MAC Address : 90:E2:BA:AF:EC:19
Speed : 10 Gbps
MTU : 9000
Virtual LAN (VLAN) : No
eth3 is the same both with HP and lenovo cables when connected between the server and the nas box.
If i need to provide more info, please let me know....
Any help appreciated
Hoping to find someone with some insights into 10Gbe networking.
I recently picked up a "cheap" G8124 in the hope I could replace my Lenovo/Intel X520-DA2 strait DAC connection between my DL160-G6 and QNAP TVS-1271U-RP with a switch and add the other Xen servers and backbone switches to the party.
The configuration I want to go for is a single switch with multiple servers and nas servers attached running Xenserver.
However, whatever I try, I can not get the 8124 to recognize my DAC cables. I have several sfp+ DAC cables, all either Lenovo 2m 00AY765 or HP 1m 487652-B21. Think the cables are OK as they report to the servers but the switch keeps telling me that the tranceivers are NOT APPROVED. The logs shows me:
Jul 5 11:09:00 phssw02 NOTICE system: Port 1 Installed Unknown
Jul 5 11:09:01 phssw02 NOTICE system: Port 1 Wrong Transceiver Type Unknown
I have tried all sorts of configurations but no luck so far. I have the feeling I need to turn on or turn off something..... or need other tranceivers/cables all together or I may have gotten a broken switch.... Anyone with an idea on how the configure the 8124 or at least confirm it's fubar.
Some info on the Switch and cards:
show version:
IBM Networking Operating System RackSwitch G8124
Switch has been up for 0 days, 1 hour, 31 minutes and 5 seconds.
Last boot: 13:20:40 Tue Jul 5, 2016 (reset by WEB-UI)
MAC address: 34:40:b5:6b:a5:00 IP (If 1) address: 192.168.49.50
MGMT-A Port MAC Address: 34:40:b5:6b:a5:fe
MGMT-A Port IP Address (if 127): 172.16.0.34
MGMT-B Port MAC Address: 34:40:b5:6b:a5:ef
MGMT-B Port IP Address (if 128): 192.168.51.50
Hardware Revision: 0
Board Revision: 2
Switch Serial No: Y350UN26S01B
Hardware Part No: BAC-00045-00 Spare Part No: BAC-00045-00
Manufacturing date: 12/27
MTM Value: 1455-24E
ESN: 23W0435
Software Version 7.11.7 (FLASH image1), factory default configuration.
Boot kernel version 7.11.7
------------------------------------------------------------------
NOTE: System dump exists in FLASH.
The dump was saved at 12:05:05 Tue Jul 5, 2016.
Use 'copy flash-dump' to
extract the dumps for analysis and 'clear flash-dump' to
clear the FLASH region.
------------------------------------------------------------------
Temperature Sensor 1: 41.5 C
Temperature Sensor 2: 37.5 C
Temperature Sensor 3: 42.75 C
Temperature Sensor 4: 46.50 C
Temperature Sensor 5: 44.75 C
Warning at 85C and Failure at 100C
Speed of Fan 1: 8307 RPM (75 PWM)
Speed of Fan 2: 8503 RPM (75 PWM)
Speed of Fan 3: 8896 RPM (75 PWM)
Speed of Fan 4: 8503 RPM (75 PWM)
Speed of Fan 5: 8640 RPM (75 PWM)
Speed of Fan 6: 8095 RPM (75 PWM)
System Fan Airflow: Front to Rear
State of Power Supply 1: On
State of Power Supply 2: On
RS G8124#show transceiver
Port Device TX TXuW RXuW TXFlt Volts DegsC Laser Approval
-------- ------- --- ----- ----- ----- ----- ----- ----- ------ ----------
Port 1 < WRONG Device Type (SFP vs SFP+) >
Detail -> Vendor: Error
Part: Error
Date: Error
S/N: Error
Port 2 < WRONG Device Type (SFP vs SFP+) >
Detail -> Vendor: Error
Part: Error
Date: Error
S/N: Error
end more like that...
X520-DA2 info
lspci -vvv -s 07:00
07:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 24
Region 0: Memory at fbc00000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at e880
Region 4: Memory at fbdf8000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at fbb80000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Vital Product Data
Product Name: Intel X520 Dual Port 10GbE SFP+ Adapter
Read-only fields:
[SN] Serial number: 11S00JY855Y050KK58N274
[PN] Part number: 49Y7960
[MN] Manufacture ID: 37 41 31 32
[V1] Vendor specific: Intel X520 Dual Port 10GbE SFP+ Adapter
[V2] Vendor specific: 00JY855
[VL] Vendor specific: 49Y7962
[VR] Vendor specific: 22
[VS] Vendor specific: 32773
[VT] Vendor specific: 9
[EC] Engineering changes: N34030
[VU] Vendor specific: 2C55B7D5DE7C435A885E592FFA246522
[RV] Reserved: checksum good, 0 byte(s) reserved
End
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number 90-e2-ba-ff-ff-ae-d4-d8
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000c0000000 (64-bit, prefetchable)
Region 3: Memory at 00000000c0100000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe
Server info (DAC between GL160 en QNAP installed):
ethtool eth2
Settings for eth2:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: external
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
and nas shows:
Interface :
Adapter 5 (10GbE)
DHCP : No
IP Address : 172.16.11.2
Submask : 255.255.255.0
Gateway : 172.16.11.1
MAC Address : 90:E2:BA:AF:EC:19
Speed : 10 Gbps
MTU : 9000
Virtual LAN (VLAN) : No
eth3 is the same both with HP and lenovo cables when connected between the server and the nas box.
If i need to provide more info, please let me know....
Any help appreciated