Movin' on up...to 40GbE

[0xbit]

X2 are not supported except by Forum and limited but useful downloads, the real trick is avoid 3rd party X2 and X3 they must be Mell genuine. Otherwise its hit or miss .. which binary is overloading the timeslots ? oh its completely fair scheduler
Besides, the firmware bugs are obvious real issue that no one here has much experience with.. and then Dell / HP / EMC / IBM all return nightmare issues that are all one off thus useless , as binary drivers differ with OS / hardware combo. Thus advice is useless unless u clone that exact topology .. almost impossible

Best price on new genuine Mell ConnectX 3 is $80 (yet my issue is its not RoCE (2) which is UDP IB connectionless or stateless (yet there are pseudo paths poss) the main thing is its TCP buffer free meaning its unreliable real time (not as bad as it sounds) the other part of this is the Ether encapsulation ( oCE ) i like the RDMA part of it, but dont need the overhead of Ethenet encap.. Im not sending it remotely, and certainly when i am , the 4 packets are going to a mobile client that cant deal with RoCE , besides RoCE cant send 4 packets , Ether cant send 4 packets , more like 300 just to get the ball rolling..

RoCE is designed for distributed datacenters . databases like Hadoop & Elastic Search all the statistics are on hundreds of remote servers that need secure Ethernet access (TCP) or https// banks for example.. Why would any private researcher need this unless dealing with the genome pool or banking.. both of these planned, a gene transplant lab in Costa Rica and deposit on a First bank in Durham City

My interest is in cutting apart the CE and exposing the UDP IB part for internal DB packet storage, over IB .. I was looking at FPGA boards to so this, like the X4 060 at $2800 ea but found quad port cards with KU115 PGA fabric .. once a ser-vice takes off its great to have a forward plan to expand past one datacenter.

Thank you the reply. I like reading through such details. It gets my mind going about interesting things to look into. It sounds like you want to get into the driver/firmware area and even play around with a software defined element in the loop that gives you sway over how things get processed and moved around at the protocol level ... i.e : implement your own protocol. I'd love to know if there are cost effective ways to do this and am surprised this isn't a bigger (thing). This made me think of SDN (software defined networking) and then I decided to search for : Software Defined NIC ...

Whereupon, it appears something just fell out the back of the mellanox truck :
Hot off the press (4 hours old)
Mellanox Announces BlueField Software-Defined SmartNIC Adaptors | StorageReview.com - Storage Reviews
http://www.mellanox.com/page/products_dyn?product_family=275&mtag=bluefield_smart_nic

I'm sure slapping a beefy soc as well as ddr is going to make this a super pricey card but it seems like its what you're after. Not sure what you can do in terms of topology by which you do the main manipulation on one node and then send more simple/protocol standard packets downstream but I get your drift. Might be worth some time to look into software defined NICs and see what kind of magic you can whip up on your power house nodes while keeping your downstream nodes on something like connectx-2 using standard protocols. I am seemingly looking at a similar situation for myself.

 

[talis]

Thank you the reply. I like reading through such details. It gets my mind going about interesting things to look into. It sounds like you want to get into the driver/firmware area and even play around with a software defined element in the loop that gives you sway over how things get processed and moved around at the protocol level ... i.e : implement your own protocol. I'd love to know if there are cost effective ways to do this and am surprised this isn't a bigger (thing). It is a big thing only in banking/ trading data-centers. I spent 1 year drilling products via parent company implementation designers and Wow all the way up to quad KU115 FPGA on a single board over 3 years ago. Not cost effective, at $45k
it all comes down to what is the mission. Datacenters need security and failsafe so the fabric focus on SSH DES Helleman et al ..

This made me think of SDN (software defined networking) and then I decided to search for : Software Defined NIC ... the X4 XLC with FPGA fabric is $2900 is nore than SDN it does any complex op at wirespeed (edge pr0cessor) up to the limits of the PGA fabric (2000 DSP slices/ 5000 MemBlocks 200,000 CLB .. its wild but complex )

Whereupon, it appears something just fell out the back of the mellanox truck :
Hot off the press (4 hours old)

b4 u get keen quantity $1500 without a BSP plus kernel dev tools like Vivado another $3000 that provide abstraction layers for compiled network stacks. Looks like the ARM load. store RISC is here again in this 16 core S0C with its shared Cache model that is addicted to mutex / spin lock design of event based processors (unfort most events happen in software) apart from browser events that this card is not a player (then they are js .. my guess starting price $2500 for 8Gb DDR

Now if only the HDMI port was bidirectional u could Mellanox an Nvidia GPU


Mellanox Announces BlueField Software-Defined SmartNIC Adaptors | StorageReview.com - Storage Reviews
http://www.mellanox.com/page/products_dyn?product_family=275&mtag=bluefield_smart_nic

I'm sure slapping a beefy soc as well as ddr is going to make this a super pricey card but it seems like its what you're after. Not sure what you can do in terms of topology by which you do the main manipulation on one node and then send more simple/protocol standard packets downstream but I get your drift. Might be worth some time to look into software defined NICs and see what kind of magic you can whip up on your power house nodes while keeping your downstream nodes on something like connectx-2 using standard protocols. I am seemingly looking at a similar situation for myself.

Reality : defeated by television, overtaken by smokescreen pantomime disguised as democracy, hopelessly lost in space between those ears that need a psychic (lawyer) to give us directions. usually in Court for a fine u did not see coming .. jus like no one opens the INline box above to read the main response .. an the human race is RoCE ted .. like immutable objects stuck in shiny headlights
Lets see if a kernel programmer enters the stage .. wont hold breath

 

[acmcool]

So i got this switch.I deleted virtul chassis ports and also added 10gbps module. But i only see 1gbps ports in Jweb

 

[whitey]

Deleting the VC ports is definitely the first step, sounds like you got that. Do you have OM3/OM4 cable and optics (or DAC's) hooked up to those 10G ports and to a NIC? I think they will not show up until you do. ge for GbE ports, xe for 10GbE ports, et for 40GbE ports.

 

[Blinky 42]

So i got this switch.I deleted virtul chassis ports and also added 10gbps module. But i only see 1gbps ports in Jweb

For some (stupid to me) reason the ports show up a ge-* until 10G (or 40G )optics are in or a link is up at > 1G.
It was quite confusing why I couldn't see the xe-* interfaces on my ex-3300's at first.

 

[whitey]

ROFL here, sorry for your pain, we ALL run into it.

 

[acmcool]

Thats resolved it...Now DHCP server wont startup

 

[whitey]

You attempting to run DHCP off of the EX4300 I assume? Have not done that myself, always use my pfSense device for that functionality but I can poke it.

EDIT: Lots of hits on google search, sorry for flood.

Configuring a Switch as a DHCP Server (CLI Procedure) - Technical Documentation - Support - Juniper Networks
Configuring a DHCP Server on Switches (CLI Procedure) - Technical Documentation - Support - Juniper Networks
Juniper Networks - How to configure the EX-series switch as a DHCP server
DHCP and Other System Services Feature Guide for EX2300, EX3400, and EX4300 Switches - Technical Documentation - Support - Juniper Networks
Example: Configuring the Device as a DHCP Server - Technical Documentation - Support - Juniper Networks
Configure DHCP Server for Multiple VLANs in JunOS

One or all of those should take care of you, RTFM time :-D

 

[acmcool]

I got configuration done...it is same as my ex3300..But it says dhcp service not running..
let me see if i can take a screenshot in jweb

 

[Blinky 42]

Do you need an enhanced / advanced feature license for the dhcp server on that model? Many things need the key to actually start (BGP, PIMd etc), but typically I have seen it complain when you try and commit config changes about needing the license. On of @whitey links above should mention it

 

[acmcool]

Good point..let me check...but my config is committed already

 

[acmcool]

I figured it out.Everything was correct except now i had to create DHCP group and assign interfaces...in ex3300 that was not the case...
I run the switch in layer 3 mode..so this was needed...

 

[whitey]

Good deal, glad you got everything working.

 

[whitey]

Since this thread is a bit hot again today could I float this one out there:

Anyone else get a unit w/ a non-functional front panel where you typically see system status where the two buttons are? Seems I may have been the only unlucky one that received a unit in that shape. It's a non-show stopper but irritating nonetheless. I know on my couple of EX3300's they all seemed to work, show firmware, allow system reboot, show status messages/alarms. Mine shows a big blank screen like a connection is not lit or just a bad LCD panel.

BOO

 

[TeleFragger]

OK the 5030 has QDR (40Gb ports) that is identical to QSFP .. agree not 56Gb (i dont care for 56)
As i want to have an ALL IB ( with packet filter steering) internal cluster only the web connects via http// (hoping to keep 100,000 client sessions open, and serviced at a rate of 4 - 8 packets (64byte) per second ( many each 5 sec ) Thus as far as ethernet, its a cable going into 2 x 1gb NICs that pass packets into a pair of session manager Servers, that can keep 100,000 sessions serviced (128GB dual node ( 4 X 2650 v1 ) From that 128GB , packet filters on either GPUs or X3- pro adapters / or both shall address translate IB access to an 8 node database held in RAM, also runs kNN decision filters for packet ordering / client updates / response back via IB to the front session manager Servers.
i may starve less, if the X2 cards work as IB handlers to and from the session server X-3 or X3pro all via the 5030 switch .. ?

Maybe noone has heard of such an arrangement, but everything i do is totally unique (including eat) most opinions i read are shallow depth , pointless and reminds me of a Whacky Plak i used to collect as a 7 yo .. it said : IF u got nothing to do .. DONT do it here.. as a highest form of American humor, along with National Lampoon. Not much changes.

Now there are a lot of IBM / HP / Dell / EMC and china X2 cards that are useless for VPI ( IB ) they mostly have SFP+ ethernet function.. but there are QDR X2 cards ( rev A2 ) with QSFP ports , that really ought to function well with IS5030 IB fabric switch. I know the X3 VPI cards do.. only i cant afford 16 X3-pro cards, at this point in time (when the bank has foreclosed on $6mil prop)
Thus maybe someone can see the sense of economy where i spend my entire monthly check on Servers, and Mellanox cards , trying to restart what the GFC ( bankers) took away ...

Whats not VPI IB about this X2 ? despite its June 2011 i realize it wont do OFED but i keep reading about great 40G IB fabrics using X2 cards ..ok the 38pin connectors are the double row shown at QSFP cable pic , are these called 4 x QSFP quad data rate QDR= 40Gb
thats another way of saying 4 x 10Gb SFP+ would be less confusing as QSFP or 4xSFP
making things worse there is QSFP // 4 x QSFP // QDR all 40Gb , then DDR = 20Gb and the below card is usually marked as DDR or 20Gb // it looks like QSFP may be a fake offering ..
http://www.mellanox.com/related-docs/products/oem/RG_Dell_SKU.pdf

Here is a mellanox cable MCC4Q28C-003 dont know if it works as a 1 : 1 4 x QSFP ( QDR )
i think the 4 x QSFP is a QDR so will connect the X2 QDR adaptor to 5030 QDR IB port
HP MELLANOX 4X QSFP INFINIBAND COPPER CABLE, in 1:1 40Gb IB cluster?

Now why wont QDR X2 cards work with QDR 5030 switches ? forgetting the 4 x 10Gb splitter on a port for port basis ? that is another thing ( how do u think this line up wont work in IB ?

the cable MCC4Q 28C / as 1 : 1 between this X2 QDR adaptor and ports of a QDR 5030 ?
only reservation is buggy *dated) drivers (2011 ) Vs the VPI X3 adaptors (that work in OFED
just dont kno how buggy this combo ( X2 / 5030 ) may be compared to X3 / 5030

Mellanox ConnectX-2 QDR 40G IB Dual Port Network Adapter
Part Number: MHRH2A-XSR
Mellanox ConnectX-2 QDR 40G IB Dual Port

ConnecX-2 VPI/InfiniBand Firmware Download Center

Version
(Current)
OPN
PSID
Download/
Documentation
2.9.1000
MHZH29B-XSR/XTR
MHZH29-XTR
MHZH29-XSR
MHRH2A-XTR
MHRH29C-XSR/XTR
MHRH29B-XTR
MHRH29B-XSR/XTR
MHRH29B-XSR
MHRH19C-XSR/XTR
MHRH19B-XTR
MHRA19-XTR
MHQH29C-XSR/XTR
MHQH29B-XTR
MHQH29B-XSR/XTR
MHQH29B-XSR
MHQH19C-XTR
MHQH19B-XTR
MHQH19B-XSR
MHQH19B-XNR
MHQA19-XTR
MHGH29B-XTR
MHGH29B-XSR
MT_0F90120008

Looking at the current driver selection across different OS/cards/medium (IB vs. ETH) it looks like the only space consistently supported by Mellanox is Linux. Indeed, for Linux you have everything:

• every card is supported (all the way from Connect-X2 to Connect-X5)
• IB and ETH and the possibility to switch from one to another for the cards that support it (VPI), and you can even use IB on one port and ETH on another
• iSER initiator is supported across the board and iSER target is supported with both LIO and SCST
• SRP initiator is supported across the board and SRP target is supported with SCST

IM NOT USING HYPERVISOR ,, but for completion

So, if you use KVM as your hypervisor , there is no problem.

However, if you want to use Mellanox IB technology in conjunction with currently the most popular hyperwisor (VMware ESXi), you're in trouble:

• there is no official support for ESXi 5.5 and up for any card older than Connect-X3
• the only VPI cards supported in IB mode are Connect-X3/Pro
• Connect-IB cards are not supported at all
• Connect-X4 cards are supported only in ETH mode
• dual-port VPI cards support only the same protocol (IB or ETH) on both ports, not a mix
• SRP initiator is no longer available
• iSER initiator is available only with 1.9.x.x drivers only over ETH and only for Connect-X3/Pro cards

ConnectX-2_VPI_Firmware: fw-ConnectX2-rel-2_9_1000-MHRH2A_A2

Release Date: 09-Jun-11

I know its an old post but doing a search in the forums, only 2 results showed up... and both were in this thread..
searched for
MHGH29B

I found out that a ConnectX-2 does exist with cx4 connector... and was trying to find out about fw, etc as mellanox site says 2.7 is latest to download and your post shows 2.9.1000!!!!!!

playing with win10 and my old ConnectX-1 cards, FW had to go to 2.8 at minimum for Win10 to work...
so ordered card and appreciate your reply from 2017!!!

albeit the card is in your list and not what you were talking about.. just blabbing that search got me this info!!! WOOOOT!!!! HAH..