$1300 to go 4x 40GbE w/ adapters and 48 1GbE Ports

[MiniKnight]

The 40GbE switches like the $800 x 18 port Gnodals are a favorite here. I know.

Here's another idea:

Get a Juniper switch that can do 40GbE for your NAS and virtualization hosts. Have 1GbE for all of the clients.

Step 1: 48x 1GbE and 4x 40GbE switch $950
Juniper EX4300-48T-AFI 48-Port Switch 2x PSU 2x Fans 4x 40GbE Ports Rack Ears | eBay

Step 2: 4x 40GbE Mellanox CX3's for NAS and virtualization hosts 649281-B21 656089-001 661685-001 HP IB FDR/EN 10/40GB 2P 544QSFP ADAPTER | eBay

Step 3: QSFP+ DACs x4 10Gtek for Juniper QFX-QSFP-DAC-1M 40GB QSFP+ to QSFP+ Ethernet Direct Attach... | eBay

Especially if you don't have a ton of need for 40gb that's killer. You could do like vSAN or w/e on the fast ports and still have IPMI and other traffic on the 1gb.

 

[zeynel]

aren't those Nics just 10Gb ethernet and 40/56 Gbit Infiniband ? Does the switch support infiniband ?

sry you're right

 

[pyro_]

I have a ex4300 24p on the way right now and also picked up one of the 4 port sfp+ modules for it. Will be interesting to see what the power draw is on it once setup. Will be able to replace two or three switches with it once it is setup

 

[pyro_]

One other possible option for qsfp cables [NEW] EMC 3M Amphenol QSFP To QSFP Infiniband Cable 038-004-067-01 811-0204-01 | eBay

 

[zeynel]

the seller of the nics accepted $55 each @ 10 pcs.

 

[_alex]

nics from this seller are basically cx354a-fcbt rev a4, got a few from him.

@zeynel: make sure to provide a phone-number / your eori, i had quite a delay from customs

 

[zeynel]

nics from this seller are basically cx354a-fcbt rev a4, got a few from him.

@zeynel: make sure to provide a phone-number / your eori, i had quite a delay from customs

thx for the tip alex.

but i let them send to my freight forworder , they are mostly cheaper and hat never issues with costoms. they have a costom service.

shipito.com

are the nic corssflashable ?


Gesendet von iPhone mit Tapatalk

 

[_alex]

Good to know customs are done right by shipi.to - thanks.

only had a very quick look on them, but i'm quite sure they can be flashed to whatever. waiting for lp brackets as thy are in fat-twin nodes and operating without bracket and cable attached is really sketchy.

 

[i386]

And more ConeectX 3 vpi cards: HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21, | eBay

 

[_alex]

Here is a list of HP Mellanox PN's that contains older NICs:
http://www.mellanox.com/pdf/products/oem/HP-Mellanox-Reference-Guide-August-2012.pdf

 

[zeynel]

nice list , thx @alex

 

[Patrick]

If you did have a small 1/2 or 1/4 rack install this would be super as well. You could put internal networking on 40GbE. Then have wan side/ management/ heartbeat NICs on the 1GbE.

Four servers with 2 data, 1 management 1GbE will use 12. Add a logging server and a firewall. you would be fine with 24 ports but unlikely to find a 24x1GbE + 4x40GbE

 

[pyro_]

Actually it is not unlikely at all they have a Poe+ and a non Poe versions of the ex4300 with only 24 1gig ports. I actually have one on the way right now and should get it sometime in the next week. Will have to wait for the 4x sfp+ 10gig module to come in hat I have ordered for it before I can really start to use it though.

Depending on noise with 24 gig, and 4 each 10gig and 40gig ports this is pretty much my perfect home lab switch

 

[i386]

According to the datasheet, the loudest 48x 1gbe poe switch from the 4300 series is about 42db(a)

 

[pyro_]

Ya I know they are not suppose to be too bad but still want to hear it before making a judgment since it is going to be living 10-15 feet from my bed

 

[whitey]

Yeah now all I need is for all you savages to not snipe those lowball bids haha j/k

 

[i386]

Seller from the op is back with more cards: 649281-B21 656089-001 661685-001 HP IB FDR/EN 10/40GB 2P 544QSFP ADAPTER | eBay

Edit: Offed $40 each for two, seller contered with $55 each

 

[Patrick]

I wonder if we can flash those to stock Mellanox firmware as well.

 

[i386]

I wonder if we can flash those to stock Mellanox firmware as well.

I followed these instructions and flashed my cards with the stock firmware: Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

 

[T_Minus]

For those of us networking n00bs when dealing with the ConnectX cards for 40Gbe is there certain model#s that give features that would be useful for 'every day' type use or are the differences very specific for some use cases? I know there's a PDF that lists features as well as networking performance but the 'features' don't mean much to me not knowing what they mean regarding networking...