Mellanox MHQH19B-XTR for half price

Patrick

Administrator
Staff member
Dec 21, 2010
11,938
4,892
113
Great price on the cards. I did a search on the HP 592520-B21 and didn't see any that low unless you OBO'd.

Would love to see details of that project.
 

wuffers

New Member
Dec 24, 2012
19
0
0
Aha, they were not bought through eBay and therefore not OBO'd. I might have bought out the supply (12 listed, bought 12), so I'm not too sure if more are available. I don't want to shoot myself in the foot just yet because I may require more, so I'll share once I determine that my needs are met.

Just a quick summary of the project:
- Mixed VMware and Hyper-V virtualization environment
- InfiniBand SAN fabric for a 100TB SAN using ZFS
- 10GbE networking for hosts

All on a relatively "tight" budget - all the SAN fabric and networking gear are all used/refurb, saving us a bundle.

More details later.
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
how will this compare to the netgear XS* 10gb 8,12,24 port switches and 10gbase-T ?
 

cafcwest

Member
Feb 15, 2013
136
14
18
Richmond, VA
Aha, they were not bought through eBay and therefore not OBO'd. I might have bought out the supply (12 listed, bought 12), so I'm not too sure if more are available. I don't want to shoot myself in the foot just yet because I may require more, so I'll share once I determine that my needs are met.

More details later.
Inquiring minds would love to know of such a source, if you find that your need has been fulfilled.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
Wuffers, have you installed your Mellanox switches yet? I could use some advice on connecting to them via serial since I do not have the original DB9 cable.

+1 on this.

I called him up for some LP brackets for the shiny new (well.. refurb) HP 592520-B21 (Mellanox MHQH29C-XTR equivalent, which is a ConnectX-2 dual port QDR adapter), which I picked up @ $100 each.

The 12x HBAs along with 3x HP Voltaire Grid Director 4036 (coincidentally, also featured in Great Deals forum) are for a work project, of which InfiniBand will be the SAN fabric.

Probably going to go with Amphenol QSFP+ cables unless someone can recommend something else. I need 22 of them of varying lengths.
 

wuffers

New Member
Dec 24, 2012
19
0
0
Sorry guys, wasn't checking the thread and was away on vacation in Europe last week.

Inquiring minds would love to know of such a source, if you find that your need has been fulfilled.
Aha, yes. I don't need any more for the project and don't mind sharing. The secret was searching for "592520R-B21", notice the R for HP's Renew program.

Unfortunately, the supplier I got them from no longer has these in stock, but I took all 12 from ACOS:



You may want to try:
Codemicro
or
Mantecho

but my suspicion is that they all worked from the same inventory pool. The latest prices I can see for refurb dual QDRs now are ~$300 each. Sorry!

holy crap,

way good prices and compatible with freebsd for anyone who runs zfs like that

freebsd-infiniband - George Neville-Neil - Re: Infiniband experiment environment setting up

how difficult is this to get up and running, i know it's not a drop in upgrade like an xr997
I have all the SAN fabric in (Infiniband) but still no servers, JBODs or drives yet, although I did get my L2ARC SSDs (Smart Optimus 400GB x4). Still trying to figure out best distro to go with but it won't be FreeBSD at all - it'll be an illumos one.

Wuffers, have you installed your Mellanox switches yet? I could use some advice on connecting to them via serial since I do not have the original DB9 cable.
Yup, got them all installed, tested and flashed latest firmware. It's just a normal null modem serial cable (DB9 F to DB9 F).
 

xnoodle

Active Member
Jan 4, 2011
259
48
28
Can I connect these to a Intel 82599 based 10gbE card (TCK99 for the C6100) using a QSA<->SFP+ converter and a SFP+ to SFP+ copper cable?
 

wuffers

New Member
Dec 24, 2012
19
0
0
how will this compare to the netgear XS* 10gb 8,12,24 port switches and 10gbase-T ?
I'm not going to get into the Infiniband vs 10GbE debate here. They're simply two different beasts depending on what you use it for.

The Netgear 8 port switch (XS708E should be going for about $1000 street price) with 10GBase-T would make things a lot simpler with cabling (and support for legacy CAT5/6 cabling and NICs) and cost per port is in the "affordable" range, versus their higher end 24 port which should probably go for $6000 street (versus say $8000 for a Brocade or Dell 10GbE switch). I think the issue would be trying to acquire 10GBase-T NICs on the cheap (most older models that are going for around $50 would be CX4 or SFP+ I believe).

I ended up purchasing two used Dell PowerConnect 8024Fs for the LAN portion of the project, for under $2500 per switch. Close to $100 per port, not factoring in cost of cabling. Add about $20-25 per occupied port (I paid between $40-45 per Cisco SFP+ cable) and that was my 10GbE networking cost.

Can I connect these to a Intel 82599 based 10gbE card (TCK99 for the C6100) using a QSA<->SFP+ converter and a SFP+ to SFP+ copper cable?
I don't see why not. I haven't tried direct connect between the two adapters but currently I have my Intel X520-DA2 (which is a 82599 based controller) connected vis SFP+ to the switch, and a ConnectX-2 VPI card connected via a QSA with SFP+ cable to the same switch. I had issues with getting link to the Infiniband adapter with the QSA and SFP+ cable, but everything just worked after installing the RoCE drivers. I'd have to test again with the Mellanox WinOF drivers as they should work. It wasn't simply "auto-detect" as told to me by a Mellanox sales engineer.

I can test this out real easily when I'm back in the office.