Mellanox MHQH19B-XTR for half price

Patrick

Administrator
Staff member
Dec 21, 2010
11,938
4,892
113
I didn't get to purchase for resale, but I did get them to play with at STH HQ. Need to get some parts inexpensively to keep costs down.
 

ehorn

Active Member
Jun 21, 2012
342
52
28
... maybe next time.
They did move quick at that price... I suspect the big boys wont wait long to move to FDR and we will see more of these QDR's hitting the market...

I hope to get a couple more of these over the coming months and am watching for deals on NIC's and QDR switches. I will post up any deals I come across.

peace,
 

cactus

Moderator
Jan 25, 2011
829
77
28
CA
Ehorn, when you talked to Mellanox, did you ask about the QDR low profile bracket only or was it more general? I need a LP bracket for the CX4 cards.
 

ehorn

Active Member
Jun 21, 2012
342
52
28
Ehorn, when you talked to Mellanox, did you ask about the QDR low profile bracket only or was it more general? I need a LP bracket for the CX4 cards.
Hi cactus,

It was specific to this QDR NIC. The guy was very helpful. I am sure they can take care of you..

Here is his contact info:

Gerry Hunt
Mellanox Technologies
+1 (978) 439-5491

Hope that helps.

peace,
 

awedio

Active Member
Feb 24, 2012
463
82
28
For anyone interested, Jose Barreto has published an updated "How-To" for a simple setup with these NICs and SMB 3.0...

http://blogs.technet.com/b/josebda/...connectx-3-using-infiniband-step-by-step.aspx

peace,
To add on to ehorn's post

Currently, SMB 3.0 is 100% Windows only and requires Server 2012 & or Windows 8

NetApp & EMC seem to be the only 2 vendors that have "announced" upcoming product. (I'm sure others are coming)

No SMB 3.0 support from the ZFS/Linux NAS & iSCSI manufacturers
 

ehorn

Active Member
Jun 21, 2012
342
52
28
To add on to ehorn's post

Currently, SMB 3.0 is 100% Windows only and requires Server 2012 & or Windows 8

NetApp & EMC seem to be the only 2 vendors that have "announced" upcoming product. (I'm sure others are coming)

No SMB 3.0 support from the ZFS/Linux NAS & iSCSI manufacturers
Indeed...

But to be clear, any SAN implementation can slip in beneath Storage spaces (via traditional RDMA) and then on to SMB 3.0 through the Windows network. If one wishes, they can have ZFS implementation as SAN and present logical "disks" to storage spaces (Microsoft LVM). This is basically what virtual storage arrays (VSA's) are. The initiator does not know (or care) who/what the target is (be it File system/OS/whatever). The initiator simply sees it as locally provisioned storage.

peace,
 
Last edited:

awedio

Active Member
Feb 24, 2012
463
82
28
Indeed...

But to be clear, any SAN implementation can slip in beneath Storage spaces (via traditional RDMA) and then on to SMB 3.0 through the Windows network. If one wishes, they can have ZFS implementation as SAN and present logical "disks" to storage spaces (Microsoft LVM). This is basically what virtual storage arrays (VSA's) are. The initiator does not know (or care) who/what the target is (be it File system/OS/whatever). The initiator simply sees it as locally provisioned storage.

peace,
ehorn,

Could you pls re-explain (maybe with some line drawings). I'm not aware of how you can present logical disks (via ZFS, Linux etc) to SS & still have SMB 3.0

I've even exchanged emails with Dr SMB (aka Jose Baretto) & I came away with the understanding that it wasn't possible.

My ideal scenario would be to have ZFS as the "backend" of an SMB 3 storage stack
 

cactus

Moderator
Jan 25, 2011
829
77
28
CA
awedio, I believe he just means using Solaris/Linux servers or NetApp/EMC filers to manage storage that is exported to a Windows box and then shared over SMB3. For example, use SRP and export a drive to Windows server, then share resources over SMB3. You can even get a semi multipath setup using multiple SRP targets and RAID0 them on the Windows sever. I feel(don't have experience directly) this is similar to how you would use a NetApp or EMC filer; export LUNs to the server which manages access.

ehorn, is Windows 8 support confirmed? To me SMB3 would be the only reason to go to W8 over W7. It would also be really nice for home setups(if host and client roles are allowed) where buying Windows Server is most of the time out of the budget.
 

awedio

Active Member
Feb 24, 2012
463
82
28
awedio, I believe he just means using Solaris/Linux servers or NetApp/EMC filers to manage storage that is exported to a Windows box and then shared over SMB3. For example, use SRP and export a drive to Windows server, then share resources over SMB3. You can even get a semi multipath setup using multiple SRP targets and RAID0 them on the Windows sever. I feel(don't have experience directly) this is similar to how you would use a NetApp or EMC filer; export LUNs to the server which manages access.

ehorn, is Windows 8 support confirmed? To me SMB3 would be the only reason to go to W8 over W7. It would also be really nice for home setups(if host and client roles are allowed) where buying Windows Server is most of the time out of the budget.
The way I understand it, the only way you can "share" over SMB 3 is both or multiple endpoints (host & client) must be Server 2012 & or Win 8
 

ehorn

Active Member
Jun 21, 2012
342
52
28
Good morning awedio/cactus,

The way I understand it, the only way you can "share" over SMB 3 is both or multiple endpoints (host & client) must be Server 2012 & or Win 8
Yes, that is my understanding as well. SMB3 needs W8 endpoints.

You would present storage to Server 2012 via iSCSI/RDMA (how it is done today). Then Storage Spaces manages the volumes and exposes them within the W8 (servers and clients) network. At this point you have SMB3 endpoints - to and from W8 (server and client) machines. I will work up a visual this afternoon.

ehorn, is Windows 8 support confirmed?
I have not confirmed it myself. I see that iSCSI initiator is still available in W8. I see no value in deprecating this and MS would be unwise to remove this interop. But I would have to test it myself (iSCSI <--> W8 Storage Spaces).

peace,
 
Last edited:

ehorn

Active Member
Jun 21, 2012
342
52
28
... he just means using Solaris/Linux servers or NetApp/EMC filers to manage storage that is exported to a Windows box and then shared over SMB3. For example, use SRP and export a drive to Windows server, then share resources over SMB3. You can even get a semi multipath setup using multiple SRP targets and RAID0 them on the Windows sever. I feel(don't have experience directly) this is similar to how you would use a NetApp or EMC filer; export LUNs to the server which manages access.
Yes, to my knowledge - that feature and ability is not going away... To be able to attach to SAN (of any flavor) via iSCSI is still present.

As you know, up until SMB 3.0, MS did not have anything to offer users looking for high-speed, low latency interconnect within the Win <--> Win stack. Engineers/Architects had the choice of ethernet, fiber or IB, but native IB was stunted in Windows as MS only offered IBoIP (Infiniband over IP) within Windows <--> Windows stack (where Infiniband sat atop IP).

Even still Infiniband was often faster than ethernet/Fiber offerings. But MS was not leveraging the true power and effeciency of IB as IBoIP was bloated with the necessary protocol translation overhead and, as such, placed a heavy demand on the hardware.

Now comes SMB3, offering RDMA via SMB Direct for Windows <--> Windows and finally giving the community high speed, low latency, and low hardware utilization. Folks can now use SMB Direct as a preferred inter-op between physical devices in a windows network.

So SMB3 is a MS implementation for a Windows environment, giving MS an effective means for high-speed, low latency interconnection. That is great news for Windows. IMHO, they are a bit late to this party. Better late than never I suppose and they appear to have brought some cool new party favors too (RSS, easy teaming, etc...) :)

But I have read nowhere that MS intends to deprecate iSCSI over RDMA as a means to attach to external storage devices. This would make no business sense as so many DC's today employ SAN appliances which rely on such a protocol.

I do not see SMB Direct as a direct replacement for iSCSI. I do see it as an alternative - if one wishes to host their storage arrays on Windows (via Storage Spaces).

But before Windows 2012, what were the options for high speed interconnects within a windows environment? Not only storage <--> compute but storage <--> storage, compute <--> compute, compute <--> workstation?

To my knowledge, there was never much of a mainstream offering there... Until now (SMB3).

What are your thoughts here?

peace,
 

awedio

Active Member
Feb 24, 2012
463
82
28
Good morning awedio/cactus,



Yes, that is my understanding as well. SMB3 needs W8 endpoints.

You would present storage to Server 2012 via iSCSI/RDMA (how it is done today). Then Storage Spaces manages the volumes and exposes them within the W8 (servers and clients) network. At this point you have SMB3 endpoints - to and from W8 (server and client) machines. I will work up a visual this afternoon.
I don't think presenting storage via iSCSI/RDMA will work (Of course, I don't have any proof since I have not tried this)

I think the only 2 valid options are:

1) Shared SAS (aka DAS) using SAS HBA

2) LSI's HA-DAS card http://www.ha-das.com/

Looking at some of the drawings from TechEd 2012, they all used some form of Shared SAS or HA-DAS

The Cluster-in-a-box setup that was shown also does the same
 

ehorn

Active Member
Jun 21, 2012
342
52
28
I don't think presenting storage via iSCSI/RDMA will work...
Perhaps, I have not yet tested it myself.

If I was MS, I would ask myself, why include iSCSI Initiator in Windows Server 2012 and Windows 8 if SMB 3.0 is the only offering for replacement?

If I was MS, I would ask myself; Who would I market Windows 2012 to? Only those Data Centers and businesses which are willing to migrate their entire storage array network to Storage Spaces?

I imagine I would not be involved in technical strategy for MS very long for promoting such ideas... ... :)
 

awedio

Active Member
Feb 24, 2012
463
82
28
SMB3 is not offered as a "replacement" to iSCSI

MS customers were asking for a simpler, easier, cost effective (cheaper than SAN) + ability to use their already in-place existing technology (file share) way to manage storage

It just so happens that MS has decided to enhance the File Share (multi channel SMB3) & make it faster & higher performance than iSCSI

btw, those enhancements are not available when using iSCSI
 

ehorn

Active Member
Jun 21, 2012
342
52
28
SMB3 is not offered as a "replacement" to iSCSI

MS customers were asking for a simpler, easier, cost effective (cheaper than SAN) + ability to use their already in-place existing technology (file share) way to manage storage

It just so happens that MS has decided to enhance the File Share (multi channel SMB3) & make it faster & higher performance than iSCSI

btw, those enhancements are not available when using iSCSI
Agreed. :)
 

awedio

Active Member
Feb 24, 2012
463
82
28