Mellanox MHQH19B-XTR

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Z

zakari

New Member
Sep 10, 2012
9
0
1
Hello,

After reading the article about this network adapter, I went ahead and bought 2 of them.

I'd like some guidelines on their basic functionnalities though; My understanding is the following :

I'm currently using Win 7 on my "working" PC & 2008 R2 on my Storagebox (Norco + areca). If I put those 2 cards in there, I will use them only as 10GB Ethernet adapter (and will have to play around with DNS or hosts file to make sure I go through the good adapter for communication between those machines).

Whenever I make the switch to Windows 8 & Server 2012, I'll be able to use SMB Direct which a sort of low level access to the servers shares (not sure I fully get how that works, but the basics are here I believe).

Thank you for your well written articles, and usefull infos in the forum it has helped me a lot building my storagebox.
 
C

cactus

Moderator
Jan 25, 2011
830
75
28
CA
For the Win7 setup you can use IPoIB(10GbE adaptor) or SRP(SCSI over RDMA, similar to iSCSI). You can also use an IP to map a network drive or browse and not have to deal with DNS/host file.

For Win8, from what I have read, SMB3 will dynamically detect RDMA/Multi links to the storage server and use them.
 
Z

zakari

New Member
Sep 10, 2012
9
0
1
Thanks for the reply, I think I'll try both SRP & IPoIB but will probably end up using the IP option which looks fairly "usable" (problems with multiple simultaneous access to one target & stuff, not sure I'm want to go this way).

Also (that might sound newbish :) ) I need that kind of cable right ? link which is called either QSFP or SFF 8436 ?
 
C

cactus

Moderator
Jan 25, 2011
830
75
28
CA
Yes you need QSFP to QSFP. I have been watching for cheap short ones for a week or so and have not found any. Got one best offer down to $35 x2 + $10 shipping. If you are ok with overly long cables this is what I think I am going to end up getting.
 
Last edited:
Z

zakari

New Member
Sep 10, 2012
9
0
1
With the p/n I was able to find the 2m version which exactly suits my needs ! thanks for that !

it's not that cheap @ 70$ but I really care for shorter length, especially on those very high bandwidth copper links
 
R

RackIt

New Member
Sep 21, 2012
12
0
1
Without a switch, would the following configuration work?

A file server with dual IB (MHQH29C-XTR) connected to two servers with a single IB (MHQH19B-XTR) ?

Would each IB on the file server need to use a different subnet?

Thanks.
 
C

cactus

Moderator
Jan 25, 2011
830
75
28
CA
Rackit, Infiniband does not use IP based traffic unless you are using IPoIB. RDMA traffic is done with Port, Node, and System image GUIDs, similar to MAC addresses on Ethernet. IPoIB adapters act similar to other kinds of network interface; excluding bonding/link aggregation/routing, each machine should only have one active IP per subnet.

With Linux you can bridge two Ethernet interfaces together to make the system act like a layer 2 switch, see bridge. Windows also has bridging capability, but I am not sure if either work with Infiniband.
 
R

RackIt

New Member
Sep 21, 2012
12
0
1
Would I need to bridge if all I want to do is have the two servers be able to talk to the file server?

Meaning:

FILESERVER-PORT1 <-----> SERVER1
FILESERVER-PORT2 <-----> SERVER2

And SERVER1 cannot communicate to SERVER2 via IB

And I was thinking of using IPoIB or is that not even needed in my case?

thanks.

cactus said:
Rackit, Infiniband does not use IP based traffic unless you are using IPoIB. RDMA traffic is done with Port, Node, and System image GUIDs, similar to MAC addresses on Ethernet. IPoIB adapters act similar to other kinds of network interface; excluding bonding/link aggregation/routing, each machine should only have one active IP per subnet.

With Linux you can bridge two Ethernet interfaces together to make the system act like a layer 2 switch, see bridge. Windows also has bridging capability, but I am not sure if either work with Infiniband.
Click to expand...
 
C

cactus

Moderator
Jan 25, 2011
830
75
28
CA
Bridging would give you full R/W to the entire tree. If that is not needed, setting up two point to point subnets would work fine.

What OS are you running on you machines?

Currently for SMB(Windows and Samba) you need to use IPoIB. Win8 and Server '12 will allow for SMB Direct(SMB over RDMA, no IP layer) with Mellanox Connect X2 or newer cards. iSER(iSCSI with RDMA read/write), SRP(SCSI RDMA Protocol), and NFSoRDMA do not need IPoIB.
 
R

RackIt

New Member
Sep 21, 2012
12
0
1
Windows Server 2012 on all three servers. I'm wanting to use IB solely because the two servers will be HyperV with the VM drives on the SAN so they can handle quick failover.

Today I just finalized on the SAN hardware. SuperMicro SC846-E26 chassis, X9SRA with Xeon E5-2609, LSI 9271-8i with CacheCade, CacheVault and recovery software. Two 256GB Samsung SSD's for the CacheCade and a bunch of 2GB SAS HDD drives.

cactus said:
Bridging would give you full R/W to the entire tree. If that is not needed, setting up two point to point subnets would work fine.

What OS are you running on you machines?

Currently for SMB(Windows and Samba) you need to use IPoIB. Win8 and Server '12 will allow for SMB Direct(SMB over RDMA, no IP layer) with Mellanox Connect X2 or newer cards. iSER(iSCSI with RDMA read/write), SRP(SCSI RDMA Protocol), and NFSoRDMA do not need IPoIB.
Click to expand...
 
Last edited:
R

RackIt

New Member
Sep 21, 2012
12
0
1
I now have SAN1 with a dual-port Mellanox 29C and then two servers SERVER1 and SERVER2 each with a Mellanox 19B.

Any recommendations on configuration of the Mellanox adapters for use with SMB 3.0? Must I use ETH (10Gb) or can I use IB (40Gb)?

Currently I have each port of SAN1 configured as ETH and for 10.10.1.1 and 10.11.1.1 but it is showing as a public network. I'm wondering if I should leave it like that or try and get it to be private. And I assume I don't want those addresses in my ActiveDirectory DNS?

I think I'm needing help on some basic NIC priority and DNS configuration issues in having the two networks.

Thanks.

p.s. In a quick test I was seeing a constant 200MB/s when copying a 20GB file over the connection for the first time. The file is then cached on the LSI CachCade (1 SSD) and subsequent transfers occur at a rate of 500MB/s. Not bad. Tomorrow I'll be configuring CacheCade with 2 SSD's so the transfers should be a bit higher.
 
Last edited:
Z

zakari

New Member
Sep 10, 2012
9
0
1
I think I'm doing something wrong here.

Both my 2 19B are showing up as "device cannot start (code 10)" on 2 different motherboards.
Either I got screwed, or I'm a newbie :)

Any ideas ? (I tried googling the issue and different openfabrics versions without succces).

Thanks !

edit : actually it's starting with ofed version 2.1.3, but no network device show up
 
Last edited:
E

ehorn

Active Member
Jun 21, 2012
342
52
28
carpenike said:
Hey Everyone,

Similar to RackIt, I'm attempting to build a Hyper-V / SMB 3.0 / RDMA setup. Having a few issues that I detailed below:

http://social.technet.microsoft.com...N/thread/5b3929f9-93a0-4b7e-afa2-2db98ca60952

Anyone know if I can get the RDMA Capable -- True or will it always just be IPoIB / 10GB / Eth?

Thanks,
Ryan
Click to expand...
Hello carpenike,

In technet link, you say; "I believe I've purchased the exact hardware models as described in the article." Can you confirm the exact model number of the HBA's you are using?
 
C

carpenike

Member
Mar 27, 2011
42
3
8
Hi Ehorn,

Certainly -- thank you for responding to my issue!

Here's a screenshot taken of the adapter information:



Based on information in this table:

http://www.mellanox.com/content/pages.php?pg=firmware_table_ConnectX2IB

The PSID MT_0D90110009 lines up with:



Which therefore gives the model number: MHQH19B-XTR

Thanks for any help you can provide!

Ryan

ehorn said:
Hello carpenike,

In technet link, you say; "I believe I've purchased the exact hardware models as described in the article." Can you confirm the exact model number of the HBA's you are using?
Click to expand...
 
E

ehorn

Active Member
Jun 21, 2012
342
52
28
carpenike said:
Hi Ehorn,

Certainly -- thank you for responding to my issue!

Here's a screenshot taken of the adapter information:



Based on information in this table:

http://www.mellanox.com/content/pages.php?pg=firmware_table_ConnectX2IB

The PSID MT_0D90110009 lines up with:



Which therefore gives the model number: MHQH19B-XTR

Thanks for any help you can provide!

Ryan
Click to expand...
np Ryan,

Just trying to stimulate some thought. The images did not show up for me. As you note, the fw does line up.

Are there any physical markings on the card itself which identifies it by Part numberings?
 
You must log in or register to reply here.
Top Bottom