basic 10gb setup for two PC's

[donedeal19]

I'm looking for a basic part list to upgrade my nic cards. I will be using windows 8 on both PC to transfer 100gb to a 500gb of movies. I would like to remove the 105MBps limit I'm stuck at currently. Both PC has two ssd configured in raid 0.
Any suggested cards and cables that can be bought off eBay easily?
I have loooked at a few cards but not sure which will play nice with win8.
A coulpe eBay links would help greatly. I read it should not cost no more than 250. I did find two 20gb nic cards for like 120 each but I'm not sure the ones recommended here. Thanks for the help.

 

[Patrick]

What about a Mellanox QDR IB card?

Here are links from another recent discussion:
MHQH19B XTR Mellanox Infiniband QDR 40 Gbps 1 Port Connectx 2 VPI MHQH19 XTR | eBay

Mellanox MHQH29B XSR 40GIGABIT Dual Port Infiniband Adapter 6111 | eBay

Benefit with these are that you can use IB also. The trick will be getting your storage to keep up.

 

[dba]

I recommend picking up two Mellanox ConnectX-2 VPI cards and a QSFP cable. Windows has built-in IPoIB drivers for these and, if you flash the firmware, you can even get RDMA under Windows 8. A QDR card will provide 3,200MB/s of maximum SMB3 file sharing throughput while a DDR card will provide 1,900MB/s - both much faster than 10GbE.

Here's the perfect DDR card for $65, if you can use low profile cards: Mellanox MHRH2A XSR HCA Low Profile | eBay. Buy two and a $40 QSFP cable and you are up to $170. This vendor has a ton of these and will put up custom eBay auctions containing exactly the number that you want. They also sell nice LSI 9201-8i (aka 9211-8i) cards for $75, so you can pick up one of these as well, if you need one.

Here is a good tall bracket QDR card, although it's more expensive at $113: http://www.ebay.com/itm/300931465691

I'm looking for a basic part list to upgrade my nic cards. I will be using windows 8 on both PC to transfer 100gb to a 500gb of movies. I would like to remove the 105MBps limit I'm stuck at currently. Both PC has two ssd configured in raid 0.
Any suggested cards and cables that can be bought off eBay easily?
I have loooked at a few cards but not sure which will play nice with win8.
A coulpe eBay links would help greatly. I read it should not cost no more than 250. I did find two 20gb nic cards for like 120 each but I'm not sure the ones recommended here. Thanks for the help.

 

[mixtecinc]

dba,

Could you give a link to the firmware to get RDMA under Windows 8? Is there any QDR switch you would recommend? Don't need 36 ports. Couple of questions:

1. What is the max length for a QSFP cable?

2. Do any of the QDR switches have Ethernet interfaces? So they can be attached to an existing network

Ideally would like to have switch in basement.

Thanks in advance.


Justin

 

[donedeal19]

Thanks for the replys, so what cable do you recommend from eBay that work with the MHQH29B XSR?
And is the configuration in w8 going to be a pain to setup? Can anyone point me in the right direction. Not sure if there is a thread that covered this already. Thanks.

 

[dba]

Any QSFP cable will work. As for IPoIB (IP over Infiniband) setup, it's very easy. Here you go:

1) Physically install the card and then boot.
2) Configure IP info (address and netmask) under the network control panel, just like a normal Ethernet card. Since it's a point-to-point connection, don't set a gateway on the IB card ports.
3) Optionally, but recommended, download and install the Mellanox driver downloaded from mellanox.com, replacing the Microsoft driver. The package is called WinOF, and the latest installer that I have is named MLNX_VPI_WinOF-4_40_0_All_win8_x64.exe
4) Win 2012 can use RDMA to gain extra speed with lower CPU utilization. I'm not sure if Win8 does the same. Assuming that it does: Optionally, but recommended, confirm that you have RDMA-capable firmware, and re-flash if you don't. A Mellanox card can be re-flashed using the Windows OFED installer while OEM cards from Sun, HP, Dell, etc. cannot. For the OEM cards, see Custom firmware enables Windows 2012 IPoIB RDMA for Mellanox OEM Infiniband Cards - www.openida.com

Thanks for the replys, so what cable do you recommend from eBay that work with the MHQH29B XSR?
And is the configuration in w8 going to be a pain to setup? Can anyone point me in the right direction. Not sure if there is a thread that covered this already. Thanks.

 

[donedeal19]

Thank you for the good article posted and info posted here in this post. I'm almost set on the hardware choices.
Can I keep my onboard nic for internet?
if so will it be hard to both at the same time? Any articles I can read on how to use the two different nic?

 

[Patrick]

Thank you for the good article posted and info posted here in this post. I'm almost set on the hardware choices.
Can I keep my onboard nic for internet?
if so will it be hard to both at the same time? Any articles I can read on how to use the two different nic?

Believe it or not, Windows is fairly good at figuring out if a NIC is Internet facing.

 

[Mapotofu]

Thank you for the good article posted and info posted here in this post. I'm almost set on the hardware choices.
Can I keep my onboard nic for internet?
if so will it be hard to both at the same time? Any articles I can read on how to use the two different nic?

Just one thing you need to do if both of your PCs are connected to your network (which I suppose will likely be the case), is to access the other PC with the IB cards' IP and not the PC's name to be sure they use The IB cards to communicate and not the GbE NICs.

 

[donedeal19]

Thanks,

Just an little update, just got one card that dba mentioned for 113$. Looking for another to pop up for around the same price.
MHQH19B XTR Mellanox Infiniband QDR 40 Gbps 1 Port Connectx 2 VPIMHQH19
XTR

So all I would need to do is update the firmware if needed correct?

Windows 2012 support smb 3 and windows 8 uses 2.2 smb is there going to be any speed difference?

 

[mrkrad]

smb3 was called smb2.2 it is smb 3 for windows 8 and 2012 (they share the same kernel pretty much).

The main benefit is design to ramp up for these faster nic's and multi-stream since single-stream will have a real hard time reaching peak potential speed.

It's real easy to make 1 connection go 2-4 gigabit (out of say 10gbe) but with smb3 it can open up multiple connections to get that full speed.

Even better 2012 has built bonding so you could take a dual port nic and use both of them. (Never use the built in bonding on an AD server, just a tidbit to keep in the back of your head).

 

[dba]

You are in luck: Mellanox MHQH19B XTR PCI E Single Port QDR Infiniband Host Bus Adapter HBA | eBay

I have several of these but have largely switched to servers with low-profile brackets.

Thanks,

Just an little update, just got one card that dba mentioned for 113$. Looking for another to pop up for around the same price.
MHQH19B XTR Mellanox Infiniband QDR 40 Gbps 1 Port Connectx 2 VPIMHQH19
XTR

So all I would need to do is update the firmware if needed correct?

Windows 2012 support smb 3 and windows 8 uses 2.2 smb is there going to be any speed difference?

 

[MiniKnight]

Just one thing you need to do if both of your PCs are connected to your network (which I suppose will likely be the case), is to access the other PC with the IB cards' IP and not the PC's name to be sure they use The IB cards to communicate and not the GbE NICs.

This is a GREAT tip! I'd tried this an faltered because I didn't do exactly this. Took me a long time to figure this out.

 

[donedeal19]

Is this your auction DBA?

If so thanks for the help. Any chance you have a spare cable to help me get running?

 

[dba]

It is my auction - I have some spare cards. I don't, however, have any spare cables.

Is this your auction DBA?

If so thanks for the help. Any chance you have a spare cable to help me get running?

 

[donedeal19]

Just an update, bought my 2nd card with latest firmware, thank you dba. Now do I need the 40gb cable?
Found this Mellanox mc2206130 40gb cable on eBay. Is the one I need to be using? Alot of different styles cables there so don't want to buy the wrong type. Thanks.

 

[dba]

That part number will work. In fact, pretty much any cable with a QSFP connector on both ends will work.

Just an update, bought my 2nd card with latest firmware, thank you dba. Now do I need the 40gb cable?
Found this Mellanox mc2206130 40gb cable on eBay. Is the one I need to be using? Alot of different styles cables there so don't want to buy the wrong type. Thanks.

 

[donedeal19]

First card just arrived and the first thing I noticed is that is an low profile card. Installed on a windows 8 PC and added new drivers. Firmware shows that is on 2.7.626.

About to try and flash it, is there anything I need to know before proceeding?

 

[dba]

Just follow: Custom firmware enables Windows 2012 IPoIB RDMA for Mellanox OEM Infiniband Cards - www.openida.com

First card just arrived and the first thing I noticed is that is an low profile card. Installed on a windows 8 PC and added new drivers. Firmware shows that is on 2.7.626.

About to try and flash it, is there anything I need to know before proceeding?

 

[dba]

dba,

Could you give a link to the firmware to get RDMA under Windows 8? Is there any QDR switch you would recommend? Don't need 36 ports. Couple of questions:

1. What is the max length for a QSFP cable?

2. Do any of the QDR switches have Ethernet interfaces? So they can be attached to an existing network

Ideally would like to have switch in basement.

Thanks in advance.


Justin

QSFP Infiniband cables are available from half a meter long up to at least 100 meters.
the Mellanox 4036E is probably the least expensive IB switch/Ethernet bridge, but it's extremely expensive - as in $12K or so.