Server to server cable and NIC for low cost max speed

[ptr727]

Hi, I have about 50TB of data to copy between two linux servers, stacked next to each other.
Over 1gbps it will take about 60 days.

What are my options for cheap 10gbps or faster speeds with crossover cables and dedicated NICs?

 

[Rock]

What are the thru-put limits of your storage drives?

Mellanox ConnectX-3 40Gbps QSFP+ cards are available pretty reasonably on ebay or this forum and can use fiber or direct connect copper transceivers.

For MultiMode Fiber cable I think you need a crossover if you are not using a switch.

 

[mshook]

MDI-X has pretty much solved the cross-cable thing for years now: most proper interfaces do the right thing when it comes to connecting 2 computers directly with a straight cable. AFAIK it's the same thing with SFP+.

As for cheap cards, you can 10Gb interfaces for something like $15 to $40 on ebay...

 

[BeTeP]

You can get 2x 40GbE cards + DAC for under $70 shipped. Just a few months ago you could have got the same kit for under $30.

 

[ptr727]

Thanks for the info.
I won't get more than ~240MBps out of the drives, so 10G would be fine, but if $ for 10G is close to $ for 40G I'll obviously go for 40G.

Any specific models I should look for, used on eBay, or off-brand new like 10gtek?

 

[BeTeP]

Get Mellanox ConnectX-3 CX354A. It is cheaper to get OEM variant like this HP 544QSFP and then flash it to retail MCX354A-FCBT to get 40/56GbE.

Passive 40Gb DAC can be found for as little as $5 sometimes. Or you can spend a bit more and get a Mellanox branded 56Gb/FDR14 capable cable instead

 

[rahulkadukar]

I think you are off by a factor of 10. This should take about 6 days.

 

[ptr727]

Yep, my bad typing 6 days, not 60 days. So by the time I'm done ordering and isntalling it is probably over, but next time I can do it in a day

 

[ptr727]

I ordered used Mellanox MCX354A-FCBT and Mellanox MC2207130-002 cards and cables of ebay.

One of the servers I'm moving data from is Unraid, and I read that Unraid does not support infiniband, so I need to run the cards in ethernet mode.
See: Mellanox ConnectX-3 support

I've never ran any infiniband equipment before, is there a difference in link speed if the card is set to run in ethernet mode?