Infiniband for dummies Part.1

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

33_viper_33

Member
Aug 3, 2013
204
3
18
Looks good so far! I look forward to seeing part 2. You should post some pictures of your lab. Once I get a working infiniband network and build the new server (going much slower than anticipated) I will do the same.
 

33_viper_33

Member
Aug 3, 2013
204
3
18
Again, awesome. Maybe add some of the commands utilized for getting infiniband drivers installed in vmware/linux/solaris and details of the subnet manager requirement would be good additions. There are some other articles out there, but it takes quite a lot of digging. Or do I hear part 3????

This was exactly what I had in mind as I learn infiniband. As soon as I have a good grasp, I was going to make a guide for dummies and post it here. It may make a good sticky thread if someone in the know could break down exactly what is required hardware and software wise, compatibility, general info to what infiniband is and the protocols utilized, and a step by step guide of the various setups. Its a big job, but it would slow down the constant infiniband questions of which I am guilty to contributing to. You are 100% correct in saying there is currently no one stop infiniband shop.

Just a nitpick, I found a spelling error and a typo in the following sentence both with quotes around them.

The software is actively being developed, with new versions almost every week, along with "am" author that is very "resposive" and will add in new features if needed.
 

TD_Trader

Member
Feb 26, 2013
63
7
8
@DarkyToo - I was just reading your tutorial, and I noticed that the speeds you listed in your tutorial were/are wrong.

You stated:
SDR: Single Data Rate, this is referred to as 10mb, when it’s really only 8. This is the cheapest speed to start with Infiniband, switches run about $200. These use a CX4 Connector
DDR: Double Data Rate, this is referred to as 20mb when it’s really only 16. DDR switches start around $400-$500. DDR Connections still use a CX4 Connector
QDR: Quad Data Rate, this is referred to as 40mb, when it’s really only 32. QDR switches start around $1000. For QDR and above connections, the use a QSFP Connector.
FDR & EDR: these run at 50mb and 100mb. These are beyond the scope of this article.

I believe you meant to say "10Gbps" (instead of 10mb), and meant to say "20Gbps" (instead of 20mb), and meant to say "40Gbps" (instead of 40mb). Also I believe FDR and EDR are 56Gbps and 100Gbps respectively.

See here: InfiniBand - Wikipedia, the free encyclopedia

SDR, DDR, and QDR use 8b/10b encoding, so you'll lose about 20% of the bandwidth due to the encoding. So yes, a 10Gbps link (4x SDR) would only be about 8Gbps of throughput and a 20Gbps link (4x DDR) would only be about 16Gbps of throughput. A 40Gbps link (4x QDR) would only be about 32Gbps of throughput.

FDR-10, FDR and EDR use 64b/66b encoding, so you're only losing about 3% of the bandwidth due to the encoding (instead of 20% with 8b/10b). So a FDR-10 40Gbps link (4x FDR-10 @1.25GB/s per link = 5GB/s = 40Gbps theoretical) would get about 38.788Gbps of throughput (with 64b/66b encoding). A FDR (4x FDR/FDR-14 @ 1.71GB/s per link = 6.84GB/s = 54.72Gbps theoretical) would get about 53.062Gbps of throughput (with 64b/66b encoding). An EDR (4x EDR @ 3.13GB/s per link = 12.52GB/s = 100.16Gbps theoretical) would get about 97.125Gbps of throughput (with 64b/66b encoding).

Keep in mind, that is just based on data rates, and none of these calculations take into account the additional physical layer overhead requirements for common characters or protocol requirements (such as StartOfFrame and EndOfFrame, etc.)

Would love to see you update your page/tutorial, and I hope this helps...

Thanks,
Mark
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
One thing that I never see anyone mentioning is the fact that IB and SAS cables are pretty similar but not interchangeable so if you're shopping for an IB cable and fail to see SAS written, you may be in for a shock. A SAS 8088 connector is physically shorter than a QSFP connector. On the CX4 world, I'm not sure if the carrying capacities of the SAS cables are the same as the IB cables so you often find the SDR CX4 cables with SAS in the name. Luckily for me, all my mistake cables cables have found uses in other projects.
 
  • Like
Reactions: Mark D

Darkytoo

Member
Jan 2, 2014
106
4
18
thank you for the feedback all! I've made the requested and suggested changes, along with adding more info on cables. I had forgotten about the whole SAS cable issue.