Have I accounted for everything needed for a 40g direct connect

[Deano]

This is my first time attempting networking anything and even after 48 hours of trying to learn new terminology, technology and acronyms i'm still worried about blind buying and ending up with incompatible parts that i'd need to return. I'm a little embarrassed asking such basic questions when a lot of the other ones posted here seemed a lot more complex, but at least asking and being embarrassed is free.

Dell MHQH19B-XTR Mellanox ConnectX-2 1-Port 40Gbps Network Adapter Card for the NAS

FUJITSU MELLANOX CONNECTX-2 40GB SINGLE QSFP NETWORK ADAPTER CA05954-1623 for the PC

FLYPROFiber 30M OM4 LC to LC Fiber Patch Cable | Length Options: 0.5m-50m, 40GB/100GB Duplex LC-LC 50/125um Multimode Fiber cable running between them

StarTech.com Arista Networks QSFP-40G-XSR4 Compatible QSFP+ Module - 40GBASE-SR4 - 40GbE Multimode Fiber MMF Optic x2
Transceiver

Did I miss anything? will a Dell and a Fujitsu branded Mellanox cards communicate with each other just fine? is that the right type of cable for the job? will a transceiver for "Arisa" networks play nicely with Mellanox cards, I could find all sorts of transceivers for "Cisco" and others but the prices started to rise and i still have no idea if they'd work for Mellanox.

Once everything is up and running assuming I didn't buy any fake items i change the network to 10.xx.xx.01 and 10.xx.xx.02 and increase the MTU / jumbo packet and it's done?

And finally did i do this the cheapest way? the two computers are probably somewhere between 8-10 meters apart, i only ever plan to have two dirrect connected and i don't mind which technology i do this with over physical cables I.E i'm fine with RJ45 if there is not much difference

 

[i386]

These connect-x2 support only 10 Gigabit/s ethernet, no 40.

Personaly I would get 2x connect-x3 vpi cards (that can do etthernet and infiniband), two used qsfp+ cisco transceivers (~$20-$30, example: Cisco QSFP-40G-SR4 40GB QSFP+ 10-2672-02 Transceiver Module >10pcs available | eBay) and a mpo cable with the desired length.

 

[NablaSquaredG]

Do yourself a favour and don't use ConnectX-2

I would recommend OEM ConnectX-3s, like Oracle 7046442 ConnectX QDR InfiniBand +40Gbs QSFP+ Network Adapter Card CX354A | eBay

You can crossflash them from QCBT to FCBT firmware and even get 56G instead of 40G (with appropriate cabling / transceivers)


Re Transceiver and cables: NO, this is not going to work
You have chosen Transceivers with an MTP / MPO connector (12 fibers, probably only 8 used), but an LC cable (only 2 fibers)

I can recommend fs.com and their cables + transceivers
e.g. Cisco QSFP-40G-SR4 Module, 40GBASE-SR4 QSFP+ Transceiver-Modul and MTP Trunkkabel 1m, 12F MTP Buchse OM4 Kabel, Typ B, Magenta (you need type-b polarity)

 

[Deano]

Oh wow, thank you guys i almost made a whole bunch of mistakes, it looks like i fell for MB vs Mb (Megabits vs Megabytes?) type of mistake when i thought the cards were 40g.

so just by going by the prices here:
£31x2 for two Oracle 7046442 Connectx's, (free delivery). Ebay
£30x2 for two Cisco QSFP-40g0SR4 Transceiver's. FS.com
£84 for a MP-12 10m Female to Female Cable (free delivery over £79). FS.com

£206 and that's everything i need? sounds great.

 

[itronin]

Oh wow, thank you guys i almost made a whole bunch of mistakes, it looks like i fell for MB vs Mb (Megabits vs Megabytes?) type of mistake when i thought the cards were 40g.

so just by going by the prices here:
£31x2 for two Oracle 7046442 Connectx's, (free delivery). Ebay
£30x2 for two Cisco QSFP-40g0SR4 Transceiver's. FS.com
£84 for a MP-12 10m Female to Female Cable (free delivery over £79). FS.com

£206 and that's everything i need? sounds great.

so just a thought. Have you searched for QSFP AOC cables? In the US a 10M cable is about $50USD.

Key differences, optics and cable are 1. So if either end or the F/O cable fails you toss it. vs. Having discrete optics (xver) on each end plus the cable.

AOC's make it a lot simpler too, FWIW.

Based on your pricing I could buy a new cable each year for 3 years and still be cheaper than discrete components.

 

[Deano]

TRANSUTON 40G QSFP+ AOC CABLE | Compatible with Cisco QSFP-H40G-AOC10M 40GBASE QSFP+ to QSFP+ Active Optical Cable for Data Centers (10M/33ft)

is this the right cable, this is the first result filtered by price low to high for £122.99 plus £9:21 shipping! phew, $50 USD sounds like you get a lot better deals!

edit:
next one up is
StarTech.com MSA Uncoded Compatible 10m/32.8ft 40G QSFP+ to QSFP+ AOC Cable - 40 GbE QSFP+ Active Optical Fiber - 40 Gbps QSFP Plus/Transceiver Module Cable (QSFP40GAO10M) at £365

 

[itronin]

TRANSUTON 40G QSFP+ AOC CABLE | Compatible with Cisco QSFP-H40G-AOC10M 40GBASE QSFP+ to QSFP+ Active Optical Cable for Data Centers (10M/33ft)

is this the right cable, this is the first result filtered by price low to high for £122.99 plus £9:21 shipping! phew, $50 USD sounds like you get a lot better deals!

edit:
next one up is
StarTech.com MSA Uncoded Compatible 10m/32.8ft 40G QSFP+ to QSFP+ AOC Cable - 40 GbE QSFP+ Active Optical Fiber - 40 Gbps QSFP Plus/Transceiver Module Cable (QSFP40GAO10M) at £365

ohhh. new/retail. sorry, I figured since you were looking at the bay for cards you were looking there for cables too.

Here's a US bay search

Here's a UK bay search - there are UK based sellers as well as US that will ship to the UK. always verify with the seller what your charges will be before ordering. Looks like US sellers are cheaper but VAT I guess. still UK prices are a little cheaper even with shipping than your AMZ pricing...

Here's another UK bay search with different terms. Interesting that items are coming from across the EU - but I guess you have inbound VAT there too.

Here's a 20m cable about 43GBP shipped from the US... Probably does not included VAT but oh did I mention its new too?

helpful hint. Once you get part numbers for what you need, also try plugging those into the ebay searches. sometimes sellers DO NOT put good "search terms" on their items... sometimes you find deals that way too...

Just trying to help you out with some different technology options, lower cost sources and your product searches ...

 

[Deano]

I'm absolutely warming up to the idea of using Ebay, it's nice when you paid full retail for something that does go wrong and you can return it, but when something you buy works for the next 6+ years then paying full retail was a bit of a waste.

I initially just wanted the Mellanox connectx-2 from ebay as you can't seem to buy that anywhere but ebay, perhaps since they were brought by nvidia i think, i have no idea, but yeah just happened to look on amazon when you mentioned it out of habit.

I remember buying something in the very early days of ebay and finding out about import item costs in a horrible way, i brought something for £500 i think, postman arrived at the door and simply told me he wanted £100 immediately to hand over my item, now to me at the time this was just insane as you don't ever just hand over cash to the postman to do his job, so he left with my item until i found out what this was all about, apparently the seller mislabel it or didn't do something and customs decided to check inside for bombs.

plus I was always worried about getting a brick in a box instead of a graphics card like you read on the internet

That last link looks like a great price, doesn't look like he ships internationally though, and definitely didn't show up in the earlier results, you're right about needing to dig deeper with part numbers

 

[itronin]

The previous posts were spot-on with CX3. btw, you can also search for the HPE variant as well. see here. As with the Oracle variant the HPE has to be x-flashed - hopefully you are comfortable with that?

I know that some folks across the pond from US (ha!) use reshippers, and I think folks on the board in the US have offered to reship from time to time as well.

I think part of this is how much of a hurry are you in.

 

[Deano]

I have no idea how to x-flash anything, from the link you provided it looks like installing the card and then running some things in cmd prompt, is that right? If so and the instructions have everything i need i think i can do it (hopefully.) sounds like good practice for when i swap my main PC over to Linux.

I'm kinda at the no rush planning stage right now, the NAS is just a saved shopping cart whilst i wait to get back and old case and power suppy to see if i can use them in the build.

 

[RageBone]

If you don't care about the cost, buying new or proper CX3, CX4 and CX5 Eth cards is the easiest and care free option.
The same can be said for cables.

IF you are on a budget, there probably are options available to make it.
the two most low cost options are cx3 OEM cards and especially HP FLEXLOM OEM Cards that need an adapter to work in normal pcie slots.
Those cards can be had for less then 10$ with similar prices for the adapter.
There are bundles available in places already.

On the Cable side, there were QSFP28 CWDM Transceivers for about 20$ with LC ports.
A friend who send them to me for testing estimated about 1$ per meter cable if i remember correctly.