Advice buying 10 Gbe NICs/Fiber

[herby]

I'm interested in moving up from GbE to 10 GbE for my home server setup but I'm unsure about SFP+ connectors, fiber, and 10 GbE NICs in general. My goal is to have a 10 GbE crossover from each of my Xenservers to my FreeNAS.

I'm looking at this listing on Ebay: Chelsio 10GB 2 Ports PCI E Opt Card LP Bracket PCI E w Transcivers 110 1088 30 | eBay

Apparently Chelsio NICs are supported by both Freebsd and Xenserver; although I'm having trouble tracking down the exact model of this NIC I think it's the N320E-SR. I don't see that model listed on Chelsio's site but there are similar listings on Ebay with that number.

Included with the NIC are Finisar Transceivers: 10GBASE-SR 300m SFP+ Optical Transceiver | Finisar Corporation

Would it then just be a matter of buying some LC multi-mode fiber like this: 10Gb Fiber Optic Cable, LC/LC, Multi Mode, Duplex - 2 Meter (50/125 Type) - Aqua - Monoprice.com
and plugging it into the transceivers?

I've read threads on some of the more popular NICs like the 1020, and the connectx-2 and threads mentioning DAC/Twinax, but in this case (since transceivers are included) just buying some LC cable would be the better move right?

 

[herby]

After a little googling I'm pretty sure the card is a N320E-SR. I found these pictures on the retailer splusdirect's website:

"110 1088 30" sticker on the front

"CC2-N320E-SR" model number on the back

I found a post on freeBSD mailing list that seems the confirm that this NIC works: Recommendations for 10gbps NIC
but mentions the card gets quite hot. Specifications I've seen give a 14W draw typical, and call for 55C max and 200 lf/M of airflow.

 

[Patriot]

I was more drawn to the scorched pcb section....

 

[TuxDude]

I was more drawn to the scorched pcb section....

Well the FreeBSD mailing list message he refers to does say the card gets quite hot. Perhaps "quite hot" is kinda an understatement.

 

[herby]

I was more drawn to the scorched pcb section....

I saw that on a 3rd party listing on Amazon too, same picture that someone lifted from someone else; but it does make me hesitant.

 

[neo]

Was there a specific reason you chose the Chelsio N3 over the Chelsio S3? I don't know much about those older generations but the S3 seems to have onboard memory for around the same price on eBay.

 

[herby]

Was there a specific reason you chose the Chelsio N3 over the Chelsio S3? I don't know much about those older generations but the S3 seems to have onboard memory for around the same price on eBay.

My freeNAS and one of the Xenservers are in 2U Chassis so I'm limited to the low profile models, that's also why the heat of these cards concerns me.

 

[neo]

From my experience Chelsio cards require server case like air flow which cool the expansion slots. I've had a Chelsio card overheat and stop functioning until it was properly cooled before.

 

[Chuckleb]

I just shot an IR heat gun on my Mellanox CX2 card's heatsink and it was about 100F (supposedly). It felt cool to the touch and this is inside a desktop chassis, so nothing fancy for air movement. If you're concerned about heat, those cards work well.

 

[herby]

From my experience Chelsio cards require server case like air flow which cool the expansion slots. I've had a Chelsio card overheat and stop functioning until it was properly cooled before.

That's better that catching on fire like the one in the picture. I could live with a temporary shut down.

I just shot an IR heat gun on my Mellanox CX2 card's heatsink and it was about 100F (supposedly). It felt cool to the touch and this is inside a desktop chassis, so nothing fancy for air movement. If you're concerned about heat, those cards work well.

Unfortunately I've read FreeBSD is picky about 10GbE adapters and I don't think the Mellanox will work.

In each my half depth 2u chassis I have a pair of old 80mm Panaflo L1A that put out 24 CFM/1900 RPM. One is pointed directly at the expansion slots and according to this airflow calcuator: Airflow Unit Conversion | Comair Rotron, 24CFM should be a hair over 275 linear feet/minute. Unfortunately I don't think it's as simple as that to ensure the NIC gets the 200 lf/m the datasheet calls for. Especially with the limited exhaust in the area.

 

[timothy.lazarus]

FreeBSD 10.1 will work with the official Mellanox drivers. FreeNAS is based on 9.x, so it isn't an option. I've moved on from FreeBSD, however, after experiencing several dropped iscsi connections that may be a bug in the iscsi target or a Mellanox driver issue...not sure.

 

[wojo]

Ran into issues with a Chelsio T420-CR in a TS140, not enough cooling at the heatsink isn't enough. Would cause the link to go down and sometimes actually lock up the entire server. Not a good card for standard desktop style cases like that.

On the other hand Mellanox Connnect-X2 and X3 cards work fine, even the dual ports. They stay quite cool compared to the T420.

 

[Pete L.]

Yes you can buy the fiber you mentioned and just plug it in to the SFP+ Modules and you should get a link as long as the SFP+ Modules they provide are "Compatible" with the cards. Some cards are "Vendor Locked" and get all pissy (I mean picky) about what SFP's you put in them. So far I can say that Dell is not picky at all, Cisco is Picky and Intel cards are very picky. Oh and Mellanox Cards have worked with evey single DAC / Twinax Cable & SFP / SFP+ Module I've tried.

As for DAC Cables they are great / easy to use but actually use more power (albeit still low) than SFP's. If you are going to order more than just a few fiber jumpers or are considering DAC Cables / SFP+ Modules then check out fs.com (Fiber Store) I am very impressed with them but you need to order enough to make the shipping worth wile.

 

[rampage666]

Given the fact this card (S320E) is max at PCI-E 1.1 X8, the total bandwidth avaialble is 2GB
so if only one port is under work, it would reach about 1.1GB/s,
but if both ports have traffics at the same time, each port's speed could be problem?

 

[av00va]

Given the fact this card (S320E) is max at PCI-E 1.1 X8, the total bandwidth avaialble is 2GB
so if only one port is under work, it would reach about 1.1GB/s,
but if both ports have traffics at the same time, each port's speed could be problem?

Think about it. Each lane = 2 gigabits per second full duplex. 2 x 8 = 16 gigabits per second In one direction. That's well over what you need. Also, 10GBe is full duplex, 10 gigabits/s = 1,250 megabytes per second. 16gigabits/s = 2,000 megabytes per second.

Sent from my SM-N910V using Tapatalk

 

[aero]

If heat is a concern, think about putting in a pci slot cooler in the slot adjacent to the card...assuming you have slots.

I've had to do this with 10gbaset cards which tend to get much hotter due to power draw.

 

[Kev]

Any conclusions on which card, low profile and suitable for FreeBSD 10? I'm planning on installing freenas 10 on a hp microserver gen8.

 

[timothy.lazarus]

I'm running FreeBSD 11 on my TS140 right now, and using an Intel E10G41AT2. Prior to switching to FreeBSD 11 a week or so ago, I was using FreeNAS 9.10 with no issues for quite some time. Its an older Intel, so I am sure power-usage is high, but it worked fine in my TS140 with stock cooling and 2 HDDs/3 SSDs for over a year. I bought it for around $100 on ebay a year ago, so there are probably newer models available now.

 

[Jerry Renwick]

As you have said above, the transceivers are already included, then the most cost-effective way is to use the LC fiber cables with the SFP+ modules. I have been using the Mellanox NIC card. And put it in the air flow area. Never have a problem !