Going 10G - RJ45 copper via Intel X540?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Hi all

I've spent the last several weeks majorly upgrading my home file server, which is now 2 x hex-core Xeon X5670 CPUs with 72GB RAM, providing 24 x 2TB 7200rpm HDD on 4 x LSI controllers, running Solaris 11.3. I might also add an SSD or two for ZIL and/or cache.

Having provided myself with all this space, and huge bandwidth (at least sequentially), I've decided I need to fit the last piece of the puzzle and provide access to it at a good speed from my workstation. Hence 10GB connectivity.

Before yesterday I knew nothing at all about 10GB ethernet, as the last time I worked professionally as a SysAdmin was 2010 and had no need of it then for the applications and servers I was working on.

I started out thinking "I'm going to need some Cat7 and a couple of cards", but then I spent the last half-day researching it and falling down the rabbit hole of fibre, transceivers, SFP, XFP, Infiniband, etc etc. I saw that the Mellanox ConnectX-2 IB cards were very popular as a very cheap option, and similarly the Chelsio (though its Solaris 11 compatibility is questionable). Later I looked at the Intel X520-DA1, another affordable, available and popular option, which is in-stock with my preferred UK vendor for used kit.

But then this morning I discovered affordable Intel X540-T1 cards, and have therefore seemingly come around full circle: back to RJ45.

It looks to me (in my ignorance), that all I would need for decent 10G workstation-server connectivity is 2 x Intel X540-T1 cards and a CAT7 patch cable?

And unlike the other options, this would allow me to easily reach distances up to 50m?

This is attractive to me because once my server is done, I would ideally like to put it in my garage in order to isolate the noise and heat. But this is at least 10m away from my desk, and probably more like 15m if I want to take a route that doesn't require drilling through many walls.

To achieve that with SFP/DA seems either impossible or very expensive. But with CAT7 it appears easy and cheap - I can get a 20 metre CAT7 cable delivered tomorrow for just £15. And then the cards also seem very simple - just a RJ45 connection, no transceivers, nothing complicated.

The Intel X540 cards are more expensive than the ConnectX-2s and other options I was looking at. But I can get them from China at not too bad a price. I can order 2 x Intel X540-T1 "OEM" cards from China for a total of about £140. Add the 20m cable and it's £155 total. This is maybe £50 more than I was expecting to spend with ConnectX-2 (which are dirt cheap in the US but rather more once I get them delivered to the UK) or Intel X520-DA1, but with the huge advantage of 20m reach (versus 3m or 5m).

I'd be grateful for any thoughts/comments/advice. Is this going to be as simple as it looks? 2 x Intel X540-T1 + 1 x 20m CAT7 cable = working 10GBase-T? Or are there reasons I should go back to thinking about the Intel X520-DA1 or other similar cards?

Thanks in advance!
 
Last edited:

voodooFX

Active Member
Jan 26, 2014
247
52
28
I see no reason you shouldn't go with the X540, especially if you are ok with the price
I would also suggest you to do a quick iperf test with the hosts side by side and another one when you have your 20m cable in place, just to be sure you are not loosing bandwidth because of interferences somewhere ;)
 

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
I see no reason you shouldn't go with the X540, especially if you are ok with the price
I would also suggest you to do a quick iperf test with the hosts side by side and another one when you have your 20m cable in place, just to be sure you are not loosing bandwidth because of interferences somewhere ;)
Thanks a lot, voodoo. Yes that is a very good point. If I go this route, I will also buy a 1m CAT7 cable so I can test performance to compare against the full cable distance.

I will be measuring carefully before I buy the cabling. I will buy as short as is usable, and maybe I don't need as much as 20m.

My office is upstairs in a converted attic/loft area, with line-of-sight to the garage. I'm planning to drill a small hole under a window and run the cable directly through the air, pulled taut like a telephone cable.

This photo shows the garage (cream building in centre) as seen from my office window - ie the exact path the cable will hopefully take. I haven't measured yet, but I'm guessing it's perhaps 10m point-to-point between the two rooms, meaning hopefully 15m will be the maximum length required to bridge the gap and reach the machines in their respective internal positions.

I haven't researched yet what sort of weatherproofing I will need, but hopefully a basic sheathe will suffice. I'll need to run at least two cables: the 10GB, and at least one 1GB for connection to the rest of the network.

Thanks again!
 

pc_doom

Member
Nov 2, 2016
50
17
8
29
Cat7 is not standard Ethernet cable, you'd better not to choose it. For 10G connection, cat6 and cat6a are both good. Ethernet cable in Monorprice is worth to buy.
 
  • Like
Reactions: TheBloke

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Cat7 is not standard Ethernet cable, you'd better not to choose it. For 10G connection, cat6 and cat6a are both good. Ethernet cable in Monorprice is worth to buy.
Oh OK, thanks. I've just read up on it a bit more and I see Cat7 is an ISO standard but not a TIA standard.

I guess either would work? But it looks like 6a is no more expensive, or even cheaper than Cat 7, so yeah I may as well choose what is specifically listed as compatible/working. I see various places selling 6A cables of decent length at good prices in the UK.

One thing I'm worried about: is it possible/realistic to buy bare cable, without RJ45 connections, and then crimp them on myself? I have done this before with Cat5 cables for 100M/1GB ethernet and it seemed to be OK - although it was only ever for low-bandwidth connections (router to media server, etc) and I never benchmarked the resulting cable.

But I am assuming the tolerances for 10GBase-T are much tighter. Can it be still done with a standard crimp tool? Or should one always buy pre-moulded cables whenever possible?

Although I hate crimping (I always find it really hard to get all those wires in the right places on the boot, and apparently Cat6 is even harder than for Cat5), it would make life easier to have bare cable - as otherwise I need to drill holes large enough to fit the RJ45 connector as well. With bare cable I can make a much smaller hole and then fit the RJ45 once the cable is pulled through.

Thanks
 

Jon Massey

Active Member
Nov 11, 2015
339
82
28
37
I'd be v suspicious of cheap "Cat7" cables from amazon, best buy good quality Cat6A from a reputable supplier (comms express have served me well in the past).

You're right that there are higher standards for 10GBase-T than for 100-/1G, and without an expensive tester you're not going to know if your crimps are up to snuff or not, other than perhaps seeing suboptimal 10G speeds. I'd certainly be doubtful of my own abilities to punch down or crimp to the required standard as I'm not a pro. On the other hand, pulling 8x OM3 MM fibre between the house and the garage was a doddle. The LC connectors are nice and small and so long as you're not brutal with it, it's easy to handle. SFP+ 10GBase-SR transceivers are dirt cheap. You could always use DACs while it's in the same room and then upgrade when you move it out to the garage.

The other advantage of running fibre outside is you don't have to worry about electrical safety (earth loops, lightning) or corrosion. If you're running either type of cable in free air, don't expect it to be self-supporting. Get yourself some catenary wire (20 quid for a kit with anchors and tensioners from toolstation), run that between anchors at either end and then ty-rap/ziptie the cable to the catenary. Also, be aware that outdoor-rated cable (of either type) has insulation that will take a lot more UV before going crispy than the indoor-rated stuff.

I ended up running 20mm conduit down the side of the house and across the gap to the shed, along the back of the shed and in through a hole in the garage wall. Provides mechanical stabilisation and protects from UV.

The main cost in doing significant cable runs is the effort, not the materials. If you're doing one run, do two, or three or 8. If I wanted to spend a fortune on MMF40G BiDi transceivers I've could have 320Gbps b/w between the house and the shed.
 
  • Like
Reactions: TheBloke

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Thanks a lot for your thoughts, Jon.

Yeah I think if I do go RJ45, I will definitely go moulded. It's just not worth the effort of doing it myself and then having a potentially unreliable cable. And yes I will buy from a better vendor than Amazon - I was just using them for a ballpark price. I will check out Comms Express in more detail, thanks. They look reasonable at first glance, eg their branded PatchSee 6A cables are £22 for 15.2m or £27 for 20m. Amusingly, their pricing is broken for long lengths of their standard/unbranded 6A cables - it increments price as normal up to 10m (about £9.50), but then on the 15m and 20m cables the price goes down to £1.86, same as for a 0.5m length :)

Thanks for your points about fibre. I do see the benefit - I only just read this afternoon about the risk of ground loops with copper, for example.

My main question/concern is whether I can get 2 x cards + 2 x SR transceivers + 15-20m of fibre cable for anything approaching affordable, whilst also ensuring Solaris 11 and Windows 10 compatability? I can get 2 x OEM X540-T1 for £145 delivered from China, then 20m CAT 6A looks to be at most £27, and potentially less (eg Comms Express' unbranded cables, and/or only 15m length.) So currently I'm looking at tops £165. I could stretch to £200 if it was definitely beneficial, but not really beyond that.

EDIT: OK, so more research has revealed more of my ignorance. When I first wrote this reply, I thought the Intel X520-DA1 card was Direct Attach / Copper only, and therefore that it couldn't work with an optical transceiver. Hence I thought I had to find yet some other card before I considered SR fibre.

I now realise that any of the Intel X520 cards can take an SR SFP+ transceiver, including the DA cards. (This is a post on an Intel forum for example, confirming my exact question re X520-DA1)

I know I can get cheap X520-DA1 cards in the UK, and in fact they're cheaper than the X540-T1's I was looking at from China.

SR transceivers I can get cheap from China, or somewhat-expensively from the UK - like £40 each in the UK, which would definitely make the whole venture quite a bit more expensive than with the X540.

So I now understand what you said about starting with DAC - I could get 2 x X520-DA1 with a cheap/short direct attach cable, to get me going and connected in my office immediately. Then I can order those Intel SFP+ SR transceivers from China (at only £13 each), and in the 10-20 days it takes for them to arrive, I'll be done with server setup and ready to do the garage move.

As for fibre, I see that's very affordable from Comms Express - eg 20m LC - LC OM3 MM fibre is £17.50 inc VAT.

I think I understand much better now, but I would be grateful if you could confirm that that's all cool and I'm not making further mistakes?

Thanks again.
 
Last edited:
  • Like
Reactions: Jon Massey

Mark Wilde

New Member
Aug 21, 2016
18
5
3
37
I recently heard of the 10G SFP+ 10GBASE-T modules offered by several vendors, is it worthy trying?
 

Jon Massey

Active Member
Nov 11, 2015
339
82
28
37
Thanks a lot for your thoughts, Jon.

I think I understand much better now, but I would be grateful if you could confirm that that's all cool and I'm not making further mistakes?
By Jove, I think he's got it! The only potential "gotcha" when you're talking about SFP+ kit (i.e. DACs and Transceivers) is that there is some vendor locking issues which you don't get with plain old twisted pair copper. So long as you're looking at Intel/"Intel Compatible" transceivers (which it looks like you're already on the case with) then you'll be fine.
 
  • Like
Reactions: TheBloke

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
By Jove, I think he's got it! The only potential "gotcha" when you're talking about SFP+ kit (i.e. DACs and Transceivers) is that there is some vendor locking issues which you don't get with plain old twisted pair copper. So long as you're looking at Intel/"Intel Compatible" transceivers (which it looks like you're already on the case with) then you'll be fine.
Thanks a lot, Jon!

Yes I had read about the vendor lock-in, and so have been looking for Intel brand transceivers. Which fortunately are available very cheap as China OEM.

Apparently the Linux driver can override this vendor lock with a simple config tweak, but I've not heard mention of this being possible in Solaris, and it's definitely impossible in Windows. So given I can get going with DAC I'll go for Intel transceivers and by the time they've arrived from China (which from past experience can be as little as 10 days) I'll be set-up and ready to move the server and use them.

I have a workman coming on Friday and Saturday to hopefully, finally waterproof my garage by putting a proper new flat roof on it. It's been leaking in various places for years, but hopefully will now finally be suitable to have some hardware in it.

Purchase-creep has now got me also looking at a cheap 10G capable switch, namely the Quanta LB4m. It's only got 2 x SFP+ 10G ports, so it wouldn't allow adding any more 10G hosts. But it would enable me to unify my 1G and 10G networks, plus providing some L2 management.

I'm awaiting a quote for the used X520-DA1 (or possibly DA2s) plus the switch and a DAC or two, and will hopefully purchase this week.
 

acquacow

Well-Known Member
Feb 15, 2017
784
439
63
42
Been running the X540-T2 cards and using standard hand-crimped cat5e at home for the two years. No issues to report connectivity-wise. My longest run is about 25ft.

At the OS layer, Intel's drivers for Windows 10 don't offer the bonding I was using in windows 7. Something about Win10's network stack is causing Intel to have to re-write that support it and it still doesn't seem to be done.

At the hardware layer, one of my 540s came with the plastic tape still on the heatsink thermal pad and it cooked itself no matter how much air I fed it. I figured when I replaced it, I would get one of the dell versions on ebay that have a fan, but that fan is SUPER loud, so I ended up putting the old large heatsink on the new dell card and just putting good thermal compound on it. It's been fine that way and has never overheated in my near-fanless desktop box.
 
  • Like
Reactions: TheBloke

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Been running the X540-T2 cards and using standard hand-crimped cat5e at home for the two years. No issues to report connectivity-wise. My longest run is about 25ft.

At the OS layer, Intel's drivers for Windows 10 don't offer the bonding I was using in windows 7. Something about Win10's network stack is causing Intel to have to re-write that support it and it still doesn't seem to be done.

At the hardware layer, one of my 540s came with the plastic tape still on the heatsink thermal pad and it cooked itself no matter how much air I fed it. I figured when I replaced it, I would get one of the dell versions on ebay that have a fan, but that fan is SUPER loud, so I ended up putting the old large heatsink on the new dell card and just putting good thermal compound on it. It's been fine that way and has never overheated in my near-fanless desktop box.
Oh that's interesting - you manage full 10G speeds on just CAT 5E? Wow, that's good to know.

I'm probably going to end up with the X520s over fibre, but that's good to hear the X540s are working well.

So were you trying to bond 2 x X540 into a 20G connection? With LACP or similar? That's a shame it doesn't work. What about SMB 3.0's multi-pathing, does that work for you at all? I just learned about that the other day, and saw it demonstrated in a (fairly crappy) YouTube video, using 4 x 1G connections between two Windows 10 boxes and getting over 450MB/s. I suppose it'd work with 2 x 10G as well, if both ends support SMB 3.0.

The X520s I will mostly like get are passive cooling, although I had wondered about cooling as my server does get rather hot in its PCI area. Once the X520 is installed, I will have all 5 PCIe slots filled, four with LSI cards (2 x 2008, 2 x 2308) and the fifth being the 10G NIC. It's only a 2U chassis and although there's a max-5000 RPM fan pointing at the cards, there's a couple of internal SFF-8087 cables that inevitably have to get in the way somewhat no matter how I try to arrange them.

I had pondered if I would maybe massacre the lid of the 2U chassis a bit, and fit a 14" 1200RPM fan directly over the PCI area, pulling air vertically out. When I test that config with no case lid at all, fan just resting on the LSI cards, the Tyan mobo reports the PCI area as no more than about 36°C, and the chassis fans slow to about 3000RPM. With the case on, that same area rises to 40-41°C, and that's with the fans blaring at ~5000RPM - and before I've added a NIC.

I'm not overly worried about noise in the server as it will hopefully move to the garage in a week or two. But I do want to keep things cool. So I might cut a square out and stick the 14" fan in - with the added benefit that it would also partially overlap the second CPU's heatsink, which tends to run hotter as it's the opposite side of the mobo from the chassis fans.

You mentioned re-attaching your NIC's heatsink - thanks, that's a good point that hadn't yet occurred to me. I have some very good quality thermal paste, one of the ones that gets highly recommended for CPU overclockers. It's no doubt better than whatever they use in the factory, so it might be worth my while removing the heatsink on the NIC - and maybe also all my LSIs, while I'm at it - and re-attaching with freshly applied, good-quality paste. Shouldn't hurt, at least.
 

acquacow

Well-Known Member
Feb 15, 2017
784
439
63
42
Cat5e will hold a 10gig signal for distances up to 50ft, cat6 gets you to 100ft.

I had bonding working in windows 7 with the intel driver, as well as multipathing using some DLLs stolen from windows server 2008. These things don't work in windows 10 at all, so I've just been running at normal single-link speeds, which is honestly fine because my normal config doesn't support much more than 1GB/sec. I simply had bonding setup for some tests to ensure I had an optimal config.

I haven't messed with SMB 3.0 stuffs, but I have windows 10 as both endpoints, so if it works in windows 10, I'll config up the 2nd link and see about that.

Yeah, definitely check the bottom of your heatsinks when you get the cards... you don't want to run into this the hard way...
upload_2017-3-23_8-37-20.png

upload_2017-3-23_8-37-33.png

-- Dave
 
  • Like
Reactions: TheBloke

n00bftw

Member
Jul 16, 2020
39
4
8
United Kingdom
Cat5e will hold a 10gig signal for distances up to 50ft, cat6 gets you to 100ft.

I had bonding working in windows 7 with the intel driver, as well as multipathing using some DLLs stolen from windows server 2008. These things don't work in windows 10 at all, so I've just been running at normal single-link speeds, which is honestly fine because my normal config doesn't support much more than 1GB/sec. I simply had bonding setup for some tests to ensure I had an optimal config.

I haven't messed with SMB 3.0 stuffs, but I have windows 10 as both endpoints, so if it works in windows 10, I'll config up the 2nd link and see about that.

Yeah, definitely check the bottom of your heatsinks when you get the cards... you don't want to run into this the hard way...
View attachment 4977

View attachment 4978

-- Dave
Thread is a tad old, but how did you remove the heatsink? I'm finding it very hard.
 

acquacow

Well-Known Member
Feb 15, 2017
784
439
63
42
Thread is a tad old, but how did you remove the heatsink? I'm finding it very hard.
It's just two plastic clips on the back side of the board. You take some needle nosed pliers and squeeze them together so that they are smaller than the hole and push them through.
 

acquacow

Well-Known Member
Feb 15, 2017
784
439
63
42
Will they expand again when putting the heatsync back on? Thanks for the reply.
Yup. And worst case, you can slip a folded up piece of paper in-between them to push them apart and keep them apart.