1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

10-Gigabit Ethernet (10GbE) Networking - NICs, Switches, etc.

Discussion in 'Networking' started by odditory, Jan 2, 2011.

  1. odditory

    odditory Moderator

    Thread for the discussion of 10-Gigabit Ethernet (10GbE)

    10-Gigabit Ethernet (10GbE) is the fastest Ethernet standard for hard-wired local area networks (LANs) since 2005. It crowns the latest in a long list of Ethernet technologies standardized by the Institute of Electrical and Electronics Engineers (IEEE) in their specification 802.3x. The 10GbE network operates at a rate of 10 gigabits per second (gbps), or ten times faster than standard GbE. Ethernet cabling is used to connect computers to hard-wired networks used in businesses, on campuses, and in homes. Over the years, the technology of transmitting digital signals across Ethernet cabling, including the cabling itself, has evolved to allow faster data transfer rates.

    Ethernet standards are often referred to in abbreviated form. The speed is indicated first, followed by the word “base” for baseband, and finally a 2-digit abbreviation for the type of cabling technology used. When referring to a level of Ethernet without specifying the cable type, a simple “X” is used to be inclusive. At a glance, this abbreviation reveals which type of Ethernet network is being referred to. For example:

    10Base-X – Standard Ethernet at 10 megabits per second (mbps)
    100Base-X – FastEthernet at 100 mbps
    1000Base-X – Gigabit Ethernet at 1,000 mbps, or 1-gigabit per second (gbps), also known as GbE
    10GBase-X – 10-Gigabit Ethernet at 10 gbps, also known as XGbE

    10GbE is ten times faster than GbE and has many competing standards that will likely shake down to a few over time. Most types of 10GbE use single or multi-mode optical fiber cabling. Single mode optical cable is used for 10GBase-SR (short range), 10GBase-LR (long range), and 10GBase-ER (extended range). Multi-mode optical fiber cable is used for 10GBase-LX4 and 10GBase-LXM. Wide area networks (WANs) have their own varietiess of 10GbE: 10GBase-SW, 10GBase-LW and 10Gbase-EW.

    For running short distances with 10GbE, there is the less expensive 10Gbase-CX4, a copper alternative. This is commonly used for bridges, such as wiring router to switch. A traditional type of 10GbE that uses twisted pair cables should be standardized by IEEE in 2006, as 10GBase-T. The twisted pair cabling used for 10GbE will have to meet technical specifications required by Category 6A cable, or “Cat-6A” for short.

    NOTE: People ask why 10GbE adapters are still so expensive and the simple reason is "the processor", which handles TOE (TCP/IP Offloading Engine) at 10 Gb/s at 10x the rate of what's required on 1GbE adapters. Without being able to offload that processing, at that data rate the processing would be significantly taxing on the host. And because these processors still aren't "commodity" yet with chinese fabs pumping out a million at a time for five cents a piece, decent 10GbE adapters will continue to be expensive.
    Last edited: Jan 2, 2011
  2. odditory

    odditory Moderator

    One of my projects this year is to outfit a few of my servers and workstations with 10GBase-T cards (10GbE over twisted pair CAT6A with RJ-45 connectors). With 10GbE switches still being pricey and not cost justifiable (or needed) for my home storage setup, the plan was to pick up a few single or dual port 10GbE adapters and direct-connect between servers. My ESXi and Hyper-V servers also stand to benefit from 10GbE, and Intel claims to have done lots in terms of optimization for virtualization scenarios with VT-c (virtualization technology for connectivity).

    Here are some of the cards I've been looking at. Not knowing enough about 10GbE adapters yet I'm partial to Intel because historically their ethernet controllers and driver stacks have been somewhat of a gold standard.

    Intel EXPX9501AT (discontinued by Intel September 2009), a.k.a. Dell XR997 (Refurbs going on ebay for $199)
    Controller: Intel 82598EB
    [​IMG]

    Intel E10G41AT2 ($525)
    Controller: Intel 82598EB
    [​IMG]

    Intel E10G42BT X520-T2 ($700)
    Controller: Intel 82599
    [​IMG]

    @Patrick weren't you playing around with some cheap Infiniband based 10G cards? Anything come of that?
    Last edited: Jan 2, 2011
  3. Patrick

    Patrick Administrator

    I was playing around with dual port Infiniband (copper based) cards. Performance is pretty good, but the Mellanox cards I picked up were awesome under Windows, but not so much under ESXi and OpenSolaris. The dual port cards were $30-40 each making the internal hardware very inexpensive.

    The other issue is that the cables are quite a bit more expensive than Cat 6A and the cost of running anything room-to-room is a bit prohibitive. If I used storage in a data center for all Windows clients, it is a decent solution, but for home use I decided not to post anything on it because it is nowhere near as user friendly as standard Ethernet controllers.

    Also, it should be noted that the performance of 10GbE is going to be better than using ten 1GbE NICs in link aggregation.
  4. odditory

    odditory Moderator

    Agreed on a single 10GbE being superior to aggregated 1GbE links. I've seen it firsthand - transferring between two fast arrays on two windows servers going at a clip of about 600MB/s and I'm pretty sure the ethernet adapter still had more headroom. While I've seen people argue that a single file transfer is *supposed* to be capable of traversing an aggregated link set by being broken up into parts, that hasn't been my finding at least between Windows servers and clients with decent Intel quadport NICs and a capable switch, and even if it were possible, I don't care for that much complexity in terms of NIC ports, switch ports and cabling that has to be tied up for it. The same as I'd prefer a single superfast SSD drive rather than 10 in raid0.
    Last edited: Jan 2, 2011
  5. john4200

    john4200 New Member

    With the third generation Intel X520-T2 dual-port adapters (E10G42BT) now selling new for about $700, I would have expected to see some good deals on the 2nd gen AT2 single-port adapters (E10G41AT2) on ebay. But all the AT2 adapters I am seeing on ebay at the moment are $500+. Maybe they will come down in the next couple months?
  6. odditory

    odditory Moderator

    Yeah these things are priced like high end raid cards. I don't really see a reason that prices *should* fall drastically any time soon, unfortunately. I think there simply aren't enough of these out there.

    I just added the Dell XR997 to the list of adapters in post #2. Tempted to get a couple to play around with since refurbs are going for around $200 shipped on ebay, and even though Intel discontinued them September 2009 they still appear supported in the latest driver set.
    Last edited: Jan 2, 2011
  7. john4200

    john4200 New Member

    $200 is not bad. But I'm probably going to wait a while before going 10GbE.

    I was curious about CAT6a cables, though. I know that 10GBASE-T will work on CAT6, but if I was going to splurge on 10GbE, I'd probably want to go whole-hog and get CAT6a. I checked monoprice.com and I was amused to see that you can get any color of CAT6a cable you want, as long as you only want blue:

    http://www.monoprice.com/products/subdepartment.asp?c_id=102&cp_id=10232#1023216
  8. Patrick

    Patrick Administrator

    That's probably because unless there is a $3,000+ box sitting there, 10GbE is most likely not needed :)

    I actually have an ebay saved search for the Intel EXPX9501AT and still haven't done it. They have basically been hitting that $200 price point consistently for months.
  9. xnoodle

    xnoodle New Member

    Seller on fleaBay with the XR997 at $149.99 + $5.25 S&H. Tempted.
  10. PigLover

    PigLover Moderator

    Odditory,

    What's your plan to connect your servers without a switch? If you are just connecting one server to one workstation it makes sense - you just run them back to back. But your post says you plan to equip "a few of your servers" with them. Things will get pretty messy pretty fast without a switch in the mix. At $150/end I am seriously tempted too.
  11. Patrick

    Patrick Administrator

    OK I picked up two. Best offer of $125/ea worked but required seller's manual approval.
  12. odditory

    odditory Moderator

    point to point connections between servers. ideally with 2-port 10GbE adapters, but worst case with my Supermicro motherboards I tend to have lots of PCIe slots free anyway, so what if a server has two to three 1-port 10GbE cards, at used ebay prices its still a fraction of a switch price.
    Last edited: Feb 2, 2011
  13. XZed

    XZed New Member

    Hello,

    So many different things at the same time : Infiniband, 10GbE RJ45, Thunderbolt ...

    Trying to setup a minimal 10GbE network between 2 hosts, for financial reasons, i asked to Patrick for this :D ...

    Hadn't never used it, i have some questions (sorry Patrick to ask it before your review :) ) :

    - How are these CX4 cards seen by the OS : as network interfaces ? After reading this article, i guess classical IP configuration is possible ?
    - So, is it possible to interconnect two hosts with this cheap package : two Mellanox cards + 1 CX4 (SFF-8470) cable, as with the crossover RJ45 cable old-way ? My goal : the cheapest possible 10GbE setup between two hosts only.
    - Finally, all cards seem to be dual-port : can it be aggregated (20Gbs between hosts ?) or it has another goal (daisy-chain ?) ?

    By the way, meanwhile 10GbE RJ45 equipments prices always prevented me to buy it, XR997 card really seems a must-have-to card (After my PERC6/i bought on Ebay, this choice seems obvious)...

    To go with the XR997, any cheap 10GbE switch lol ? OK, i'll keep with host-to-host cable :).....

    (Hesitating between buying XR997 cards or directly SuperMicro mobos w/ 10GbE integrated.... :confused: )

    Thanks again for your advices.

    Sincerely,

    XZed

    P.S. : obviously, i already considered bottleneck case about hdd transfer speeds : it will serve to RAID to RAID transfers :)
    Last edited: Feb 25, 2011
  14. nitrobass24

    nitrobass24 Moderator

    Which SM boards have builtin 10gbe?
  15. XZed

    XZed New Member

  16. Patrick

    Patrick Administrator

    Those are the exact cards I am using, although I have another one with 128MB onboard which cost a bit more. What I will note however, is that Mellanox also has working drivers which is good. Also important to note, Solaris needs cards with onboard memory as you cannot use the memory free ones which utilize system memory instead. Finally, FreeBSD and OpenSolaris were ROUGH to set up.

    BTW nitrobass, I have a few new toys to review in the next week or two (one is a dual port 10GbE (SFP+ though) adapter. Another barely made it as I didn't bring the truck:
    [​IMG]

    Yea it was a scary ride with a 4U chassis hanging over the shifter.
  17. XZed

    XZed New Member

    I suppose 1TC/1SC references ?

    By the way, i was thinking about another use of the dual ports :

    2 computers linked @10GbE to a "central" computer while using on this the 2 ports ?

    Waiting to read about your new toys :).

    Nice ride !
    Last edited: Feb 26, 2011
  18. XZed

    XZed New Member

    Related to the Dell XR997 :

    What about the Dell RK375 ?

    (Don't know how Broadcom BCM957710 Chipset performs compared to the Intel one)
  19. XZed

    XZed New Member

    Well, after many researches, i'll surely buy Mellanox products.

    But, i'm facing with a problem which will be present whatever my final choice will be :

    I only have PCI-E 1x ports left on my mobos :/.

    Obviouslly, i'm aware of the impact of using a 1x lane for such a setup but, for the moment, forgetting the performance part, i'd like to setup it.

    So, i wanted to know if you think that using this adapter, it could work (at 1x speed) ?

    I'm not worrying about performance but about functionality.

    By the way, unless i'm wrong, i understood that using such an adapter, forces the use of a low profile card.

    But after reading that (page 57), i understood that they are low profile natively and i only have to change bracket (don't knowing where to find it, in first steps, it will be bracketless) in order to plug it in the adapter.

    Well, if someone could confirm it before buying :) ..

    I guess i have to seriously think about mobo replacement instead :) ...

    Thank you,

    Sincerely,
  20. iceboi714

    iceboi714 New Member

    Nice toys patrick...but it looks like you are crushing you laptops :p

Share This Page