Dual NICs on PCIE x1 slot recommendation

Discussion in 'Networking' started by sofianito, Jan 20, 2012.

  1. sofianito

    sofianito New Member

    Joined:
    Nov 9, 2011
    Messages:
    14
    Likes Received:
    0
    Hi,

    I am looking to fill the second and last PCIE x1 slot of my HP MicroServer MB with a dual nics 1gb card that support:

    - Solaris 11 natively (required)
    - Link aggregation (required)
    - VLAN (required)
    - Jumbo Frames (Required)
    - TCP checksum offload (would be a plus)
    - ESX5i (would be a plus)

    Any recommendations are welcome :)
     
    #1
  2. Metaluna

    Metaluna Member

    Joined:
    Dec 30, 2010
    Messages:
    64
    Likes Received:
    0
    Check out the Intel E1G42ET dual-port card. This has the 82576 chip, which is on ESXi 5's compatibility list. Not sure about Solaris 11, but it's on Solaris 10's HCL, so I'd say it's pretty likely to be supported.

    The card is PCIe x4 but I think it will fit in a x1 slot.
     
    #2
  3. sofianito

    sofianito New Member

    Joined:
    Nov 9, 2011
    Messages:
    14
    Likes Received:
    0
    After googling a bit, it seems that a part of the StarTech which seems not supported on Solaris, there are not a lot of choices. Most of dual NICs need an x4 PCIe slot...

    However, according to the specs of the HP NC360T, a four lane (x4) PCIe, it is compatible with x1, x4, x8, and x16 PCIe slots!. In order to plug it in the x1 slot of the MicroServer, a hardware modding needs to be done. Basically you either dremel the x1 slot of the motherboard or the card pins. In my case, I don't want to mess with the motherboard, so I'll choose option 2 ;). The end result would look like this picture (courtesy of NoGi's Random Tech Blog):
    [​IMG]

    The only doubt I have is whether the HP MicroServer's PCIe v2 x1 would get saturated by two gigabit ports running at full duplex...
     
    #3
    Last edited: Feb 7, 2012
  4. Metaluna

    Metaluna Member

    Joined:
    Dec 30, 2010
    Messages:
    64
    Likes Received:
    0
    That Startech card looks like it uses two Realtek chips. Definitely not a good choice for ESXi support, and probably not great for Solaris either. That HP card uses an Intel 82571 chip, so probably a good choice if you can get it for less than the Intel one I mentioned. If it's about the same price, go for the Intel as it's a newer generation chip, probably has a more powerful processing/offloading engine, and uses less power.

    That mod looks pretty scary. Look how close the upper left corner of the cut area is to the signal traces and vias for the remaining part of the connector. Even if you don't physically cut into that area, if your cutoff wheel generates too much heat you could delaminate the traces and break them.

    As for whether a 1x slot can support two gigabit ports without losing performance, I don't know. You're cutting it pretty close (no pun intended) bandwidth-wise. Neither of the Intel chips I mentioned support 5.0GT/s PCIe transfers, so the card would need to sustain 2.0Gbit/sec over a 2.5Gbit/sec link, including PCIe protocol overhead, etc. My guess is you'll probably lose a little performance but not anything too dire. Maybe 10 or 20% less.
     
    #4
    Last edited: Jan 21, 2012
  5. cactus

    cactus Moderator

    Joined:
    Jan 25, 2011
    Messages:
    825
    Likes Received:
    76
    I did this to two video cards and one came out working and the other I must have messed up with ESD. I had an Intel Dual Port ET in a 1x port that I cut the back out of so it will for sure work just be careful when handling the card. The Intel 82571 is only PCI-e v1 so it is cutting it close for throughput. If you are really concerned there are some Intel chips that support PCI-e v2 link.
     
    #5
  6. sofianito

    sofianito New Member

    Joined:
    Nov 9, 2011
    Messages:
    14
    Likes Received:
    0
    @Metaluna: The Intel E1G42ET looks great with a lot of features for a virtualized environment, but it is expensive... Also being an x4 PCIe card, I am afraid it doesn't fit OOTB in the x1 slot. It would need some modding same as the HP NC360T... Although it has less features, the pros of the HP card over the intel one is that it is less than half of the price in ebay, and if mess with it I guess I would not pull my hair as hard as if it were the intel one... :p

    @cactus:
    Was the cut similar to the picture I posted? Do you have a picture of your modding?
     
    #6
    Last edited: Jan 21, 2012
  7. cactus

    cactus Moderator

    Joined:
    Jan 25, 2011
    Messages:
    825
    Likes Received:
    76
    I have 3 of the dell branded E1G42ETs from ebay for $50US each. I did not mod any of my Intel NICs, I cut the back out of the x1 PCI-e slot on the motherboard sothe 4x card would fit. The two video cards I did cut down were cut at the edge of the solder resist.
     
    #7
  8. sofianito

    sofianito New Member

    Joined:
    Nov 9, 2011
    Messages:
    14
    Likes Received:
    0
    Lucky you! :). Are you willing to sell me one :p. Actually, the lowest price I found in ebay is around $140 shipping cost not included...

    Did you use a dremel or a knife such as X-Acto for modding?

    At the edge did you you cut along the longitudinal or lateral axis?

    Forgot to mention that in my case, if I decide to mod the mdb slot x1, I would need to do 2 cuts! because in the HP MicroServer mdb, there is an internal PCIe v2 x4 slot aside thex1 slot. The x4 slot is for an internal HP IPMI card. Here is a picture:
    [​IMG]
     
    #8
    Last edited: Jan 22, 2012
  9. sofianito

    sofianito New Member

    Joined:
    Nov 9, 2011
    Messages:
    14
    Likes Received:
    0
    I finally bought a dual link HP NC360T and modded it to fit the PCIe x1 slot on my HP MicroServer :D. I used an X-Acto X235 saw to do the cut. It was very precise and dit the job perfectly :cool:

    [​IMG]
    [​IMG]
    [​IMG]
     
    #9
  10. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,425
    Likes Received:
    4,367
    Wow... interesting mod! Does everything work?
     
    #10
  11. cactus

    cactus Moderator

    Joined:
    Jan 25, 2011
    Messages:
    825
    Likes Received:
    76
    sofianito, I used a fine tooth saw similar to the one you used.

    How is everything working out?
     
    #11
  12. sofianito

    sofianito New Member

    Joined:
    Nov 9, 2011
    Messages:
    14
    Likes Received:
    0
    So far, so good :D

    Both nics are recognized under Solaris 11:


    # dladm show-phys
    LINK MEDIA STATE SPEED DUPLEX DEVICE
    net0 Ethernet up 1000 full bge0
    net3 Ethernet up 1000 full e1000g0
    net1 Ethernet up 1000 full e1000g1

    # dladm show-link
    LINK CLASS MTU STATE OVER
    net0 phys 1500 up --
    net3 phys 1500 up --
    net1 phys 1500 up --

    # for i in 0 1 3; do dladm show-linkprop net${i} | /usr/xpg4/bin/grep -E 'LINK|speed|duplex|mtu'; done
    LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
    net0 speed r- 1000 1000 --
    net0 duplex r- full full half,full
    net0 mtu rw 1500 1500 1500

    net1 speed r- 1000 1000 --
    net1 duplex r- full full half,full
    net1 mtu rw 1500 1500 1500-9216

    net3 speed r- 1000 1000 --
    net3 duplex r- full full half,full
    net3 mtu rw 1500 1500 1500-9216

    # ipadm show-addr
    ADDROBJ TYPE STATE ADDR
    lo0/v4 static ok 127.0.0.1/8
    net0/v4 static ok 172.16.1.4/24
    net1/v4 dhcp ok 172.16.1.22/24
    net3/v4 dhcp ok 172.16.1.19/24
    lo0/v6 static ok ::1/128


    I am planning to test the following:
    * Link aggregation: I need to figure out how to configure link aggregation on Solaris 11 and HP Procurve 1810g-24 switch.
    * Jumbo Frames
    * Measure iSCSI performance between the ESXi5 node and the HP MicroServer SAN

    I got a lot of stuff to learn, but I am motivated :)

    Cheers

    [​IMG]
    [​IMG]
     
    #12
    Last edited: Feb 8, 2012
  13. alderaan

    alderaan New Member

    Joined:
    Apr 4, 2012
    Messages:
    2
    Likes Received:
    0
    Hi i've a NC364t Quad-port Hp and i'd like to use it with my hp microserver. I can do the same thing with this card?
     
    #13
  14. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,879
    Likes Received:
    835
    Probably can. I think pci express specs say everything up to x16 can work in a x1 slot.
     
    #14
  15. cactus

    cactus Moderator

    Joined:
    Jan 25, 2011
    Messages:
    825
    Likes Received:
    76
    You should be ok, but you are going to be hurting for bandwidth.
     
    #15
  16. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,441
    Likes Received:
    209
    Assuming he is using these all at once. Using 2 max is not bad. PCI express 2 = 500MB/s for an x1.
     
    #16
  17. Metaluna

    Metaluna Member

    Joined:
    Dec 30, 2010
    Messages:
    64
    Likes Received:
    0
    Not all multi-port controllers support the PCIe 2.0 speeds though, so he might be limited to 250MB/s.
     
    #17
  18. alderaan

    alderaan New Member

    Joined:
    Apr 4, 2012
    Messages:
    2
    Likes Received:
    0
    Ok, done everything. So used 2 nic:

    - NC364t 4xlan connected in pci-e x16
    - intel 39Y6127 2x lan cutted and connected in pci-e following this thread.

    Thank.
     
    #18
  19. RADCOM

    RADCOM New Member

    Joined:
    Aug 27, 2012
    Messages:
    11
    Likes Received:
    4
    IBM variant of the pro/1000PT I count 14 traces after the cut on my card, yours has only 13 or is my eyesight really that bad and I have no business cutting this card anyway lol Did you cut through one?
     
    #19
  20. TvdH

    TvdH New Member

    Joined:
    Jul 17, 2013
    Messages:
    5
    Likes Received:
    0
    Would it not have been easier to use something like this instead of cutting the card's PCI-E interface?

    [​IMG]
     
    #20

Share This Page