10Gb SFP+ single port = cheaper than dirt

Discussion in 'Great Deals' started by MiniKnight, Sep 17, 2015.

  1. Keljian

    Keljian Active Member

    Joined:
    Sep 9, 2015
    Messages:
    429
    Likes Received:
    71
    For anyone wondering, these are pretty much plug and play with ESXi 6.0U1- you just need to set the rate (10000mbit)

    A curiosity is that I had the option of setting 40gbit, didn't try it as I don't have any 40gbit hardware.
     
    #121
    Last edited: Oct 8, 2015
  2. Keljian

    Keljian Active Member

    Joined:
    Sep 9, 2015
    Messages:
    429
    Likes Received:
    71
    Grumble- they bumped shipping to Australia to $50-60 ..
     
    #122
  3. xbliss

    xbliss Member

    Joined:
    Sep 26, 2015
    Messages:
    68
    Likes Received:
    0
    As these are tempting at the price even though I have/ would not be able to use them for a while.
    Should I just wait for when I am ready as the prices will stay or go down. Would the prices of these stay at this level, or go down in the future?

    I guess I wont be able to saturate until I have a Fast RAID or major SSDs that surpass SATA bandwidths.
     
    #123
  4. Keljian

    Keljian Active Member

    Joined:
    Sep 9, 2015
    Messages:
    429
    Likes Received:
    71
    Better/newer cards will drop in price as time goes on, naturally. These cards likely won't drop much more.

    Nvme drives can saturate 10gbit, and a pair of conventional SSDs could also. (Remember we are talking 1 gigabyte per sec, most SSDs can get close to 6gigabit (550 megabytes) a sec)

    Latency is the key though. You can have a 1000gb link, but if you have big latency, then when you trigger something off, it will take a while.
     
    #124
  5. J Hart

    J Hart Active Member

    Joined:
    Apr 23, 2015
    Messages:
    143
    Likes Received:
    97
    Latency is the main problem with using SFP+ DAC cables in the end. Unlike 10G-SR SFP+ modules(~100ns), the DACs take a certain amount of time to gain carrier(500ns). It adds up especially if your applications are sending many small frames. Of course it is no where near as bad as 10GB-T which takes substantially longer(2000-2500ns). And like you said, 1000base-T is even worse in this department at ~4000ns. Anyway, I always find these sorts of things interesting especially because you get situations where it is faster to read the memory of another machine than to wait for a local SSD to return some data(SSD latency ~100us).

    http://www.datacenterknowledge.com/...benefits-of-deploying-sfp-fiber-vs-10gbase-t/

    http://www.anandtech.com/show/8104/...ew-the-pcie-ssd-transition-begins-with-nvme/3

    And the ever popular latency comparisons guide from Dean and Norvig

    https://gist.github.com/jboner/2841832
     
    #125
    lmk, spazoid, Keljian and 1 other person like this.
  6. Keljian

    Keljian Active Member

    Joined:
    Sep 9, 2015
    Messages:
    429
    Likes Received:
    71

    Thanks for your post! I haven't tested fibre to fibre yet- still waiting on a second functional card, but gigabit to 10gig fibre latency is about 0.5 ms across the mikrotik crs226
     
    #126
  7. Fritz

    Fritz Well-Known Member

    Joined:
    Apr 6, 2015
    Messages:
    1,902
    Likes Received:
    396
    I may have set a new record yesterday. Picked up 5 Chelsio S310E-CR for $70 with free shipping. That's $14 apiece.
     
    #127
    Keljian likes this.
  8. Keljian

    Keljian Active Member

    Joined:
    Sep 9, 2015
    Messages:
    429
    Likes Received:
    71
    Not going to beat anyone here: I got a T420 for $80, shipping is taking ages though, I ordered about a month ago, with an estimated arrival of 2 weeks from now.
     
    #128
    Last edited: Oct 23, 2015
  9. ehfortin

    ehfortin Member

    Joined:
    Nov 1, 2015
    Messages:
    56
    Likes Received:
    5
    Does this apply to the hp 671798? If so, do we have to upgrade the firmware to an original Mellanox?

    I'm asking because I noticed that there are a few Connectx-2 supported (mnph29d-xtr, mt26478) with esx 6U1 and the mnpa19-xtr is not on the list.

    Thank you.

    Ehfortin
     
    #129
  10. Keljian

    Keljian Active Member

    Joined:
    Sep 9, 2015
    Messages:
    429
    Likes Received:
    71
    The mpna19-xtr was the card I used. Had a much easier time with it than the Chelsio t420(which I needed to install drivers for)
     
    #130
  11. DonJon

    DonJon Member

    Joined:
    Apr 9, 2016
    Messages:
    52
    Likes Received:
    5
    #131
  12. ehfortin

    ehfortin Member

    Joined:
    Nov 1, 2015
    Messages:
    56
    Likes Received:
    5
    That's what I ordered a few months ago. Didn't try to flash them but they are working fine with Windows Server 2012 R2 and VMware ESXi 6.x.
     
    #132
  13. Peanuthead

    Peanuthead Active Member

    Joined:
    Jun 12, 2015
    Messages:
    765
    Likes Received:
    124
    They should work fine. Buy me a couple and I'll tell you. :)
     
    #133
    Pete L. likes this.
  14. DonJon

    DonJon Member

    Joined:
    Apr 9, 2016
    Messages:
    52
    Likes Received:
    5
    Thanks for your inputs. Got some of those HP branded cards. Works flawlessly. The firmware was at ver 2.9.1000. But the boot rom is old which takes to a CLI when <ctrl-b> is entered. the newer FlexBoot version has a gui and supports iSCSI boot. Here is the documentation for it.

    https://www.mellanox.com/related-docs/prod_software/FlexBoot_user_manual.pdf

    But the problem I have is, these cards have a Device ID and PSID of HP's. So cannot update the rom with the image file downloaded from Mellonax (Can never find the fw updates from HP site). Tried to update the firmware by using the Mellonax .ini config with HP PSID and Device ID of Mellonax card, but after that there is no PXE boot screen. Below is the doc on how to customize these card for OEM FW.

    http://www.mellanox.com/page/custom_firmware_table

    It will be great to have iSCSI boot option on these cards. Has anyone tried and succeeded in updating the boot rom/firmware on these cards?
     
    #134
  15. Sleyk

    Sleyk Well-Known Member

    Joined:
    Mar 25, 2016
    Messages:
    901
    Likes Received:
    305
    I actually saw the auction Fritz was talking about before. I got 5 of those cards for $75 total. Fritz, my man, you beat me by $5 bucks!
     
    #135
  16. DonJon

    DonJon Member

    Joined:
    Apr 9, 2016
    Messages:
    52
    Likes Received:
    5
    The Device ID for the generic Mellanox and the HP branded card is 26448. The boot option GUI for iSCSI is included with FlexBoot version 3.4.460 ROM image available at this link under Archive.
    http://www.mellanox.com/page/products_dyn?product_family=34

    But the problem is the compiled rom images in .tgz file for 3.4.460 does not have the image for ConnectX-2 cards (i.e. Device ID 26448). But they are available in the prior version package 3.4.306 (But the boot GUI was not included on this version, only PXE boot available with iSCSI thru PXE). So no luck in getting the iSCSI boot config pages with this version.

    But the source code for FlexBoot 3.4.460 is available for download from the same link. Has anyone got experience in compiling the source for this specific card? ConnectX-2 with device id 26448? Any pointers are welcome. Hope to get iSCSI configurations directly on the boot options rather than depending on PXE server.
     
    #136
  17. Jerry Renwick

    Jerry Renwick Active Member

    Joined:
    Aug 7, 2014
    Messages:
    202
    Likes Received:
    37
    really a nice price!
     
    #137
  18. Boddy

    Boddy Active Member

    Joined:
    Oct 25, 2014
    Messages:
    767
    Likes Received:
    143
    How many SSDs would you need running to saturate a 1oGb? Just for interest sake.

    PS. There are some 10Gb cards with 3 meter cables for $20 available on EBay that seller is accepting offers for $18

    Mellanox Single Port with 3m DAC
     
    #138
  19. Sleyk

    Sleyk Well-Known Member

    Joined:
    Mar 25, 2016
    Messages:
    901
    Likes Received:
    305
    Well, if you raido two samsung evo 850 250gb ssd's, you would get a transfer speed of alittle over 1000MB/s if you use your onboard intel 6gbps controller. This is mighty close to 1GB/s which is close to the approximate limit of a 10gb link, which is theoretically 1250MB/s. I tested this in an effort to see what it would take to saturate my 10gb link to my htpc. It really isnt hard though. If you use one good ssd, like the samsung evo, you would still transfer at around 400-500MB/s. I test this with a ramdisk using my 120gb samsung evo 850. That little beast transfers at 430MB/s. The 250GB versions can do more.

    Hey Donjon, I wish i had abit more experience in compiling drivers for these Mellanox cards to lend a hand, but I confess I have little experience with this. Mellanox does seem open to answering questions about their drivers though. I base this on their high support of different OS'es. Maybe they can offer a tidbit of sustenance for you? hope you can get the cards working for you!
     
    #139
    Last edited: May 19, 2016
    Boddy likes this.
  20. Keljian

    Keljian Active Member

    Joined:
    Sep 9, 2015
    Messages:
    429
    Likes Received:
    71
    1x950pro - on 4xpci3
     
    #140
    Boddy likes this.

Share This Page