100GbE Mellanox 180 USD

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

MiniKnight

Well-Known Member
Mar 30, 2012
3,072
973
113
NYC
That heatsink isn't a heatsink. It's a block and heatpipe without a heatsink. You'll need to change that out to use these
 

fossxplorer

Active Member
Mar 17, 2016
554
97
28
Oslo, Norway
Just incredible speeds these cards offer:
  • 1/10/20/25/40/50/56/100Gb/s speeds
I've been doing some bench on 40GbE on ConnectX-3 VPI cards, and with RDMA you can push close to line rate, i wonder if it's the case with 100GbE too.
 

MikeWebb

Member
Jan 28, 2018
92
29
18
53
I couldn't begin to fathom the hardware required to push these speeds. Backbone speeds yes, but a storage server? JBOF?
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
Backbone speeds yes, but a storage server?
Think about use cases where you serve cached/hot data from ram. ;)

Also 100gbe or better qsfp28/sfp28 have lower latencies (compared to qsf14/qsfp+ or sfp+) which can be enough to justify upgrades to faster networking.
 
  • Like
Reactions: MikeWebb

forroden

Active Member
Jan 1, 2017
53
46
28
  • Like
Reactions: gigatexal

funkywizard

mmm.... bandwidth.
Jan 15, 2017
848
402
63
USA
ioflood.com
being pcie x16 is somewhat interesting, vs the x8 on the 2x40. being only a single port limits flexibility.

I guess you could attach a water block onto what's still on the card there.

Interesting, but cheap for a reason I would say.
 

Myth

Member
Feb 27, 2018
148
7
18
Los Angeles
how hard would it be to install a fan on these cards? Or maybe adjacent to them? We have big fans in our chassis, I wonder if we just left it as is, and put a 2,000 or 3,000 120mm fan pointing directly on them, if that would keep them cool.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
being pcie x16 is somewhat interesting, vs the x8 on the 2x40. being only a single port limits flexibility.

I guess you could attach a water block onto what's still on the card there.

Interesting, but cheap for a reason I would say.
You need a minimum of PCIe 3.0 x16 for a single port 100GbE card. Dual port 40GbE cannot push 80gbps on a PCIe x8 slot. That is also one of the big reasons some companies pushed for 50GbE. You can push 50GbE on a single PCIe 3.0 x8 which maximizes throughput per x8 in a server and gives you 2x the server density per switch since 50GbE is 1/2 100GbE.
 

LukeP

Member
Feb 12, 2017
183
21
18
44
after researching more on the Eth side, i found discussion that these guys dont run anywhere near as fast as advertised. IB is another fish.

i have several Solarflare and CX-2 / CX-3 cards but no CX-4 yet to really test.
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
after researching more on the Eth side, i found discussion that these guys dont run anywhere near as fast as advertised.
Probably bottlenecked by the cpu, 40gbe is about 85% cpu load on a 1630v4 without rdma
 

llowrey

Active Member
Feb 26, 2018
167
138
43
I'm the sucker who ordered one of these. It's still on the slow boat from China though.

My use case is, perhaps, unusual. I have a 32-core dual Opteron that is PCIe2 only. So, an x8 card is going to max out at about 25Gbps. In order to go faster I need an x16 card since I only have the one slot free and can't do 2 bonded x8 cards. I doubt I'll get even 50Gbps out of this but if I can do better than 10Gb single-stream or better than 20Gb aggregate, I'll be happy.

Now, what to do about the heatpipe... Since I'm only going to lightly load this thing, I wonder if the "cold" end of the bare heatpipe will be able to radiate enough heat and/or conduct enough away.

I understand that it is possible to monitor the temperature of ConnectX cards. How does one read the temp sensor? I've poked around the /sys tree for my ConnectX-3 card but couldn't find anything that looked like a temperature sensor. Does the inbox driver expose the sensor or does one need to use an MFT utility to get at it (mget_temp)?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Run "mget_temp" from the mellanox firmare tools.
Just installed mft 4.10 to my esx but no mget_temp unfortunately :(
Quite weird because there even is a man page for it ...

Code:
 ./mst version
mst, mft 4.10.0.104, built on Jul 01 2018, 16:52:28. Git SHA Hash: 9999fe74e30f7bbd4166f3efcd5273aa64c43694

/opt/mellanox/bin] ls -l
total 87204
-r-xr-xr-x    1 root     root          1747 Jul  1 14:01 flint
-r-xr-xr-x    1 root     root       9832400 Jul  1 14:01 flint_ext
-r-xr-xr-x    1 root     root        144432 Jul  1 14:01 mcra
-r-xr-xr-x    1 root     root       1204560 Jul  1 14:01 mdevices_info
-r-xr-xr-x    1 root     root       5325120 Jul  1 14:01 mlxconfig
-r-xr-xr-x    1 root     root          1749 Jul  1 14:01 mlxdump
-r-xr-xr-x    1 root     root      12925288 Jul  1 14:01 mlxdump_ext
-r-xr-xr-x    1 root     root      13285472 Jul  1 14:01 mlxfwmanager
-r-xr-xr-x    1 root     root          1749 Jul  1 14:01 mlxlink
-r-xr-xr-x    1 root     root      12324940 Jul  1 14:01 mlxlink_ext
-r-xr-xr-x    1 root     root           691 Jul  1 14:01 mlxmcg
-r-xr-xr-x    1 root     root          1748 Jul  1 14:01 mlxreg
-r-xr-xr-x    1 root     root      11980684 Jul  1 14:01 mlxreg_ext
-r-xr-xr-x    1 root     root          1750 Jul  1 14:01 mlxtrace
-r-xr-xr-x    1 root     root       9685348 Jul  1 14:01 mlxtrace_ext
-r-xr-xr-x    1 root     root         15902 Jul  1 14:01 mst
-r-xr-xr-x    1 root     root           691 Jul  1 14:01 mstdump
-r-xr-xr-x    1 root     root        151376 Jul  1 14:01 mtserver
-r-xr-xr-x    1 root     root          1748 Jul  1 14:01 wqdump
-r-xr-xr-x    1 root     root      12365068 Jul  1 14:01 wqdump_ext
 
Last edited:

LukeP

Member
Feb 12, 2017
183
21
18
44

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
@i386 - Ever seen parts of the mft missing on ESXi? Or never checked probably?;)

@All - could anyone who has mft installed on ESXi verify? Feedback appreciated (ESXi/mft version, present or not) ..
Thanks