Put this together....

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
I thought someone may enjoy this one. I was putting together two more 2P nodes for the STH hosting cluster (need more Ceph OSDs/ ZFS mirrors so I can sleep).

Arriving yesterday were also three quad port SFP+ cards. Since the middle riser in the Intel R2208WTTYS was occupied by the NVMe card for Intel A2U44X25NVMEDK, the chassis still had three open slots. Why not?

Intel X710 quad port 10Gb x3 Intel S2600WTT.JPG

Enough 10GbE for a 14 port appliance (2 onboard):

Intel X710 quad port 10Gb up lspci.JPG

And in Proxmox it became quite hard to tell which NIC (exactly) I wanted to use:
Intel X710 quad port 10Gb up Proxmox VE 4.0.JPG

Certainly some fun times with this system! I think the cards will come out when I take it to Fremont though.
 
  • Like
Reactions: Mrlie

Lost-Benji

Member
Jan 21, 2013
424
23
18
The arse end of the planet
Sorry, I fail to see the the need for anymore than 1x 10GbE to tie physical machines. The Internet connection not likely 10G, router/firewall won't handle 10G, and machines in VM's so it is software networking.

Too many dollars.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Lost-Benji is hell bent on peeing in someone's cheerio's today lol, sorry bro, just read your bite tongue thread and am laughing here.

On a thread related note...Yeah that many 10G connection is overkill (but certainly a fun scientific project), if Patrick is using it between Proxmox nodes certainly not a waste and can/will be used I assume...temporarily at least, if it's single node then yes the majority of the communications would be intra-node/leverage OVS or whatever bridge garbage Proxmox 'may' still be using :-D
 
  • Like
Reactions: Patriot

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
I have applications running today that use 4x10 LAGs and keep a second 4x10 LAG to a second router for redundancy. Why its not just a single 40g is a rather long story that shockingly makes sense when you consider the whole thing.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Sorry, I fail to see the the need for anymore than 1x 10GbE to tie physical machines. The Internet connection not likely 10G, router/firewall won't handle 10G, and machines in VM's so it is software networking.

Too many dollars.
So when that single 10GbE connection fails what happens? What about if the switch it is connected to needs to go down? Also - the Ceph cluster was easily able to saturate 1x 10GbE with only 12 drives four of which were hard drives.

This is not going to be a long term cluster solution, but it was fun to do.
 
  • Like
Reactions: Patriot

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
Ya - a single 10G is not very much anymore, if considering "large" 2P servers and virtualization (large is in quotes because 2P is still small - but large for a 2P which are the most common in virtual environments). Start with 2x10 as a bare minimum for redundancy, and a server with a couple 14-core chips and a TB of RAM can easily run enough VMs to keep that link busy before you even consider live-migrating those VMs to other hosts. And if all your storage traffic is going over those links as well (SMB/NFS/iSCSI/etc.) the bandwidth needs start to add up fast. 14 links in a single server is a little excessive, but just 1 is not nearly enough.
 

Lance Joseph

Member
Oct 5, 2014
82
40
18
Has anyone been able to get a hold of the XL710-QDA2 cards? Or any 40G Intel QSFP+ port adapters?
I've seen a Supermicro rebrand on Newegg and a couple of options from miscellaneous sellers on Amazon.
Are these in stock anywhere or are they sold out? Are they having issues keeping these in stock? If so, why?
 

Naeblis

Active Member
Oct 22, 2015
168
123
43
Folsom, CA
Has anyone been able to get a hold of the XL710-QDA2 cards? Or any 40G Intel QSFP+ port adapters?
I've seen a Supermicro rebrand on Newegg and a couple of options from miscellaneous sellers on Amazon.
Are these in stock anywhere or are they sold out? Are they having issues keeping these in stock? If so, why?

i have one NIB listed on ebay right now. New Intel Ethernet Converged Network Adapter X710-DA4 FH 10GB 4 SFP+ quad port

it became irrelevant when i went to FDR

Edit oops just read that it was QSFP+