400Gb/s? Now that 40g is dirt cheap, has anybody yet started using 400/200/100g at home?
- Thread starter Tsugi
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I see every now and then dual port 100gbe/edr mellanox cards* for ~500€ each, but I don't have any x16 slots left in my workstation or servers.
The cables (qsfp+ 28) are damn expensive, almost 120€ for 1m.
*Edit: flom versions, not standard add in cards
The cables (qsfp+ 28) are damn expensive, almost 120€ for 1m.
*Edit: flom versions, not standard add in cards
The majority of advanced users is moving to 40gb only recently (in force last 6 months or maybe a year I'd say) with the availability of cheaper options.
The next step is still a while out - especially since you need reasonably cheap hardware as well as the backend elements (storage etc) to use it.
O/c there are always the really early adopters but its quite an investment in both network hw and storage
The next step is still a while out - especially since you need reasonably cheap hardware as well as the backend elements (storage etc) to use it.
O/c there are always the really early adopters but its quite an investment in both network hw and storage
Cheapest I've been able to get offered in on Mellanox cards to do 100Gb ~$500-600 each. Which sucks when you want 2-4 of them.
Even if a switch with 2-4 100gb and 6+ 40gbe is $2k or less the cost-per-node is still $$$.
A bit pricey IMHo still for home Although I am looking ha ha.
Even if a switch with 2-4 100gb and 6+ 40gbe is $2k or less the cost-per-node is still $$$.
A bit pricey IMHo still for home
Although I am looking ha ha.
I also think until prices for qsfp28 cabling and switches dont come down there are not many people willing to go for home usage. Even somehow affordable hca/nic doesn't mean the rest of the fabric will be in the same range.
See the dirt cheap cx3, but good luck with finding fdr capable cables and ethernet switches. (i'm going through this atm, upgrading from QDR infiniband and 10gbe sfp+ to a single fdr fabric and spent way more time and money than i ever imagined it would take)
Also, there is for sure the question if there is anything that really requires single or multiple EDR (or even faster) links and also how urgent the need to do so is. Massive amount of nvme or sas3 ssd's maybe. But for this getting a dual port cx3 would (letting pcie 3 Limits out) give 80-112 GBit, depending on linkspeed, too - for a much lower total.
A lot of the currently used QDR/FDR Equipment came from OEM like Dell, hp, EMC, IBM that was obviously upgraded to EDR. So my bet is that there will be another 3-4 years until this will be upgraded and give interesting options for home use.
See the dirt cheap cx3, but good luck with finding fdr capable cables and ethernet switches. (i'm going through this atm, upgrading from QDR infiniband and 10gbe sfp+ to a single fdr fabric and spent way more time and money than i ever imagined it would take)
Also, there is for sure the question if there is anything that really requires single or multiple EDR (or even faster) links and also how urgent the need to do so is. Massive amount of nvme or sas3 ssd's maybe. But for this getting a dual port cx3 would (letting pcie 3 Limits out) give 80-112 GBit, depending on linkspeed, too - for a much lower total.
A lot of the currently used QDR/FDR Equipment came from OEM like Dell, hp, EMC, IBM that was obviously upgraded to EDR. So my bet is that there will be another 3-4 years until this will be upgraded and give interesting options for home use.
Sure
Im ready and prepared !
bottom: 2x 36x100Gb as spine's ( core )
middle: 2x 10/25 access ( 48 SFP+ ), 6x 40 Gb ( 4 blue are 100Gb ) leafs
top: router for that ( 48x10G SPF's ) , 6x 100Gb.
waiting for few MTP/MTO cables...( QSFP's are ready ) - on top of that there is one 100G DAC..
so leafs can be scaled to mentioned uplink : 4x100G...
and wating for rack...
Im ready and prepared !
bottom: 2x 36x100Gb as spine's ( core )
middle: 2x 10/25 access ( 48 SFP+ ), 6x 40 Gb ( 4 blue are 100Gb ) leafs
top: router for that ( 48x10G SPF's ) , 6x 100Gb.
waiting for few MTP/MTO cables...( QSFP's are ready ) - on top of that there is one 100G DAC..
so leafs can be scaled to mentioned uplink : 4x100G...
and wating for rack...
@mstrzyze this is 'home lab' use? If so I REALLY wanna know what type of stg/hosts/infra you have behind that BEAST of a fabric setup? :-D
He asked for something faster than usuall ..
Anyway , idea is for EVPN / BGP / Vxlan fun..there arent too much 25 g nic's now..yet
Mostly leaf/spine architecture ..and pretty fast core switches..
@whitey ..hmmm what is behind ..i dont want to steal this thread ..(3 racks: 1st is storage, 2nd networking, 3rd is compute..this leaf/spine probably will be in 4th rack..mixed as now). but this is "fast one"..maybe i can take few pics,but there is mess with cabling..
Anyway , idea is for EVPN / BGP / Vxlan fun..there arent too much 25 g nic's now..yet
Mostly leaf/spine architecture ..and pretty fast core switches..
@whitey ..hmmm what is behind ..i dont want to steal this thread ..(3 racks: 1st is storage, 2nd networking, 3rd is compute..this leaf/spine probably will be in 4th rack..mixed as now). but this is "fast one"..maybe i can take few pics,but there is mess with cabling..