Celestica Silverstone DX400 32x 400GbE Switch - Information Requested

[werewolf191]

Hi,

I recently started to setup a home lab and get some hands on experience configuring an Arista 7050 TX 72. This forum was real helpful to find the right EOS firmware version that was compatible with the switch.

I recently came across Celestica Silverstone DX400 32x 400GbE and looking to get it for running AI/ML work across different servers i have. The information about this switch was hard to come by and I could not find any except for a data sheet for the switch.

I would like to seek guidance from this group if anyone has any experience or familiar with the switch. I would really appreciate if someone can throw some light on it.

I am looking to install SONiC on it eventually.

Thank you.

 

[necr]

Hi werewolf, wondering what you're planning for NIC connectivity. ConnectX-6? Or CX7 with SR8 or DACs? Note that QSFP-DD DAC will not work on NIC's side (NICs require normal QSFP28/56/112).

 

[werewolf191]

Currently, My NICs are ConnectX-5, and i am going to use 100G only for now. So going with QSF-DD to QSF28 adapter cable for now if i end up getting the switch. There are a few NICs that directly support QSFP-DD like Nvidia Mellanox ConnectX-7 400GbE CX71343DA CX71343DAC_WEAF_FB. Currently, they are going at $400 on ebay...

By the way, Do you have any experience with the above switch? Alternatively, I am also considering OCP Wedge 400G as well as alternative but it has a few missing parts such as HDD, Battery, CPU/COMe module, BSM, RAM. While I found all except the BMC Storage Memory... Any help on finding M.2 Type A key 2260 for BSM would also be helpful...

 

[necr]

has a few missing parts such as HDD, Battery, CPU/COMe module, BSM, RAM

sure looks like a rabbit hole, can't buy Minilake that easily, Facebook has not retired all Minipacks yet. But it seems they're throwing quite some early CX7s out.

There are a few NICs that directly support QSFP-DD

I'm still waiting to see one that supports QSFP-DD, I'm fairly sure it's QSFP112 based. (Upd. found some BCM957608)

By the way, Do you have any experience with the above switch? Alternatively, I am also considering OCP Wedge 400G as well as alternative

DX400 looks like a better bet, Wedge 400G is not on SONiC HCL, and that design hasn't been fully onboarded by OCP. Some Meta-specific storage, no spare parts, no firmware downloads whatsoever, Celestica who's not so responsive...sounds like my next HW project lol

 

[werewolf191]

Thanks for your detailed response... I wanted to try out Wedge 400G just because of the price and the spares are available and cheap except the BCM Storage Unit which I am unable to figure out the spare. So going to be tough... And also your view that it is not fully onboard by OCP makes total sense since I could not find documentation for it on OCP and was wondering why that would be...

Sent an email to Celestica support team on DX400, they said it is an old model and out of warranty which is expected but can support any open NOS. They asked me to consider standard DS4000 which is in production now but something definitely out of budget for a homelab in my opinion.

Let me see if i end swinging for the Celestica DX400 or not...

 

[Layla]

Where did you find the type 7 COMe module for Wedge400? Or minilake? I couldn't find any at all.. Also no luck on BSM so far. Are you guys having better luck with Google than I am?

 

[Layla]

There are a few NICs that directly support QSFP-DD like Nvidia Mellanox ConnectX-7 400GbE CX71343DA CX71343DAC_WEAF_FB.

You're sure these support QSFP56-DD? I also thought they were vanilla QSFP112.. though if they were made for Facebook QSFP56-DD support would make some sense given Wedge400..

 

[werewolf191]

You're sure these support QSFP56-DD? I also thought they were vanilla QSFP112.. though if they were made for Facebook QSFP56-DD support would make some sense given Wedge400..

I stand corrected. Yeah you are right it is vanilla QSFP112 and might need a 400G OSFP Flat Top to 400G QSFP-DD Passive Direct Attach Copper Twinax Cable. These cables seem to be available in FS.

 

[werewolf191]

Where did you find the type 7 COMe module for Wedge400? Or minilake? I couldn't find any at all.. Also no luck on BSM so far. Are you guys having better luck with Google than I am?

While I found type 7 COMe that does not seem to be minilake and may not be suitable afterall.

 

[Layla]

Any updates?

 

[aosudh]

Instead of worrying about whether the hardware can be purchased, it is better to think about whether the software on it can run smoothly, whether the ports are normal, and what the noise and power consumption will be like after purchase. The most important thing is whether Sonic can be installed at all. Even if there are corresponding port files and ONIE, will Sonic be easy to use after installation?

 

[necr]

Instead of worrying about whether the hardware can be purchased, it is better to think about whether the software on it can run smoothly, whether the ports are normal, and what the noise and power consumption will be like after purchase. The most important thing is whether Sonic can be installed at all. Even if there are corresponding port files and ONIE, will Sonic be easy to use after installation?

Yep, all of that plus firmware, I did not pull the trigger.

 

[Jelmer]

Even if there are corresponding port files and ONIE, will Sonic be easy to use after installation?

If just for switching, maybe a vlan without routing, yes. Any other features will not be easy with SONiC.

 

[werewolf191]

Apologies for delayed response. I did not buy 400G but ended up getting Plexxi 100G switch (nothing but Celestica DX010 100G) that was produced after the Intel ATOM clock issue was corrected. Installed SONiC and had basic network configuration done.... Haven't had much time recently... but will let you know how the Switch has been working in a bit.

 

[tritron]

I wonder if facebook and celestica use same bsm module ?

 

[Pureness2611]

Ive also been hunting for the BSM for Wedge 400. Been in contact with a board mfg to see if they reproduce it. COTS chips. No resistor or capacitor values makes it tough.

 

[Kadrit]

Ive also been hunting for the BSM for Wedge 400. Been in contact with a board mfg to see if they reproduce it. COTS chips. No resistor or capacitor values makes it tough.

https://www.opencompute.org/documents/wedge400-ocp-specification-v0-3-pdf This may be enough to make a run of new BSMs. I also have located all of the chips that the design specs call for including the discontinued chip. In the other thread there is a pretty good set of information and supporting links to the software end of things which is where I pulled this document from.

 

[Layla]

Any progress here for anyone?

 

[Kadrit]

Any progress here for anyone?

Ive got a company that will prototype but need to get a gerber file made for them to make it. Right now Im looking at hiring one.

 

[Foray]

Any progress here for anyone?

Yes.