32x40GbE $400 OBO if you are crazy

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
so tempted to pick one of these up and try to get arista's EOS running on them. can 100% guarantee they're trident II, only question is the management plane

edit: after some googling it seems the management plane on these is intel x86 (Intel Rangeley, 4-core, 2.4 GHz) (not the AMD x86 SOC EOS expects), so it would take some work, but possible, especially if arista publishes their coreboot sources like they promised via email
 
Last edited:

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
so tempted to pick one of these up and try to get arista's EOS running on them. can 100% guarantee they're trident II, only question is the management plane

edit: after some googling it seems the management plane on these is intel x86 (Intel Rangeley, 4-core, 2.4 GHz) (not the AMD x86 SOC EOS expects), so it would take some work, but possible, especially if arista publishes their coreboot sources like they promised via email
Hm...I dunno, man. Arista EOS is only meant for Arista switches, and this switch supposedly will run well with Cumulus Linux (I heard that Penguin computing sold an OEM clone called the Arctica 3200XLP).
If you do a search for the Arctica 3200XLP on the Cumulus Linux HCL, you'll see the Arctica model listed.

Celestica looks like one of those too-big-for-its-own-good IT firms that does not know what its component subsidiaries are up to, and it's not readily apparent where you'll go for support if something goes belly up.
 

SPCRich

Active Member
Mar 16, 2017
256
137
43
42
Hm...I dunno, man. Arista EOS is only meant for Arista switches, and this switch supposedly will run well with Cumulus Linux (I heard that Penguin computing sold an OEM clone called the Arctica 3200XLP).
If you do a search for the Arctica 3200XLP on the Cumulus Linux HCL, you'll see the Arctica model listed.

Celestica looks like one of those too-big-for-its-own-good IT firms that does not know what its component subsidiaries are up to, and it's not readily apparent where you'll go for support if something goes belly up.
the Arctica 3200XLP's are great, and Cumuls Linux is pretty easy to navigate around... last job used nothing but penguin switches + CL.
 
  • Like
Reactions: nedimzukic2

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Hm...I dunno, man. Arista EOS is only meant for Arista switches
meh, EOS is just x86 fedora with a lot of custom shit on top making BRCM API calls to the ASIC. if it were a different ASIC I would say no, but both these switches are the exact same Trident II. Should just be a recompile of coreboot for the intel SoC board, an edit of the mgmt board definition files so it knows what PCI address to talk to the ASIC on, and maybe an edit of the I2C addresses so it knows where to find the temperature sensors and fans (they're probably addressed differently on the intel board). I know EOS couldn't care less what it's running on, I've even stuck the switch images in VMs and it boots fine (just complains about not finding the chassis SPI data etc)

I'd wager it'd take me less work than the quanta > brocade crossflash did (having nice python and bash to work with is night and day versus digging through disassembled PowerPC binaries). Just a matter of finding the motivation and free time, I probably won't have any until after the holidays
 
Last edited:

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
meh, EOS is just x86 fedora with a lot of custom shit on top making BRCM API calls to the ASIC. if it were a different ASIC I would say no, but both these switches are the exact same Trident II. Should just be a recompile of coreboot for the intel SoC board, an edit of the mgmt board definition files so it knows what PCI address to talk to the ASIC on, and maybe an edit of the I2C addresses so it knows where to find the temperature sensors and fans (they're probably addressed differently on the intel board). I know EOS couldn't care less what it's running on, I've even stuck the switch images in VMs and it boots fine (just complains about not finding the chassis SPI data etc)

I'd wager it'd take me less work than the quanta > brocade crossflash did (having nice python and bash to work with is night and day versus digging through disassembled PowerPC binaries). Just a matter of finding the motivation and free time, I probably won't have any until after the holidays
Yeah, but what's the point/benefit of running EOS on non-Arista hardware? You'll still ostensibly need access to an Arista support contract to get EOS, you'll still have to hack each and every subsequent update to make it work, and the vendor might/might not implement some dealbreaker check that'll cripple your ability to use it down the line. It's like running Hackintosh machines - it's fun for the sake of hobbyist/learning, but after doing it after the second/third time, it really is the textbook definition of repeating something fully expecting different results. The same applies hacking up a JunOS image from a QFX5100 or OCX1100 to make it work - that's a legit Broadcom Trident2 switch connected to a quadcore Xeon v2, so in theory it should work somewhat - Those things are FreeBSD 11 based and it's not that hard to hack the installer scripts to work on a VM. Of course, both switches don't really have a bootloader (like ONIE) that allows it to run something from a competitor.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Yeah, but what's the point/benefit of running EOS on non-Arista hardware? .
Probably the same "point" that's driving more than a hundred downloads a month of my quanta > brocade firmware a year after it was released - you get new, maintained firmware with the featureset of a $1000+ switch on a cheap OEM box that's readily available for $300.

Everything you're saying surely applies if you're purchasing for an enterprise environment, in which case what are you doing in the servethehome great deals section? Buy a switch that's not 5+ years old and run cumulus, done
 
  • Like
Reactions: nedimzukic2

Tiberizzle

New Member
Mar 23, 2017
25
11
3
124
Hi there, crazy checking in here I suppose. I purchased one of these a couple weeks ago and went ahead and purchased a second when I saw you lot found them :p

They are remorselessly whitebox x86 with the well-supported Trident II ASIC as noted above and will run Cumulus, Open Net Linux, probably any other similar switch image or even possibly a vanilla distro with switchdev or something built around one of the open source HALs. They are fantastically constructed and the chassis is a stunning unbadged and unfinished stainless steel.

The first came pre-installed with ONIE and the Ubuntu 14.04 LTS / Broadcom ICOS x86 distribution, hardware stats below. This means they are still supported for security updates to the included base OS (but not the Broadcom ASIC components, which I have not been able to find software updates for), and that you can painlessly install whatever else you have an image for.

In my opinion, these are the most switch you can get for your dollar right now or any time in recent memory. Other switches of this class range 3-10x as much and many of those lack ONIE support. ONIE support together with this chipset's positioning as a forerunner of the open networking "movement" should mean the switch ages very well.

Code:
System Description............................. Smallstone XP 56850 System, 3.2.2.24, Linux 3.16.0-29-generic
Machine Type................................... Smallstone XP 56850 System
Machine Model.................................. BCM-56850
Serial Number.................................. CES02173700010
Maintenance Level.............................. A
Manufacturer................................... 0xbc00
Burned In MAC Address.......................... 00:E0:EC:7C:44:E1
Software Version............................... 3.2.2.24
CPLD version................................... 0x13
Operating System............................... Linux 3.16.0-29-generic
Network Processing Device...................... BCM56850_A2
Additional Packages............................ BGP-4
                                                QOS
                                                Multicast
                                                IPv6
                                                Routing
                                                Data Center
                                                OpEN API
                                                Prototype Open API
Code:
# dpkg -l | grep icos
ii  icos-net-snmp                       1.0                                        all          The snmpd config files to enable proxy support on ICOS.
ii  icos-rxp-sxp                        1.3.2.2.24                                 amd64        Networking software for Broadcom ethernet switches.
# cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 77
model name    : Intel(R) Atom(TM) CPU  C2538  @ 2.40GHz
stepping    : 8
microcode    : 0x121
cpu MHz        : 1200.000
cache size    : 1024 KB
physical id    : 0
siblings    : 4
core id        : 0
cpu cores    : 4
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 11
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms
bogomips    : 4800.19
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 77
model name    : Intel(R) Atom(TM) CPU  C2538  @ 2.40GHz
stepping    : 8
microcode    : 0x121
cpu MHz        : 1200.000
cache size    : 1024 KB
physical id    : 0
siblings    : 4
core id        : 1
cpu cores    : 4
apicid        : 2
initial apicid    : 2
fpu        : yes
fpu_exception    : yes
cpuid level    : 11
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms
bogomips    : 4800.19
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor    : 2
vendor_id    : GenuineIntel
cpu family    : 6
model        : 77
model name    : Intel(R) Atom(TM) CPU  C2538  @ 2.40GHz
stepping    : 8
microcode    : 0x121
cpu MHz        : 1200.000
cache size    : 1024 KB
physical id    : 0
siblings    : 4
core id        : 2
cpu cores    : 4
apicid        : 4
initial apicid    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 11
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms
bogomips    : 4800.19
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor    : 3
vendor_id    : GenuineIntel
cpu family    : 6
model        : 77
model name    : Intel(R) Atom(TM) CPU  C2538  @ 2.40GHz
stepping    : 8
microcode    : 0x121
cpu MHz        : 1200.000
cache size    : 1024 KB
physical id    : 0
siblings    : 4
core id        : 3
cpu cores    : 4
apicid        : 6
initial apicid    : 6
fpu        : yes
fpu_exception    : yes
cpuid level    : 11
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms
bogomips    : 4800.19
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

# cat /proc/meminfo
MemTotal:        4028112 kB
MemFree:         1596104 kB
MemAvailable:    2943332 kB
Buffers:          186012 kB
Cached:          1046856 kB
SwapCached:            0 kB
Active:          1612164 kB
Inactive:         589664 kB
Active(anon):     750596 kB
Inactive(anon):      320 kB
Active(file):     861568 kB
Inactive(file):   589344 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        750488 kB
Mapped:            71472 kB
Shmem:               428 kB
Slab:             160360 kB
SReclaimable:     130520 kB
SUnreclaim:        29840 kB
KernelStack:        6160 kB
PageTables:         4824 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2014056 kB
Committed_AS:     862280 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       75728 kB
VmallocChunk:   34359605880 kB
HardwareCorrupted:     0 kB
AnonHugePages:    409600 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       72708 kB
DirectMap2M:     4102144 kB
root@core-rdu-1:/etc# lspci
00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 03)
00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 03)
00:02.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 2 (rev 03)
00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 03)
00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 03)
00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 03)
00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 03)
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 03)
00:17.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA2 Controller (rev 03)
00:18.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA3 Controller (rev 03)
00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 03)
00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 03)
01:00.0 Ethernet controller: Broadcom Corporation Device b850 (rev 03)
# lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 14.04 LTS
Release:    14.04
Codename:    trusty
 
Last edited:

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
Probably the same "point" that's driving more than a hundred downloads a month of my quanta > brocade firmware a year after it was released - you get new, maintained firmware with the featureset of a $1000+ switch on a cheap OEM box that's readily available for $300.

Everything you're saying surely applies if you're purchasing for an enterprise environment, in which case what are you doing in the servethehome great deals section? Buy a switch that's not 5+ years old and run cumulus, done
Okay, several points, and this is not meant as a personal attack against you and the wonderful work you do for the STH community here.

Running a brand name firmware (like modified IOS/JunOS/EOS) on OpenSwitch devices are a headache. Never mind the legality of grabbing name brand firmware, hacking it up to make it work (could be fun as an academic/hobbyist exercise, I used to do that in the Hackintosh/Hackintosh VM side of things), or the distribution of such an artifact, you are also depending on a single subject matter expert (SME), namely you, to hack-patch-release it. What happens if your work/personal life kicks in and free time goes to naught? Well, all those people who depended on you are SOL. When I buy hardware I don't plan to only keep it until the guy who release the software gets married and disappear - I prefer to buy something that doesn't have a single source for these kinda things, which is why I am glad i went with my Arista. Even if I go Cumulus it's still $799/year for access to software maintenance (which allows me to get the binary blobs that actually sets up the switch hardware, otherwise its just a large heavy noisy serial terminal), so mathwise, it doesn't make too much sense, even if the hardware is quite interesting. I might change my tune if Broadcom ICOS gets packaged into an RPM or pkg form, or if switchdev is a Broadcom hardware supported thing (it's only a Mellanox thing at the moment).

BTW, not everyone on the Enterprise side thinks whitebox switches are a great idea or are as promising as advertised. It is if you are Google/Facebook/Amazon scale, buys switches by the thousands, have a large devops team and have dedicated data center and remote hands staffing on the payroll 24/7. I don't have a large budget nor do I keep an inventory of spare parts, my staff can barely cope with the stuff we deal with already, and remote hands at my colocs are billable at $200/hour with 1 hour minimums. I would rather buy the name brand switch at the markups, get access to the long term support firmware versions, and then pay out the 27/4/365 support contracts so I can sleep better at night knowing that if something happens to my network equipment, it's not attributable to any failings on my part. My sales guys will also rip me a new one if the contract stipulates well known vendors and fully supported environments.

So no, I would rather snap up a Juniper or Arista on the cheap when they come on the secondary market (Would love a QFX5100 with that beefy Quadcore Ivy Xeon), and leave the whiteboxes alone for now.
 
Last edited:

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
Hi there, crazy checking in here I suppose. I purchased one of these a couple weeks ago and went ahead and purchased a second when I saw you lot found them :p

They are remorselessly whitebox x86 with the well-supported Trident II ASIC as noted above and will run Cumulus, Open Net Linux, probably any other similar switch image or even possibly a vanilla distro with switchdev or something built around one of the open source HALs. They are fantastically constructed and the chassis is a stunning unbadged and unfinished stainless steel.

The first came pre-installed with ONIE and the Ubuntu 14.04 LTS / Broadcom ICOS x86 distribution, hardware stats below. This means they are still supported for security updates to the included base OS (but not the Broadcom ASIC components, which I have not been able to find software updates for), and that you can painlessly install whatever else you have an image for.

In my opinion, these are the most switch you can get for your dollar right now or any time in recent memory. Other switches of this class range 3-10x as much and many of those lack ONIE support. ONIE support together with this chipset's positioning as a forerunner of the open networking "movement" should mean the switch ages very well.

Code:
System Description............................. Smallstone XP 56850 System, 3.2.2.24, Linux 3.16.0-29-generic
Machine Type................................... Smallstone XP 56850 System
Machine Model.................................. BCM-56850
Serial Number.................................. CES02173700010
Maintenance Level.............................. A
Manufacturer................................... 0xbc00
Burned In MAC Address.......................... 00:E0:EC:7C:44:E1
Software Version............................... 3.2.2.24
CPLD version................................... 0x13
Operating System............................... Linux 3.16.0-29-generic
Network Processing Device...................... BCM56850_A2
Additional Packages............................ BGP-4
                                                QOS
                                                Multicast
                                                IPv6
                                                Routing
                                                Data Center
                                                OpEN API
                                                Prototype Open API
Code:
# dpkg -l | grep icos
ii  icos-net-snmp                       1.0                                        all          The snmpd config files to enable proxy support on ICOS.
ii  icos-rxp-sxp                        1.3.2.2.24                                 amd64        Networking software for Broadcom ethernet switches.
# cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 77
model name    : Intel(R) Atom(TM) CPU  C2538  @ 2.40GHz
stepping    : 8
microcode    : 0x121
cpu MHz        : 1200.000
cache size    : 1024 KB
physical id    : 0
siblings    : 4
core id        : 0
cpu cores    : 4
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 11
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms
bogomips    : 4800.19
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 77
model name    : Intel(R) Atom(TM) CPU  C2538  @ 2.40GHz
stepping    : 8
microcode    : 0x121
cpu MHz        : 1200.000
cache size    : 1024 KB
physical id    : 0
siblings    : 4
core id        : 1
cpu cores    : 4
apicid        : 2
initial apicid    : 2
fpu        : yes
fpu_exception    : yes
cpuid level    : 11
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms
bogomips    : 4800.19
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor    : 2
vendor_id    : GenuineIntel
cpu family    : 6
model        : 77
model name    : Intel(R) Atom(TM) CPU  C2538  @ 2.40GHz
stepping    : 8
microcode    : 0x121
cpu MHz        : 1200.000
cache size    : 1024 KB
physical id    : 0
siblings    : 4
core id        : 2
cpu cores    : 4
apicid        : 4
initial apicid    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 11
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms
bogomips    : 4800.19
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor    : 3
vendor_id    : GenuineIntel
cpu family    : 6
model        : 77
model name    : Intel(R) Atom(TM) CPU  C2538  @ 2.40GHz
stepping    : 8
microcode    : 0x121
cpu MHz        : 1200.000
cache size    : 1024 KB
physical id    : 0
siblings    : 4
core id        : 3
cpu cores    : 4
apicid        : 6
initial apicid    : 6
fpu        : yes
fpu_exception    : yes
cpuid level    : 11
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms
bogomips    : 4800.19
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

# cat /proc/meminfo
MemTotal:        4028112 kB
MemFree:         1596104 kB
MemAvailable:    2943332 kB
Buffers:          186012 kB
Cached:          1046856 kB
SwapCached:            0 kB
Active:          1612164 kB
Inactive:         589664 kB
Active(anon):     750596 kB
Inactive(anon):      320 kB
Active(file):     861568 kB
Inactive(file):   589344 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        750488 kB
Mapped:            71472 kB
Shmem:               428 kB
Slab:             160360 kB
SReclaimable:     130520 kB
SUnreclaim:        29840 kB
KernelStack:        6160 kB
PageTables:         4824 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2014056 kB
Committed_AS:     862280 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       75728 kB
VmallocChunk:   34359605880 kB
HardwareCorrupted:     0 kB
AnonHugePages:    409600 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       72708 kB
DirectMap2M:     4102144 kB
root@core-rdu-1:/etc# lspci
00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 03)
00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 03)
00:02.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 2 (rev 03)
00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 03)
00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 03)
00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 03)
00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 03)
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 03)
00:17.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA2 Controller (rev 03)
00:18.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA3 Controller (rev 03)
00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 03)
00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 03)
01:00.0 Ethernet controller: Broadcom Corporation Device b850 (rev 03)
# lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 14.04 LTS
Release:    14.04
Codename:    trusty
Hm. Can you do an lspci -vvv and see if there are any support for SRIOV, and can you alos do a dmidecode -t 17? I want to see if that Celestica have RAM slots within, or is it just soldered RAM?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Okay, several points, and this is not meant as a personal attack against you and the wonderful work you do for the STH community here.

Running a brand name firmware (like modified IOS/JunOS/EOS) on OpenSwitch devices are a headache. Never mind the legality of grabbing name brand firmware, hacking it up to make it work (could be fun as an academic/hobbyist exercise, I used to do that in the Hackintosh/Hackintosh VM side of things), or the distribution of such an artifact, you are also depending on a single subject matter expert (SME), namely you, to hack-patch-release it. What happens if your work/personal life kicks in and free time goes to naught? Well, all those people who depended on you are SOL. When I buy hardware I don't plan to only keep it until the guy who release the software gets married and disappear - I prefer to buy something that doesn't have a single source for these kinda things, which is why I am glad i went with my Arista. Even if I go Cumulus it's still $799/year for access to software maintenance (which allows me to get the binary blobs that actually sets up the switch hardware, otherwise its just a large heavy noisy serial terminal), so mathwise, it doesn't make too much sense, even if the hardware is quite interesting. I might change my tune if Broadcom ICOS gets packaged into an RPM or pkg form, or if switchdev is a Broadcom hardware supported thing (it's only a Mellanox thing at the moment).

BTW, not everyone on the Enterprise side thinks whitebox switches are a great idea or are as promising as advertised. It is if you are Google/Facebook/Amazon scale, buys switches by the thousands, have a large devops team and have dedicated data center and remote hands staffing on the payroll 24/7. I don't have a large budget nor do I keep an inventory of spare parts, my staff can barely cope with the stuff we deal with already, and remote hands at my colocs are billable at $200/hour with 1 hour minimums. I would rather buy the name brand switch at the markups, get access to the long term support firmware versions, and then pay out the 27/4/365 support contracts so I can sleep better at night knowing that if something happens to my network equipment, it's not attributable to any failings on my part. My sales guys will also rip me a new one if the contract stipulates well known vendors and fully supported environments.

So no, I would rather snap up a Juniper or Arista on the cheap when they come on the secondary market (Would love a QFX5100 with that beefy Quadcore Ivy Xeon), and leave the whiteboxes alone for now.
Again, you are taking this way out of context. Nobody is suggesting to do this in an enterprise environment, or somewhere where the lack of a new version release means the end of the world if I disappear. Given that there are zero updates available for these by default (unless you have an icos license etc), even a single new release from somebody is already an improvement for people who have bought these. Same goes for the quanta > brocade release, I could die tomorrow and the hundreds of lb6m owners would still be better off than not, given they now have a 2018 fastiron build versus the garbage 2011 build of faspath they were originally stuck with. If you want even more examples, look at the SX6XXX thread, 10+ pages of people using even more thoroughly hacked firmware to get mellanox running on EMC devices. That one even involved editing actual binaries, way more than would be necessary for something like this.

I completely understand you would rather spend two to three times as much on a switch that comes with EOS out of the box, I would too if I were using it in a cash flow generating context (and indeed do at the day job). What I've been trying to tell you is that preference puts you in the minority on servethehome, the same forum that has people dremelling out holes in the back of servers to save $30. The amount of interest in making this happen that I've received in pm's, in irc, and on my discord just from my single post above drives home that fact. In fact, a few have even offered to help. So again, it's not going to be technically challenging, and it will happen regardless (just need to find a few free days), so you are welcome to ignore it when it does :)
 
Last edited:

Tiberizzle

New Member
Mar 23, 2017
25
11
3
124
Hm. Can you do an lspci -vvv and see if there are any support for SRIOV, and can you alos do a dmidecode -t 17? I want to see if that Celestica have RAM slots within, or is it just soldered RAM?
Code:
# dmidecode -t 17
# dmidecode 2.12
SMBIOS 2.8 present.

Handle 0x003E, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x003C
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 4096 MB
    Form Factor: DIMM
    Set: None
    Locator: DIMM0
    Bank Locator: BANK 0
    Type: DDR3
    Type Detail: Synchronous Unbuffered (Unregistered)
    Speed: 1600 MHz
    Manufacturer: <BAD INDEX>
    Serial Number: <BAD INDEX>
    Asset Tag: <BAD INDEX>
    Part Number: <BAD INDEX>
    Rank: 1
    Configured Clock Speed: 1600 MHz

Handle 0x0040, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x003C
    Error Information Handle: Not Provided
    Total Width: Unknown
    Data Width: Unknown
    Size: No Module Installed
    Form Factor: DIMM
    Set: None
    Locator: DIMM0
    Bank Locator: BANK 1
    Type: DDR3
    Type Detail: Synchronous Unbuffered (Unregistered)
    Speed: Unknown
    Manufacturer: NO DIMM
    Serial Number: NO DIMM
    Asset Tag: NO DIMM
    Part Number: NO DIMM
    Rank: Unknown
    Configured Clock Speed: Unknown

Handle 0x0041, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x003C
    Error Information Handle: Not Provided
    Total Width: Unknown
    Data Width: Unknown
    Size: No Module Installed
    Form Factor: DIMM
    Set: None
    Locator: DIMM1
    Bank Locator: BANK 0
    Type: DDR3
    Type Detail: Synchronous Unbuffered (Unregistered)
    Speed: Unknown
    Manufacturer: NO DIMM
    Serial Number: NO DIMM
    Asset Tag: NO DIMM
    Part Number: NO DIMM
    Rank: Unknown
    Configured Clock Speed: Unknown

Handle 0x0042, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x003C
    Error Information Handle: Not Provided
    Total Width: Unknown
    Data Width: Unknown
    Size: No Module Installed
    Form Factor: DIMM
    Set: None
    Locator: DIMM1
    Bank Locator: BANK 1
    Type: DDR3
    Type Detail: Synchronous Unbuffered (Unregistered)
    Speed: Unknown
    Manufacturer: NO DIMM
    Serial Number: NO DIMM
    Asset Tag: NO DIMM
    Part Number: NO DIMM
    Rank: Unknown
    Configured Clock Speed: Unknown
# lspci -vvv 00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Tr - Pastebin.com (lspci -vvv, too long to post)

If that dmidecode is accurate it's only using 1 channel of the dual channel memory controller. I'll take a look inside and see if slots actually exist when the next one arrives, utilizing an open second channel could be a pretty decent and low risk boost to control plane performance.
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
Code:
# dmidecode -t 17
# dmidecode 2.12
SMBIOS 2.8 present.

Handle 0x003E, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x003C
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 4096 MB
    Form Factor: DIMM
    Set: None
    Locator: DIMM0
    Bank Locator: BANK 0
    Type: DDR3
    Type Detail: Synchronous Unbuffered (Unregistered)
    Speed: 1600 MHz
    Manufacturer: <BAD INDEX>
    Serial Number: <BAD INDEX>
    Asset Tag: <BAD INDEX>
    Part Number: <BAD INDEX>
    Rank: 1
    Configured Clock Speed: 1600 MHz

Handle 0x0040, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x003C
    Error Information Handle: Not Provided
    Total Width: Unknown
    Data Width: Unknown
    Size: No Module Installed
    Form Factor: DIMM
    Set: None
    Locator: DIMM0
    Bank Locator: BANK 1
    Type: DDR3
    Type Detail: Synchronous Unbuffered (Unregistered)
    Speed: Unknown
    Manufacturer: NO DIMM
    Serial Number: NO DIMM
    Asset Tag: NO DIMM
    Part Number: NO DIMM
    Rank: Unknown
    Configured Clock Speed: Unknown

Handle 0x0041, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x003C
    Error Information Handle: Not Provided
    Total Width: Unknown
    Data Width: Unknown
    Size: No Module Installed
    Form Factor: DIMM
    Set: None
    Locator: DIMM1
    Bank Locator: BANK 0
    Type: DDR3
    Type Detail: Synchronous Unbuffered (Unregistered)
    Speed: Unknown
    Manufacturer: NO DIMM
    Serial Number: NO DIMM
    Asset Tag: NO DIMM
    Part Number: NO DIMM
    Rank: Unknown
    Configured Clock Speed: Unknown

Handle 0x0042, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x003C
    Error Information Handle: Not Provided
    Total Width: Unknown
    Data Width: Unknown
    Size: No Module Installed
    Form Factor: DIMM
    Set: None
    Locator: DIMM1
    Bank Locator: BANK 1
    Type: DDR3
    Type Detail: Synchronous Unbuffered (Unregistered)
    Speed: Unknown
    Manufacturer: NO DIMM
    Serial Number: NO DIMM
    Asset Tag: NO DIMM
    Part Number: NO DIMM
    Rank: Unknown
    Configured Clock Speed: Unknown
# lspci -vvv 00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Tr - Pastebin.com (lspci -vvv, too long to post)

If that dmidecode is accurate it's only using 1 channel of the dual channel memory controller. I'll take a look inside and see if slots actually exist when the next one arrives, utilizing an open second channel could be a pretty decent and low risk boost to control plane performance.

Huh. That Celestica have a virtualization capable routing engine which can also do SR-IOV, and if I read it correctly, that is conencted to an onboard i354 quadport GigE NIC. I wonder if it's inside some breakout in the chassis that can be exploited...
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY


Here's what they look like on the inside. It does appear to have an open SO-DIMM slot, and an mSATA slot for storage.
Huh. That looks like DDR3L notebook RAM - my guess is that you can do 32GB of RAM max using a pair of 16s. Is it ECC? I don't see any labels on the DIMM module and the dmidecode didn't say anything helpful. Anyways, Newegg is having a sale on the 128GB Vaseky mSATA drives, you might want to stock up if you plan to put some other stuff into the switch besides its base OS image.
 

Kneelbeforezod

Active Member
Sep 4, 2015
529
122
43
46
Considering that there are 3 small diameter fans blowing between the 2 power supplies, I would say that it is likely VERY noisy.
That's what i thought - has anyone replaced these fans yet as folks have done with other enterprise switches?
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
If you can set the fans to 30% pwm like arista switches then they will be still too loud for an office, but okayish for a dedicated homelab or "server" room.