I've had a server cabinet at my house since 1999. It is in a constant state of change, and I occasionally document it on my web site. I've posted a few pictures here previously, usually to illustrate some point or to answer a question. A few months ago I realized I never posted the whole build, so I asked Patrick and he said I was welcome to post it here. So, without further ado:
The networking gear is in the top half of the left-hand rack:
From the top down are:
Most people can make the front of a rack look good without too much effort. Having the back also look good is a bit more of a challenge:
All of the cables (except for fiber jumpers) are custom-made by me to the exact length needed. I can get fiber jumpers so inexpensively (I pay less for an assembled and tested duplex cable) than I would for one of the four connectors I'd need to make the jumper myself, so I haven't custom-made any fiber jumpers in probably 10 years or so. I still do termination of bulk fiber (you simply can't get a 100,000-foot piece of 192-strand fiber with the ends pre-connectorized) for work.
The rest of the equipment in the racks is documented in lots more detail on my web site - this post just goes into the networking parts.
Over the years the equipment has changed. The switches have been (in order, as I remember them):
Here are some pictures of the cabinet(s) from various earlier times:
January, 2000:
February, 2003:
September, 2005:
The door was permanently removed from the server cabinet (for cooling reasons) in 2001. The single cabinet was switched out to a dual cabinet (replacing the older-style single cabinet) in September, 2013 after quite a few months of planning. Here is an image saved from Visio which shows the plan at the time - some equipment and locations changed before the switch-over and some of it after:
There was also a detailed planning document, called "The Big Plan 2" (the original Big Plan was for a major FreeBSD software + hardware update). Here is a slightly-redacted version:
Edited 9-Apr-18 to fix typo and use correct Visio design image, as well as adding information about RANCID, MRTG, and IPMI monitoring.
The networking gear is in the top half of the left-hand rack:
From the top down are:
- Fiber patch panel [advertising link]. 12 strands of fiber to one of my former off-site backup locations terminate in this panel. While that fiber is no longer in use, the patch panel [adverising link] provides a nice place to neatly coil up excess fiber.
- Cisco 3845 router. Its primary purpose is to provide out-of-band serial connectivity (hence the "OOB1" label) to other equipment in this rack via a NM-16A card. A secondary function is 14 VoIP termination ports using an EVM-HD-8FXS/DID card with an EM-HDA-6FXO expansion module. Right now I am only using 4 of the 14 ports. Also included are an NME-AIR-WLC8-K9 wireless LAN controller card and an NME-NAM-120S network analysis module. Rounding things out are 4 T1 ports (no longer in use) on a pair of VWIC-2MFT-T1-DI cards, a VPN module and a compression module. Before I upgraded to dual ASR1001 routers, this unit was my primary gateway.
- Dell PowerConnect 8024F and 8024 10 Gigabit Ethernet switches. These are stacked via 4 DAC cables, which you can see on the right hand side of the switches. This switch stack is trunked to the Catalyst 4948-10GE below via a pair of multimode fiber links for a total capacity of 20Gbit/sec. VLANs are used to keep traffic separated into griups. This leaves 38 ports for connecting various pieces of equipment. Most ports are available for future use - at present, the only 10 Gigabit links to end systems are the 4 RAIDzillas and the Dell PowerEdge [advertising link] R710 (both described below). At some point I will extend the 10 Gigabit network to other locations in the house.
- PowerDsine PD-9024G/ACDC/M/F PoE (Power over Ethernet) injector. This is a device which is inserted between a network switch and client devices to power them over the Ethernet (rather than using "wall wart" power bricks). While this is a 24-port unit, I have only cabled 8 ports to the Catalyst 4948-10GE below. The first 4 ports provide power to four Cisco Aironet 702i access points, while the 8th port powers the clock in the rack to the right. This is the managed version of the unit, so it also has its own Ethernet connection to the Catalyst 4948-10GE (you can see this cable on the right). Using the management interface I can remotely control power to each connected device as well as perform other management tasks. This is a Gigabit Ethernet unit and can provide 36W of power to each of the 24 connected devices.
- Cisco Catalyst 4948-10GE switch. This is my core network switch [advertising link]. It is connected to the Dell PowerConnect switches above via a pair of 10 Gigabit Ethernet multimode fiber links. It also connects to 3 other Catalyst 4948 switches throughout the property, via Gigabit Ethernet (the 10 Gigabit Ethernet is only used within these racks). The cables are color-coded - black is for network cables within the racks, blue is for network cables that go outside of the racks, green is for the serial console and yellow is for special purposes (this one connects to the Network Analysis Module in the Cisco 3845 router above).
- A pair of Cisco ASR1001 routers. These connect my equipment to the outside world via Gigabit Ethernet and a 100Mbit/sec backup Ethernet. They are both trunked to the Catalyst 4948-10GE via a pair of Gigabit Ethernet cables. Those cables carry a half dozen or so VLANs. The routers use HSRP for redundancy, so either can fail (or be reloaded) without affecting connectivity.
Most people can make the front of a rack look good without too much effort. Having the back also look good is a bit more of a challenge:
All of the cables (except for fiber jumpers) are custom-made by me to the exact length needed. I can get fiber jumpers so inexpensively (I pay less for an assembled and tested duplex cable) than I would for one of the four connectors I'd need to make the jumper myself, so I haven't custom-made any fiber jumpers in probably 10 years or so. I still do termination of bulk fiber (you simply can't get a 100,000-foot piece of 192-strand fiber with the ends pre-connectorized) for work.
The rest of the equipment in the racks is documented in lots more detail on my web site - this post just goes into the networking parts.
Over the years the equipment has changed. The switches have been (in order, as I remember them):
- Allied Telesyn (or Allied Telesis, depending on when you looked - 10BASE-T hub, not a switch)
- N Base NH208 (an unpleasant switch from an unpleasant company who wasn't interested in patching responsibly-reported security vulnerabilities)
- Cisco 2900M (couldn't walk and chew gum at the same time)
- Cisco 2900XL (same problems as the M, but add-in cards were faster)
- Cisco Catalyst 5505/RSM/NFFC (worked fine, but was a power pig, had an integrated RSP2-class router)
- Netgear GSM7248 (an attempt to get a Catalyst 4948-class switch, but on a budget)
- Cisco Catalyst 4948 (gave up on the Netgear, bought the switch I should have purchased in the first place)
- Cisco Catalyst 4948-10GE plus Dell PowerConnect 8024
- Cisco Catalyst 4948-10GE plus Dell PowerConnect 8024 and 8024F (current)
- Cisco 2501
- Cisco 2600 family - 2610, 2611, 2651, 2651XM
- Cisco Catalyst 5500 RSM (integrated router in switch)
- Cisco 2800 family (2811, 2821)
- Cisco 3800 family (3825, 3845 - the 3845 is still in use for some ancillary functions)
- Ubiquiti Edgerouter 8 Pro (played with, never put into production)
- Ubiquiti Edgerouter Infinity (likewise, beta unit)
- Cisco ASR1001 (quite a big upgrade!)
- Cisco ASR1001 * 2 (for redundancy)
Here are some pictures of the cabinet(s) from various earlier times:
January, 2000:
February, 2003:
September, 2005:
The door was permanently removed from the server cabinet (for cooling reasons) in 2001. The single cabinet was switched out to a dual cabinet (replacing the older-style single cabinet) in September, 2013 after quite a few months of planning. Here is an image saved from Visio which shows the plan at the time - some equipment and locations changed before the switch-over and some of it after:
There was also a detailed planning document, called "The Big Plan 2" (the original Big Plan was for a major FreeBSD software + hardware update). Here is a slightly-redacted version:
Code:
Any time:
X Blow dust out of both air conditioners from outside
X Clean filters on both air conditioners
X Make sure all FreeBSD systems have latest ports, kernel+world
X Create new port assignments for new-switch1 (4948-10GE) and new-switch2 (8024)
X Put that config on new-switch1
Sun 9/6:
X Remove log devices (SSDs) from rz1m/2/3
X Make sure RAID battery on gate charged properly; if not grab the one from the
prior gate and make sure it gets installed before gate gets racked
X Dismount rz1/1m/2/3 from gate NFS and comment out those fstab entries
X Shut down rz1/1m/2/3
X Shut down ns0/2
X Shut down old-www
X Shut down sv
X Shut down TL4000
X Remove power, network, console, etc. cables from shut down systems
X Move rz1m/3, ns0/2, old-www, sv, tl4000 out of room
X Remove batteries from Symmetra XR packs
X Detach cables from XR packs
X Move XR packs to near bookshelf stand, (temporarily) re-install batteries
X Disconnect and safely store wap4, clock1
X Bring in proper socket from garage tools for attaching side rails
Mon 9/7:
X De-rack remaining systems and stack on floor of dining room
X Remove remaining network, power, console, etc. cables
X Remove batteries, power modules from Symmetra and set aside
X De-rack Symmetra and (temporarily) re-install batteries, power modules
X Move old cabinet to living room (temporarily)
X Deinstall old air conditioner from behind rack and place on back stairwell landing
X Install new air conditioner in existing sleeve - measure total depth for future
removal w/o needing to move new racks
X Slide wooden base frame away from window until even w/ door frame
X Vacuum the whole room (or as much of it as can be reached), including ceiling
X Bring new cabinets from entryway and place on wood frame
X Install feet on cabinets, ensure level and room for wires under racks
X Bolt cabinets together
X Do final position adjustments of cabinets (make sure we have A/C clearance)
X Position front & rear rails, install 3rd set of rails (rear-facing)
X Bring old black & blue cabinets to entryway
X Install new SSDs in rz1/1m/2/3, install 10GE card in rz2
Tue 9/8:
X Install equipment in racks, starting with Symmetra in left and XR's in right
X Verify cabinet position, rail positions, cabinet level, A/C clearance
X Continue installing equipment in cabinets per drawing
X Party time!
Aftermath:
X Cable all of this stuff up again 8-(
X Move old cabinet (plus others) to entryway and call for pickup
X Vent fan in rear window (parts from basement / last year's failed project)
X 10GbE cabling
X Console (VGA / KVM) cabling
X Position temperature probes
X Krone blocks and frame, clean up wiring, mount in 2nd cabinet
X Try to neaten Ethernet / Cable Modem / Phone / whatever wires that enter racks
so all nice like blue solid CAT 5.
X Put sides on
X Need powerd configured on rz3, rz4
X Connect audio extension cable from SV to speakers on bookshelf
X Run a couple of permanent Ethernet cables for test systems (wall jacks?)
X Install USB extensions for keyboard and mouse, so lap holdable
X Pictures!
Notes:
o RPB current draw with all systems powered (RPB1/2/3/4): 8/7/5/5
o UPS runtime with all systems powered: ~ 90 Minutes
Last edited:
