Stepping up in the world - Upgrading the rack.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
I am recording my current plans for upgrade here which I hope can help others in the future.

I will fill out this post but just want to get the two pictures up for now.

This is my home upgrade.

Proposed Network.



Storage organisation (so far).



RB
 
Last edited:

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Guessing that since your ISP entrance is on an ONT that you are FiOS (or one of the FiOS markets Verizon sold of recently)?

In that case...consider bypassing the "service provider router". The one that they provide is a serious bottleneck for your network. its slow, has a very limited NAT table, etc.

Call them as ask them to activate the RG-45 connector on the ONT (and clear the DHCP so that you can pull a new address on it). You have to keep their router in place so that your other services (like TV I presume) still work so you just reverse the order a bit and put their router inside your LAN (go into their router - they should have given you the admin password - disable their NAT and set it into passthrough mode which will change it from a Router into a switch).

You'll be much happier, especially if you upgrade to any of their 100mbit or higher services since their "service provider routers" normally don't even have GigE ports on them.
 

Biren78

Active Member
Jan 16, 2013
550
94
28
PigLover - I don't think he is in the US.

Idea --- can we make a common template for peeps to fill out with there setup diagram? Would be a great way to share these and then inform newbies (like me) on the different setups.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Oops - not paying close attention.

Of course, other PON-based FTTP may have the same issues FiOS does (yes - that is a stretch intended to cover for my inattention) :)
 

Mike

Member
May 29, 2012
482
16
18
EU
It's probably needed for voip with his contract. Or add asterisk for that matter :)
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
You can always run the ISP router behind your pfsense/Untangle/Sophos/etc. I did that on FIOS for a while to make MoCA simple for the set-top boxes. A difference with FIOS compared to how RimBlock has his diagram setup is FIOS separates the phone at the ONT no on the router.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Oops - not paying close attention.

Of course, other PON-based FTTP may have the same issues FiOS does (yes - that is a stretch intended to cover for my inattention) :)
:D.

Yep in Singapore.

I am on FTTH and running a 300Mbps (37.5MB/s) local have seen 8MB/s international (average between 7MB/s & 3MB/s depending on the time of day).

The initial router after the ONT has a single GbE port and 4x 100Mb. The 100Mb are fine for the STBs for service providers IPTV offering. Having my own DHCP server on the same network as the STBs kills their service, hence the seperate subnet. I have since learnt that the IPTV stuff is registering with a couple of vLANs from the service provider and that a second IP address is also assigned to each customer but is hidden. Whilst I could make use of these, the service provider will not provide a solution is an issue occurs sometime down the line. The two subnet setup works just fine for new ;). The main change is for the C6100 and the storage.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
;)
Rim block, how are you setting up your DAS? Also, how are you connecting it to your C6100 and what case are you using?
The DAS is a Norco 4120 (aww nuts, need to ammend the storage diagram again for 20 bays rather than 24). I will be connecting via a LSI 9202-16e SAS controller from the 1st C6100 node. The controller has 4x 4 lane channels (4 connectors). Two will go direct to the drives (for full speed SSD if needed) and the other two will go to the expander (when that is sorted out with NitroBass24 ;) ;) ) for mechanical hard drives. The Norco will have sata -> SAS internal cables, PCI SAS int to ext brackets and then there will be external SAS cables to the controller.

The big questions I currently have are;
1. Disk layout / Drive choice for my storage.
2. Disk configuration for my ESXi servers (VM store & swap on node / swap on node & VM store on the SAN / all on SAN).

I am tempted to go for the Seagate CS or ES.2 2TB drives. The CS are cheap entry level Enterprise but the ES.2 have an extra 2 years warranty, lower failure rate and are only around US$35 more expensive. 2TB seems to be a fairly good point for me on price, capacity and being able to make use of multiple drive for speed / redundancy.

RB
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
PigLover - I don't think he is in the US.

Idea --- can we make a common template for peeps to fill out with there setup diagram? Would be a great way to share these and then inform newbies (like me) on the different setups.
Any idea on the template contents ?.

RB
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Well my current server is now sold so that takes the pressure off a bit in this upgrade.

I am now just waiting for DBAs switch to arrive, Nitrobass24s Intel expander and my 2x C6100 servers.

I am playing around with getting one of the Rackable Systems DAS chassis. At US$175 bare, it seems pretty good. The ones I am looking at seem to be the later models after the SGI merger. They also tend to not look quite as ugly.

I already have a Supermicro JBOD power board which would go well with it.

Still not 100% sure on it yet though.

I also managed to spot some 8mtr DDR Infiniband CX4 to CX4 cables. Just what I needed and only US$15 each plus shipping (new is around US$80 for 3mtrs each).

Just waiting for the postman to come knocking.

RB
 

McKajVah

Member
Nov 14, 2011
59
10
8
Norway
I am playing around with getting one of the Rackable Systems DAS chassis. At US$175 bare, it seems pretty good. The ones I am looking at seem to be the later models after the SGI merger. They also tend to not look quite as ugly.

RB
I've got two of the SE3016's. I bought it from the same place as the link, but paid $149 i think... Also the pictures where the same as in your link, but are actually the "old ugly" type. It's rather loud, so I've thinking of swapping the two 80mm fans with 120mm "low" speed ones... The powersupply fan is also loud and should be changed. Both I got were SUPER clean inside and out.

I've not really tested the speed as I run Snapraid, but the expanders are 3Gbps only. I also had to upgrade my firmware on the Dell SAS2008 controller for some strange reason.

Some people over at "[H]" are not so happy with the SE3016's, but for the price there is nothing else out there and I've been really happy with them.


Edit: Here is the auction link for my purchase http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=130831824906
 
Last edited:

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Hi,

Yeah I had seen the thread at [H] and it was partly the reason I was even considering.

The only other options were the HP MSA70 but it is 2.5" drives and is SAS 3Gb/s but SATA 1.5Gb/s (sata I) only. The DELL FS12-NV7 FILE SQL STORAGE SERVER was also an option. It seems these were custom builds just like the C6100s but they come with dual 6 core AMD cpus and DDR2 ram. They are also US$399 which is a bit over budget especially as I didn't want the board / CPUs or ram. There is also the MSA60 but at that price I am getting closer to the new price for a Supermicro CSE-936A-R900B.

Good to hear about your experience with the Rackable though. I will check on the picture being of the actual unit if I decide to get one. The ones with the silve metal levers on the drive trays looks... well it is just not to my taste ;).

RB
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
The first C6100 arrived yesterday night with one Infiniband QDR mez card. The unit was well packaged and apart from a few very light scuffs (probably racking and de-racking related) looked to be in perfect condition. All the parts appeared to be as expected, all the cables were neat and tidy and even all the black shield pieces were there.

First impressions.
1. The top removable panel it pretty thin and flimsy. A little surprised at this as the Supermicto chasses and even the Norco units seem to be quite a bit stronger. On the other hand, how strong does it need to be :).

Top tip: Top middle where the removable 'lid' meets the top front of the chassis is a screw. This needs to be removed before the top 'lid' section can be slid off. Took me a while to spot it ...

2. The drive trays are terribly cheap. Again, compared to Norco or Supermicro they are just cheap. Easy to bend, just two 'rails' to attach the drive to with no bottom (no chance of mounting a 2.5" drive without an adapter).

3. The rails that came with it seem to be for racks with round mounting holes. Mine has square holes and the rails just do not seem to be able to fit. This is disappointing but not a big problem.

4. As other have said, this unit is noisy, very noisy. I will be doing the fan mod PigLover has detailed in his thread here. It also does not help that one or more of my fans are pulsing.

5. The motherboard sleds are secured with a single screw for each. Again, it took a few moments to work out why the sleds were not coming out until I finally found these.

On a more positive note, all CPUs recognised, all ram looking good with the option of doubling to 48GB per node with more 4GB modules (only half the ram slots are populated).

Infiniband.
I also received DBAs Infiniband DDR switch which is in great condition. I am not sure how I will rack it yet as it is around four times the depth of my HP 1810-24G and heavy all the way (weight is distributed along its full depth). A set of ears for the rear of the switch (the front came with ears) and a mid point rail in the rack would probably do the job very well but I currently have neither. Plugging in my cable from the C6100 Infiniband Mez card to the switch seemed to go well with a green light appearing on that port but I am not yet in a position to test anything (nothing else connected to the switch and no OS on any of the C6100 nodes yet). Looking good so far though.

IPMI.
Due to issues that have been reported after setting static IP addresses for the BMC I left it on DHCP but then needed to find a way of working out which IP had been assigned. I found that Supermicros IPMIView utility will detect the BMCs when doing a scan and enable login, sensor readings and power on/off etc but not any of the graceful reboots or console. This is not much of an issue as when you have the IP address you can just HTTPS://[ipaddress] and there is a web interface for management. The only issue now is that I can only seem to find two of the nodes. I suspect it is a switch config issue but unfortunately we had a power cut the night before and I cannot now locate the IP address to manage the Procurve (not default or any of the two addresses I have assigned in the past). I will probably need to reset the switch but that is no big deal.

Flirting with storage appliances.
I wanted to give ZFS a go so thought I would try Nexenta Community Edition. i downloaded the image (fast download) and burnt it to a DVD. I had a Seagate ES 500GB drive I recently purchased from a friend second hand so put it in the tray and booted up. It booted as expected, started the install and then after selecting the 500GB drive I hit the first problem. "FDisk cannot write partition layout to the drive". The installer then become unresponsive and for all intents and purposes hung. I pulled the drive, took it out of the sled and slotted it in to my trayless hotswap bay on my PC case. The issue was that the drive had been used in a Mac and had a 200MB EFI partition. Had to spin up Diskpart and remove it, put ti back int he sled and then reboot the node and start the installer again. This time it went through to the end, displayed the new hostname etc details but then appeared to hang again at the "press ok to reboot" screen. This is not instilling my with the greatest confidence. I will continue with it tonight.

One other thing I has seen is that the drive activity lights seem to be very weak. I can barely make out that the drive is being used. Anyone else see this ?.

RB
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Infiniband.
I also received DBAs Infiniband DDR switch which is in great condition. I am not sure how I will rack it yet as it is around four times the depth of my HP 1810-24G and heavy all the way (weight is distributed along its full depth). A set of ears for the rear of the switch (the front came with ears) and a mid point rail in the rack would probably do the job very well but I currently have neither. Plugging in my cable from the C6100 Infiniband Mez card to the switch seemed to go well with a green light appearing on that port but I am not yet in a position to test anything (nothing else connected to the switch and no OS on any of the C6100 nodes yet). Looking good so far though.
Gruber rails (or similar) to mount as a suggestion.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Yeah, def an option.

I even tried my Supermicro rails but the holes are in the wrong positions. They also do not fit my Norco chassis...

Cant have other manufacturers producing low cost rails and hitting other manufacturers high profit margins for their own versions.

Of course I could just put a few more holes in the Supermicro rails. Not too many though :D.

RB
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
For IPMI addresses I used DHCP 'reservations' in my DHCP server. Basically it reserves a specific IP based on the requesting MAC. In fact, wherever possible, I don't use any true statics anymore. It's so much easier to manage from one central place...and I don't have to rely on my feeble memory. I can also force rational addressing, for example using 192.168.200.181..183 for the IPMI on nodes 1..4 of my C6100. Also very easy on the memory.

You really do want it get the IPMI working. With remote console and ISO redirection you can put that DVD drive in the drawer and never burn another disk to load an OS again! And you can do it all sitting in your easy chair with a laptop while watching the game. It will change how you manage your servers and you'll wonder what you did before you had it.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Too bad you didn't end up getting the real rails Dell made for the C6100. They fit both square and round hole racks (though they might not fit threaded hole racks), work like the Gruber rails and are tool less install.