Sharing my growth in the datacenter

[Yarik Dot]

Hi everybody,

I found some pictures on my phone and I have decided to share them with you. I will update this post with pictures and I will create another posts with pictures of hardware that is not mine and I only saw/managed it.

This post will be updated gradually, so we start at 2011 and you will see the bussiness growth. Just for your information - this hadn't been my primary source of money till now, it was only a second job and I provided the service mostly to friends of mine.

I hope, you enjoy it.

2011

By the end of 2011, I have collected several servers, put them to the datacenter to a shared rack with friend of mine and started providing some services on them. ONLY THOSE BLUE MARKED ARE MINE. The rest is friend's of mine.
- 1 monitoring server - the first on top at the back of the rack
- 2 customer dedicated servers - 4 drive server and twin right
- 1 backup server - 2 drive server
- 1 virtualization server - twin left - used KVM virtualization

2012

Nothing much happened in 2012. Some servers were replaced around me.

2013

Nothing much happened again. I spent a whole year abroad therefore I didn't have much time for this. Even that I was at least able to:
- replace backup for 4 drive server (instead of 2 drive)
- replace one socket 771 server for Dual socket Nehalem
- add one Dual socket Nehalem for virtualization (yes, more virtuals, yes)
- add mini 1U monitoring server with GSM gateway connected via COM port
- moved servers a little, so I had the bottom part of the rack and friend of mine had the top part of the rack

Only supermicro servers are mine. The rest is friend's of mine.

2014

Slowly, growing and adding more and more servers. Again only supermicro servers are mine.

However, one big event happened this year. I bought and resold a lot of servers. All of them were Dual Nehalem L5640, 24G RAM. I kept few of them for myself.


Continues further in the topic as I hit the limit of 10 uploaded files.

 

[Patrick]

Eagerly awaiting this series!

 

[Yarik Dot]

The main post has been updated.

Some nice cable management I found in one datacenter.

 

[Yarik Dot]

Added growth in 2013 and 2014. 2015 was juicy as I left my work and started providing officialy managed servers on my own. I will update the thread hopefuly soon.

 

[Yarik Dot]

As I can only attach 10 files I have to create a new post here.

2015 - Datacenter #1

I did some shopping and we had to move from 1/2 rack to full 42U rack. I had top part of it and friend of mine the bottom.


- Just added few managed/virt/web/... servers
- Added one 48x 1Gbps switch for servers
- Added one 24x 100Mbps for private management VLAN for IPMI
- Upgraded the backup server to 12 drives machine

I played a little bit with cabeling (thx softlayer for inspiration), so you can see the progress - all was done during a day without major outage (just a few seconds for cable replacement):


The goal was to have 1 power cable per U and 3 (IPMI, Public, Internal VLAN) per U (except the one right above the switches).

 

[Yarik Dot]

2015 - Datacenter #2

In April 2015, one bigger project came in and I needed cheaper connectivity (it was very expensive in the first datacenter). So I decided to start the second housing - I ordered 10U of space, 1kW of power and 4x1Gbps uplinks.

Firt of all I built a single server and moved it there.



It didn't take much time to add a new switch and another server:


And we were getting more and more servers online and by the end of 2015, the second datacenter looked like:



The amount of servers grows, so does the traffic:

 

[Yarik Dot]

Early 2016 - Datacenter #2

To be honest most of my work is now done in this datacenter (and datacenter #3, where I don't have any pictures from). Datacenter #1 is just running, but very soon I am going to buy new hardware and do some migrations.

Anyway, the project with high traffic requirements grows and I needed to move to get more space, so I did upgrade from 10U to 1/2 rack + 1.5kW power + 8x1Gbps LACP.

First of all I prepared cabeling. I don't know why people in the datacenter decided to put PDUs like this, but I didn't have time to remake that as I just arrived at the beginning of planned maintenance and needed to get things done as quickly as possible.


And then let's move the servers:

Hmm, quite empty, right. So, let's add some storages:


The current plan is to have only 2U servers below the switch, one 2U or 4U server above the switch and the rest of the rack fulfilled with 1U servers. Therefore I could afford to prepare power cabeling like this.

What concerns the switch - there are currently 2 switches:
- 24 port cisco 3560G used for public traffic (VLANs and IPv6 static routing)
- 10 port mikrotik RB2011 for IPMI VPN/IPv6 firewall

Servers are some Nehalem (L5640, E5640) servers, storage servers are E3v3/E3v5-1230, 32G, 12x4TB drives and sometimes some SSDs.

TODO:
- first of all I want to replace the switch with Dell N3048
- then as money allows I would like to start replacing those 1U servers with Xeons E3/E5.

 

[Yarik Dot]

This time a little bit different post.

One of my customers is a video sharing portal and my job is to provide as cheap as possible solution (HW and SW) for them. There is one testing server as a part of the CDN platform I do for them. The server is special, because it is meant to have lowest possible configuration and push as much as possible traffic out.

We have hit the 10G limit on that server today.



The server configuration is following:
- 2U Supermicro 825TQ (8SATA/SAS chassis)
- Intel Xeon E3-1230v5
- X11SSL-F motherboard with 1x 16G of RAM
- Some Supermicro 10G PCI-E card
- Only 3 SSDs (1x 512GB Samsung Pro, 2x Intel 535 480G SSDs) directly connected to the motherboard

Unfortunately, all SSDs are at their limits, therefore more traffic will mean better solution. Really Samsung can't handle more traffic, but I believe Intels could do a little bit more.

Anyway - it is nice to see, that 3 SSDs can push 10G of traffic in a real live scenario.

Update: The best part of all of this is that the system consumes only 70W @220V with redundant power supplies.

 

[T_Minus]

That is some impressive traffic @Yarik Dot

Is it local mainly to you or INT or certain countries mainly ?

 

[Yarik Dot]

@T_Minus There are currently no limits in terms of countries, but I buy connectivity only for few coutries. There is either no traffic to other destinations or there is a traffic shaping as the connectivity to those destinations is expensive.

 

[Yarik Dot]

We have been under quite nice DDOS last several days and I am going to share some insights with you.

There were several attacks going on, so I will describe one by one. Let's start with some graph.


Unfortunately, first attack took the site down. It was typical UDP flood. Interesting was that it aimed to port 80. However, problem in fact wasn't that server couldn't handle the load. The problem was that IRQ generated by network card used only 1 CPU core, which hit it's limits. Therefore there was no space for other requests. I will create a test scenario for this later and implement fix. Anyway, the traffic was generated from some transit ISP that doesn't have correctly setup BGP filters, so the traffic was from randomly generated IP addresses. As we don't need UDP on this server I just blocked all incoming UDP traffic on switch and problem dissapeared (I love those smart switches, they can handle so much).

The second attack was more interesting as it was generated by thousands of wordpress instances (12 875 to be exact). Somebody used this bug: #4137 (Pingback Denial of Service possibility) – WordPress Trac and hit our servers with tons of HTTP requests. As all of them went to the homepage, it took the site down as PHP overloaded CPU.


There were almost 400 000 requests per minute hitting the site. The fix was very simple this time:
if ($http_user_agent ~* (wordpress)) { return 404; }

Meanwhile, there were some other UDP/NTP attacks going on, but they were blocked by the switch.

I am currently eager to see other attacks from this mother ****er! Let's see if you can take the site down again.

 

[Yarik Dot]

Just a quick update. We hit the ceiling with the proxy server at 12.5Gbps. I have created a serverfault question to find out how to optimize the system: 10/20/40Gbps nginx large files caching webserver

 

[Yarik Dot]

An upgrade just arrived today. Time to get rid of some Nehalem servers.



All servers are dual E5-2428L, 48GB DDR3 RAM. Power consumption is hell good (75W idle, 128W full load without drives). Performance will be tested soon, but I expect similar to dual L5640.

 

[Yarik Dot]

Another upgrade today - this time the caching proxy.



HW: 2x E5-2620v4, 64GB (4x16GB - I will buy another 4x16G as soon as I have money), 2x LSI 2308.

 

[Yarik Dot]

20Gbps throughput achieved with the beast above:



Only one cpu of two was used:



Next goal? Gimme 30Gbps, baby ) I am going to connect 2 new 10Gbps ports to this server - I want to test dual port 10G supermicro network card. So, in a real live scenario 30Gbps is next achievement. But out of peak, I would like have 40Gbps of ab traffic.

 

[soamz]

Another upgrade today - this time the caching proxy.

View attachment 3130 View attachment 3131 View attachment 3132

HW: 2x E5-2620v4, 64GB (4x16GB - I will buy another 4x16G as soon as I have money), 2x LSI 2308.

Which model of supermicro is this ?
I see lots of bays.

I guess you use it as caching ?

We are looking for a similar caching server alone with atleast 24 bays.

 

[T_Minus]

20Gbps throughput achieved with the beast above:

View attachment 3148

Only one cpu of two was used:

View attachment 3149

Next goal? Gimme 30Gbps, baby ) I am going to connect 2 new 10Gbps ports to this server - I want to test dual port 10G supermicro network card. So, in a real live scenario 30Gbps is next achievement. But out of peak, I would like have 40Gbps of ab traffic.

That's awesome!

 

[Yarik Dot]

Which model of supermicro is this ?
I see lots of bays.

I guess you use it as caching ?

We are looking for a similar caching server alone with atleast 24 bays.

I think it is this one: Super Micro Computer, Inc. - Products | Chassis | 2U | SC216BA-R920LPB or some very similar.

 

[Yarik Dot]

2016 - End of the year - Datacenter #2

It's been a while I have posted pictures of our progress, but I think it's worth it. We got out of space, therefore, we moved all storage servers 1U down and let the PDU lie without mounting it. We also misscalculated an amount of PDUs, so there is a little help down there (DC doesn't like it).

Switch has been up(down)graded to N3048. Downgraded because ... although it is newer than our old cisco, it has less SW features. Hopefuly, we will switch that for some higher serier of the switch and this one will go to the first datacenter. Currently thinking of Dell S3048 as a replacement.

Servers are now all at least E3 or E5, except the most bottom storage, which is still Nehalem server. Everytime we got a replacement, customer called us that he needed more video encoding performance. :\

 

[maze]

Clean. Better looking than a lot of dc space i'v been in