1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Sharing my growth in the datacenter

Discussion in 'General Chat' started by Yarik Dot, Jun 13, 2015.

  1. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    Hi everybody,

    I found some pictures on my phone and I have decided to share them with you. I will update this post with pictures and I will create another posts with pictures of hardware that is not mine and I only saw/managed it.

    This post will be updated gradually, so we start at 2011 and you will see the bussiness growth. Just for your information - this hadn't been my primary source of money till now, it was only a second job and I provided the service mostly to friends of mine.

    I hope, you enjoy it.

    2011
    20111202_001_smaller.jpg 20111202_002_small.jpg

    By the end of 2011, I have collected several servers, put them to the datacenter to a shared rack with friend of mine and started providing some services on them. ONLY THOSE BLUE MARKED ARE MINE. The rest is friend's of mine.
    - 1 monitoring server - the first on top at the back of the rack
    - 2 customer dedicated servers - 4 drive server and twin right
    - 1 backup server - 2 drive server
    - 1 virtualization server - twin left - used KVM virtualization

    2012
    20120321_006.jpg 20120321_007.jpg

    Nothing much happened in 2012. Some servers were replaced around me.

    2013
    13060015.jpg 13060022.jpg 13060023.jpg

    Nothing much happened again. I spent a whole year abroad therefore I didn't have much time for this. Even that I was at least able to:
    - replace backup for 4 drive server (instead of 2 drive)
    - replace one socket 771 server for Dual socket Nehalem
    - add one Dual socket Nehalem for virtualization (yes, more virtuals, yes)
    - add mini 1U monitoring server with GSM gateway connected via COM port
    - moved servers a little, so I had the bottom part of the rack and friend of mine had the top part of the rack

    Only supermicro servers are mine. The rest is friend's of mine.

    2014
    14110002.jpg 14110004.jpg
    Slowly, growing and adding more and more servers. Again only supermicro servers are mine.

    However, one big event happened this year. I bought and resold a lot of servers. All of them were Dual Nehalem L5640, 24G RAM. I kept few of them for myself.
    DSC_0598.JPG

    Continues further in the topic as I hit the limit of 10 uploaded files.
     
    #1
    Last edited: Mar 11, 2016
    Alfa147x and Patrick like this.
  2. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    10,429
    Likes Received:
    3,540
    Eagerly awaiting this series!
     
    #2
  3. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    The main post has been updated.

    Some nice cable management I found in one datacenter.
    12040010.jpg 12040009.jpg
     
    #3
    T_Minus likes this.
  4. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    Added growth in 2013 and 2014. 2015 was juicy as I left my work and started providing officialy managed servers on my own. I will update the thread hopefuly soon.
     
    #4
    Patrick likes this.
  5. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    As I can only attach 10 files I have to create a new post here.

    2015 - Datacenter #1

    I did some shopping and we had to move from 1/2 rack to full 42U rack. I had top part of it and friend of mine the bottom.
    15060015.jpg

    - Just added few managed/virt/web/... servers
    - Added one 48x 1Gbps switch for servers
    - Added one 24x 100Mbps for private management VLAN for IPMI
    - Upgraded the backup server to 12 drives machine

    I played a little bit with cabeling (thx softlayer for inspiration), so you can see the progress - all was done during a day without major outage (just a few seconds for cable replacement):
    v0.jpg v1.jpg v2.jpg v3.jpg v4.jpg

    The goal was to have 1 power cable per U and 3 (IPMI, Public, Internal VLAN) per U (except the one right above the switches).
     
    #5
    Quasduco, Fzdog2 and Patrick like this.
  6. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    2015 - Datacenter #2

    In April 2015, one bigger project came in and I needed cheaper connectivity (it was very expensive in the first datacenter). So I decided to start the second housing - I ordered 10U of space, 1kW of power and 4x1Gbps uplinks.

    Firt of all I built a single server and moved it there.

    15100004.jpg

    It didn't take much time to add a new switch and another server:
    15100007.jpg 15100008.jpg

    And we were getting more and more servers online and by the end of 2015, the second datacenter looked like:

    WP_20151222_16_37_44_Pro.jpg

    The amount of servers grows, so does the traffic:
    uplink.png
     
    #6
    Last edited: Mar 14, 2016
    Chuntzu, whitey and Fzdog2 like this.
  7. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    Early 2016 - Datacenter #2

    To be honest most of my work is now done in this datacenter (and datacenter #3, where I don't have any pictures from). Datacenter #1 is just running, but very soon I am going to buy new hardware and do some migrations.

    Anyway, the project with high traffic requirements grows and I needed to move to get more space, so I did upgrade from 10U to 1/2 rack + 1.5kW power + 8x1Gbps LACP.

    First of all I prepared cabeling. I don't know why people in the datacenter decided to put PDUs like this, but I didn't have time to remake that as I just arrived at the beginning of planned maintenance and needed to get things done as quickly as possible.
    WP_20160209_18_03_34_Pro.jpg

    And then let's move the servers:
    WP_20160210_04_34_06_Pro.jpg
    Hmm, quite empty, right. So, let's add some storages:
    WP_20160220_09_20_16_Pro.jpg WP_20160220_09_22_10_Pro.jpg

    The current plan is to have only 2U servers below the switch, one 2U or 4U server above the switch and the rest of the rack fulfilled with 1U servers. Therefore I could afford to prepare power cabeling like this.

    What concerns the switch - there are currently 2 switches:
    - 24 port cisco 3560G used for public traffic (VLANs and IPv6 static routing)
    - 10 port mikrotik RB2011 for IPMI VPN/IPv6 firewall

    Servers are some Nehalem (L5640, E5640) servers, storage servers are E3v3/E3v5-1230, 32G, 12x4TB drives and sometimes some SSDs.

    TODO:
    - first of all I want to replace the switch with Dell N3048
    - then as money allows I would like to start replacing those 1U servers with Xeons E3/E5.
     
    #7
    Fzdog2 and Chuntzu like this.
  8. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    This time a little bit different post.

    One of my customers is a video sharing portal and my job is to provide as cheap as possible solution (HW and SW) for them. There is one testing server as a part of the CDN platform I do for them. The server is special, because it is meant to have lowest possible configuration and push as much as possible traffic out.

    We have hit the 10G limit on that server today.

    proxy_traffic.png

    The server configuration is following:
    - 2U Supermicro 825TQ (8SATA/SAS chassis)
    - Intel Xeon E3-1230v5
    - X11SSL-F motherboard with 1x 16G of RAM
    - Some Supermicro 10G PCI-E card
    - Only 3 SSDs (1x 512GB Samsung Pro, 2x Intel 535 480G SSDs) directly connected to the motherboard

    Unfortunately, all SSDs are at their limits, therefore more traffic will mean better solution. Really Samsung can't handle more traffic, but I believe Intels could do a little bit more.

    Anyway - it is nice to see, that 3 SSDs can push 10G of traffic in a real live scenario.

    Update: The best part of all of this is that the system consumes only 70W @220V with redundant power supplies.
     
    #8
    Last edited: Apr 12, 2016
    niekbergboer, spazoid, lmk and 2 others like this.
  9. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,102
    Likes Received:
    1,216
    That is some impressive traffic @Yarik Dot

    Is it local mainly to you or INT or certain countries mainly ?
     
    #9
  10. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    @T_Minus There are currently no limits in terms of countries, but I buy connectivity only for few coutries. There is either no traffic to other destinations or there is a traffic shaping as the connectivity to those destinations is expensive.
     
    #10
  11. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    We have been under quite nice DDOS last several days and I am going to share some insights with you.

    There were several attacks going on, so I will describe one by one. Let's start with some graph.
    ddos_switch.png

    Unfortunately, first attack took the site down. It was typical UDP flood. Interesting was that it aimed to port 80. However, problem in fact wasn't that server couldn't handle the load. The problem was that IRQ generated by network card used only 1 CPU core, which hit it's limits. Therefore there was no space for other requests. I will create a test scenario for this later and implement fix. Anyway, the traffic was generated from some transit ISP that doesn't have correctly setup BGP filters, so the traffic was from randomly generated IP addresses. As we don't need UDP on this server I just blocked all incoming UDP traffic on switch and problem dissapeared (I love those smart switches, they can handle so much).

    The second attack was more interesting as it was generated by thousands of wordpress instances (12 875 to be exact). Somebody used this bug: #4137 (Pingback Denial of Service possibility) – WordPress Trac and hit our servers with tons of HTTP requests. As all of them went to the homepage, it took the site down as PHP overloaded CPU.
    ddos_nginx.png

    There were almost 400 000 requests per minute hitting the site. The fix was very simple this time:
    if ($http_user_agent ~* (wordpress)) { return 404; }

    Meanwhile, there were some other UDP/NTP attacks going on, but they were blocked by the switch.

    I am currently eager to see other attacks from this mother ****er! Let's see if you can take the site down again.
     
    #11
    Last edited: Apr 25, 2016
    lmk likes this.
  12. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    #12
    Markus likes this.
  13. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    An upgrade just arrived today. Time to get rid of some Nehalem servers.

    WP_20160708_15_44_09_Pro.jpg WP_20160708_17_24_23_Pro.jpg WP_20160708_17_25_48_Pro.jpg WP_20160708_15_58_41_Pro.jpg

    All servers are dual E5-2428L, 48GB DDR3 RAM. Power consumption is hell good (75W idle, 128W full load without drives). Performance will be tested soon, but I expect similar to dual L5640.
     
    #13
    Patrick and Markus like this.
  14. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    Another upgrade today - this time the caching proxy.

    WP_20160727_14_58_29_Pro.jpg WP_20160727_14_58_35_Pro.jpg WP_20160727_15_17_10_Pro.jpg

    HW: 2x E5-2620v4, 64GB (4x16GB - I will buy another 4x16G as soon as I have money), 2x LSI 2308.
     
    #14
    Patrick likes this.
  15. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    20Gbps throughput achieved with the beast above:

    proxy_traffic.png

    Only one cpu of two was used:

    load.png

    Next goal? Gimme 30Gbps, baby :)) I am going to connect 2 new 10Gbps ports to this server - I want to test dual port 10G supermicro network card. So, in a real live scenario 30Gbps is next achievement. But out of peak, I would like have 40Gbps of ab traffic.
     
    #15
    T_Minus and Patrick like this.
  16. soamz

    soamz New Member

    Joined:
    Aug 15, 2016
    Messages:
    18
    Likes Received:
    0
    Which model of supermicro is this ?
    I see lots of bays.

    I guess you use it as caching ?

    We are looking for a similar caching server alone with atleast 24 bays.
     
    #16
  17. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,102
    Likes Received:
    1,216
    That's awesome!
     
    #17
    Yarik Dot likes this.
  18. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    #18
    Patrick likes this.
  19. Yarik Dot

    Yarik Dot Active Member

    Joined:
    Apr 13, 2015
    Messages:
    108
    Likes Received:
    73
    2016 - End of the year - Datacenter #2

    It's been a while I have posted pictures of our progress, but I think it's worth it. We got out of space, therefore, we moved all storage servers 1U down and let the PDU lie without mounting it. We also misscalculated an amount of PDUs, so there is a little help down there (DC doesn't like it).

    Switch has been up(down)graded to N3048. Downgraded because ... although it is newer than our old cisco, it has less SW features. Hopefuly, we will switch that for some higher serier of the switch and this one will go to the first datacenter. Currently thinking of Dell S3048 as a replacement.

    Servers are now all at least E3 or E5, except the most bottom storage, which is still Nehalem server. Everytime we got a replacement, customer called us that he needed more video encoding performance. :\

    WP_20161217_01_35_43_Pro.jpg WP_20161217_01_36_37_Pro.jpg
     
    #19
    CreoleLakerFan, maze and Patrick like this.
  20. maze

    maze Active Member

    Joined:
    Apr 27, 2013
    Messages:
    359
    Likes Received:
    50
    Clean. Better looking than a lot of dc space i'v been in :)
     
    #20

Share This Page