ISP and 10G Upgrade Thread

[JustinH]

Hi All,
Posting here although its strictly not a server build, but slightly related.

about 3 weeks ago, we had a PC show here in Singapore, and lucky for us, a lot of the ISP's are in a bandwidth war at the moment. They are all trying to out do each other with either high speeds, lower costs, or lots of freebies.

Lucky for me, I'm with one of the smaller ISP's here in Singapore thats pretty aggressive, having rolled out the first 2G Fibre plan outside of Japan recently and are very "tech friendly". When I signed up with them 2 years ago, I signed up to a 200Mb plan, since then, they have bumped me (free of charge as well!) from 200Mb, to 400Mb to 500Mb and most recently to 1G down/500M up plan.

So just before the PC show, they sent a email letting me know my contract was about to expire and offering a promotion - 2G down/1G up, + free VPN (for Netflix etc) + a free Andriod Media Player + 6 months free subscription for around $US60/mth. I'm currently paying around $US110/mth on my current plan. Of course you can guess that I signed up very quickly. (I also get a additional 5 Static IP's from them that I host some websites etc on)

So whats a guy gonna do with a Gig Ethernet Network, but a 2G ISP link? Time to upgrade the Network of course! In this thread I'll post some updates over the next week or so on my experiences doing the upgrade.

First - I currently have the following servers/network at home
1 EdgeRouter Lite
1 Dell R510 acting as a Storage Server and oVirt Node
1 Dell PE 850 that acts as a Backup Server
1 Intel 2216 being built up (waiting on CPU's/Memory at the moment)
1 Cisco SG200 Switch

So, the EdgeRouter lite, while a awesome little box, isn't rated for 2G, and additionally, I'll most likely be doing LACP from the ISP's GPON Modem to my router - and the EdgeRouter series doesn't do any hardware offloading on bridged connections, so it would fall into the 200-300Mb throughput range. Not Acceptable.

Solution - MicroTik CCR1009-8G-1S-1S+ - Its one of their lower range routers but has a SFP and SFP+ port on it, and based on the performance results, for most setups minus 64byte packets, it will easily handle 2G from my ISP. It cost me around $US400 as a bundle deal through my ISP and is due to be delivered next week.
I also ordered 1 SFP and 1 SFP+ from Mikrotik as well. That should arrive early this week.

Next - The SG200 Switch - Not a bad little guy. 18 Ports, supports VLANS and also has 2 combo SFP/Ethernet Uplinks Ports. It had been my "core switch" upto now, but with my servers, and all the gadgets in the house, there was no way I could start doing LACP etc to other devices (to get the 2Gig bandwidth), as I was pretty much using all the ports already.
Ironically, right around that time, my Iphone Ebay up threw up a notification about "1 new search result for Nexus 5010" that I had saved. I previously had looked at the N5K switches, as they get be got for around $US1000 on ebay, but loads of people had told me Cisco lock the SFP's you can use in their gear, and Genuine Cisco SFP+'s are not cheap at all (thus turning the 1K switch into a 2K switch assuming you can get 20 Cisco SFP+ at $50/each (possible, but now not so attractive anymore).
The Ebay result was a real tease though, as it was for a N5K with a single power supply for $US800 BIN with BO option. Add frieght forward shipping to Singapore at 200-300, the total price was about what most N5K's were going for on Ebay.
So I went on a search for Cisco SFP+ deals, and found a little gem when researching Finisar FTLX8571D3BCL-E5 in Cisco Gear - the Undocumented command: "service unsupported-transceiver". Some more research and it showed that people were using these SFP+'s in Nexus gear without a problem.
So, I found a good deal and ended up getting 10 of these SFP+'s for $US100 shipped.

I also picked up some 1Gig Fibre SFP's (the first 8 ports in the N5K can do 1G/10G) and some 1 Gig Ethernet SFP's.

Network Cards wise, I got 2 Intel 520-SR2 cards (dual port SFP+'s) - Again confirmed with the Finisar SFP+ and a bunch of OM2 and OM3 Fibre cables.

Final purchase so far was 2 Cisco SFP's for the SG200 Switch. I'd continue to use that as a edge switch, but wanted to link it with Fiber to the N5K with LACP. I got geniune SFP's here, as the SG200 doesn't have a CLI, so I wasn't sure how it would react to finsair SFP's.

Now the hard part - After buying it all, it was gonna take a couple of weeks to reach Singapore. I was nervous about shipping damage, faulty/DOA etc and of course, was reading Nexus Documentation, and getting really worried that I had brought something that I wasn't gonna be able to make work. So on friday, the friendly courier man delivered all the goods. My next post I'll put up some details on getting the N5K up and running and some specs.

 

[Patrick]

Sad that I live a quick walk to Google, Symantec, LinkedIn, Intuit and etc here in Mountain View, CA and I get excited @ 100mbps speeds.

May I make 2 requests?
1. I am really interested in your MikroTik results and what you think about it.
2. Pictures!!!!

 

[Yarik Dot]

Agree. Pictures.

 

[JustinH]

So, since Friday, the WAF factor has taken a huge hit. After she woke up to see all the deliveries made, and everything laid out on the living room floor, I knew I was gonna have plenty of quiet time to get things upgraded!

The N5K arrived in perfect condition, and included 2 power supplies and 2 fan trays that brand new (still in antistatic bags with Cisco Seals unopened), while the N5K was used. It had a few minor scratches, but was otherwise very clean. Unfortunately no blanking plate for the Expansion Module at the front (back?), but later a bit of cardboard cut nicely fixed up any airflow issues there

I powered it up, and was greased with a Jumbo Jet, although it takes a while to boot, and during the initial boot, just the Power Supply Fans run. Once its fully operational, the fans in the Fan Trays seem to go full speed, then dial back to a lower RPM. But even in a nice cool 22 Degree living room, the fans were noisy as hell.(great way to wake up the kids from their afternoon nap!)


That measurement is taken about 30cm from the fan trays.I don't know how accurate that app is, but you get the idea. Its not a "desktop switch" by any stretch!

Luckily, the final location of the switch, I wont have to worry about noise to much, but I decided to take apart one of the Fan Trays to see what fans it uses, and later I can research other fan options (but more on that later).


For the fans, as you can see in the photo its 6 fans in 3 rows. The way the sticker is applied, and if you can see in the first photo, the back to back fans appear to be 1 unit, although you can separate them. each fan is a 4 wire fan, and connects to the plugin board on the fan tray. I also wanted to see if the fans could be reversed, as they way it would end up in store room, it would be blowing hot air on the same side as the air intakes on my R510. (more on that later).

Power Usage wise, with a single Power Supply, and only a few SFP's inserted, but nothing connected, was a bit of a disappointment, but not totally unexpected:

It draws about the same amount as my fully populated Dell R510 with 12 Spinners and Dual 5620 CPU's.
Each Power Supply is rated for 550W, and there are N5K's out there with 750W power supplies as well. It didnt consume much more with both power supplies connected as well.


So, I did some basic testing on the living room floor, and trying to remember the various Cisco IOS commands, got a very basic config up and running so I could test the SFP compatibility and the magic "service unsupported-transceiver" command

So the SFP's that I had brought for the N5K were as pictured:


From top to bottom:
1 Gig LC SFP
1 Gig Ethernet SFP
10 Gig LC SFP+

The 1 Gig LC SFP was on the supported list, so I was ok there. The 2nd one was advertised as a Cisco brand, but obviously not. (Nerves start setting in) and the last one is obviously not Cisco, but Finisar manufacture 10Gig SFP's for Cisco (Cisco OEM them!) so I was quietly confident there.

Upon inspecting the interfaces with the 1Gig Ethernet and 10Gig SFP's all showed as "down - unsupported SFP" in nexus. Issue the "service unsupported-transceiver" and bingo, they all came up (but link down obviously as nothing was connected to them). Using a laptop on the 1Gig Ethernet SFP's showed that they came up and worked as expected. Bonus! I think I saved hundreds here over using genuine Cisco 10Gig SFP's!

Next up was to install the 10Gig cards in my servers. Unfortunately only 1 card arrived (the others are stuck at Customs for some reason!). I put the first Intel 520 Card with those same Finsair SFP+s into my R510 that runs Centos 7. Upon booting, I was greeted with another "unsupported sfp" message, but after making a quick config change and reloading the kernel module, it was also happy.

Time to move it all into its final location. Thats the next Post

 

[JustinH]

Sad that I live a quick walk to Google, Symantec, LinkedIn, Intuit and etc here in Mountain View, CA and I get excited @ 100mbps speeds.

May I make 2 requests?
1. I am really interested in your MikroTik results and what you think about it.
2. Pictures!!!!

hehe. Trust me, 2 years ago, the best I could hope for was 10Mb ADSL. Then the bandwidth wars started and Singapore wired up the entire country for Fiber.

Whats really interesting - One of the big ISP's here has just announced a 10Gig trial starting later this year. I'd love to know what router they are using thats consumer friendly, and support 10Gig throughput!

1) yeah - I'm also interested. I've been very happy with my edgerouter, but my ISP have said the Microtik is the only affordable router that can get close to 2Gig throughput. I should get that router later this week, and will start posting updates then. (but I am thinking I might already have to upgrade the Microtik to a high model, as it will also now have to route my 10Gig VLANs etc)
2) 1st set of Pictures up. 2nd set very shortly

 

[legopc]

So ehm whats the average price of a house there?

 

[JustinH]

So here in Singapore, every apartment/house is mandated to have a bomb shelter included. (hangover from WWII). Of course, everybody uses it as storage space (a 1x2m room, what else are you going to do with it). Of course, for me, it makes a perfect "server room" (but I have to fight with the Wife and Maid on that aspect - So Server cum, storage room) as can be seen in the photos below!

One of the challenges is that its at the back of our apartment, which is open to the rear of our apartment, thus, no air-conditioning. If you know Singapore, we have 2 seasons, "Hot and Rains", and "Just Hot", and temperature range is usually 32-34 during the day, and will be lucky if it drops to 30 at night.

With the existing gear in the room, it usually averages around 36-37 degrees. I pulled two fans from a old Sun V400 (I think) and did a bit of handy work to push air into the room when the door was closed:


These are surprisingly quiet but push a fair bit of air. They also draw about 0.8Amps at 12Volts though. Without them, then the room can rise upto 39 degrees though. Leaving the door open isn't really a option as it blocks a passageway, and also the noise of the R510 starts getting annoying (this room is next to a wet kitchen/laundry area).

As you can see, its cramped inside:


Ladders, chairs, toilet paper and network/server gear! Ironically, the money I've spent on this upgrade was intended for a 18 or 22U rack to go at the back. I guess thats on hold now!

Directly in the center is the SG200 Switch, a APC PDU, EdgeRouter, and the ISP GPON Modem. Right middle is the R510 and PE850 and above it is the new home of the N5K:



I originally had the N5K sitting on top of the R510, but after a day sitting there, I started getting some really high temps in on the R510 and in the room in general.

Needless to say, the N5K runs HOT. Without that cardboard covering the expansion slot, I was getting temp warnings on Outlet 1 and 2 that were in the 50-52 Degree range. I'm now thinking that switching fans might not be the best option given how hot this guy runs. And right now, its not really passing any traffic. I don't know if my meager setup at home will stress it (its rated to push something like 500GBytes/Sec) but I might put that plan on hold right now.

nexus5010# show environment temperature
Temperature
-----------------------------------------------------------------
Module Sensor MajorThresh MinorThres CurTemp Status
(Celsius) (Celsius) (Celsius)
-----------------------------------------------------------------
1 Outlet-1 60 50 47 ok
1 Outlet-2 60 50 47 ok
1 Intake-1 60 50 37 ok
1 Intake-2 60 50 37 ok
1 Intake-3 50 40 37 ok
1 Intake-4 50 40 37 ok
1 PS-1 60 50 38 ok
1 PS-2 60 50 43 ok​

So, its almost pushing up the temp by 10 degrees. My Poor R510 was ramping up her fans to cope as well, and the room temp started rising as well. These high temps were even with the door open while I was walking in/out:


This graph is over 7 days and is the inlet temp on the R510. You can see on the afternoon of the 20th, the temp start fluctuating. That was as I was powering up and down over the next two days getting things setup on the N5K. This was NOT good for the life of my server (well, not as good as a average temp of 35!) Hence, moving the N5K to the shelf above it helped reduce the temp to 35-36 in the room.

I also have done a test tonight. The Fans above the door, were originally pushing air into the room (as the outside temp was always cooler than inside the room). I spoke with a friend and he said its probably more effective to pull air out as the at the height of those fans, the air temp would be warmer, so this evening I reversed the fans, and I'll monitor over the next few days. I kinda expected a temp increase (considering its drawing almost 200W but not that much!

Overall, with everything but my PE850 running, I'm now pushing 2.4Amps. (from around 1.2Amp previously):


I wonder how much of the monthly savings from my ISP's promotion is now going to go towards the Electricity Bill (especially when the 2216 gets fired up!)

The N5K linked up with Bonded connections to the R510 with no problems either. I'm very relieved my gamble on "unsupported SFP+'s" worked out!

Next I tried to create a LACP bond to the SG200 switch via the 1G SFP's. Here i'm running into challenges. As individual links, it works fine, as trunked links, it works fine, as a LACP bond without trunking it works fine, but with a LACP bond with all the VLAN's on the SG200 set to trunk to the N5K, I start getting link flaps and either one link will eventually die (with a linkflap error on the N5K) or both will die. As soon as I disable Trunking though, the LACP link stays up and reports 2Gig bandwidth.

Unfortunately the N5K doco is pretty bare. It basically just documents the various commands and little else. I upgraded the firmware on the SG200 (as it was outdated by about 2 years) and still had issues. I then find on a random FTP site a much more recent version (5.2) of NXOS for the N5K, which I installed last night, but havn't attempted configuring the bond yet.

But apart from that bond to the switch (which I can probably live without - as after all it will just end up as a edge switch for the rest of the house - which is not bandwidth intensive) its working well. I'm still waiting for the other 10Gig cards (and of course the Mikrotik from my ISP) so I can fully test that at least my servers get their 20Gig bandwidth and my pipe can be used to its full potential.

Overall, I think as long as you can live with the noise and power of the N5K, then it might be a viable option for 10Gig switches. With it supporting SFP+ from finsair makes it a bit more affordable. Additionally, you can pick up N2K Fabric Expanders. These are basically 1Gig "expansion" modules that hook up to the N5K via 10Gig links (1-4 links). The N5K still does all the switching, so the N2K isn't standalone, but they offer you 24-48 ports of 1Gig Ethernet. I've seen the N2K go for around $200-300 on ebay.

License wise - Everything you need to do basic switching is included in the base OS. The N5K can also do FCoE and the expansion port can take FC SFP's. But for that, you need a license, and they run for 1-2K on ebay. (if you trust buying a paper license with no return policy on ebay!)

There is also a "Enterprise Lan License" that enables the N5K todo some advanced L3 routing (BGP, OSPF etc) I think the N5K base license allows you to do basic static lan routing, which I'm really hoping for, and will do some testing when the rest of the Netcards arrive. (the doco on the Net is confusing in this aspect, a lot of it refers to the Nexus 5500 series, but not the 5010 or 5020. I can add static routers on my N5K now, and it accepts them, but I can't test that out yet.

If I can route between vlans on the N5K it means I don't have to get the Mikrotik to do that (and although it has a SFP+ port, its highest throughput on the performance specs is only around 7Gig, and thats without any firewalling etc). I know I could probably run pfSense or something todo the intervlan routing (if required) but I don't like making my network dependant upon my virtual servers being up. Right now, I can turn off everything but the edgerouter and SG200 switch, and still have internet access from the rest of the house.

 

[JustinH]

So ehm whats the average price of a house there?

Government Subsidized Apartments range from about 400K to 800K. (apartments - Not houses)
Private Apartments (with facilities like pools etc) are generally over a million

Houses - as in landed homes - you would be lucky to get for under 4 million.

Singapore is tiny. something like 60Km wide, and 30km high. Land is really scarce around here.

 

[legopc]

How are you planning to make use of that 10gig connection web servers, torrenting or?

 

[T_Minus]

I'm jealous! I wish we could get 20mbit, 50mbit or even 100 but 2gig....

 

[JustinH]

You mean my ISP connection? That's 2G Down/1 G up (the 10G I mentioned is a trial one of the ISP's is expected to start soon).

Honestly I don't know. Upto a couple of months ago I was working from home and my job involved building fairly large OS images for digital signage. I've since moved to a proper office and moved a few of the servers and DAS there as well.

After the move, I've barely stressed the 1G connection. I do host a few websites but it's all related to open source and fairly low traffic.

But seeing as this promotion means cheaper subscriptions for 2G over what I was paying for 1G the geek in me took over

I do run VPN's to both my office and some Virtual servers at rackspace and azure and I might setup some more storage at home and use it as offsite backups for them. Otherwise I doubt in day to day usage I'll be pushing 2G.

 

[T_Minus]

Yep, 1gig is insane let alone 2 or 10

I think even in the city right now Comcast Business is ~150mbit max. Which can come to residential for very affordable, but that's still not 1Gig !!!

 

[JustinH]

I'm jealous! I wish we could get 20mbit, 50mbit or even 100 but 2gig....

Actually I hope you guys can catch up soon. The problem with us getting huge bandwidth over here is lack of hardware choices. (Your typical linksys/asus/netgear residential routers fall over at about 500MB.) then it's either Ubiquity or mikrotik , but hardly consumer friendly ( no wifi built in etc)

I can usually tell when downloading from a server in the US who might be colo with 100Mb connection, cause the download will peak around that speed, and any attempts to open new pages on the site while the download is ongoing are leggy etc!

I had a friend get banned from him companies FTP server cause he wanted to mirror it locally so he didn't always have to "wait" for the downloads when he needed needed to grab updated software. Unfortunately his mirroring satuated the link!

 

[JustinH]

Yep, 1gig is insane let alone 2 or 10

I think even in the city right now Comcast Business is ~150mbit max. Which can come to residential for very affordable, but that's still not 1Gig !!!

I though Google fiber was offering some decent speeds where it's deployed?

 

[T_Minus]

Not widely available

ATT just pulled Fiber into my small town for the 4G Cell tower, and are offering it commercially but we're talking $750/mo for 20mbit, which, in the end I may have to go for if I'm within distance even!!

 

[JustinH]

That's crazy!

It's doesn't add up if you compare to bandwidth charges that AWS/Rackspace/Azure offer though!

750/mth on those virtual server places gets you a beast of a box and terrabytes of outbound traffic at really decent speed.

I have a 12 core virtual server 128G ram with rackspace, about 600G of storage and push about 3TB traffic a month for less than 500!

 

[T_Minus]

Yeah, and it's even more crazy what $750-1200 a month gets you (42U Full Rack + 20A Power + 1Gig) at some CoLo locations.

I already pay out the ass for a T1 at my house, we have 0 other internet options but $$ satellite with less bandwidth and more latency. In reality that 20mbit isn't even 2x the price of a T1, and if I wanted a 2nd T1 it's ~$625/mo for 3mbit/3mbit so yeah, things are pricey when you're not in the city near the big pipes

I run multiple businesses so I need it at my home too.

 

[JustinH]

I guess the upside is 2nd hand/off lease market there in the US. It's non-existent here in SG so I always get jealous when looking at the great deals section knowing it would cost at least another $40 min for shipping to SG and then running the risk of DOA etc!

All up this purchase cost $400 in freight forwarding/customs and insurance for me. But given most of the N5K's I've seen on ebay run at $1200 I keep telling myself I've "saved money!" (Lol!)

 

[JustinH]

1. I am really interested in your MikroTik results and what you think about it.

Took a while, sorry, but here is what I think of the MikroTik after using it for a couple of months....

Its a CCR1009-8G-1S-1S+-PC model.

Overall, I'm satisfied with it, but not blown away by it. Its got a few quirks and issues that drive me up the wall.

Firstly, the 1G SFP port - I can't get it to link despite using the official Mikrotik SFP (and I also tried other 3rd party SFPs). Thought it was hardware, and had both the router and SFP replaced. Still no luck. No combination of configurations would bring it alive, but the 10G SFP+ works fine though.

2nd - It does get a bit warm. This is the Passively cooled version, with a large heatsink out the back. It reports a internal Temp of 50 Degress C, and a CPU temp of 67 Degress C. And its mostly Idle.

Performance wise - it doesn't break a sweat on my 2Gig Connection. It has a CPU graph option - but it hasn't nudged off 0% since I've used it. I've even got it routing via 2 VLAN's on the 10G SFP+ connection and it easily pushes 5Gig of traffic (only a single SFP+ port, so its "in and out" traffic).

Power Consumption wise - It reports about 10W at 12V. I'm sure much lower at the actual outlet though.

Interface wise - Meh... The web interface feels a bit dated (after coming from a EdgeOS router). And some things are not that intuitive (eg, you expect the "interfaces" tab to contain IP address options, but it doesn't. IP addresses are configured in separate screen), but just about everything I needed to configure on this box (with some pretty complex Mangle rules etc) I was able to do via the Web interface, unlike a few others where the web interface doesn't lacks features that the CLI might have (eg, EdgeRouters)

The CLI Interface is also a bit Meh... they have tried to copy the Cisco CLI, but with their own touches. Its a mix of old style DOS directory categories and Cisco Autocomplete. I'm a old school CLI guy, but their implementation just bugs the hell out of me.

Its also got a Windows GUI. I've only used it once, and it is almost identical to the Web Interface.

The Manual is actually a Wiki - and it appears to lag behind a on the actual features. Its also very terse. There is a ton of "howto's" on the Wiki, but again, its just more like "click this, do this, do that" rather than explaining what its trying to do. The forum seems next to useless for support, with most posts being a "me too" type response, and very little activity by the Mikrotik guys (at least to my problems with the SFP port)

For my setup, my 2Gig ISP connection is actually 2x1G connections, with 2 Public IP addresses. I also have a additional /29 subnet.

I thought I could do WAN load balancing, but the /29 subnet threw some real spanners in the works. The traffic from the /29 has to also go via a single WAN connection, so I was trying to configure policy based routing with WAN Load balancing for my "normal" network. Its debugging options just suck here, (ie, Log files from firewall/NAT/Mangle rules etc) in that you would have to page through each log file from the start to see the most recent entries, and if you tried to view through the GUI, well, the browser would just start hanging after it had downloaded thousands of lines of Log Files.....

It does have a touch screen color LCD panel on the top of the box, but comes with ears to mount in a rack. If you mounted it, then you wouldn't be see the LCD screen. The LCD screen can show per interface stats, overall stats (bandwidth and packets per second info).

From the hardware perspective, its got 8 Ethernet, 1 SFP, 1 SFP+, RS232 (for console) and a MiniUSB (for external HDD, modems, etc)

4 of the Ethernet connections are actually internally connected to a switch, and it can do some basic filtering etc on the switch, but if the CPU needs to get involved, there is only a single 1G connection back to the CPU. So if you have anything more than 1Gig of traffic on those first 4 ports, its likely to be a bottleneck.
the remaining 4 ethernet are standalone and connect direct to the CPU.

It does Hardware Offloading as well, but the minute you start doing things like NAT, or VLANs etc, it disables the hardware offload and goes software only. Its a shame, as the EdgeRouter could do hardware offload even with firewall/nat rules. But considering its not even breaking a 0% CPU utilization, it doesn't seem to matter at the moment.

I do think its got some bufferbloat going on here compared to the EdgeRouter. They haven't adopted much in the way of dealing with BufferBloat, and you can't even modify the hardware queue sizes etc to help alleviate it.

It does support installing 3rd party packages, but I can't find many at all. Its based on their own Version of Linux, and a not so popular hardware platform called Tile. At least with the EdgeRouters - You could install 3rd party packages from the official Debian repositories!

Feature wise, it has what you would expect from a prosumer router. all the Routing Protocol Support, DHCP Servers and Relays, even Squid Proxy Server (but its internal storage is only 128Mb, so you would need a external HDD via a USB2.0 connection... which runs at 480Mb, so that seems kinda dumb for a router that can push 10G traffic!). It took me a while to figure out how to get IPv6 (install a separate package from their download pages)

Overall, I like the Hardware, but the OS itself needs some polish. It would be awesome if something like the EdgeOS or pfSense could be ported to it. (after all, its OS is called RouterOS - based on Linux).

 

[Patrick]

I think the CCR1036 was 34-36w last time I checked.