Questions about colocating

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fragar

Member
Feb 4, 2019
32
0
6
I am looking for a colocation facility and have a few questions.

Background and requirements:

1. This will be my first time colocating.
2. I need a rack of around 10 kW, which I will fill with 4u 1 kW inference servers (hardware discussion here: https://forums.servethehome.com/index.php?threads/choosing-a-server-chassis-for-gpu-workload.28360/).
3. Price is very important.
4. I have pinged a few data centers about prices but have not tried to negotiate.
5. I live in Budapest but also spend significant time in Prague and Warsaw.
6. I don't expect to need much physical access to my servers. My plan is to set them up, have them run for 5 years, and then replace them, while adding new servers maybe once per year.

Pricing observations:

In the discussion below I have converted all prices to Euros, excluded VAT, excluded internet costs, excluded any "loyalty" discounts, and assumed 24/7 operation.

From among the minority of data centers which offer 10 kW racks, the pricing schemes vary wildly but the total price seems to always work out to about the same amount, ie. ~€160 per kW per month. For example:

atw.hu (Budapest): €196.90/month for a 10kW rack and then €0.192 per kWh used, which works out to €157.80/kW/month.
servergarden.hu (Budapest): €478.59/month for a 10kW rack and then €0.163 per kWh used, which works out to €165.50/kW/month.
ttc-teleport.cz (Prague): €1070.11/month for a 10kW rack and then €0.096 per kWh used, which works out to €176.09/kW/month.
verneglobal.com (Iceland): €1100.90/month for a 10kW rack and then €0.046 per kWh used, which works out to €142.49/kW/month.

Verne Global also offers a "direct power" service for a much lower price. This service basically excludes UPS's and a few other amenities, which sounds ok. I would not mind a few hours of downtime every quarter or whatever. I'm not sure they'd be ok with me writing the price here as it took several emails for them to reveal it, but the main problem is that to get that price you need a minimum of 90 kW and that is way beyond my current needs.

Questions:

1. How negotiable are these prices and is there some trick for approaching that (other than telling them, truthfully, that I am very price sensitive)?
2. Are there any other ways to reduce colocation costs?
3. How important is it to be close to one's data center? Is it realistic to only show up at the data center when installing new servers?
4. How reasonable is it to run inference servers on a "direct power" service?
5. Does it make any sense to hit the Verne Global 90 kW "direct power" minimum by sharing a contract with someone else (ie. I take 10-20 kW and find someone who needs 70-80 kW)? Is that common?
 
Last edited:

Blinky 42

Active Member
Aug 6, 2015
615
232
43
48
PA, USA
At the $job - we have servers all over the planet that we have never seen in person. Configured by our vendors and direct shipped to the colos and installed by colo staff. It is pretty common and you can negotiate that rack & stack work into your quote.
If your hardware is stable and you don't need to make adjustments, seeing the site to setup and then not going back for years is very possible.
If you can commit to longer terms you can get more wiggle room in pricing, but do read the fine print and see if how power prices are adjusted based on market rates. And see if there are any other annual increases built in.
How quickly will you fill the rack and use 10kw? Is it cheaper to look at colos that are geared for 6kW cabinets and expand to 2+ racks over time?
If you can tolerate the downtime running w/o UPS isn't unreasonable.You could always put management / network / storage servers on a UPS in a your rack or elsewhere in the facility and then the power hungry servers on their own.

Sharing a contract with another business is probably not going to be encouraged / allowed by the colo but you may be able to basically sub-let a rack from a larger player. Depending on your business goals, consider the risks and liability the arrangements will put you in and get all the terms and expectations in writing regardless.
 
  • Like
Reactions: fragar

fragar

Member
Feb 4, 2019
32
0
6
Thanks. That all makes sense.

Verne Global lets you ship your equipment directly to them from the vendor and will hold it for you for free in a secure bay. Traveling there once every year or two would be no big deal.

6 kW cabinets make sense when they're available. TTC Teleport offers a 6 kW rack for €664.02 per month compared to €1070.11 per month for their 10 kW rack. This works out to €179.78/kW/month in the 6 kW rack vs €176.09/kW/month in the 10 kW rack. That's a no-brainer for the 6 kW racks regardless of how quickly I ramp up past 6 kW.

Verne Global and servergarden.hu only offer 10 kW racks though, and those are the better deals (per kW).

I'd love to get those Verne Global "direct power" prices but am stumped about how to handle the 90 kW minimum. Indeed, splitting the 90 kW could be risky. Another idea is to run bitcoin mining rigs just to fill up that 90 kW but that might turn into a massive rabbit hole, and it's risky as well.

EDIT: one more question - what are the drawbacks to the longer (multi-year) commitments? It's not like I am going to be switching data centers after a year.
 
Last edited:

fragar

Member
Feb 4, 2019
32
0
6
One more question: are there any other cheap "direct power" services out there which have a lower minimum kW than the 90 kW minimum of Verne Global?

Is it possible to rent a single 10 kW rack somewhere without UPS for significantly less than €160/kW/month?
 

MiniKnight

Well-Known Member
Mar 30, 2012
3,072
973
113
NYC
If you've got 1 rack, make it close to you. You should also see if you can negotiate to add racks at the same price. What sucks is when you've negotiated a space, but adding more costs 50% more because the know it costs too much to move the rest of your setup.
 

fragar

Member
Feb 4, 2019
32
0
6
Does this mean that it's possible to negotiate 50% off?

It seems to me that every data center is optimized for a certain density. TTC Teleport is optimized for 6 kw racks (but offers 10 kW racks at the same price per kW for people who really insist on that). Verne Global and servergarden.hu are optimized for 10 kW racks, they don't even offer anything else.
 

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,059
1,478
113
I doubt you'd be able to negotiate much, though I'm only familiar with the local market and have no experience with Europe. There was a bit of wiggle room when I was shopping around, but certainly not 50%. I've not really seen too many places that offer high density. 30A (208/240) redundant service is pretty common here and above that is rare, though it does exist. At my previous employer, we actually had 50kW+ per 50U rack, which is the highest I've ever seen. The quick disconnects and cables on 60A 415V 3 phase drops (two per rack) are hilariously big.
 
  • Like
Reactions: fragar

fragar

Member
Feb 4, 2019
32
0
6
The data center I'm talking to now (servergarden) specifies their 10 kW racks as "redundant power (2x)3x16A", which is more than 20 kW when both redundant halves are working.

I wonder if it's possible to run the servers in that rack at the full 20 kW, and when one of the redundant halves goes down, throttle down to 10 kW for the duration of the "half-outage". Would the data center allow that? Can servers be configured to handle that ok?

(Alternatively, could 6 kW racks be stretched to 12 kW in this way?)
 

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,059
1,478
113
It depends if the datacenter will allow it. There's only so much cooling capacity that most can provide. Otherwise, there's no reason you couldn't do that. Just expect a hard power loss on half your servers should there be an outage (there's no alternative unless you have a UPS).
 

fragar

Member
Feb 4, 2019
32
0
6
Makes sense, I'll ask. The cooling capacity is probably a lot more flexible than the power.

Would it make sense to hook up the two redundant power supplies on each server to the two different redundant power sources from the data center, so that when one of those two data center power sources goes down, each server would get half of its expected power? What happens when a server suddenly loses half of its power like that?

This could cut colocation costs by up to 40% so I bet it's either common or there is a catch.
 
Last edited:

fragar

Member
Feb 4, 2019
32
0
6
Servergarden.hu got back to me about pulling more than 10 kW from a 10 kW rack with a non-committal answer: each rack is guaranteed to get 10 kW, and "in practice" the maximum "might be 10 kW". This data center only offers 10 kW racks and claims to be the densest in Budapest. Each rack gets redundant (2x)3x16A.

Does anybody here have any experience with using more kW from a rack than the advertised number?
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
That all seems to alien to me those power, is that 16a 3-phase ??
Normally would get around 14kw/21kw racks with 4 x 32amp single phase or 2 x 32amp 3-phase.

they say 10kw is the most dense in Budapest, I am surprised but I guess I am dealing in way more dense environments. 20+kw free air cooling and more for better contained or water cooled solutions.

back on topic the rates your seeing are a bit above what I would be used to seeing for a single rack but not major so, I wound guess it will be hard to negotiate much better but try. Maybe just tell them it’s a sealed bid, best price wins.
 
  • Like
Reactions: fragar

fragar

Member
Feb 4, 2019
32
0
6
I am not sure about the power specs but 3*16A * 220V would be 10.56 kW.

Budapest seems to have mostly 3 kW racks, a few facilities which go to 6 kW/rack, and two facilities which go to 10 kW/rack, at least from what is publicly available. Maybe I'm missing something.

Here is the page for Servergarden: Szerver hosting és szerver elhelyezés - garantált biztonság. They've got 122 10 kW racks. Good online reviews. Relevant quotes (in English):

  • Knürr 46U 600x1200 rack cabinet
  • Maximum 10kW load capacity
  • Redundant (2x) 3x16A power supply
  • Separately metered and billed power consumption
  • Electricity fee 58.- HUF / kWh
  • Without internet connection
and

  • Hungary's highest power density server room (300m2, 1MW useful power, 3.33 kW / m2)
  • Redundant (1 + 1) uninterruptible power supply (2x 500kW)
  • Two-way, independent route power supply (2x 2.5MVA)
  • serving up to 10kW / rack power
  • 5 MVA Diesel Aggregator with 24 hours of diesel supply and continuous supply
  • Assumed availability: 99.999%
  • Actual availability in the last seven years: 100%
 

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,059
1,478
113
Makes sense, I'll ask. The cooling capacity is probably a lot more flexible than the power.

Would it make sense to hook up the two redundant power supplies on each server to the two different redundant power sources from the data center, so that when one of those two data center power sources goes down, each server would get half of its expected power? What happens when a server suddenly loses half of its power like that?

This could cut colocation costs by up to 40% so I bet it's either common or there is a catch.
If you need redundancy, you need to ensure that the each power drop can handle the full load. That means you will be running each drop at half capacity max.

If you lose one of the power drops, the servers will immediately draw the full load from the other one. They don't just run at half power. If you're over capacity, you can expect to trip a breaker and lose everything in your rack.

One other thing to note is that at least locally, one is expected to only have 80% continuous utilization on their drops (so, if you have 30A, only use 24A). This is to ensure that any temporary spikes do not exceed 100%.

You will not be able to run over capacity. This isn't like overclocking a CPU when you can just go over a bit.
 
  • Like
Reactions: fragar

fragar

Member
Feb 4, 2019
32
0
6
Ok, so I'd have to hook up half the servers in each rack to one redundant power source and the other half to the other source, effectively giving up redundancy.

My power consumption is extremely steady. I just watched my GPU power consumption via nvidia-smi for a few minutes and it fluctuated between 150 Watts and 170 Watts per GPU (at a nominal setting of 160 Watts), so +6.25% above nominal in the worst case. The peak for all of the GPUs in a full rack wouldn't be more than maybe 2% above nominal. This 80% utilization rule is for spikier workloads, right, or am I missing something?

Also, if a data center advertises (2x)3x16A, would that have some buffer built in, or would it trip at exactly 48.0A (during a drop)?
 
Last edited:

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,059
1,478
113
You would want to order dual drops instead of requesting redundant ones. The assumption with redundant is that you'd only be using half the total capacity and therefore you would be billed higher for dual drops since your electricity usage would be double. They'll quickly catch on if you have redundant drops that are both running at full load.

I imagine you're correct about most workloads having more fluctuations, but I would check with the datacenter regarding their policies on usage. They may advertise 16A per phase, when in reality it's rated for 20A (80% thing like I mentioned). The PDU breakers should trip if you exceed the total capacity of any phase, not the combined total, so potentially as low as 17A. Be sure to split the load evenly between all 3 phases.
 
  • Like
Reactions: fragar

fragar

Member
Feb 4, 2019
32
0
6
Thanks. Do you mean that if all of my servers are the same, then with 3 phases it would be best to have the number of servers in a rack be a multiple of 3? Or is there some other way to balance the load between the 3 phases?

And yes, if I want to sacrifice redundancy to double the power draw per rack, I should probably clear that with the data center first. This idea might be too crazy anyway, especially for my first rack.
 

Blinky 42

Active Member
Aug 6, 2015
615
232
43
48
PA, USA
Many of the colos won't let you use the full power on a circuit either due to the electrical codes in place in the local area and how they built out the circuit - they will size it so that it is legal but for continuous use you can only draw 80% of the power you would expect. Be sure to check the quotes accordingly.
For example, here in the US when I get a 30A 208V power feed at any of the major providers I can't draw the full 30A 24x7, we can draw 5kW on it instead of 6.24kW constantly. The % varies a bit by location based on how they designed and built out the power distribution, UPS and the cooling of the room your are in, and what is standard varies more by country. Facilities that are newer and built out for 15-20kW per cabinet size the power distribution and cooling so it isn't an issue, but the older facilities wouldn't spend the extra $ to overbuild the physical plant because customers who needed that much power in a rack were doing it themselves at the time in their own datacenters or leasing entire floors where they would retrofit to their needs.

Also as others mentioned above, you will need to get dual feeds into your cabinet to pull the full load on each - the standard redundant feed setup is designed to support 100% of the contracted load on either side when one is down but have it close to 50/50 split power usage under normal circumstances because they need to balance the power draw across each phase for efficiency.

The more together sites we are in will regularly audit the power draw in cabinets and charge you overages, good sites will also go around and audit the connectors and power distribution to make sure none are runnign hot with and IR camera and replace (or make you replace) equipment that is running hot and showing signs of failure to avoid fires.
 
  • Like
Reactions: fragar

fragar

Member
Feb 4, 2019
32
0
6
Thanks. Sounds like I'll need to verify this at every data center. Servergarden in their first email to me wrote "- we can serve performance exceeding 10kW/ rack" and later clarified that they "guaranteed" 10 kW. Now I understand why they worded it that way.

Good point about the power efficiency of running each feed at full load. That's another reason to use the redundant feeds as they are intended.