Tossing around an idea for STH 2016 - Shared lab facility?

[Patrick]

With the new Sunnyvale colocation, we now have the ability to keep a lot of hardware up and running. The pipe is fairly big (200mbps commit on a 1Gb connection right now) but very low utilization. We also have the ability to allow remote testing, for example, William, myself and a few others can VPN in and have access to all the hardware in the lab. The idea thus far has been to provide access for doing our hardware reviews. Someone challenged me this weekend on whether this should continue being the case.

One idea I had was to allow people (maybe forum members) onto the infrastructure to test hardware/ software for short periods of time.

I am not 100% sure what this would look like (very early on). Here is what I was thinking:

• Rotate an interesting set of hardware through the lab.
• Allow people to buy "tokens". I am not exactly sure how much but maybe $10 or $25/ machine and it gives you something like a day or so with a box. There is a very non-negligible cost to setting up/ operating this infrastructure so would need to have a way to recoup those costs.
• We can give people local Linux Desktops and just spin them down when not in use. That would allow people to remote into the VM and have a local resource, making it easier to use sitting on a beach with a lower bandwidth connection. It would also allow people to restart their VM on their next session and have all of their tools/ configurations available.
• Can give access to the power metering infrastructure so you can see power consumption on your workload/ hardware on a per-outlet basis.
• Likely would give Operator IPMI access. So basically full IPMI access minus the ability to change the Administrator credentials and reset the BMC.
• All 10Gb/ 40Gb internal network. Likely could have some sort of shared storage.
• Stretch goal (a bit) is to be able to image machines after they are done being used so that they can be re-deployed on new hardware, minimizing the need to re-setup everything each time they deploy a machine.

The goal is basically giving people some bare metal hardware new/ used to test drive. I could probably talk to some of the vendors that send hardware for STH to review and see if we can keep the hardware a bit longer for people to run their own tests on.

Certainly scaling out in this manner I would likely need the help of folks on here to get this setup. I did want to at least float the idea since it could be an awesome capability for STH: read the review then get on the actual hardware & a previous generation of hardware to run your own tests. The benefit is that the environment is vendor neutral and unlike a partner, we are not selling hardware (and being pushed by vendor incentives.)

I think there is a long list of reasons people would NOT want to use this. On the other hand, if folks can use it as a low cost option to test things for work (e.g. spend $100 and a day setting up a proof of concept and testing it) and home labs, maybe it would be worthwhile. Also, when life changes and it is no longer tenable to have a 70dba 2kW rack in the basement, one could sell the hardware and use this as an on-demand bare metal lab.

I would be extremely interested in hearing thoughts/ opinions on this one. I am also interested in hearing thoughts on the best way to set this up. We are using RADIUS authentication, OpenVPN and a lot of manual configs right now.

 

[Blinky 42]

Interesting thought. Having a day or so to test new things out, even remote could be helpful in several cases.

I have done the same in vendor-specific situations in the past with IBM & HP, but being able to test out a new CPU or CPU/Mobo combo for a little while from other vendors like Super Micro without having to roll the dice and wait for general supply availability and purchasing demo units would have been nice.
For example a few hours to test out code on a Xeon-D is enough to see if it is worth dropping the $ on 1 or more for more deeper experimentation or passing and waiting for the next thing. The same ability to run comparisons on recent AMD & NVidia GPUs would have been quite helpful in past projects. Slogging through sites to get benchmark numbers can rule out some platforms, but the tricky things like picking between the SKU with the extra cores or the SKU with faster cores is best done running the actual code or benchmarks developed in-house on that hardware.
Staying vendor-neutral or at least multi-vendor could also be quite compelling in other situations. Heck even quick A/B comparisons that are a pain to do on your own like showing the practical differences on current IPMI / Remote Access implementations from Dell, HP, Lenovo/IBM and Super Micro to help sway those who are paying the bills would be super handy.

The tricks will be to avoid things that one can check out by spinning up an AWS or other cloud provider instance, and potentially letting folks have dedicated access to the server for a period of time to really test whatever they needed.

You could basically lease access to gateway VM's that people can stage what they need into the colo, and then when their turn comes up configure the network to expose the bare hardware to the staging VMs to run whatever testing they wanted You could do the same with physical servers too - send in a 1U server that you colo and grant access to your new fun hardware pool as needed.

 

[MiniKnight]

I like the idea and what @Blinky 42 said. If it was like $50 to go try Xeon D it's totally worth it to us. Buying a $1200 system is only part of the cost to getting our own hardware. We'd have to install it in our racks too. The labor there is not insignificant. For us, and I'm sure other orgs, $50 I can expense with a receipt. $1200 I need to go get approvals.

I think Op access is fine on IPMI.

 

[legopc]

Thrust is a big thing with something like this, I can think of a couple of ways to abuse that internet connection and hardware. A person could also willingly brick the hardware by messing with the BMC or BIOS. But it is a great way to check out new stuff without buying it and would love to see something like it.

 

[Jeggs101]

We'd use it just to not have to deal with tracking equipment. I wouldn't say we'd use it daily but once or twice a quarter.

 

[TechIsCool]

I agree with @legopc there is a lot that could be done to the hardware. Its having trust in the people that use it. I like the idea just not sure how to make it work fully. A self provision portal would make your lives much easier. Also some way to orchestrate stuff.

 

[MatrixMJK]

This sounds really interesting. I know I would use it, but how often would be the bigger question.

Might be a good use case for OpenStack or other 'Cloud' management system that can provision at the bare metal layer, like Ironic or Fuel with some extra work. The cloud management would give you metering and stats for usage and once setup, make it relatively easy to tear down one instance and bring up a new one. It could also help with the user accounts/token use, and should provide some protection from abuse. Of course, if someone gets admin access to the BMC it could still be bad.

 

[Diavuno]

I like the idea and what @Blinky 42 said. If it was like $50 to go try Xeon D it's totally worth it to us. Buying a $1200 system is only part of the cost to getting our own hardware. We'd have to install it in our racks too. The labor there is not insignificant. For us, and I'm sure other orgs, $50 I can expense with a receipt. $1200 I need to go get approvals.

I think Op access is fine on IPMI.

Agreed, for $50 or less I can write it off as a business expense... Cant say I like having to write off more.

The herd of NUCs in my basement is the result of not having a system to test before I buy.

 

[Charlie Wilbur]

Are Vendors OK with tests drives like this? Not having the latest firmware etc...could make or brake a sale? Being the Devil's Advocate here.

 

[Patrick]

Are Vendors OK with tests drives like this? Not having the latest firmware etc...could make or brake a sale? Being the Devil's Advocate here.

Somewhat depends right? The goal would be to use the latest firmware.

For products that the vendors send for this purpose, we would be getting the vendor's blessing and therefore they would have to be OK with it. For hardware that we purchase and make available, then vendors would not have a lot of say.

 

[wsuff]

It's quite interesting idea to consider that could work out well for the community as long as it could be managed in a way that isn't too burdensome to the parties involved. I look forward to see what STH and the community as a whole does in 2016. Thank you to all for the helpful information available here from everyone's experiences. Long time lurker but finally joining to contribute.