New home NAS and Hypervisor setup
- Thread starter TallGraham
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
just run raid-50 bro, two axles allow for two disk failures (one per raid-5).
Raid-50 is the fastest protected raid for disk restores which is why I use it for D2D storage. When you have to restore that 1TB VM, over 10/20/40gbe - It works best to restore with raid-50 or you end up taking all day.
10 disk is the maximum size for raid-5 still, or 20 disk raid-50. Always put your OS on something else.
For extra queue depth, you can slice up the disk. Put the slower tracks .. faster tracks. That's how 3par works they rock 1000's of tiny raid spanned so you get short stoke performance via (Adaptive Optimization).. It works.
It's good to see another Lefthand user!! 11.0 is awesome sauce! The VSA installer is so brain dead simple to install now (because it works!).
I was wondering what if you were to put some drives in raid-10 and some in raid-6 then make tier-0 the raid-10 and tier-1 the raid-6? In theory that would allow you to be flexible.
Lefthand/Storevirtual is not about speed, it is definitely about safety. Being able to stripe a network raid across 2 SITES is awesome for FT fault tolerance.
Raid-50 is the fastest protected raid for disk restores which is why I use it for D2D storage. When you have to restore that 1TB VM, over 10/20/40gbe - It works best to restore with raid-50 or you end up taking all day.
10 disk is the maximum size for raid-5 still, or 20 disk raid-50. Always put your OS on something else.
For extra queue depth, you can slice up the disk. Put the slower tracks .. faster tracks. That's how 3par works they rock 1000's of tiny raid spanned so you get short stoke performance via (Adaptive Optimization).. It works.
It's good to see another Lefthand user!! 11.0 is awesome sauce! The VSA installer is so brain dead simple to install now (because it works!).
I was wondering what if you were to put some drives in raid-10 and some in raid-6 then make tier-0 the raid-10 and tier-1 the raid-6? In theory that would allow you to be flexible.
Lefthand/Storevirtual is not about speed, it is definitely about safety. Being able to stripe a network raid across 2 SITES is awesome for FT fault tolerance.
Wow! So many replies. Thank you everyone 
I have 3 Adaptec RAID cards I will be using.
1. Adaptec 6405 with 4 x Crucial M4 128GB SSDs (RAID5) and I have a spare drive to hand. This will hold OS and bits (Windows 2012R2, AD, DHCP, DNS, WDS, MDT, Serviio)
2. Adaptec 5805 with 8 x 1TB Western Digital REDs (RAID6). This will be for all my iSCSI links for my VMs
3. Adaptec 5805 with 8 x 1TB Western Digital REDs (RAID6). This will be for all my other data.
I also have a 1TB Red as a spare too. So I think I should be OK for resilience.
Our old Lefthand SANs at work are quite old now, but I absolutely love the network raid. Our College has 2 x 10Gb dark fibre links between our two largest campuses. I am just going to start the process of moving the elements of the network raid across the sites. For some stupid reason they were both installed on separate floors of one campus!!!!! I have been reading about the new StoreVirtual software that allows basic servers to be used and network raided, plus it can install on Hyper V or ESXi servers. Not done much other than reading at present though
Now as for backing all my stuff at home up..... I'd love to play with the HP StoreVirtual stuff in some way. I doubt I'll get the chance though. So maybe get an off the shelf NAS box or something. Would be nice to get something Like Veeam for ESXi or the same for Hyper V as well. Can you get cheap versions of these for home use or educational use?
I do have another question for you guys about all your VMware ESXi clusters you set up. How do you do it with the standard free versions of ESXi you can download from their site? At work we have the vCenter licenses but they are super expensive.
I plan to try and run 2 x diskless/headless Virtual Host servers. Both will boot from iSCSI, so hopefully I can easily switch them between either Hyper V or ESXi. I really want to cluster them and play about with some high availability (HA) stuff with both these and the Microsoft server apps.
I have 3 Adaptec RAID cards I will be using.
1. Adaptec 6405 with 4 x Crucial M4 128GB SSDs (RAID5) and I have a spare drive to hand. This will hold OS and bits (Windows 2012R2, AD, DHCP, DNS, WDS, MDT, Serviio)
2. Adaptec 5805 with 8 x 1TB Western Digital REDs (RAID6). This will be for all my iSCSI links for my VMs
3. Adaptec 5805 with 8 x 1TB Western Digital REDs (RAID6). This will be for all my other data.
I also have a 1TB Red as a spare too. So I think I should be OK for resilience.
Our old Lefthand SANs at work are quite old now, but I absolutely love the network raid. Our College has 2 x 10Gb dark fibre links between our two largest campuses. I am just going to start the process of moving the elements of the network raid across the sites. For some stupid reason they were both installed on separate floors of one campus!!!!! I have been reading about the new StoreVirtual software that allows basic servers to be used and network raided, plus it can install on Hyper V or ESXi servers. Not done much other than reading at present though
Now as for backing all my stuff at home up..... I'd love to play with the HP StoreVirtual stuff in some way. I doubt I'll get the chance though. So maybe get an off the shelf NAS box or something. Would be nice to get something Like Veeam for ESXi or the same for Hyper V as well. Can you get cheap versions of these for home use or educational use?
I do have another question for you guys about all your VMware ESXi clusters you set up. How do you do it with the standard free versions of ESXi you can download from their site? At work we have the vCenter licenses but they are super expensive.
I plan to try and run 2 x diskless/headless Virtual Host servers. Both will boot from iSCSI, so hopefully I can easily switch them between either Hyper V or ESXi. I really want to cluster them and play about with some high availability (HA) stuff with both these and the Microsoft server apps.
The last bits I was waiting for turned up yesterday from Scan Computers.
Intel Xeon E3-1220LV2 CPU (17W)
Seasonic 520W Fanless power supply
Silverstone NT01-P Passive Heatsink
Happy Graham
Intel Xeon E3-1220LV2 CPU (17W)
Seasonic 520W Fanless power supply
Silverstone NT01-P Passive Heatsink
Happy Graham
So I started the build proper this afternoon. I love putting hardware together and never rush it. Here are some pictures so far. Starting with a few of the X-Case RM420 Pro Case.
First job was to swap the power supply plate over from a rackmount PSU plate to the standard ATX PSU plate. Picture shows new plate in place and the old one next to it.
The next few shots are of the case showing the Hot swap back planes, 3 x 120mm fans, and the access ports on each side for cabling. I hope these will help future people trying to make decisions on cases.
OK onto the motherboard fit. The first thing I noticed when I was adding the post-screws into the case was there was no room for one on the motherboard under where the expansion cards would be. This would be an issue when trying to seat cards. See below
(Had to break post because of too many pictures)
First job was to swap the power supply plate over from a rackmount PSU plate to the standard ATX PSU plate. Picture shows new plate in place and the old one next to it.
The next few shots are of the case showing the Hot swap back planes, 3 x 120mm fans, and the access ports on each side for cabling. I hope these will help future people trying to make decisions on cases.
OK onto the motherboard fit. The first thing I noticed when I was adding the post-screws into the case was there was no room for one on the motherboard under where the expansion cards would be. This would be an issue when trying to seat cards. See below
(Had to break post because of too many pictures)
I had a look at the board and then noticed this circle underneath on the silicon.
It lined up perfectly with the mounting points on the case so I thought I could just add a post-screw and then rest the board on to it. But I was a bit worried about the brass straight onto the board. Luckily I found some old little round stickers and put one of them on the top to help try and protect the motherboard a bit when I was pushing in expansion cards.
Next job was to fit the mounting plate to the bottom of the motherboard for the heatsink. I ended up having to fit the processor first though because the plate structure on the top of the motherboard covered the CPU release clip.
Now to seat the motherboard in the case. Fitted the Supermicro IO shield plate and then seated the motherboard. I always like to try and use the "old skool" cardboard washers when screwing in a motherboard. This one in particular was pretty damn expensive and I have a habit of overtightening things.
Lastly, before dinner I fitted the heatsink.
I'll be back later with a further update and more pictures
It lined up perfectly with the mounting points on the case so I thought I could just add a post-screw and then rest the board on to it. But I was a bit worried about the brass straight onto the board. Luckily I found some old little round stickers and put one of them on the top to help try and protect the motherboard a bit when I was pushing in expansion cards.
Next job was to fit the mounting plate to the bottom of the motherboard for the heatsink. I ended up having to fit the processor first though because the plate structure on the top of the motherboard covered the CPU release clip.
Now to seat the motherboard in the case. Fitted the Supermicro IO shield plate and then seated the motherboard. I always like to try and use the "old skool" cardboard washers when screwing in a motherboard. This one in particular was pretty damn expensive and I have a habit of overtightening things.
Lastly, before dinner I fitted the heatsink.
I'll be back later with a further update and more pictures
Hi Jeggs101
I wanted a passive heatsink and to be honest it was the only one I could find that was listed by the manufacturer as OK to run passive. Probably massive overkill for a 17W CPU as well.
I have had to change it's orientation as it covered the back of one of my RAID cards. So I couldn't get the cables into it. More pictures to come in the next post of this.
I wanted a passive heatsink and to be honest it was the only one I could find that was listed by the manufacturer as OK to run passive. Probably massive overkill for a 17W CPU as well.
I have had to change it's orientation as it covered the back of one of my RAID cards. So I couldn't get the cables into it. More pictures to come in the next post of this.
So I came back from dinner last night and decided to start adding in the RAM and the controller cards. I immediately hit on a problem. The heatsink is so wide it goes over the back of the RAID card and stopped me from being able to put the cables in. Solution was to turn the heatsink around 90 degrees. New position is show below.
There is still a nice gap between the top of the memory and the side of the motherboard, so happy with that
The next little problem was the motherboard header leads wouldn't quite reach the location on the Supermicro motherboard. These are over by the processor and not by the expansion slots as on many other motherboards.
So I added the other controller cards in and then set about re-routing the cables to come out the gap on the other side of the case.
Took the cover off the top of the disk enclosure and re-routed cables back, doubling up a small part of the way to tidy the slack up nicely.
There you go. Nice and neat header leads.
There is still a nice gap between the top of the memory and the side of the motherboard, so happy with that
The next little problem was the motherboard header leads wouldn't quite reach the location on the Supermicro motherboard. These are over by the processor and not by the expansion slots as on many other motherboards.
So I added the other controller cards in and then set about re-routing the cables to come out the gap on the other side of the case.
Took the cover off the top of the disk enclosure and re-routed cables back, doubling up a small part of the way to tidy the slack up nicely.
There you go. Nice and neat header leads.
Next I decided to add all of the SFF-8087 leads and thread them through the chassis. Not much room normally, but luckily this case has removable hot-swap fans so I just pulled the end one out and "hey presto!" loads of room to work. Even for my huge fingers and thumbs
Here they are all cabled in. And then with the fans back in place too.
Last job was now the power supply. Seasonic Fanless 520W. This is the first time I have used a modular power supply. The fanless one does cost a hell of a lot. Then you open the box and the power supply comes in its own little velvet bag and the leads in their own little pouches with a velcro seal. A bit overkill, but hey.
So here is the power supply screwed into the case. Lots of nice space all around it for airflow which is good.
Here is a picture of all the power cabling complete. Needed a couple of power splitters as you need 7 (seven) molex plugs for this case. 5 for the hard disk bays and 2 for the fans and controller. Please note again how the hot swap fan is taken out to easily make room for working on the case. Got to love this feature
Last job was to replace the hotswap fan and plug the fan controller header into the motherboard. And it was done!
Here they are all cabled in. And then with the fans back in place too.
Last job was now the power supply. Seasonic Fanless 520W. This is the first time I have used a modular power supply. The fanless one does cost a hell of a lot. Then you open the box and the power supply comes in its own little velvet bag and the leads in their own little pouches with a velcro seal. A bit overkill, but hey.
So here is the power supply screwed into the case. Lots of nice space all around it for airflow which is good.
Here is a picture of all the power cabling complete. Needed a couple of power splitters as you need 7 (seven) molex plugs for this case. 5 for the hard disk bays and 2 for the fans and controller. Please note again how the hot swap fan is taken out to easily make room for working on the case. Got to love this feature
Last job was to replace the hotswap fan and plug the fan controller header into the motherboard. And it was done!
Well this has been very eventful so far. First went to power system up. Pressed power button on the front of the case and..... nothing
Tested with the reset button instead operating as power and it worked. A bit of head scratching and then I turned the 2 pin power connector around. And it worked! They must have wired it up wrong. Solved that one anyway
Next issue was the fans in the case. Started really slowly and then alarm would start bleeping at about 1,000,000 decibels and the fan failure light kept coming on. This would swap between different fans. I tried looking in the Supermicro BIOS but there was nothing about fan speed settings that I could see.
As the bleeping was becoming so annoying I just unplugged the fan controller header from the motherboard. Fans now run at maximum speed, which is also noisy but not as annoying as that bleeping. No fan failures so must be something to do with how the motherboard is sending info to the controller.
And onto the next problem...... yes there is more
I have 1 x Adaptec 6405 card and 2 x Adaptec 5805 cards. On boot up the 2 x 5805 cards were visible and detecting their disks but the 6405 was not even being detected. Tried taking out cards one by one and when the 6405 is in there on it's own it detects perfectly and sees all of it's disks.
All 3 cards work perfectly on their own. This is driving me nuts!!!!!
Tested with the reset button instead operating as power and it worked. A bit of head scratching and then I turned the 2 pin power connector around. And it worked! They must have wired it up wrong. Solved that one anyway
Next issue was the fans in the case. Started really slowly and then alarm would start bleeping at about 1,000,000 decibels and the fan failure light kept coming on. This would swap between different fans. I tried looking in the Supermicro BIOS but there was nothing about fan speed settings that I could see.
As the bleeping was becoming so annoying I just unplugged the fan controller header from the motherboard. Fans now run at maximum speed, which is also noisy but not as annoying as that bleeping. No fan failures so must be something to do with how the motherboard is sending info to the controller.
And onto the next problem...... yes there is more
I have 1 x Adaptec 6405 card and 2 x Adaptec 5805 cards. On boot up the 2 x 5805 cards were visible and detecting their disks but the 6405 was not even being detected. Tried taking out cards one by one and when the 6405 is in there on it's own it detects perfectly and sees all of it's disks.
All 3 cards work perfectly on their own. This is driving me nuts!!!!!
The fan issue, I thought this case has it's own temperature control? So the only pin which should be connected to the motherboard is the speed sensor (yellow)?
If you have the X9SCM board you need PWM fan's to enable speed control, I had no luck with normal fans and had to buy a PWM adapter.
Regarding the RAID cards, have you tried them in different PCIe slots?
Might be worth disabling the SMBus on the board to see if that changes anything (theres a header on the motherboard)
Also this case looks interesting, the central(ish) fan holder does this not meet the top of the whole case, as in the photos it looks to be 1-2" away. My lower end model creates two 'compartments' which are separated only by the fans, which really does help to improve temperatures of the HDDs by forcing air over them.
If you have the X9SCM board you need PWM fan's to enable speed control, I had no luck with normal fans and had to buy a PWM adapter.
Regarding the RAID cards, have you tried them in different PCIe slots?
Might be worth disabling the SMBus on the board to see if that changes anything (theres a header on the motherboard)
Also this case looks interesting, the central(ish) fan holder does this not meet the top of the whole case, as in the photos it looks to be 1-2" away. My lower end model creates two 'compartments' which are separated only by the fans, which really does help to improve temperatures of the HDDs by forcing air over them.
@nry
Hi there. Thanks for the suggestions. The fan controller lead coming from the case is a four pin one. All three hot swap fans plug into this controller which in turn has the 4 pin lead going to the motherboard.
I've attached a picture here of the top part of the case. As you can see there is a foam bar that goes across and seals the case into two "compartments" as you mention. They had to fit this as they lowered the fans quite a bit. I think so they blow more directly onto the hardware that is lower down like the motherboard.
Yes I have the X9SCM-iiF motherboard. Motherboards slots are laid out as follows: Slot 4 | Slot 5 | Slot 6 | Slot 7 | CPU
I have tried the RAID cards in various slots. Here is what I have found so far.
They all work perfectly on their own in any slot.
Both of the 5805 controllers will work together.
If the 6405 is in a higher slot that the 5805 card then it will be detected and run, but the 5805 card then fails reporting a firmware error of "FF"
I have switched the Boot option in the BIOS to be "Legacy Only" to try and rule out an UEFI issue. No luck there.
The firmware is of course OK. I have also upgraded all of the Adaptec cards to the latest firmware on the Adaptec website. I know the 5805 and the 6405 card will work together in the same system as that is how I had them running before.
I will have a look at the SMBus bit now and see what it does
Hi there. Thanks for the suggestions. The fan controller lead coming from the case is a four pin one. All three hot swap fans plug into this controller which in turn has the 4 pin lead going to the motherboard.
I've attached a picture here of the top part of the case. As you can see there is a foam bar that goes across and seals the case into two "compartments" as you mention. They had to fit this as they lowered the fans quite a bit. I think so they blow more directly onto the hardware that is lower down like the motherboard.
Yes I have the X9SCM-iiF motherboard. Motherboards slots are laid out as follows: Slot 4 | Slot 5 | Slot 6 | Slot 7 | CPU
I have tried the RAID cards in various slots. Here is what I have found so far.
They all work perfectly on their own in any slot.
Both of the 5805 controllers will work together.
If the 6405 is in a higher slot that the 5805 card then it will be detected and run, but the 5805 card then fails reporting a firmware error of "FF"
I have switched the Boot option in the BIOS to be "Legacy Only" to try and rule out an UEFI issue. No luck there.
The firmware is of course OK. I have also upgraded all of the Adaptec cards to the latest firmware on the Adaptec website. I know the 5805 and the 6405 card will work together in the same system as that is how I had them running before.
I will have a look at the SMBus bit now and see what it does
You might take a look at "PCI Express Port - Gen x" bios setting under integrated i/o. I ran the same mb with 2x adaptec 5805 + 2x pci nics, and had to change from auto to gen2 to eliminate adaptec bios display getting scrambled during boot. I suspect my problem was that the nics were older generation - Intel dual PT - but it can't hurt to try.
Thanks for the input poto
I had just found these two Adaptec support documents. I'm posting the links here in case anyone else finds them useful. I'm just going through the Gen X settings you mentioned right now, no joy so far though
Adaptec controllers and UEFI motherboards
Problems with Adaptec RAID Controller installed in Supermicro X9SC motherboards
I also have a quad port Intel PT card, so thanks for the info. I've taken it out of the system right now whilst I am trying to get the RAID cards to work.
I had just found these two Adaptec support documents. I'm posting the links here in case anyone else finds them useful. I'm just going through the Gen X settings you mentioned right now, no joy so far though
Adaptec controllers and UEFI motherboards
Problems with Adaptec RAID Controller installed in Supermicro X9SC motherboards
I also have a quad port Intel PT card, so thanks for the info. I've taken it out of the system right now whilst I am trying to get the RAID cards to work.
Maybe you can try to disable all unnessicary features and controllers in the bios. I wonder if the bios is running out of option ROM memory . Not usually a problem in a Supermicro board but I have seen issues similar to this on consumer boards that run raid controllers. Just for kicks try turning off all the Sata controllers USB ports FireWire serial or anything else you don't specifically need. If still no luck then I would give supermicro a call on Monday they have great support.
Thanks for the reply DBayPlaya2k3
I have been looking at the Adaptec firmwares for both cards. I found this in the readme.txt file for one of the previous 6405 firmware releases.
"4. Known Limitations
4.1 Adaptec Series 5 Forward Compatibility
Adaptec Series 5 firmware is not forward compatible with newer Adaptec controller models. In a server with two controllers on the backplane--a
Series 5 in Slot 3 and a Series 6 in Slot 5--the system will detect only the Series 5. If the slots are reversed, with the Series 6 in the lower slot, both
controllers are detected. Additionally, on a PCIe Gen3 backplane, with the Series 6 in the lower slot and the Series 5 in the upper slot, the Series 5
firmware may crash."
Now I had a 6405 and a 5805 working perfectly before in an Asus P5Q3 motherboard. These same two cards even that I am trying to use now. I have tried downgrading the firmware on both cards and sadly the same thing happens as before. Gone back a few versions now as well. This is more than a little bit annoying now.
I have emailed Supermicro support to ask for their advice. I'll email Adaptec support now as well I think and see what they say. I really don't want to have to buy new RAID controller(s) as they cost an arm and a leg. And it is very difficult to sell on your old ones too
I have been looking at the Adaptec firmwares for both cards. I found this in the readme.txt file for one of the previous 6405 firmware releases.
"4. Known Limitations
4.1 Adaptec Series 5 Forward Compatibility
Adaptec Series 5 firmware is not forward compatible with newer Adaptec controller models. In a server with two controllers on the backplane--a
Series 5 in Slot 3 and a Series 6 in Slot 5--the system will detect only the Series 5. If the slots are reversed, with the Series 6 in the lower slot, both
controllers are detected. Additionally, on a PCIe Gen3 backplane, with the Series 6 in the lower slot and the Series 5 in the upper slot, the Series 5
firmware may crash."
Now I had a 6405 and a 5805 working perfectly before in an Asus P5Q3 motherboard. These same two cards even that I am trying to use now. I have tried downgrading the firmware on both cards and sadly the same thing happens as before. Gone back a few versions now as well. This is more than a little bit annoying now.
I have emailed Supermicro support to ask for their advice. I'll email Adaptec support now as well I think and see what they say. I really don't want to have to buy new RAID controller(s) as they cost an arm and a leg. And it is very difficult to sell on your old ones too
That makes sense. Such a strange design tho. Wonder why they didn't just make it a normal tower.
Maybe I missed but what case is this? I've never seen those red fans.
Hi Jeggs101
The heatsink is by silverstone and designed specially to fit into one of their desktop/HTPC cases. Hence the funny shape. It does look odd in the middle of this huge case
The case itself is from a company in the UK called X-case (Welcome to Xcase) I'm sure they just buy it in and re-brand it. They call it their X-Case RM 420 Pro. It is a 4U rackmount case. The fans are hotswap ones that come with the case and clip into a fan controller PCB that runs along the bottom of the case.
Very nice design, but a bit pricey. The fans really move the air though.
The heatsink is by silverstone and designed specially to fit into one of their desktop/HTPC cases. Hence the funny shape. It does look odd in the middle of this huge case
The case itself is from a company in the UK called X-case (Welcome to Xcase) I'm sure they just buy it in and re-brand it. They call it their X-Case RM 420 Pro. It is a 4U rackmount case. The fans are hotswap ones that come with the case and clip into a fan controller PCB that runs along the bottom of the case.
Very nice design, but a bit pricey. The fans really move the air though.
I use the X-Case RM424pro with a Supermicro X9SRL-F, had a X9SCM-iiF in there before. Fan control works, there's a setting in the IPMI Web GUI (fan control). I believe you can set optimal, full and standard. Standard works well for me.Well this has been very eventful so far. First went to power system up. Pressed power button on the front of the case and..... nothing
Tested with the reset button instead operating as power and it worked. A bit of head scratching and then I turned the 2 pin power connector around. And it worked! They must have wired it up wrong. Solved that one anyway
Next issue was the fans in the case. Started really slowly and then alarm would start bleeping at about 1,000,000 decibels and the fan failure light kept coming on. This would swap between different fans. I tried looking in the Supermicro BIOS but there was nothing about fan speed settings that I could see.
As the bleeping was becoming so annoying I just unplugged the fan controller header from the motherboard. Fans now run at maximum speed, which is also noisy but not as annoying as that bleeping. No fan failures so must be something to do with how the motherboard is sending info to the controller.
And onto the next problem...... yes there is more
I have 1 x Adaptec 6405 card and 2 x Adaptec 5805 cards. On boot up the 2 x 5805 cards were visible and detecting their disks but the 6405 was not even being detected. Tried taking out cards one by one and when the 6405 is in there on it's own it detects perfectly and sees all of it's disks.
All 3 cards work perfectly on their own. This is driving me nuts!!!!!
Chris