Advice on NAS for backups, $2k budget

Discussion in 'DIY Server and Workstation Builds' started by JayG30, Feb 23, 2015.

  1. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    I'm not going to lie, what you just wrote confused me. Something about how the sentences are worded makes it really difficult for me to understand what you are trying to say.

    Per the motherboard manual HERE, page 2-39. The 2xSATA3.0 and 4xSATA2.0 ports are supported by the Intel C222 PCH chip. In addition 8xSAS connectors are supported by the LSI 2308 controller.

    The SATA ports aren't "useless". They just can't be used with a SAS expander. Drives could still be connected to them directly with a SATA cable instead of connecting to the backplane. The drives could be placed somewhere within the case. A good use would be SSD's for ZIL/L2ARC. This is what TuxDude had mentioned and inline with my thinking as well.
     
    #21
  2. lmk

    lmk Member

    Joined:
    Dec 11, 2013
    Messages:
    128
    Likes Received:
    20
    lol :)

    ^ what I said = what you said, except for certain

    just making sure everyone knows that the 'other' (Intel) SATA ports cannot be used with the reverse breakout+expander
     
    #22
  3. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    Hey guys, looking for some info. As I mentioned already I have these parts here or coming;

    Code:
    Case: SuperMicro CSE-826E16-R1200LPB: $399.95
    Motherboard: SUPERMICRO MBD-X10SL7-F-O $249.99
    CPU: Intel Xeon E3-1231V3 $256.99
    RAM: Crucial CT2KIT102472BD160B (2 x 8GB) $169.99
    
    The case was new from ebay (see HERE). It was setup as a JBOD case but I won't be using for that purpose. I don't think this is a big deal, but I'm coming across some connection questions that I'm just not seeing explained in the supermicro PDF's.

    So the way the case came is with the CSE-PTJBOD-CB2 power card in it. Cables that were connected to it were the 16 pin control cable, 24pin power cable, Power SMB (System Management BUS) coming from the PSU, and a 3-wire I2C cable that runs into the backplane.

    On the motherboard (x10SL7-F) I'm able to connect everything expect the 3 wire I2C cable that runs from the BPN-SAS2-826EL1 backplane. I think this is going to result in a problem because I believe the I2C connection is what would allow the board to control the backplane (fans, leds, etc), but I'm not 100% sure. And the documentation on the backplane for this chassis doesn't even show or talk about this 3 wire I2C connection or any SGPIO alternative.

    Also the fans in the case are all wired to the BPN-SAS2-826EL1 backplane. This is nice for wire management, but I'm curious is this will work correctly and allow them to spin up/down in response to CPU temps and so on? My guess is that it will as long as I use the correct cabling from the backplane to motherboard. Which points me back to my last question on the I2C connections.

    From what I can tell in the manual for the motherboard, it has SGPIO connections (6-SGPIO1/2 & T-SGPIO1/2), Power SMB (JPI2C1), but no 3 wire I2C connector like the CSE-PTJBOD-CB2 power board has. Alternatively the BPN-SAS2-826EL1 backplane doesn't mention anything about SGPIO connections or I2C connectors, although it clearly must have an I2C connector.

    In the end, what if any wiring might I be missing or need to connect the BPN-SAS2-826EL1 backplane to the X10SL7-F motherboard so that everything functions "correctly"?
     
    #23
  4. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    So it looks like the I2C cable I'm seeing is CBL-0102L which is optional and part of the JBOD kit. So I don't think I need this for my setup. Interesting I can see the connector on the backplane pictures but it doesn't make note of what it is anywhere. I managed to stick a camera in the case and take a picture where I could see where it was labeled "PRI_I2C1" or something like that.

    So I think what I'm still unsure of is if I need to wire the fans to the MB or can I leave them on the backplane and still have them function correctly (spin down and up based on load). And if I need any type of SGPIO cabling since the backplane doesn't seem to provide that anywhere (just on the MB).
     
    #24
  5. britinpdx

    britinpdx Active Member

    Joined:
    Feb 8, 2013
    Messages:
    348
    Likes Received:
    151
    I'm pretty sure that there is no logic on the SM backplane to sense temperature and control fan speed via PWM. The SAS2-826EL backplane users guide on the SM web site (published in 2010 ) indicates that the fan connectors are 3 pin, in which case there would be no speed control, just on or off.

    I'm pretty sure from the eBay pictures that the fans themselves are 4wire PWM, so you can leave them on the backplane (likely fixed speed) or move them to the motherboard and get PWM control. I'd personally be taking this route, and in the motherboard BIOS set the fan control to "optimal" for lowest noise.

    If you do move the fans to the motherboard then the backplane manual indicates that there are jumpers "FAN_MONITOR_DISABLE" and "FANFAIL_LED_DISABLE" which are both open by default so that the backplane can monitor fan condition in a JBOD application. Seems like both of these jumpers should be installed if you move the fans to the motherboard to disable backplane monitor and error reporting.
     
    #25
  6. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    I had thought the same when looking at the BPN-SAS2-826EL1 backplane manual (I think it says 3 pin) but it has a 4 pin connector as see here;

    [​IMG]


    So as a JBOD setup with the CSE-PTJBOD-CB2 power board, I wonder does it control fan speed? If so, how? Perhaps via the I2C connector? If it is over I2C than I guess I have to wire the fans to the motherboard instead since it doesn't appear the motherboard has an I2C connector (unless I'm mistaken). In fact I don't think I've ever seen a motherboard with an I2C connector. Just the Power SMB (I2C) connector which I thought was just for control of the power supplies. But perhaps it does more than I think in this instance?
     
    #26
  7. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,772
    Likes Received:
    1,116
    The SAS2 backplanes definitely have 4-pin fan connectors. But I've never found any jumper or setting that allows the backplane to slow the fans. AFAICT, the pwm lead is simply tied to 100% duty cycle
     
    #27
  8. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    I'm thinking that perhaps they might be controllable via the I2C connection on the CSE-PTJBOD-CB2, perhaps? But I'm really not sure. Unless even in JBOD there is no fan control at all.

    I guess it couldn't hurt to try, see if the fans are controllable, and if not rewire them to the MB. Just have to check if all the cables are long enough to read the small X10Sl7-F MB or if I need to extend them.

    From what I've seen in images, with a MB in the case they all wire the fans to the motherboard. So probably going to be what I end up doing as well.
     
    #28
    Last edited: Mar 2, 2015
  9. britinpdx

    britinpdx Active Member

    Joined:
    Feb 8, 2013
    Messages:
    348
    Likes Received:
    151
    Most of the newer SM chassis that I've owned that were equipped with PWM mid chassis fans (823, 825, 836, 846) have cables that are long enough to work on Micro ATX (X9SCM), ATX (X9SCA) and EE ATX (X9DR3-LN4F) layouts.

    For older SM chassis with 3 pin fan cables I've replaced the cables with the SM CBL-0216L which is 200mm long, which I believe is the cable used in the newer chassis.

    I've not yet used the X10Sl7, but looking at the manual the left chassis fan ought to reach the FANA header, the chassis fan ought to reach the FAN2 header, and I bet the center fan cable will reach the FAN3 header.

    I've learned to greatly respect the designs that Supermicro come up with, and they really are very modular in their "generic" designs. They have unique motherboards that work hand in hand with unique chassis, but I think you will have no issues to deal with.

    SM chassis are built like tanks and once you have pulled them apart a couple of times, you begin to realize how easy it is to swap cables, backplanes, power supplies, power distributors, motherboards, fans etc.
     
    #29
    PigLover likes this.
  10. NeverDie

    NeverDie Active Member

    Joined:
    Jan 28, 2015
    Messages:
    299
    Likes Received:
    26
    The OP talks about starting with 4 disks but wanting a 12 bay chasis as room for growth, so might as well get the SAS backplane, etc. Is it just me, or isn't that a lot of room for growth? If that growth will take years to happen, I wonder whether the motherboard and backplane and everything else will be relevant enough by the time that growth happens?

    Also, regarding the backplane, what do you see as the primary advantages of that? Is it the ease of swapping drives, the multipath, the less cabling (so better airflow), and/or something else? I can see why datacenters would use them, especially where perhaps unskilled labor on the graveyard shift is trained how to swap drives when failures occur, but I'm less clear about the cost-justification in a home lab.
     
    #30
  11. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    I am the OP, I already purchased the chassis with the E1 backplane.
    Yes it is a lot of room for growth, but the issue is that right now it is to put backup data on. A few TB of storage should be fine for them to hold a few months worth of backups. But they don't currently have an "archiving", which is something I'm 99% sure they need due to regulations. Thus that data will probably grow quite a bit. Plus it will find use as a replication point from another site. You can begin to see how the storage could grow.

    Plus, it isn't costing me much extra for the ability to grow. a few hundred buck to have the capability is worth it IMO. If I could have found something that made sense for drastically cheaper then I might have went with a smaller setup.

    Why the backplane? To reduce SAS/SATA cabling through use of SFF 8087. Why expander? Because from what I have gathered a SAS2 expander has plenty of bandwidth for 12 mechanical HDD and that allows me to drive all 12 disks via 1 cable (reverse breakout SFF8087 or regular SFF8087). That gives me another 4 SAS lanes to drive another set of disks in the same way. Plus free's up the SATAIII ports on the motherboards for SSD disks. If I didn't use this particular backplane and instead went with a TQ or A style backplane I'd have to use basically all the SAS/SATA ports available on the motherboard to drive the 12 disks (14 ports available on MB). First I didn't like the idea of mixing the SAS/SATA ports and second if I wish to expand that would mean I'd have to get another HBA. Not a horrible thing, but it just didn't make sense to me just to save a few hundred bucks (I think the difference would be $175).
     
    #31
    NeverDie likes this.
  12. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    I can use FAN1/2/3 for the 2 fans. The third fan furthest away from the PSU can only reach the FANA header like you mentioned.

    I know this has been discussed all over the internet but my understanding is that some motherboards have 2 controllers for the fans. So FAN1-4 might be controlled by system temp and FANA & B might be controlled by CPU temp.
    For reference:
    Super Micro Computer, Inc. - FAQ Entry
    Super Micro Computer, Inc. - FAQ Entry
    Super Micro Computer, Inc. - FAQ Entry

    Does anyone know with regards to this Motherboard? It seems that using FANA shouldn't be an issue...
     
    #32
  13. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    Ok, so did some reading on SGPIO and 8087 cables to try to figure out how this will work.

    If I was using an HBA, I could just use a standard SFF-8087 (mini-SAS) to SFF-8087 (mini-SAS) cable (36-pin connectors). Supermicro makes these (CML-0108L-02, CML-0109L-02, CML-0110L-02) and they carry sideband (SGPIO). You can see that in the image;

    [​IMG]
    The issue I have is that the motherboard doesn't have an SFF-8087 port, it has individual SAS ports. This requires the use of a reverse breakout cable (4 port "host" to 1 SFF-8087 "target"). However I don't have any idea if this provides the drive activity information to the backplane or not.

    The motherboard has an SGPIO connection for the SAS and SATA ports, but I have not seen a single "reverse breakout cable" that also has a sideband connector (ie. 4port+sideband "host" to 1 SFF-8087 "target"). And even if it did I'm not so sure that would work anyway.

    So unless this "just works" and I'm missing something, it might make more sense to just get a simpler motherboard and add an HBA that you can wire up to the backplane using an SFF-8087 cable.
     
    #33
  14. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    So while I wait of a few parts to show up, I had some spare RAM laying around, so I booted up the server.

    I noticed that the CPU temp seems rather high for idle. Just booted into BIOS and looking at sensors with IPMI, but CPU temp is reading 48-50C. Stock Intel cooler on a E3-1231v3. CPU fan shows that it is running at 900RPM. System temp was low 20's (~22C). Room has a dedicated cooling unit (Data Aire). I checked and it the cooler seems to be pushed all the way down. I removed and replaced the TIM as well. Didn't really help.

    There are other SuperMicro servers running in there for a long time that aren't showing this high of temps. Have a E3-1230V3 with a passive cooler in a 1U SC813MT-350CB right near it. CPU temp showing 29C.

    EDIT:
    Temps seem to fall to 31-32C once OS is booted and idle. So I guess that is ok. Still surprised how low the idle is on my Passive 1U server in comparison.
     
    #34
    Last edited: Mar 12, 2015
  15. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    I have a question regarding hot swapping.
    This is my hardware;

    Code:
    Case: SuperMicro CSE-826E16-R1200LPB
    Motherboard: SUPERMICRO MBD-X10SL7-F-O
    CPU: Intel Xeon E3-1231V3
    RAM: Crucial CT2KIT102472BD160B
    HDD: 6 x Toshiba MG03ACA300 3TB Enterprise SATA
    Norcoo SFF8087 reverse breakout cable
    The drives are all hooked through the SFF8087 reverse breakout cable to the onboard LSI 2308 ports.

    I was trying to attempt hotswapping hard drives in FreeNAS 9.3 (so FreeBSD). However it seems to not work. If I attempt to remove a disk, say the one in bay 3, and than put it back in, it doesn't seem to work until you restart the system. The OS reports this;

    Code:
    da3 at mps0 bus 0 scbus0 target 11 lun 0
    da3: <ATA TOSHIBA MG03ACA3 FL1A> s/n            53L8K729F detached
    cam_periph_alloc: attempt to re-allocate valid device da3 rejected flags 0x118 re
    daasync: Unable to attach to new device due to status 0x6
    It sounds to me like this might be an issue with FreeBSD that they are working on but have not finalized. Getting that from HERE, HERE, and HERE.

    Does this seem right? This setup SHOULD support hot swapping, correct? FreeBSD seems to be the issue?
    Anyone have any input on what OS will support this best for ZFS? I assume a Solaris derivative would work best? Anyone tried hot swapping under ZFS on Linux?
     
    #35
  16. TeeJayHoward

    TeeJayHoward Active Member

    Joined:
    Feb 12, 2013
    Messages:
    374
    Likes Received:
    106
    I've got a SC846E16, X10SL7, reverse breakout setup. If I recall correctly, using ZFS on Linux I can hot swap SAS drives with no issue. I know for a fact I could do it on Solaris 11.1. Right now I'm using an X9 motherboard and a 2308 PCIe card, otherwise I'd pull out a drive and test it for ya.
     
    #36
  17. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    Thanks!

    I tested the other day with OmniOS and it worked. Removing the disk without offlining it puts it in the removed status. After plugging it back in I had to online the drive. But it was detected and no complete resilvering of the disk and no reboot necessary.

    I'm going to test freebsd as well. Version 10 I think has improvements to this. Freenas is on version 9.3 so perhaps a test with freebsd 9.3 would be worthwhile to see if it is a freebsd or freenas issue. My guess is freebsd just isn't very good with hot swap at this point. Not good considering ZFSonLinux can do it!
     
    #37
  18. JayG30

    JayG30 Active Member

    Joined:
    Feb 23, 2015
    Messages:
    226
    Likes Received:
    46
    FYI, hot swap is working in FreeNAS 9.3 now. Not sure why it wasn't before. It might have been being caused by an issue with Offlining and Onlining disks using FreeNAS. Turns out you HAVE to use the GUI to offline a disk. If not the identifiers get confused and you have to restart to fix it. Someone was more likely to drop to CLI to offline a disk considering the GUI didn't offer a online feature. So I submitted a feature request and they have added it coming in the next update.
     
    #38
    MiniKnight likes this.
  19. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,941
    Likes Received:
    857
    Always good to report that stuff.
     
    #39
Similar Threads: Advice backups
Forum Title Date
DIY Server and Workstation Builds Need Advice Building a 16+ ZFS NAS Oct 17, 2019
DIY Server and Workstation Builds First Time Builder Needs Advice Aug 3, 2019
DIY Server and Workstation Builds Quiet Xeon LGA 3647 Workstation build, seeking advice Jul 24, 2019
DIY Server and Workstation Builds Advice for remote gaming rig Apr 24, 2019
DIY Server and Workstation Builds Looking for advice on build, storage spaces server Mar 4, 2019

Share This Page