New Nas from scraps - in need of a hba or something like that

[jmoll]

Hi,

I have an old computer which I am converting to the fileserver / NAS.

-Specs-
OS: Debian->Openmediavault / i7-3770k, 16gb ddr3, asus p8z77-v lx mobo, small ssd for OS ,ASM1061 cheap sata cards 2x plugged in, 8x 2T enterprice grade hdd's, 2X 1T consumer ssd's

-The Plan-
The plan is to make new budget/quickish file server to saturate 10Gbps network. I will make two raids, A) fast, 2 ssd disk raid 0. B) slow, 8 disk raid 10 made of hdd's. (raid (A), will be mirrored on old NAS server I have so redundancy is not a problem)

-The question-
The question is: what would you do? Everything I previously listed I already have so I would rather spend the minimun cash from now on. In mobo I have 2x 6Gb/s satas for the SSD's but with the two cheap sata cards I do still miss one sata port for 8 drive raid. I do still have one long pci-e slot unoccupied so what should I put in there ? buy another cheap sata card ? or throw all of these out and get proper HBA ? Are there any good budged friandly ones. I just need the throughput... I am planning to go for software raid

Thnx

 

[OBasel]

For Sale: H220 LSI 9205-8i Controllers

Not too pricey at all.

Do you pay for power? That is a lot of small drives. To saturate 10g it's still hard using 2x sata or sas2 ssds. You might be able to get cheap SAS3 ssds and saturate 10g if you get a sas3 controller but those cost an extra 100 or 150 for the controller.

 

[whitey]

Those or any ole' LSI 2008 gen (most cost effective and rock solid still) 6Gbps HBA and be done w/ it. Cheapest is likely H310 (don't pay more than $40 for this one, have to crossflash so don't get that one if you're not comfortable w/ that process), the H220 will likely need one as well unless you are fine w/ the v15 IT mode that they said are on them now. LSI 9210/9211-8i's are gonna be drop dead simple to flash.

 

[jmoll]

Hi,

Thank you for the replies. I did order a LSI 9210-8i from ebay... I will figure out the flashing when it arrives.

I am paying for the power, but it is not really a problem. This NAS is for the home office and it is bringing all of the income to household so costs are acceptable. During winter I am heating with electricity anyways so half of the year you can consider it running "cost free". I don't have to saturate the 10gbe fully I figured that the I/O of 2 disk ssd raid 0 will be enough and the disks I already have so I don't want to invest any more money to those. I am personally only one using this thing, but I have several computers accessing the data. Anyhow my I/O is getting huge upgrade I was previously running a 1gbe network + very lazy nas

thnx

 

[whitey]

9210-8i is simple to flash once you create your USB flaher stick, two cmds.

sas2flsh -o -e 6
sas2flsh -o -f 2118it.bin

 

[jmoll]

9210-8i is simple to flash once you create your USB flaher stick, two cmds.

sas2flsh -o -e 6
sas2flsh -o -f 2118it.bin

Thank you! Yes looks simple enough

 

[fractal]

A few things to consider. You are not going to saturate 10G with a couple of SSDs. You will be well north of 1G but short of 10G.

Think about how you connect your drives. Enabling / burning the bios to your HBA makes boots take F O R E V E R. Well, not that long but it feels like it. You may want to boot from your motherboard SATA ports and disable the bios on your HBA if boot time is any concern to you.

Finally, as with all storage servers, be sure to invest in a UPS with proper shutdown software.

 

[jmoll]

A few things to consider. You are not going to saturate 10G with a couple of SSDs. You will be well north of 1G but short of 10G.

Think about how you connect your drives. Enabling / burning the bios to your HBA makes boots take F O R E V E R. Well, not that long but it feels like it. You may want to boot from your motherboard SATA ports and disable the bios on your HBA if boot time is any concern to you.

Finally, as with all storage servers, be sure to invest in a UPS with proper shutdown software.

Hi, Thanks for your reply. Didn't know that the HBA would increase the booting time, but in my case its not a problem the system is already installed on small ssd plugged in onboard sata slot. HBA will be occupied by 8 hdd raid 10.

I've my workstation plugged in to Eaton 5P UPS Its not a monster ups but better than nothing. Planning to put also this file server behind it. I haven't figured out yet how it works with linux/openmediavault... but shouldn't be too much trouble getting it shutdown neatly with it.

I am still running my old fileserver so I am totally open for suggestions. Waiting for the hba to be delivered. I thought that I did pick one from Europe... but it turned out that it was shipped from Hong Kong . I hope avoiding customs. Those can double up the price of the cheap HBA

 

[nk215]

i7-3770k does not support vt-d. In your case it may not matter but it may limit the future options.

 

[jmoll]

I did today get the hba, VERY long delivery but I did avoid paying the customs so did save couple of euros. Didn't yet have change to flash or test if it is working. Tomorrow I am going to do it and final build and if things work out.... migrate the necessary data and functionality of the file server over weekend.

One thing I added to the build is that I did buy a new efficient/supposedly reliable psu. I thought that would hopefully make the system bit more fail proof even though I do have it plugged in to ups.

 

[jmoll]

Hi,

I did face some troubles and don't really understand what is going on. Maybe someone can give me some direction where to troubleshoot (Networking is not really my strongest feat).

-setup/problem-
ssd raid 0 plugged in to the motherboard sata 6gb/s ports, two cifs/smb shares on same raid and I am directly copying files from one to another. I am experiencing this throttling and I am not even getting the speed of one ssd drive. So something is off here and I don't know whats is going on. Sorry I am no really experienced with this stuff. would be great if someone could shed some light on the matter.

see the image:

 

[jmoll]

This is starting to get way off from the HBA gategory but I think I am having some other problems here. When I try to copy from one of these shares to my workstation I am getting ~250mb/s copy speed... not enough.

 

[jmoll]

This is starting to get way off from the HBA gategory but I think I am having some other problems here. When I try to copy from one of these shares to my workstation I am getting ~250mb/s copy speed... not enough.

-Network setup-

LB6M switch, 3 meter DAC's, Mellanox MNPA19-XTR ConnectX-2 nics on each computer.

Thnx, every bit of help is welcome

 

[Kai Amundsen]

@jmoll What speeds are you expecting? Those seem pretty good and about what I would expect. 516MB/s is basically the max speed of a 6Gb/s SSD (note, there is a HUGE, 8x to be exact, difference between Gb and GB with small b being bit and large being bytes). With network/OS/Speed of your workstation drive, getting 250MB/s is reasonable although if you have a fast SSD in the workstation maybe a little slow, that is ~2Gb/s, so double gigabit ethernet speed already.

 

[ttabbal]

A 2-disk RAID0 is not going to saturate 10Gb/s networking. Maybe with some really high end nvme drives, but not with a pair of most SSDs. And if you are writing to a local drive, that's a factor as well. You can only receive as fast as you can write. And Windows has been known to need a little work to get full speed on 10Gbe...

Test first with iperf or similar. That way, you know if your network can even move data that fast.

Then benchmark the drives, they are likely the limiting factor here. Next up would be filesystem overhead and OS level issues. Note that the highest numbers are not real life numbers. That's for simple sequential access, likely no other access on the disk at the time.

With good enterprise SSDs like the Intel 3700 series, I would expect you could saturate 10Gb/s in a stripe of 5 or so. Lesser devices will need much wider stripes. My 10-wide spinner array can almost do it, if the system is quiet and access is sequential. My traffic is read-heavy so with 10 mirrors it tends to do pretty well. It doesn't take much for IOPS to kill the speeds though.

 

[jmoll]

Guys thnx for your replies,

My problem is mostly that I don't get steady copy speed. Sometimes there is loads of variation in the speeds. I don't know really what is the problem with that. Maybe it has more something to do with network, cifs/smb share and windows explorer copy than actual disk io/hardware/network throughput. I did do some test with ftp and it was working much more steady and very quick. With caches I do saturate the 10gbe which is already helping alot. I have done iperf3 tests and run several tools I do get decent throughput with network.

ssd raid is somewhat a waste atm. I am considering adding another ssd on my computer to have similar striped raid. I was hoping I would get better performance from copying stuff from share to another on same ssd raid. I have a feeling than I had better performance with single ssd disk than two ssd disks in raid 0.

Other than that I am quite happy to the new fileserver it was almost free and everything is now working quite well. Flashing/updating firmw on the hba was very easy and went smooth. The 8 hdd drive software raid 10 with hba in it mode is working great.

Does anyone know good benchmark software for nas? something which does read/write to/from ram to disks in network. I have tried few and those are somehow using the caches and only testing really the network throughput. Maybe I should have adjusted the settings so that the sent files wouldn't fit in the memory or something like that.

thnx for all of the input. HBA is working perfect so I did get good input for my initial question.

 

[jmoll]

Here is one bad performance example.
Maybe some one could explain what is happening here ?

In this copy dialog the "workfolder" is a share on ssd raid 0 and I am copying files to share "test" on hdd raid 10. It is copying large files +100 mb. The speed at first is very quick but then slows waaaayyyyyy down.

 

[jmoll]

Here is CrystalDiskMark test to ssd 0 share

-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :  1185.858 MB/s
  Sequential Write (Q= 32,T= 1) :   745.153 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   142.112 MB/s [ 34695.3 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :   255.553 MB/s [ 62390.9 IOPS]
         Sequential Read (T= 1) :   870.839 MB/s
        Sequential Write (T= 1) :   589.588 MB/s
   Random Read 4KiB (Q= 1,T= 1) :    55.589 MB/s [ 13571.5 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :    54.935 MB/s [ 13411.9 IOPS]

  Test : 32768 MiB [W: 8.9% (170.5/1924.2 GiB)] (x5)  [Interval=5 sec]
  Date : 2017/01/24 23:19:18
    OS : Windows 10 Professional [10.0 Build 14393] (x64)

 

[jmoll]

Here is CrystalDiskMark test to hdd raid 10. I don't completely understand this either... shouldn't be this quick right ?

-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :  1203.856 MB/s
  Sequential Write (Q= 32,T= 1) :   604.247 MB/s
  Random Read 4KiB (Q= 32,T= 1) :     5.147 MB/s [  1256.6 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :   249.275 MB/s [ 60858.2 IOPS]
         Sequential Read (T= 1) :   891.230 MB/s
        Sequential Write (T= 1) :   617.180 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     8.881 MB/s [  2168.2 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :    51.974 MB/s [ 12689.0 IOPS]

  Test : 32768 MiB [Z: 0.4% (31.0/7391.8 GiB)] (x5)  [Interval=5 sec]
  Date : 2017/01/24 23:31:56
    OS : Windows 10 Professional [10.0 Build 14393] (x64)

 

[jmoll]

Here I am copying from hdd raid share to local folder. Speed is as expected and working very well