Overdue Servers/Network Build Log

[kidchunks]

Latest Pictures!

New edgerouter 1U Bracket

ATP USBe fits like a glove!

Wiring === fun

Keeping that fiber card nice and cool..

Rack completed..for now. Ignore the workstation and amp .

Also, considering another ESXi host with the following:

Mobo: Supermicro X10DRL-i
CPU: 1xE5-2630v3
RAM: 4x16GB DDR4 ECC Samsung
SSDs: Still searching..
Chassis+PSU: Still searching..

 

[T_Minus]

I'm not seeing any pics? Did they not post right

 

[kidchunks]

I'm not seeing any pics? Did they not post right

Did you try clicking "Spoiler"?. I didn't want the thread to scroll too far down .
Removed spoiler button for now.

 

[maze]

Have you thought about rack mounting the edgerouter?

WaveGuard WG-UB-RM1 – Indoor enclosures – ISP Supplies

Gotta get me one of these !

Gonna check what shipping is to denmark

Edit: yikes.. $45 to ship.. guess i'll have to go play around with some bending metal myself.

 

[neo]

Gotta get me one of these !

Gonna check what shipping is to denmark

Edit: yikes.. $45 to ship.. guess i'll have to go play around with some bending metal myself.

It shouldn't cost more then $25 or so if they use USPS

If you really want it, I can probably reship it to you.

 

[T_Minus]

Very high end USB sticks May I ask where you got them? I couldn't find any info other than Manufacture and digiKey. SLC USB? MMMhmm

 

[kidchunks]

Very high end USB sticks May I ask where you got them? I couldn't find any info other than Manufacture and digiKey. SLC USB? MMMhmm

I used the manufacturers flyer as a guide. I went with the SLC model, look at that TBW!
http://www.atpinc.com/downloadlog/16ab9ce2fcfcee0b

Highest capacities are the 16GB and 8GB. Digikey was the best place I could find the 8GB model.
AF8GSSGH-OEM ATP Electronics, Inc. | AF8GSSGH-OEM-ND | DigiKey

They also sell the 16GB...but you need to purchase........1,000
AF16GSSGH-OEM ATP Electronics, Inc. | AF16GSSGH-OEM-ND | DigiKey


I'm happy with the speed and performance. Definitely recommend it as a boot drive if you don't need much space and want reliability.

 

[maze]

It shouldn't cost more then $25 or so if they use USPS

If you really want it, I can probably reship it to you.

Thanks neo, i'll check up on it when im back from vacation! - Apreciate it

 

[kidchunks]

Finally had some time to update this thread. Only managed to snag these while parts being delivered and building the NAS.

No introduction needed

Favorite Fans (noctua industrial 120mm)

PM863s with 3M thin sata cables (these are fragile!)

Semi-tight fit but enough air-flow

In the meantime, here is some benchmarks for my new VM storage (4xPM863 480GB in raidz on a flashed dell h200)


Test was done with the VM attached to the storage ( NFS, 10gbe and MTU 9000).

zpool dataset Settings:
compression=off
sync=disabled
primarycache=metadata

Arc cached was not used

-----------------------------------------------------------------------
CrystalDiskMark 5.1.2 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 1096.495 MB/s
Sequential Write (Q= 32,T= 1) : 362.453 MB/s
Random Read 4KiB (Q= 32,T= 1) : 47.691 MB/s [ 11643.3 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 11.788 MB/s [ 2877.9 IOPS]
Sequential Read (T= 1) : 346.657 MB/s
Sequential Write (T= 1) : 259.825 MB/s
Random Read 4KiB (Q= 1,T= 1) : 5.866 MB/s [ 1432.1 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 3.041 MB/s [ 742.4 IOPS]

Test : 4096 MiB [C: 45.8% (14.6/31.9 GiB)] (x5) [Interval=5 sec]
Date : 2016/05/20 11:35:45
OS : Windows 7 Ultimate SP1 [6.1 Build 7601] (x64)

4K looks a little slow..curious if anyone with these drives in a similar configuration experienced something similar?

Update: Just tested locally (using bonnie and nmon for monitoring iops)..Something else is going on..either the network or the VM...4k speeds should not be that low.


Update #2:

Originally when I did my first round of testing I had too many variables in place. MTU changes, different blocksizes, only testing VM's disk. So, I honed my test and after a few days of testing (wont show it all)...decided on using ISCSI and 64k blocksize for the zvol datastore.

NAS Specs/Config:
OS: Debian Jessie
iSCSI Target: iSCSI Enterprise Target
RAM: 8GB
CPU: Xeon-D 1518
Disks: 4xPM863
Network: 10GBe (MTU 1500)

Pool/Dataset Config:
RAIDZ
sync=always
compression=off
primarycache=all

Testing pure disk speeds (Reduced max arc to 2GB to avoid hitting it)
I used arcstat.sh to make sure I wasn't hitting the arc.

IOMETER Config:
Test unformatted iscsi disk
4GB disk size
2 Workers, 16 Outstanding I/Os (per worker) 2 mins per test

Test Patterns

1. VDI - 4K 20% Seq/80% Random (80% Write / 20% Read)
2. SQL Server 16K - 100% Random (34% Write/ 66% Read)
3. SQL Server 64K - 100% Random (34% Write/ 66% Read)
4. Other patterns were included in with IOMETER.

Results:

LAT(ms) -- Avg IOPS -- Avg MBps

VDI
3.24 -- 4934 -- 19
SQL Server 16K
2.78 -- 5746 -- 89
SQL Server 64K
3.17 -- 5050 -- 315
64K Seq Read
1.29 -- 12404 -- 775
64K Seq Write
2.55 -- 6263 -- 391
256K Seq Read
4.78 -- 3344 -- 836
256K Seq Write
8.53 -- 1876 -- 469


Takeaways:

1. I didn't see much performance gains with striped mirrors (2 spread vs 3 spread raidz). That's a given I would think as one would need a few more striped vdevs to really see the power of striped mirrors. If you have only 4 SSDs..stick to raidz.
2. zvol blocksize makes a difference. I tested 8k, 16k, 64k, 128k and found 64k to be the right balance for my storage needs (16k close second).
3. NFS fell short when sync=always was set where as iSCSI performed very well. Stick to iSCSI(with sync=always) if you can.
4. Need more ram but will wait till ARC miss ratio hits < 85%

Look forward to hearing others experiences and feedback.