Fusion-io ioDrive 2 1.2TB Reference Page

[acquacow]

Yeah, not sure if I did anything wrong, but I did it twice to confirm...

Here are my results:

4 disks by themselves:

4-disk with a 2-column stripe for ~double speeds

Built an ssd tier and even had write caching enabled with the ssds and the perf was worse than a 4-disk mirror with striping...

This is how I built it... anything wrong you can see in there?

$pd = (Get-PhysicalDisk -CanPool $True | Where MediaType -NE UnSpecified)
New-StoragePool -PhysicalDisks $pd –StorageSubSystemFriendlyName “Windows Storage*” -FriendlyName “StoragePool”

$ssdTier = New-StorageTier -StoragePoolFriendlyName "StoragePool" -FriendlyName SSDTier -MediaType SSD
$hddTier = New-StorageTier -StoragePoolFriendlyName "StoragePool" -FriendlyName HDDTier -MediaType HDD

New-VirtualDisk -StoragePoolFriendlyName "StoragePool" -FriendlyName TieredSpace -StorageTiers $ssdtier, $hddtier -StorageTierSizes 1090GB, 7260GB -ResiliencySettingName Mirror -WriteCacheSize 1GB

Get-VirtualDisk TieredSpace | Get-Disk | Initialize-Disk -PartitionStyle GPT
Get-VirtualDisk TieredSpace | Get-Disk | New-Partition -DriveLetter “E” -UseMaximumSize
Initialize-Volume -DriveLetter “E” -FileSystem REFS -Confirm:$false

Thoughts?

 

[i386]

PS C:\WINDOWS\system32> Get-PhysicalDisk -CanPool $True | ft FriendlyName,OperationalStatus,Size,MediaType

FriendlyName          OperationalStatus          Size MediaType
------------          -----------------          ---- ---------
Fusion ioCache 1200GB OK                1205000000000 SSD
Fusion ioCache 1200GB OK                1205000000000 SSD
Fusion ioCache 1200GB OK                1205000000000 SSD

Did you change the media type for the iodrives manually or were they automatically detected as ssds?

 

[acquacow]

Did you change the media type for the iodrives manually or were they automatically detected as ssds?

Auto-detected as SSDs.

 

[TLN]

I'm trying to see what options I can tune with Fusion IO and what I can get.
According to this: https://www.hpe.com/h20195/v2/GetPDF.aspx/c04111683.pdf
It says I need 530Mb per 100Gb of storage. Which means I need 6.3Gb memory for 1.2Tb drive. I have that, but it does looks like "too much". In case I want to use 512block size I need what.... 47Gb of RAM. WTF? And I guess if I want to pass-through this card to VM (just to try) I need to provide VM with all that memory.

Dell document here: Fusion ioMemory Support Downloads :: SanDisk
says that I need to use 0.5% of available memory. So for 1205Gb x 0.005 = 6Gigs, which is closer real-life values. It also says that ESXi VMFS requires 512b sector.

So, for use as VM-storage, do I need to worry about sector size and/or memory reservations?

 

[T_Minus]

Well, I doubt you're using it as HP intended as an IO accelerator... also, I was under the impression that when they say "storage" they mean the actual total storage, and not the IO accelerator capacity but I could be reading it wrong too.

Either way I've yet to see anyone post here about having to allocate #MB of RAM because of their Fusion IO setup, and many many here seem to get these / deals over the last 2-3 years.

I believe @Marsh uses these often, and may have some other input. I've only used the original Fusion-IOs, but have a couple IODRIVE2s in the mail here so I can let you know in a week or so if I had to do anything, but I doubt it.

 

[TLN]

I don't know why HP listed this table, because vmware recommends 512 size block for VMFS and with 1205 "capacity" it is ~49gig ram. If we use "storage" as actual data stored on device, say 500gig (wasting 700gig on pricey device) it will be 20Gig.

0.5% makes more sense to me. It results in ~6gig for 1.2Tb device, and it we use only amount of data that we have, for 500gig it will be less then 3 gig, which is absolutely doable.

I see @aquacow was working there, may be he can help with that question.
For now, I'm mostly curious, how do you guys format it (I guess 512), do you use full capacity(yes, why not?).

 

[i386]

I have about 650gb on my iodrive2 and firefox is still the process consuming most ram

-

I got two of them this week and these drives are awesome!

 

[acquacow]

Memory usage in the VSL depends 100% on what blocksize you are WRITING. The memory consumption is just to hold pointers in memory to quickly lookup where data is on the card.

If you are writing at 512B, you will consume a LOT more memory than if you are writing at 4k.

A lot of DB apps/etc require 512B block sizes on the backing store, but they tend to write at 32-64K or larger, which uses very little memory footprint in the VSL.

 

[Maritime]

I have about 650gb on my iodrive2 and firefox is still the process consuming most ram

-

I got two of them this week and these drives are awesome!

On W Server 2012 R2, ioDrive2 1.2TB takes about 33-35 GB with 512B block size. I'm using it as standard drive, without VMs or so. Drivers are SanDisk latest - 3.2.14.1675. If drive is heavily written to, RAM allocation increases 1-2 GBs.
However, I have turned off Windows virtual memory in control panel so this might lead to higher RAM allocation.

 

[acquacow]

On W Server 2012 R2, ioDrive2 1.2TB takes about 33-35 GB with 512B block size. I'm using it as standard drive, without VMs or so. Drivers are SanDisk latest - 3.2.14.1675. If drive is heavily written to, RAM allocation increases 1-2 GBs.
However, I have turned off Windows virtual memory in control panel so this might lead to higher RAM allocation.

Well, if you are just using it as NTFS formatted, it's likely your NTFS blocksize is 4k, so that is the size of your smallest writes, which will save you a lot of memory.

 

[Maritime]

Well, if you are just using it as NTFS formatted, it's likely your NTFS blocksize is 4k, so that is the size of your smallest writes, which will save you a lot of memory.

These cards are formatted by their own formatting command (fio-formate) and by default block size is 512B. I don't have memory issues since cards are working on server with 384GB RAM but yes, it might be wise to reformat it to 4k block size.

 

[acquacow]

Low level formatting the cards to 4k will definitely decrease your worst-case memory consumption if that is a concern, but if you aren't writing at 512B to begin with, it won't really do much. All of the higher capacity 3.2TB and 6.TB ioDrives default format to 4K block sizes.

 

[NetWise]

I'm seriously considering grabbing one of these to use as a Starwind VSAN cache disk. Wondering how it'd perform on a 10gb network for small-scale virtualization clustering.. I'm thinking I'm going to try it. Worst case is it ends up as my vmware workstation scratch disk..

I don't know if the Nytro Warp 1.2TB I am using for that exact purpose is on feature parity with these drives here or not - but it sure is doing the trick as a cache disk

 

[i386]

Low level formatting the cards to 4k will definitely decrease your worst-case memory consumption if that is a concern, but if you aren't writing at 512B to begin with, it won't really do much. All of the higher capacity 3.2TB and 6.TB ioDrives default format to 4K block sizes.

Maybe a stupid question: does the drive support other block sizes than 512b or 4k? And do bigger block sizes mean the driver needs less ram?

 

[acquacow]

Maybe a stupid question: does the drive support other block sizes than 512b or 4k? And do bigger block sizes mean the driver needs less ram?

I've only ever used 512B and 4K, but I believe you can set it larger, but your write amplification will likely go up if you are doing smaller writes than your block size.

I'm not sure how performance will be either, you'd have to benchmark it at each block size to see what is best for your application.

 

[warlockedyou]

Has anyone here installed the drivers for Linux? I am trying to setup my 1.2TB Fusio IO as an iSCSI target on CentOS7 box. They have supported drivers for it, but I can't fix the error about incompatible driver with the firmware of the drive.

Error in question:"The firmware on this device is not compatible with the currently installed version of the driver"

Reference Code:

[root@localhost]# fio-status -a

Found 2 ioMemory devices in this system with 1 ioDrive Duo
Driver version: 3.2.14 build 1675

Adapter: Dual Adapter
        1.28TB High IOPS MLC Duo Adapter for IBM System x, Product Number:xxx, SN:xxx
        ioDrive Duo HL, PN:xxx
        External Power: NOT connected
        PCIe Bus voltage: min 11.99V max 12.05V
        PCIe Bus current: max 0.93A
        PCIe Bus power: max 11.16W
        PCIe Power limit threshold: 24.75W
        PCIe slot available power: 25.00W
        PCIe negotiated link: 4 lanes at 5.0 Gt/sec each, 2000.00 MBytes/sec total
        Connected ioMemory modules:
          fct0: Product Number:xxx, SN:xxx
          fct1: Product Number:xxx, SN:xxx

fct0    Status unknown: Driver is in MINIMAL MODE:
                The firmware on this device is not compatible with the currently installed version of the driver
        IBM ioDIMM 640GB, SN:xxx
!! ---> There are active errors or warnings on this device!  Read below for details.
        IBM ioDIMM 640GB, PN:xxx
        Located in slot 0 Upper of ioDrive Duo HL SN:xxx
        Powerloss protection: not available
        PCI:07:00.0, Slot Number:2
        Vendor:1aed, Device:1005, Sub vendor:1014, Sub device:3c3
        Firmware v5.0.7, rev 107053 Public
        Geometry and capacity information not available.
        Format: not low-level formatted
        PCIe slot available power: 25.00W
        PCIe negotiated link: 4 lanes at 2.5 Gt/sec each, 1000.00 MBytes/sec total
        Internal temperature: 40.36 degC, max 40.85 degC
        Internal voltage: avg 1.03V, max 2.38V
        Aux voltage: avg 2.44V, max 2.38V
        Rated PBW: 10.00 PB
        Lifetime data volumes:
           Physical bytes written: 0
           Physical bytes read   : 0
        RAM usage:
           Current: 0 bytes
           Peak   : 0 bytes

        ACTIVE WARNINGS:
            The ioMemory is currently running in a minimal state.

         
fct1    Status unknown: Driver is in MINIMAL MODE:
                The firmware on this device is not compatible with the currently installed version of the driver
        IBM ioDIMM 640GB, SN:xxxxxx
!! ---> There are active errors or warnings on this device!  Read below for details.
        IBM ioDIMM 640GB, PN:xxxxxx
        Located in slot 1 Lower of ioDrive Duo HL SN:xxxxxx
        Powerloss protection: not available
        PCI:08:00.0, Slot Number:2
        Vendor:1aed, Device:1005, Sub vendor:1014, Sub device:3c3
        Firmware v5.0.7, rev 107053 Public
        Geometry and capacity information not available.
        Format: not low-level formatted
        PCIe slot available power: 25.00W
        PCIe negotiated link: 4 lanes at 2.5 Gt/sec each, 1000.00 MBytes/sec total
        Internal temperature: 40.36 degC, max 40.36 degC
        Internal voltage: avg 1.00V, max 2.38V
        Aux voltage: avg 2.44V, max 2.38V
        Rated PBW: 10.00 PB
        Lifetime data volumes:
           Physical bytes written: 0
           Physical bytes read   : 0
        RAM usage:
           Current: 0 bytes
           Peak   : 0 bytes

        ACTIVE WARNINGS:
            The ioMemory is currently running in a minimal state.
[root@localhost]#

I was hoping someone here would have some advise for me on how I should proceed with this. Also, do I need to get "External Power"? Why? Why not?

Thanks

 

[T_Minus]

How many firmwares have you tried?

Have you updated the drivers to latest?

 

[warlockedyou]

How many firmwares have you tried?

Have you updated the drivers to latest?

I have only tried 3.2.11(the latest available):
SanDisk Dell Support

File:

dell_iodrive_3.2.11-20150618.fff

I dont know how to acquire new drivers for CentOS to match something that is compatible with the disk firmware. It is my understanding that firmware number(v5.0) does not have to match the driver version.

Am I understanding this correct? How should I proceed on updating the drivers?

This is my first time purchasing a drive that needs "special" drivers and thus the struggle. =/

EDIT: 3.2.11 firmware version

EDIT2:
Seeing that the drive has "Firmware v5.0.7", I wasn't able to change the firmware on the drive to anything else.

[root@localhost]# fio-update-iodrive /root/dell/dell_iodrive_3.2.11-20150618.fff
WARNING: DO NOT TURN OFF POWER OR RUN ANY IODRIVE UTILITIES WHILE THE FIRMWARE UPDATE IS IN PROGRESS
  Please wait...this could take a while

 / Updating - fct0: 100% | fct1: 100%
Error: Firmware file does not contain firmware for device 'fct0', part 'xxxxxxxxxxx'.
Error: Firmware file does not contain firmware for device 'fct1', part 'xxxxxxxxxxx'.
[root@localhost]#

 

[T_Minus]

I'm only familiar because I just did this a week or so ago too, and there's a thread on it here.

I'll look for it and link it here if you can't find it before I post

 

[warlockedyou]

I'm only familiar because I just did this a week or so ago too, and there's a thread on it here.

I'll look for it and link it here if you can't find it before I post

This is the only one I found so far.
Fusion-io ioDrive II - 1.2TB+ drives <0.50/GB