Enterprise SSD "small deals"

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
M

miraculix

Active Member
Mar 6, 2015
154
52
28
MSameer said:
They are but they consume a LOT of power.
Click to expand...
Just to be clear... a lot more power compared to SATA SSDs of similar capacity, or compared to NVMe drives of similar capacity?

It doesn't show power consumption by capacity, but the old Samsung data sheet reports 11W active / 4.5W idle. I can only speak to my own limited experience, but that's not far off from my Intel/Solidigm D7-5600s and P4610s.

Having said that... still tempting but I'd rather hold out for higher capacity NVMe drives and make better use of the PCIe lanes on the extra motherboards I have in mind.
 
jhartbarger

jhartbarger

Member
Jan 5, 2022
69
79
18
miraculix said:
Just to be clear... a lot more power compared to SATA SSDs of similar capacity, or compared to NVMe drives of similar capacity?

It doesn't show power consumption by capacity, but the old Samsung data sheet reports 11W active / 4.5W idle. I can only speak to my own limited experience, but that's not far off from my Intel/Solidigm D7-5600s and P4610s.

Having said that... still tempting but I'd rather hold out for higher capacity NVMe drives and make better use of the PCIe lanes on the extra motherboards I have in mind.
Click to expand...
Yes when compared to SATA, you are also correct that power usage between SAS and NVME is similar. The advantage to SAS is that its twice as fast as SATA but draws twice the power, other advantages are density compared to nvme (unless your platform has tons of PCIe lanes) and cost when compared to sata when you get past 3.84TB drive sizes.

Go compare 1DWPD SATA, SAS, and NVME 7.68TB drive prices and the picture becomes clear as to "why" one would choose SAS drives.
 
  • Like
Reactions: blakwolf, miraculix, b3rrytech and 4 others
L

luckylinux

Well-Known Member
Mar 18, 2012
1,441
438
83
ca3y6 said:
Agree. To me they only make sense if either you need the higher performance but don't have the PCIe lanes to go U.2; or you need drives larger than 7.64TB (max size of SATA enterprise SSDs).
Click to expand...
It cannot be that it consumes more Power than NVME ...

The P4608 I just received claim up to 44W of Power at full Load. I still need to test them though. I guess/fear that ASPM is not going to work.
 
C

ca3y6

Well-Known Member
Apr 3, 2021
657
640
93
You are taking a bit of an extreme. A typical samsung SAS SSD will idle at 5w and consume 12w full, 18w for SAS4. Samsung U.2 will idle at 7w and full power are 20-25w. So consumes a bit more than SAS, but in the same order of magnitude vs SATA. SATA typically idles around 1w and full load 3-4w. And that's for enterprise SSD. Retail SATA is way less.
 
Last edited:
  • Haha
Reactions: luckylinux
L

luckylinux

Well-Known Member
Mar 18, 2012
1,441
438
83
ca3y6 said:
You are taking a bit of an extreme. A typical samsung SAS SSD will idle at 5w and consume 12w full, 18w for SAS4. Samsung U.2 will idle at 7w and full power are 20-25w. So consumes a bit more than SAS, but in the same order of magnitude vs SATA.
Click to expand...
I'd be interested to see how much it Idles or "almost" Idles the P4608 I mean.

I still need to make up my Mind about which RAID/Mirror Configuration to use since it's 2 Drives in 1. I would guess 2x2 Mirror or possible RAIDZ-2 with 2 of these ? Or maybe just a single Drive Striped + a HDD for automatic Backup on the same Machine ?
 
L

luckylinux

Well-Known Member
Mar 18, 2012
1,441
438
83
ca3y6 said:
per specs, I think P4608 idles at 5w
Click to expand...
Do you trust Datasheets :D ?

It all depends what exactly is meant by "Idle" and "Load". I personally wouldn't expect much power Draw for an Idle System that is only writing Systemd Logs to Disk at a Rate of say 5MB/s but let's see.
 
E

ericloewe

Active Member
Apr 24, 2017
344
167
43
32
The P4608 is far from representative, because it's actually two SSDs and a PCIe switch in a trench coat. The latter alone probably probably needs 5-7W just to keep the links going at full speed.
The bad old days when all the relevant platforms did support bifurcation but vendors had not yet flipped the bit in the setup menu to show the bifurcation options to end-users...
 
Last edited:
  • Like
Reactions: blakwolf, luckylinux and nexox
F

foureight84

Well-Known Member
Jun 26, 2018
371
309
63
Just bought a 6 (2.5) bay enclosure for my salvaged Precision 5820. Hopefully more SATA drives will popup.
 
Last edited:
L

luckylinux

Well-Known Member
Mar 18, 2012
1,441
438
83
  • Like
Reactions: MSameer
L

luckylinux

Well-Known Member
Mar 18, 2012
1,441
438
83
So ... I just quickly tested the 3 x Intel / Oracle P4608 I got for around 140 EUR each.

They all came with running Firmware QDV1RF35, while on the Label it was QDV1RD28.

Idle Power Consumption, based on the Supermicro PSU I2C Reporting, went from around 78W to around 110W. So around 10W Idle each.

The Good News is that these old Devices seem to be able to support ASPM:
Code: 
88:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] (prog-if 02 [NVM Express])
    Subsystem: Oracle/SUN NVMe PCIe 3.0 SSD 6.4TB AIC (P4608)
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 32 bytes
    Interrupt: pin A routed to IRQ 48
    NUMA node: 1
    IOMMU group: 19
    Region 0: Memory at fb910000 (64-bit, non-prefetchable) [size=16K]
    Expansion ROM at fb900000 [disabled] [size=64K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI-X: Enable+ Count=129 Masked-
        Vector table: BAR=0 offset=00002000
        PBA: BAR=0 offset=00003000
    Capabilities: [60] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <64us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25W
        DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <64ns
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM L0s Enabled; RCB 64 bytes, Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s, Width x4
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range BCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
             EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [150 v1] Virtual Channel
        Caps:    LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:    ArbSelect=Fixed
        Status:    InProgress-
        VC0:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:    Enable+ ID=0 ArbSelect=Fixed TC/VC=01
            Status:    NegoPending- InProgress-
    Capabilities: [180 v1] Power Budgeting <?>
    Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap:    MFVC- ACS-, Next Function: 0
        ARICtl:    MFVC- ACS-, Function Group: 0
    Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-50-49-f9-f1
    Capabilities: [2a0 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [2d0 v1] Latency Tolerance Reporting
        Max snoop latency: 0ns
        Max no snoop latency: 0ns
    Capabilities: [310 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
        L1SubCtl2:
    Kernel driver in use: nvme
    Kernel modules: nvme
Obviously I have some other Issues with the Supermicro BIOS on the Supermicro X10DRi Motherboard, so ASPM is NOT working, in the best case I just need to upgrade BIOS & BMC Firmware, otherwise I'll need to patch FDT ACPI Tables as I already did in the Past, as indicated by lspci -vvv:
Code: 
00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 01)
    Subsystem: Super Micro Computer Inc Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 0
    NUMA node: 0
    IOMMU group: 84
    Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00
        DevCap:    MaxPayload 128 bytes, PhantFunc 0
            ExtTag- RBE+
        DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 128 bytes, MaxReadReq 128 bytes
        DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
            ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed unknown, Width x0
            TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        RootCap: CRSVisible-
        RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
        RootSta: PME ReqID 0000, PMEStatus- PMEPending-
        DevCap2: Completion Timeout: Range BCD, TimeoutDis+ NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- LN System CLS Not Supported, TPHComp+ ExtTPHComp- ARIFwd-
             AtomicOpsCap: Routing- 32bit+ 64bit+ 128bitCAS+
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd-
             AtomicOpsCtl: ReqEn- EgressBlck-
        LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [e0] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
    Capabilities: [144 v1] Vendor Specific Information: ID=0004 Rev=1 Len=03c <?>
And dmesg:
Code: 
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.12.22+bpo-amd64 root=UUID=afadc67e-4b3e-40e8-bf24-9b1c9a25f39a ro acpi.debug_layer=0x2 acpi.debug_level=0xffffffff pcie_aspm=force quiet pci=realloc=off mpt3sas.msix_disable=1 acpi.debug_layer=0x2 acpi.debug_level=0xffffffff pcie_aspm=force
[    0.029713] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.12.22+bpo-amd64 root=UUID=afadc67e-4b3e-40e8-bf24-9b1c9a25f39a ro acpi.debug_layer=0x2 acpi.debug_level=0xffffffff pcie_aspm=force quiet pci=realloc=off mpt3sas.msix_disable=1 acpi.debug_layer=0x2 acpi.debug_level=0xffffffff pcie_aspm=force
[    0.029810] PCIe ASPM is forcibly enabled
[    0.029902] PCIe ASPM is forcibly enabled
[    0.718283] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.818764] acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.821404] acpi PNP0A03:02: FADT indicates ASPM is unsupported, using BIOS configuration
[    0.827852] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.829228] acpi PNP0A03:03: FADT indicates ASPM is unsupported, using BIOS configuration
[    0.846275] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.847150] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
[    0.870725] pci 0000:00:03.2: ASPM: current common clock configuration is inconsistent, reconfiguring
[    0.879224] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.880171] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
EDIT 1: I actually forgot to mention the most Important Part :D . BOTH 3.2TB "Drives" on each of the 3 Physical Drives were Detected (6x3.2TB Total) :).

I guess the best might be a RAIDZ1 across the 3 Drives on the 1st 3.2TB Device striped together with another RAIDZ1 across the 3 Drives on the 2nd 3.2TB Device.
 
Last edited:
  • Like
Reactions: slow.oar7486, onose, b3rrytech and 2 others
W

whoknew123

Member
Jun 21, 2025
30
18
8
Does anyone happen to have/willing to share a list of drive models/capacity and what you consider a good deal?

I’m only looking for 2-3 drives so I don’t mind paying a 5-10% premium if it means I have them in hand sooner. With all the options out there it’d take me ages to put together a list.
 
L

luckylinux

Well-Known Member
Mar 18, 2012
1,441
438
83
whoknew123 said:
Does anyone happen to have/willing to share a list of drive models/capacity and what you consider a good deal?

I’m only looking for 2-3 drives so I don’t mind paying a 5-10% premium if it means I have them in hand sooner. With all the options out there it’d take me ages to put together a list.
Click to expand...
Except that it's NOT only 5-10% different between a Great Deal, a Good Deal and a "Normal" (or normal-ish) Price. Typically it's around 100% Difference, possibly even 150-200% :p .

It depends also where you Live, if you include VAT or not (if you live in e.g. the EU), which Drive Technology (SATA, SAS, M.2, U.2, PCIe) and which Capacity you are looking for.

The best Deals I had so far were (I mean, not really a secret, most were announced in this Thread, plus the Netapp X357A from another Thread in this Section of the Forum):
- Intel P4608 6.4TB PCIe (Oracle branded): approx. 140 EUR/piece including Shipping & including VAT
- Intel P4500 4.0TB U.2: approx. 103 USD / piece + Shipping + VAT
- Micron 7300 MAX 3.2TB U.2 including Adapter: approx. 70 EUR/piece including Shipping & including VAT
- TOSHIBA PX05SRB384 (Netapp branded X357A) SAS: approx. 82 USD / piece + Shipping + VAT

NOT that great, but this is what I bought before the Thread started, this is already what I would consider "reasonable" (half the Price compared to used List Price in eBay Germany):
- Samsung PM983 MZQLB1T9HAJR-00007 1.92TB U.2: approx. 90 EUR / Piece including Shipping & including VAT
- Samsung PM983 960GB M.2: approx. 50 EUR / Piece including Shipping & including VAT

EDIT 1: of course, some of the Prices needed to be Negotiated (send an Offer). Problem of course being that the "hotter" the Deal is, the more likely somebody else will "snatch" it while you wait for your Offer to be approved. So some of them you take the "Risk" send an Offer an Wait, some of them you Buy as listed Price :D .
 
Last edited:
  • Like
Reactions: slow.oar7486
You must log in or register to reply here.
Top Bottom