EXPIRED New Samsung PM983 7.68TB 2.5" U.2 <$490 (edit: All gone)

Almighty

Active Member
Oct 27, 2019
238
160
43
It seems like I'm hitting 16Gbps ceiling transfer rate with these drives. Should be a lot faster. Could be custom firmware limited? FW EDB56W0Q. There might be other more plausible reasons too but it will save me a lot of trouble if someone can confirm faster transfers on these.
How are they hooked up? Any other u.2 drives to test in their place?

Mine come in tomorrow and I'm hooking them into a PCIe 3.0 x4 slot for each one using a u.2 to PCIe adapter from 10Gtek.
 
Last edited:
  • Like
Reactions: Samir

jei

Member
Aug 8, 2021
35
28
18
Finland
Supermicro X11SPM-TPF (Gen3 4x4x4x4 bifurcation) -> Supermicro AOC-SLG3-4E4T -> Oculink -> ICY DOCK ToughArmor MB699VP-B V3

No other U.2 drives available. I tested now with Samsung 980 Pro NVME M.2 boot drive and it's also hitting 16Gbps so hmm :confused:

I've been testing with
Code:
dd if=/dev/nvm0ns1 of=/dev/null bs=1M status=progress
Also RAIDZ-2 with 8 U.2 drives seem to be hitting the same bottleneck.
 
  • Like
Reactions: Samir

Almighty

Active Member
Oct 27, 2019
238
160
43
Supermicro X11SPM-TPF (Gen3 4x4x4x4 bifurcation) -> Supermicro AOC-SLG3-4E4T -> Oculink -> ICY DOCK ToughArmor MB699VP-B V3

No other U.2 drives available. I tested now with Samsung 980 Pro NVME M.2 boot drive and it's also hitting 16Gbps so hmm :confused:

I've been testing with
Code:
dd if=/dev/nvm0ns1 of=/dev/null bs=1M status=progress
Also RAIDZ-2 with 8 U.2 drives seem to be hitting the same bottleneck.
I forget exact commands, but I think you can use lspci to check on actual link speeds of the devices connected.
 
  • Like
Reactions: Samir

jei

Member
Aug 8, 2021
35
28
18
Finland
Everything seems to be ok here:

lspci -nvv
Code:
67:00.0 0108: 144d:a808 (prog-if 02 [NVM Express])
    Subsystem: 144d:a801
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 255
    Region 0: Memory at e0c10000 (64-bit, non-prefetchable)
    Expansion ROM at e0c00000 [disabled]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl:    CorrErr- NonFatalErr- FatalErr+ UnsupReq-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 8GT/s, Width x4, ASPM not supported
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s (ok), Width x4 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
             EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
        Vector table: BAR=0 offset=00003000
        PBA: BAR=0 offset=00002000
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
    Capabilities: [158 v1] Power Budgeting <?>
    Capabilities: [168 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [188 v1] Latency Tolerance Reporting
        Max snoop latency: 0ns
        Max no snoop latency: 0ns
    Capabilities: [190 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
        L1SubCtl2:
 
  • Like
Reactions: Samir and Almighty

Almighty

Active Member
Oct 27, 2019
238
160
43
No transfer issues here.

Code:
> sudo dd if=/dev/nvme2n1 of=/dev/null bs=128k status=progress
61436723200 bytes (61 GB, 57 GiB) copied, 17 s, 3.6 GB/s^C
481394+0 records in
481393+0 records out
63097143296 bytes (63 GB, 59 GiB) copied, 18.8724 s, 3.3 GB/s

I did notice these are both logical and physical 512b block size drives - would have assumed at least 4k physical? Any issues with that?
smartctl -x /dev/nvme2n1 revealed the 4k physical sector size - for whatever reason fdisk/parted (using ioctl?) show 512/512.
 
Last edited:
  • Like
Reactions: Samir

jei

Member
Aug 8, 2021
35
28
18
Finland
Thanks. Oof.. I did format them to 4k before doing any further tests. Because Samsung 980 Pro is showing pretty much the same speed cap maybe it's not the sector size tho. I tried to adjust CPU performance settings, disable power saving completely, etc, all it did was go slower! BIOS "Power saving" = 1800MB/s~, "Maximum Performance" = 1400MB/s. o_O

Code:
>nvmecontrol identify nvme1ns1|grep LBA
Number of LBA Formats:       2
Current LBA Format:          LBA Format #01
LBA Format #00: Data Size:   512  Metadata Size:     0  Performance: Best
LBA Format #01: Data Size:  4096  Metadata Size:     0  Performance: Best
edit: Booted into Ubuntu Live (FreeBSD based TrueNAS 13 normally), got about 2.5GB/s. So.. it's over the 16Gbps limit atleast.
 
Last edited:
  • Like
Reactions: Samir

TonyArrr

Member
Sep 22, 2021
40
14
8
Straylia
Just curious... when they say these are "NVMe SAS U.2", is that just filler to make them appear in more searches, or do they actually work as SAS drives?

Cause I've been looking for SAS drives this big and they are all twice this price, while NVMes all seem much cheaper per gig. So if there are combo SAS/NVMe drives out there and they are on the lower NVMe side of pricing, this makes my day :p
 
  • Like
Reactions: Samir

sparx

Active Member
Jul 16, 2015
317
114
43
Sweden
I think thats just some typo from the ebay seller. No such thing as a NVMe and SAS drive at the same time. And I'd love to be proven wrong! :D
 
  • Like
Reactions: Samir

TonyArrr

Member
Sep 22, 2021
40
14
8
Straylia
I think thats just some typo from the ebay seller. No such thing as a NVMe and SAS drive at the same time. And I'd love to be proven wrong! :D
So would I, it’d be a great bridging option in my next upgrades to my array could be carried over to a new server in a few years.

What got me was when I searched the model number there were a few different sites listing it the same way. Clearly we have some copy pasta going on here
 
  • Like
Reactions: Samir

jei

Member
Aug 8, 2021
35
28
18
Finland
No transfer issues here.

Code:
> sudo dd if=/dev/nvme2n1 of=/dev/null bs=128k status=progress
61436723200 bytes (61 GB, 57 GiB) copied, 17 s, 3.6 GB/s^C
481394+0 records in
481393+0 records out
63097143296 bytes (63 GB, 59 GiB) copied, 18.8724 s, 3.3 GB/s
Well.. I got the H12SSL-i and EPYC 7302. There might be some power management foolery going on yet and only 4/8 memory channels populated atm, but it's now hitting about 3GB/s. Everything else is a lot faster too. Maybe the 2nd Gen Intel had one too many mitigations applied.. (there are dozens of mitigations in the BIOS changelog over the years).
 
  • Like
Reactions: Samir