1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Intel Xeon D-1500 Series Discussion

Discussion in 'Processors and Motherboards' started by Patrick, Mar 9, 2015.

  1. mackle

    mackle Member

    Joined:
    Nov 13, 2013
    Messages:
    153
    Likes Received:
    22
    I'm interested in getting an '08 for a 12TB (4x4TB Z1) FreeNAS setup - my only concern is whether it'll bottleneck on 10GBps links.

    Does anyone have any experience?
     
    #1341
  2. jgreco

    jgreco New Member

    Joined:
    Sep 7, 2013
    Messages:
    28
    Likes Received:
    14
    Putting on my FreeNAS cap:

    1) RAIDZ1 is not recommended unless the data is not particularly important; RAIDZ1 loses redundancy in a drive loss scenario, and any URE's of pool data are unrepairable unless maybe they happen to be metadata.

    2) 10Gbps with FreeNAS is a little dicey even on a well-resourced (read: MUCH larger) platform, and one needs to be very careful about defining things. It is possible to get over a Gbps out of a small setup. On the other hand, I can make a large setup crawl at a few Mbps by applying pathological traffic patterns (think, especially, random seeking of small blocks). Defining the use case is critical.

    2a) Your underlying pool is incapable of 10Gbps. Today's fast drives can sustain perhaps 225MB/sec, three of those pounding out sequential data at top speeds would be 775MBytes/sec, or 0.775GBytes/sec, something shy of 10Gbps, and experience suggests that even the 775MB/sec is extremely wishful thinking.

    3) To a certain extent, an environment with parallelism is beneficial in that you are more likely to be able to hit your pool's actual I/O limits. It is important to define what sort of protocol you are using to access the filer, and what sort of concurrency there might be.

    I'd bet you'd see two to four Gbit/sec for ISO or other large sequential file storage as long as you kept occupancy and fragmentation down to reasonable numbers (80%/20%), tuned for large TCP buffers on both sides, and used a Chelsio T420 or T520 on the FreeNAS side, along with a well-designed machine and OS on the client side. I would be pleased to be underestimating that. :)
     
    #1342
  3. J--

    J-- Active Member

    Joined:
    Aug 13, 2016
    Messages:
    178
    Likes Received:
    40
    Any datapoints on the max IOPS and throughput if you were to populate all of the onboard SATA?
     
    #1343
  4. jgreco

    jgreco New Member

    Joined:
    Sep 7, 2013
    Messages:
    28
    Likes Received:
    14
    Sorry, no, that's kind of like asking "how fast could my car go."

    Performance on a low clock ZFS platform will always be somewhat limited, so I can generally say anything built on the Pentium D-1508 will be towards the slower side, but filling all your ports with a bunch of 4TB 2.5" laptop HDD's in a big RAIDZ3 is going to be very slow compared to filling all your ports with a bunch of 850 Pro's in mirrors, which will be expensive but definitely fairly zippy as far as these things go, and the IOPS potential of the SSD platform could easily be a hundred times greater if built correctly.

    The D-1508 is very challenging for a software storage platform like ZFS where the parity computations are being done by the CPU. With only two cores on the 1508, you can be burning one doing RAIDZ parity and burning the other doing Samba. Even if you're optimistic and looking at that as threads, four threads at 2.2GHz isn't that much. Getting rid of the parity calculations and other overhead by moving to mirrors and increasing potential random IOPS by going to SSD means that you can have more parallelism in the system. Having multiple clients accessing the system simultaneously is more likely to result in an optimal situation where you're making full use of the potential of the NAS.
     
    #1344
    SSS likes this.
  5. evancox10

    evancox10 New Member

    Joined:
    Nov 12, 2015
    Messages:
    8
    Likes Received:
    2

    I think that the max SATA throughput question is valid. There could be a bottle-neck between the "South Bridge" in the Xeon-D SiP, where the SATA ports hang off of, and the main die, such that 6 SSD's going full bore could saturate the link. Six SATA3 ports have an effective transfer rate of 28.8 Gbps (have to take of 20% due to 8b/10b encoding), which is about equivalent to 3.65 PCIe Gen3 lanes (~7.88 Gbps/lane). I know a lot of the consumer motherboards with PCIe M.2 slots connected off the SB were running into bandwidth limits because the connection to the CPU was basically only 4x PCIe Gen3.

    But in general, you need more context, specific workloads, specific druves, etc., before you can answer that.
     
    #1345
  6. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    9,301
    Likes Received:
    2,860
    @evancox10 the difference between the M.2 on consumer motherboards is that the Xeon D SATA ports are basically through an on-die PCH.
     
    #1346
  7. jgreco

    jgreco New Member

    Joined:
    Sep 7, 2013
    Messages:
    28
    Likes Received:
    14
    Not to put too fine a point on it, but in the context of the question being asked, why would you care? If you're building a NAS, you're going to be limited by the network speeds. Even if you managed to get two 10Gbps connections with LACP and a well-balanced high traffic workload, there's no chance you're going to get "6 SSD's going full bore". Plus there isn't the CPU oomph to do any sort of significant software defined storage (ZFS etc) at those speeds, and using simpler technology (ext3/FFS without all the overhead, and NFS or Samba to serve) will generally start to bottleneck well before the 20Gbps, so you're not even really being limited by the network speeds, but by the overall platform's inappropriateness to the task.

    That's not to say that it isn't a mildly interesting question, but in the larger picture you probably need to at least move upwards on cores for it to even begin to matter. Even the midrange Xeon D's are I/O-rich, CPU-shy parts. It's easy to build contention scenarios especially if you "go wide" by taking something like the 2116 on one of the X10SDV storage boards:

    x10sdv-block-diagram.png

    Fine so there's 16 6Gbps ports there and if you were to SSD them all, theoretically you have around ~8GBytes/sec of storage crossing ~4GBytes/sec of PCIe, but can you actually generate even that much traffic in a meaningful scenario (meaning via processing or file service, not dd /dev/zero)?

    If you really want to be able to max out your peripheral devices, the Xeon D probably isn't the right choice. These things are aimed more at being "the new Atom" and at that they do some jaw-dropping-ly amazing stuff - for an Atom/Avoton replacement.

    I really like the Xeon D as a pragmatic choice for deployments where you don't need dozens of cores and all the drama, but it is still a limited platform, and a lot of that limit is in the form of the CPU itself, and most of the rest is in the form of the limited expansion options. For the things you're likely to use this platform for, you're probably going to smash into CPU as being an issue before any realistic chance of hitting I/O bottlenecks.
     
    #1347
    evancox10 likes this.
  8. evancox10

    evancox10 New Member

    Joined:
    Nov 12, 2015
    Messages:
    8
    Likes Received:
    2

    I agree with all your points; it's an academic question for a NAS. However I think you missed that you were responding to two completely different users. j--, who asked about using all 6 SATA ports, is not running a NAS application, but rather some database/VM, with a CPU much beefier than the 1508. (He/she talked about it in another thread)
     
    #1348
  9. J--

    J-- Active Member

    Joined:
    Aug 13, 2016
    Messages:
    178
    Likes Received:
    40
    Ha, thanks for reading my other thread. :)

    I was just asking for theoretical speeds to try to judge what the PCH is capable of, irrespective of CPU. I wouldn't use this for a DB server; that would be a bit ambitious.
     
    #1349
  10. evancox10

    evancox10 New Member

    Joined:
    Nov 12, 2015
    Messages:
    8
    Likes Received:
    2
    I think the SATA, Gen 2 PCIe, and USB ports are on a separate die integrated into the package. Source:

    Intel gives the ‘v3’ treatment to new 14 nm Xeon D processors

    Edit: Point being, there's still some interconnect, and if all they did was put an existing die in the package it's going to be the same interconnect as previously.
     
    #1350
    Last edited: Sep 28, 2016
  11. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    9,301
    Likes Received:
    2,860
    That is a strange article. The block diagram shown and provided by Intel has the PCH integrated in the SoC. The article also makes it sound like the 10GbE is fully integrated. The 10GbE ports do not have PHYs integrated for example so the full solution is not integrated.

    BTW if you want some more of the slides Intel provided - Intel Xeon D SoC - Changing the low end with Broadwell-DE

    ...and more detail on the PCH
    Broadwell-DE / Intel Xeon D-1540 SoC PCH Information

    We did the majority of the early Broadwell-DE reviews/ pieces and made public some of the major bugs in early silicon.

    One thing we have not done is physically opened the package. @William - maybe a fun project de-lidding a package and snapping some photos?

    I do not think you are going to get PCH SATA speed limited.
     
    #1351
    William likes this.
  12. evancox10

    evancox10 New Member

    Joined:
    Nov 12, 2015
    Messages:
    8
    Likes Received:
    2
    Cool, thanks for the info Patrick. Also I'm familiar with your excellent coverage, having read most of it! Thank you for that, by the way :)

    The PCH separate die thing seems to be something they aren't exactly trumpeting. But it's been confirmed, see the post by the author halfway down this comment thread

    Intel's 1st Xeon SoC Twists ARM | EE Times

    Not that it really affects the end user either way, it doesn't make the product any less great! I think I'll be getting Supermicro's mini-tower with the 1541 soon
     
    #1352
    Patrick likes this.
  13. RobertC

    RobertC New Member

    Joined:
    Apr 8, 2016
    Messages:
    1
    Likes Received:
    0
    Has anyone booted this with any of the Solaris clones, like SmartOS, Illumos, or OpenIndiana? I know these do not support USB3 (it is currently in alpha) and am concerned that the USB 2 ports are implemented via the USB3 controller.

    If you HAVE booted this from a flash stick, does the booted OS allow you to use a standard USB keyboard, mouse, and a USB2 audio adapter?

    Thanks in advance!
     
    #1353
  14. Laugh|nGMan

    Laugh|nGMan New Member

    Joined:
    Nov 27, 2012
    Messages:
    13
    Likes Received:
    3
    Guys! Anyone have info for Broadwell-DE with Windows server 2016/windows 10:
    1. idle watts with Balanced power plan and idle cpu frequency?
    2. idle watts with Performance power plan and idle cpu frequency?

    Just curios, how it compares with Sandy/IvyBridge ATX size 6-8core platforms. I still don't see any point to upgrade. Broadwell-DE cpu unswappable, pair PCI-E slots for expansion, pricey. Somehow i feels with Broadwell-DE my hands are tied, no room to expand.
     
    #1354
  15. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    9,301
    Likes Received:
    2,860
    This just made my Google Keep list. I may use WS2016 Essentials on a few machines.

    I do think with NVMe pricing becoming affordable that if you are thinking of building a larger platform, the E5 V4 route is the way to go. More RAM expansion, more PCIe lanes, more SATA 3.0 lanes and etc.

    At the low end, (if you never want more than 128GB/ 8 cores) I think the Xeon D makes a lot of sense. If you are power constrained (1U 1A 110V hosting) then the Xeon D is a category killer.

    With Windows Server 2016 licensing moving to a per-core model, higher clock speed E5 V4's are going to be preferable over Xeon D save in the WS2016 Essentials role.
     
    #1355
  16. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    9,301
    Likes Received:
    2,860
  17. evancox10

    evancox10 New Member

    Joined:
    Nov 12, 2015
    Messages:
    8
    Likes Received:
    2
    Hey, so I just noticed here that the latest BIOS update (1.1c) for the SuperMicro X10SDV platform contains this:
    Anyone know what the available options are? Sounds like you may not need a PLX adapter now, if the chipset supports bifurcation. Even just two M.2 would be nice.
     
    #1357
  18. pyro_

    pyro_ Active Member

    Joined:
    Oct 4, 2013
    Messages:
    613
    Likes Received:
    138
    Wonder if they are adding it to add support for their dual M2 add in card
     
    #1358
  19. dave

    dave Active Member

    Joined:
    Apr 2, 2015
    Messages:
    162
    Likes Received:
    33
    Came across your helpful post when searching for Samsung DDR4-related naming info, and your DDR3 link led me to Samsung's DDR4 naming explanation FYI: http://www.samsung.com/semiconductor/global/file/product/DDR4_Product_guide_May15.pdf
     
    #1359
  20. WenleZ

    WenleZ New Member

    Joined:
    Aug 9, 2016
    Messages:
    2
    Likes Received:
    1
    YEs, you can passthrough the onboard Intel SATA controller. You must either install the ESXi on a USB drive, or on a NVMe SSD.
    What you need to do is:
    add the following in /etc/vmware/passthrough.map ( or passthru.map in ESX6.5 ):

    # INTEL Lynx Point AHCI
    8086 8c02 d3d0 false
     
    #1360
Similar Threads: Intel Xeon
Forum Title Date
Processors and Motherboards Intel’s Skylake Xeon E5 2699 v5 32 Cores Leaked Benchmark Mar 25, 2017
Processors and Motherboards Intel Xeon Gold Series 18 core chips Feb 12, 2017
Processors and Motherboards Intel Xeon engineering sample: Should I get one? Jan 10, 2017
Processors and Motherboards Intel Xeon D-1541 or Xeon E5-2630L V3 What wins? Jan 9, 2017
Processors and Motherboards Curiosity - Intel Xeon Phi Processor Nov 14, 2016

Share This Page