Help with motherboard and CPU for new FreeNAS build

[zack$]

You only need a GPU for plex if you are transcoding your media..

IPMI is a life saver, I would not buy a MB unless it has it. You will find out soon.

You don't need a 1TB drive to boot, it's gonna be wasted space with ESXI. You don't need anything bigger than 16GB to boot. Further, you should get a drive that is qualified with your motherboard, as there are sometimes issues with booting from an unqualified drive: M.2 List | Supermicro

You will also need a drive to store your TrueNAS VM so that it can boot up and then offer shares to your servers (using your hdd or ssd pools). If you are using all the data port and the lsi 3008 for storage in TrueNAS, you will probably need to use a pcie slot or one of the oculink ports to get storage.

If I were you, I would get one of the p3605 in the great deals section and throw that in the x4 slot for local VM storage. That will hold you for local VM storage for a while.

 

[rskoss]

You don't need a 1TB drive to boot, it's gonna be wasted space with ESXI. You don't need anything bigger than 16GB to boot. Further, you should get a drive that is qualified with your motherboard, as there are sometimes issues with booting from an unqualified drive: M.2 List | Supermicro

Well that's a little scary.

This one will be a little less wasteful and at half the cost: Micron M.2 256GB

You will also need a drive to store your TrueNAS VM so that it can boot up and then offer shares to your servers (using your hdd or ssd pools). If you are using all the data port and the lsi 3008 for storage in TrueNAS, you will probably need to use a pcie slot or one of the oculink ports to get storage.

If I were you, I would get one of the p3605 in the great deals section and throw that in the x4 slot for local VM storage. That will hold you for local VM storage for a while.

I'll probably have half a dozen VM's. All smallish - Windows will be the largest for Blue Iris.

This one: 1.6TB p3605 ?

 

[Markess]

I've been following this thread with interest and thought I'd weigh in here from the "I've been pretty much where you are" standpoint. A couple things to consider before you finalize your setup. Smart folks...please jump in if I mess any of this up.

• FreeNAS, TrueNAS and their siblings are based on software designed to handle enterprise levels of data and users while minimizing risk of data loss & downtime even in extreme situations. That doesn't mean they aren't a good choice for small business/home users. But, it does mean with them that the "bar is set pretty high" already. Even a simple system with a "minimal" feature set is going to be pretty darn robust for a home/small business user. Wanting FAST is something separate from features though, and probably more than anything else will dictate the range of hardware that will work for your needs.
• You mentioned "Not many CPU choices with that socket " For Intel, If you haven''t already, take a look at Intel's advanced product search for available products in a given CPU line. It lets you dial in the exact socket and features you're looking for (i.e. not discontinued, core count range, etc.) and gives you a list of CPUs that fit the criteria. Intel product specifications .
• One FYI on the free version of ESXi: If you're the kind of person that sits down with "the manual" and cranks through, its pretty well documented. But, if you think you'll be relying at all on web tutorials, or Googling for troubleshooting answers, then just be aware that much of what you find will be based on the paid product line (with vSphere etc.) and sometimes you'll need to "translate" what you read/see from one product to another. Inevitably, you will find that something you want to do won't be available in "Free", sometimes only after wasting some time trying to make it work. But, the vast majority of features you'd want for a single system are either going to be there, or already have a documented workaround somewhere on the web.

 

[rskoss]

• You mentioned "Not many CPU choices with that socket " For Intel, If you haven''t already, take a look at Intel's advanced product search for available products in a given CPU line. It lets you dial in the exact socket and features you're looking for (i.e. not discontinued, core count range, etc.) and gives you a list of CPUs that fit the criteria. Intel product specifications .

I did not know about that page, but alas, it doesn't cover the socket on this board. I think it's too new.

• One FYI on the free version of ESXi: If you're the kind of person that sits down with "the manual" and cranks through, its pretty well documented. But, if you think you'll be relying at all on web tutorials, or Googling for troubleshooting answers, then just be aware that much of what you find will be based on the paid product line (with vSphere etc.) and sometimes you'll need to "translate" what you read/see from one product to another. Inevitably, you will find that something you want to do won't be available in "Free", sometimes only after wasting some time trying to make it work. But, the vast majority of features you'd want for a single system are either going to be there, or already have a documented workaround somewhere on the web.

Yeah, unfortunately I found that out. Every search on ESXi is all vsphere this and vcenter that.

But, I have to start somewhere. While I want a production system that works well, the reason I'm building this instead of just buying a storage server is because I like learning new things.

 

[Markess]

I did not know about that page, but alas, it doesn't cover the socket on this board. I think it's too new.

The motherboard you listed, X11SPH-NCTF is socket 3647, which is actually a few years old (mid-2017). On their website(s), Intel tends to list list sockets by their full acronyms, in this case FCLGA3647. So a search for "Launched" products with that socket returns over 130 diffrerent CPUs. Here's a link to a search with those filters selected:

Intel product specifications

Looking at Supermicro's motherboard product page, it lists only the original (1st Gen) and 2nd Gen Xeon Scalable (but not 3rd Generation) for Socket 3647, so maybe stick to just the processors on these two pages for best compatability: There's still a lot to choose from:

Xeon Scalable Processors (1st Gen): Intel® Xeon® Scalable Processors Product Specifications
2nd Generation Xeon Scalable: 2nd Generation Intel® Xeon® Scalable Processors Product Specifications

Personally, I'd think that anything in the Socket 3647 Scalable family is overkill for a single user set-up. In your case though, looking for all new parts with a ton of SATA + 10G onboard, your selection is more limited. If nothing else, it will give you a lot of headroom for expansion.

Yeah, unfortunately I found that out. Every search on ESXi is all vsphere this and vcenter that

I wasn't trying to discourage you. I was just mentioning it, so you know going in that the screenshots and command sequences you'll see on tutorials and etc. won't necessarily match what you'll be seeing on your own screen. I know I found a few helpful pages/videos on ESXi "free" when I was using it. Its been a while though, and I don't have links handy, as I recently switched to a mix of Unraid and Linux with QEMU/KVM.

 

[rskoss]

Ah - I started my search with LGA and didn't see the number. Bummer, I just ordered the Silver 4110 yesterday morning. If I had seen this list of processors, I may have chosen one with 12 cores. Why 12? That's how many my next desktop workstation is going to have. But then I saw the prices of these new processors and I think I'm happy with my choice. Or I'm going to learn to be happy with my choice ;-)

 

[Markess]

Yeah, its not an inexpensive ecosystem to buy into. Even used are quite expensive because its still too new for much corporate "retired" hardware to be hitting the used market. Glad you were able to get sorted for now though. Good luck with the build!

 

[rskoss]

All the parts are finally here - power supply came yesterday.

The CPU cooler and CPU placement on the motherboard are much more complicated than the youtube videos I watched on building a PC.

Just keeping anybody interested in the loop.... <wink>

 

[petreza]

I did not know about that page, but alas, it doesn't cover the socket on this board. I think it's too new.



Yeah, unfortunately I found that out. Every search on ESXi is all vsphere this and vcenter that.

But, I have to start somewhere. While I want a production system that works well, the reason I'm building this instead of just buying a storage server is because I like learning new things.

Don't forget the VMUG Advantage recommendation above. It gives you the whole shebang for home lab use for $200/year ($180 if you use a coupon) - limited to 32 cores total.

 

[rskoss]

Hi all,

3 back surgeries later, and I think I figured out where all the wires attach. I turned it on and I got the memory not detected beep code. Well, at least I don't have to wonder if it works - now I know it doesn't.

It appears that I have wasted $676 dollars by buying the wrong memory. I discovered a footnote in the mobo manual that says the 2933 memory that I bought is only supported by 82xx/62xx processors, and I have a Xeon Silver 4110 processor.

So I know what doesn't work, but nowhere can I find any kind of processor/motherboard/memory matrix that tells me what to buy.

The board is a Supermicro X11SPH-NCTF. The processor is an Intel Xeon Silver 4110 Tray Processor 8 Corel 2.1 GHz. Can somebody help me figure out what to buy?

 

[Markess]

Hi all,

3 back surgeries later, and I think I figured out where all the wires attach. I turned it on and I got the memory not detected beep code. Well, at least I don't have to wonder if it works - now I know it doesn't.

It appears that I have wasted $676 dollars by buying the wrong memory. I discovered a footnote in the mobo manual that says the 2933 memory that I bought is only supported by 82xx/62xx processors, and I have a Xeon Silver 4110 processor.

So I know what doesn't work, but nowhere can I find any kind of processor/motherboard/memory matrix that tells me what to buy.

The board is a Supermicro X11SPH-NCTF. The processor is an Intel Xeon Silver 4110 Tray Processor 8 Corel 2.1 GHz. Can somebody help me figure out what to buy?

Hope you're recovering well!

Often, those disclaimers refer to operating speed, and not necessarily that it won't work at all. But, this could be the exception of course.

But, before you give up on it, have you tried starting up with just a single stick in DIMMA1? Also, it would help if you posted back here with the brand and model number of the memory you're using. There may be some issue with DIMM type, rank count, or data width that someone here knows about (not me though, I don't have any Xeon Scalable gear).

 

[rskoss]

Hi,

Thanks for replying.

The wording from the mobo manual is, "Note 1: 2933MHz memory is supported only by the 82xx/62xx series processors."

Is it safe to start it with only a single stick of ram? I thought they always had to be used in pairs and in specific slots.

Here is a link to newegg for the memory I purchased: SuperMicro (M393A8G40MB2-CVF) 64GB 288-Pin DDR4 SDRAM ECC Registered DDR4 2933 (PC4 23400) Server Memory Model MEM-DR464L-SL01-ER29 - Newegg.com

There is a table in the mobo manual for 41xx processors, but I can't figure out what the pieces of the table mean or how they translate to something I can purchase on Amazon/Newegg.

 

[Rand__]

Silver first gen only does 2400, but memory should clock down.

That board is a bit finicky with memory in my experience (and with certain dac's in the NICs), but else its nice (if you can get it run.
The advice with trying with a single module is the best start. Additionally if you can get it to boot, you might want to update the bios.
I dont think the memory you got was available when the board got on the market so it might not be known to the bios.

 

[Rand__]

Its fine to run with 1 module, pair's has been long gone, nowadays its triplets (for best memory bandwith)

Edit: You know on the Supermicro page there is a list of validated memory modules that you can use if you're unsure what to pick ?

 

[Markess]

Here is a link to newegg for the memory I purchased: SuperMicro (M393A8G40MB2-CVF) 64GB 288-Pin DDR4 SDRAM ECC Registered DDR4 2933 (PC4 23400) Server Memory Model MEM-DR464L-SL01-ER29 - Newegg.com

Okay, so I see that this is 64GB 2Rx4 (2Rx4 is also listed as DRx4 some places..like the manual for your motherboard). You have a 4110 CPU, so you need to look at the memory population chart labeled "DDR4 Memory Support for 81xx/61xx/51xx/41xx/31xx Platform" on page 33 of the motherboard manual. If I read that right, RDIMM Dual Rank x 4 (DRx4) only works in Capacities of 16 & 32GB (look in the DIMM Capacity columns).

If you look at the next chart down "DDR4 Memory Support for 82xx/62xx/52xx/42xx/32xx Platform" on page 34, 64GB is listed as compatible. But you don't have this series CPU.

So, it may be that those won't work for you. But, if they don't I, think its because they're 2Rx4 @64GB and not because they are 2933mhz.

Memory is soooo much harder to match with motherboards/CPUs these days than it used to be. For example, your memory IS listed on Supermicro's site as compatible. But with the caveat in small print that its only with Cascade Lake CPUs. You have a Skylake SP CPU. It can be super frustrating if you're just trying to build a single system and don't want to have to memorize Intel's catalog of model numbers and code names.

@Rand__ 's advice is good. Go with the validated memory table, BUT ALSO you need to run down if any of the exceptions apply to you AND consult the matrix in the manual if there is one, for best results.

But, If you find something not on the list that's a great deal though, you can always try googling it along with the motherboard model number (i.e. M393A8G40MB2-CVF X11SPH-nCTF) to see if anyone else has had success....or not...with the combination.

 

[Rand__]

Yeah. Its always a gamble with memory nowadys. Got a couple of 128G RDimm modules working on this board. They'd never make the compatibility list (8R), but they work (most of the time - actually since updating to 3.4 I didnt have any single issue with them, I think128G modules finally were officially added [for higher clocks, but hey, something])

 

[Markess]

Yeah. Its always a gamble with memory nowadys. Got a couple of 128G RDimm modules working on this board. They'd never make the compatibility list (8R), but they work (most of the time - actually since updating to 3.4 I didnt have any single issue with them, I think128G modules finally were officially added [for higher clocks, but hey, something])

When I was getting ready for my most recent build, I ran across a new LGA3647 motherboard (Tyan) on Amazon for ~$80. I considered it, but after a google search and seeing all the pain and anguish people were having with matching memory, torquing down heatsinks, and all the rest, I opted to give 2011-3 one last go, with a new-old-stock motherboard and a Broadwell CPU from one of the "Great Deals" posts.

 

[Rand__]

I phased out my last 2011-3 build last week (for now). vSan runs now on oldish e3's, Gaming/Workstation use a W3200 and Truenas on the -nctpf.
Would have loved to keep that on -3, but no go with NVDimms. But Skylake+ is a PowerHog, so one 24/7 box is enough.

Wise decision to stay on -3

 

[ullbeking]

You mention 4K video editing... Are you planning an SSD pool to feed your 10Gb network connection? If so I'd probably look at SAS over SATA and honestly nvme would probably be the better route - at least for the clips you are actively working on...

I AM trying to implement something like this. I have many SSD's (capacities vary all over the place from 100 GB HGST SLC SAS SSD's to 1.6 Intel DC P3700). In any case I can create consistent sets of 2, 4, 8, 12, etc so they are still very useful.

Could you please say what you have in mind by "SSD pool" and what technologies you would use to implement this?

I'm looking at everything from ZFS, Btrfs, and bcache/fs, to the good old classic: LVM+RAID+XFS. In all cases where multimedia editing might be a use case for that NAS, I plan to install LOADS of RAM. But I feel like this is oversimplified or that I've missed something.

Could you please tell me how you would implement an SSD storage pool to make multimedia editing fast, or by inserting such a structure between a "regular" NAS and a 10 GbE connection or directly to the machine that is running the editing application? Such information would be immensely useful for me. Thank you so much!!

 

[rskoss]

Edit: You know on the Supermicro page there is a list of validated memory modules that you can use if you're unsure what to pick ?

Yep, saw that. Where I went astray was not seeing the footnote in the manual about what works with my processor. The Supermicro page only shows what works with their board. Remember, this is all new to me.

Now that I picked the right speed, I see the part number for this memory on Amazon: Samsung M393A4K40BB2-CTD 32GB DDR4-2666 LP ECC REG DIMM at Amazon.com

Can you (or anybody) please take a look at that and give me a, "Yup, that should work." Not that when I click on the "Make sure this fits" link on the product page, my board isn't listed.

Should I get a single stick to see if that will get me to the bios, or go for the gold and buy 4 sticks?