No a format is not the same thing. Badblocks test reads and writes for every sector, and verifies that each read is correct. All this should also happen in a time frame that is similar for drives of that size.
Help a ZFS Noob select SAS drives for ZFS pool?
- Thread starter Breezy2428
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
If you already have 3-2-1 right?
And honestly from my perspective I'd much rather blow the drive before its in service. I think a lot of folks simply balk at the wall clock time to do the burn-in and so eh "I'll just deal with it". personally I use this as good practice so I maintain that good practice in my day job and pass that along to the folks that I mentor (gods help them all
).However if you are scratch building and have no pre-existing storage resources that are currently keeping you going during the burn-in then I can see your point.
your 846 also may have internal studs on the power supply wall so you can attach drive sleds to if you don't care about hot swap. If it does then your 846 is STUD-LY.
Use dummys as @mattventura says. Your 846 may well be a studly dummy too.
When sparse populating the bays I like to place the drives in patterns that while blinking create a cosmic sense of calmness. When I do that I think that I have an 846 from Marin county...
If I have a cosmic studly dummy then I name it lloyd, the backup server is then named Harry...
I could keep going on and on and on...
This is suddenly quite relevant, I am hoping someone can help guide me here. I will apologize in advance her for the wall of text. Succinct writing eludes me.
So I was not specifically seeing the 2TB Intel NVME m.2 in the bios, from previews threads there can be problems with NVME and the X9 boards, depending on exactly which board and bios version but it was not very clear what my outcome would be. there is a generic "Hard Drive" listing in the boot order section I was hoping that was it, Last night I booted to Debian 12 live USB. the live session loaded drivers for the M.2 NVME drive (only drive in the system besides the USB drive) and installed to it just fine without error.
removed the USB, Reboot = no drive present,
board came with BIOS 3.3 I had already updated to what supermirco's website calls 3.48 but shows as 3.4 in the bios itself ( X9DR3P9.B20 ) according to older posts as a work around one can use windows to modify the BIOS file to add in the NVME driver, this really feels like the edge of my abilities, that is exactly where one learns but usually by making a giant mess and hours of pain figuring out how to fix said mess. plus if there are any issues down the road mentally it will be a troubleshooting distraction, Is it the modified Bios? and for a third strike some report slow speeds and lost connections after getting it to "work".So fall back from M.2 NVME to 2.5" SATA (or SAS?) SSD it is.
But I have an issue, most of the talk about 846 2.5" drive compatibility revolves around "B" & "E" models, the sticker starboard side of mine just says
P/N: CSE-846
FACTORY CODE: ABC-02
with no suffix letter codes...
is that a B model or an E model??
The internal fixed x2 2.5 mount MCP-220-84603-0N is cheaper $28 https://www.amazon.com/Supermicro-MCP-220-84603-0N-Dual-Fixed-Tray/dp/B004I8YFYC and would be SATA only.
Picture here of one installed in an 846
Project: Infrastructure – Need Not to Know
Posts about Project: Infrastructure written by Zac
zacfu.com
Note the 846 in that photo has 4 standoffs making two install positions along the top edge of the PSU case
my 846 instead has three stand offs there,
From the front stand off to the middle of the three is 4 5/8" center to center, matching the forward position he did not use in the picture above, below this front set in the floor are 2 small conical studs. the studs are laid out with the same center line as the stand offs above but I am not exactly sure if this works with the drive case, the drive cage appears to instead use rubber feet on a flat surface.
My motherboard, X9DRI-LN4F + with its dual sockets and obscene number of DIMM slots is very wide, some say the internal x2 2.5" fixed cage does not work with the extra wide boards. looking at the picture above referencing the motherboard standoffs it looks like the drive mount would interfere with the power connectors at the edge of the motherboard. so I think the internal route is out sans modifications.
The nicer but more expensive option is the x2 external hot swap bays,MCP-220-84610-0N, this appears to have the advantage (or does it?) of also having a miniature SAS backplane for SAS or SATA 2.5" drives.
But will it fit?
I do have a blanking plate next to the PSU that I believe was originally intended for an optical drive, and the features of my 846 appear to match and the needed standoffs from the installation manual https://www.supermicro.com/manuals/chassis/drive_kit/MCP-220-84610-0N.pdf. but my motherboard comes right up to the edge of this optical slot giving me concerns it will interfere.
Second question on the hot swap bay, Its says SAS/SATA but all the connections are SATA plus a SGPIO. my motherboard does have 2 T-SGPIO headers, does that SGPIO make connections allow SATA headers to run SAS drives? I have an extra 4 port SAS connection on my motherboard but it is labeled SCU SAS, and does not appear to work with this hot swap bay as it is SFF-8087
Attachments
Warning - I'm opinionated - - yikes I wrote a really long reply. hopefully understandable with some funnies as this is a pretty dry topic.
Just split off power for it and run your SATA cables.
Based on the age of that Motherboard and the pictures you sent you probably have an older SAS2 OEM 846. FWIW you can always go to the supermicro e-store site use the chat - in general I've had very positive experiences figuring out what I have and what options I can get for chassis. If they can't help you directly they usually have an answer within a few hours. Also depending on the item sometimes its actually "cheaper" per item to buy from SM e-store than e-bay. However SM e-store charges full rate on shipping and that can make a single item too expensive. Seller TM_SPACE on the bay takes advantage of that fact in their SM markups. Pretty sure they have a direct line in to the SM logistics chain and can get some things that you cannot purchase through th e-store.
end of the day velcro works and a cheap low cost boot drive (used or new) is not going to use a lot of power or produce a lot of heat. it should be fine as long as the velcro adhesive doesn't wear out!
Use the chat function at the SM e-store to try and find out what you ahve and what parts will work with the chassis you have. If you can find a S/N and its not from an reseller/OEM but a true SM sold system then the e-store can probably look it up.
NO. SGPIO provides out of band (not in the SATA or SAS channel) control/mgmt functions to whathever backplane you have. Example if you just velcro a drive and connect a sata cable you have to backplane to control and no need for an SGPIO.
TBC using SGPIO does not magically allow SATA connections to be used with SAS drives. SAS SATA are storage protocols. they are related to, somewhat intertwined with but still different from the physical connection aspect so don't confuse SAS with SATA based on cable, connector etc.
I can have motherboard on board sata connectors that are SAS and I can have motherboard SFF-8087 that is all SATA...
Personally I avoid using the intel SCU SAS stuff. Get yourself a nice LSI 2008 SAS 2 IT mode HBA and use that to connect to your expander backplane. Or get an LSI 3008 SAS3 IT mode HBA (cause they certainly are cheap now) and then buy some SFF-8643 to SFF-8087 supermicro cables to connect your HBA to your expander. Tastes great - less filling and less likely to get system heartburn.
disable the intel SAS SCU on the motherboard. Make sure you keep your base chipset SATA ports enabled and use two for your boot pool mirror. you only need 2.
If you buy the fancy 2 bay SFF hot swap unit for the back of your 846 that will have an SGPIO port on it and you may want to take advantage of that for the blinkies. NB: if you use SAS then a proper 8087 to SATA breakout should also ahve the out of band cable, ribbon cable with IDC connector on it. But honeslty what's your use case for the bays? boot drives? velcro I say and full speed ahead with and damn the torpedos - you may fire when you are ready gridley. etc. etc.
IDK if your pcie slots support bifurcation though an article says maybe SM will grant you a beta bios that was never released ... but you can definitely do simple x4 nvme if you want. Good for SLOG, good for ARC device (but really max your memory before you try and do a dedicated ARC device cause memory is cheap). For pete's sake you have so many rdimm slots. max your memory. All that said more memory = more power consumption blah blah and I know you seemed interested/worried/take note of your drive power consumption etc. so maybe don't max your memory? IDK. Please do have fun with all this though!
- yikes I wrote a really long reply. hopefully understandable with some funnies as this is a pretty dry topic.you have diagnosed this correctly and what you are seeing is the expected behavior. IMO unless you specifically enjoy modd'ing BIOS files or can find one already built for what you want to boot nvme - is it worth it? - just use a sata boot drive (or TWO in mirror) to bring up your system. All the x9's I've played with can handle an x4 NVME just fine. Bifurcating a slot for multiple NVME X9 depends on the mobo. your user experience won't be that different booting from sata ssd as nvme. maybe 10-15 seconds of time saved?
I wouldn't get too hung up on it. Instead of a sled you can also VELCRO a pair of low costs SATA boot drives to your PSU wall and forgo the sleds.
Just split off power for it and run your SATA cables.
Based on the age of that Motherboard and the pictures you sent you probably have an older SAS2 OEM 846. FWIW you can always go to the supermicro e-store site use the chat - in general I've had very positive experiences figuring out what I have and what options I can get for chassis. If they can't help you directly they usually have an answer within a few hours. Also depending on the item sometimes its actually "cheaper" per item to buy from SM e-store than e-bay. However SM e-store charges full rate on shipping and that can make a single item too expensive. Seller TM_SPACE on the bay takes advantage of that fact in their SM markups. Pretty sure they have a direct line in to the SM logistics chain and can get some things that you cannot purchase through th e-store.
end of the day velcro works and a cheap low cost boot drive (used or new) is not going to use a lot of power or produce a lot of heat. it should be fine as long as the velcro adhesive doesn't wear out!
so the internal sled doesn't care whether the drive in it is sata or SAS, its a passive hunk of metal. Just like velcro doesn't care. It matters what cables you use to hook it up. you can break individual SAS/SATA lanes from SFF-8087 - just requires correct cable. My 2cents use your onboard SATA ports with normal single channel SATA cables. I would not really worry about hooking up GPIO.The internal fixed x2 2.5 mount MCP-220-84603-0N is cheaper $28 https://www.amazon.com/Supermicro-MCP-220-84603-0N-Dual-Fixed-Tray/dp/B004I8YFYC and would be SATA only.
Picture here of one installed in an 846
Project: Infrastructure – Need Not to Know
Posts about Project: Infrastructure written by Zac
Note the 846 in that photo has 4 standoffs making two install positions along the top edge of the PSU case
my 846 instead has three stand offs there,
From the front stand off to the middle of the three is 4 5/8" center to center, matching the forward position he did not use in the picture above, below this front set in the floor are 2 small conical studs. the studs are laid out with the same center line as the stand offs above but I am not exactly sure if this works with the drive case, the drive cage appears to instead use rubber feet on a flat surface.
Velcro
velcro.
Use the chat function at the SM e-store to try and find out what you ahve and what parts will work with the chassis you have. If you can find a S/N and its not from an reseller/OEM but a true SM sold system then the e-store can probably look it up.
so much to reply to here.
NO. SGPIO provides out of band (not in the SATA or SAS channel) control/mgmt functions to whathever backplane you have. Example if you just velcro a drive and connect a sata cable you have to backplane to control and no need for an SGPIO.
TBC using SGPIO does not magically allow SATA connections to be used with SAS drives. SAS SATA are storage protocols. they are related to, somewhat intertwined with but still different from the physical connection aspect so don't confuse SAS with SATA based on cable, connector etc.
I can have motherboard on board sata connectors that are SAS and I can have motherboard SFF-8087 that is all SATA...
Personally I avoid using the intel SCU SAS stuff. Get yourself a nice LSI 2008 SAS 2 IT mode HBA and use that to connect to your expander backplane. Or get an LSI 3008 SAS3 IT mode HBA (cause they certainly are cheap now) and then buy some SFF-8643 to SFF-8087 supermicro cables to connect your HBA to your expander. Tastes great - less filling and less likely to get system heartburn.
disable the intel SAS SCU on the motherboard. Make sure you keep your base chipset SATA ports enabled and use two for your boot pool mirror. you only need 2.
If you buy the fancy 2 bay SFF hot swap unit for the back of your 846 that will have an SGPIO port on it and you may want to take advantage of that for the blinkies. NB: if you use SAS then a proper 8087 to SATA breakout should also ahve the out of band cable, ribbon cable with IDC connector on it. But honeslty what's your use case for the bays? boot drives? velcro I say and full speed ahead with and damn the torpedos - you may fire when you are ready gridley. etc. etc.
IDK if your pcie slots support bifurcation though an article says maybe SM will grant you a beta bios that was never released ... but you can definitely do simple x4 nvme if you want. Good for SLOG, good for ARC device (but really max your memory before you try and do a dedicated ARC device cause memory is cheap). For pete's sake you have so many rdimm slots. max your memory. All that said more memory = more power consumption blah blah and I know you seemed interested/worried/take note of your drive power consumption etc. so maybe don't max your memory? IDK. Please do have fun with all this though!
Not necessarily. Some drives act correct, but are 10x-100x slower. This week I found 2 bad drives that would write data as fast as normal drives, but reading it back was slow, they would probably spend up to a month to read all data back. They started fine, but one started to halt at 8GB read, another started to halt at 2.2TB . No errors, they just spend A LOT of time reading data.
If one drive completes badblocks a lot slower than normal working drives, I would strongly advice against having it running in your raid.
Would be fun to create a pool, write tons of TB's and realize later that you will spend a month reading it all back.....
To check if your chassis is compatible with the 2.5" rear hotswap cage, check if you have a vertical cutout neat the power supplies. It should be approximately the size of two 2.5" drives side-by-side.
The internal mount isn't "SATA only" per se, it's just the only connection that works without an adapter (since SAS and NVMe drive-side connectors don't have the split in the middle).
X9s don't natively support NVMe booting. However, you can use a low capcity drive (or even a SATA DOM if you want to get fancy) for your /boot and /boot/EFI partitions and you're good to go - once the kernel has booted, it would load its own NVMe drive, so your actual OS and whatever else can be on NVMe. In addition, if you plan to do a ZFS root, you can use ZfsBootMenu as your bootloader, and in that case, you only need the EFI system partition to be readable by the BIOS.
re: SGPIO - it's irrelevant for a connection to an expander backplane, since the expander itself provides the management functionality. It's only needed for direct-attach backplanes (such as the 2x rear, or the 'A' model front backplane without expanders).
The internal mount isn't "SATA only" per se, it's just the only connection that works without an adapter (since SAS and NVMe drive-side connectors don't have the split in the middle).
X9s don't natively support NVMe booting. However, you can use a low capcity drive (or even a SATA DOM if you want to get fancy) for your /boot and /boot/EFI partitions and you're good to go - once the kernel has booted, it would load its own NVMe drive, so your actual OS and whatever else can be on NVMe. In addition, if you plan to do a ZFS root, you can use ZfsBootMenu as your bootloader, and in that case, you only need the EFI system partition to be readable by the BIOS.
re: SGPIO - it's irrelevant for a connection to an expander backplane, since the expander itself provides the management functionality. It's only needed for direct-attach backplanes (such as the 2x rear, or the 'A' model front backplane without expanders).
Yes, this comes first.
Yes and I mean you will have the scenario at some point with a broken disk in the pool anyway. It can happen directly after burn-in, with new ones or used.
Yes you are right. I also had a case with the slow hard drive once, it was then noticed with
gstat Had throughput like a floppy drive. Experienced & opinionated replies are exactly what I am looking for here. I have good to intermediate "computer guy" experience with desktop and laptop hardware and this is more of the same ideas but in other ways it has more depth.Warning - I'm opinionated -
Ahhh ok, thank you.
So let me see if I got this right, you can plug a SAS drive into the rear hot swap backplane, it connects upstream to to a specialized blue motherboard header that is physically just like SATA and uses a "SATA" cable but since both ends know how to talk SAS over these wires that makes these bays SAS capable. but since I do not have any of these special blue "SATA"/(SAS) headers it will just be SATA in my X9 system.
that clears a lot of that up.
BTW I hate Velcro and would never have thought to use it on purpose but that hate flows from a very different application. its probably quite appropriate in this case. a some of our interior "pretty" bits are held together with Velcro, after 20 years or so of freezing cold, cycling to deep into the triple digits & extremes of humidity and vibration the adhesive gives up and lets go. Someone will try a quicky glue or epoxy job, and that works for a few flights and when it falls down then someone else will throw more glue on top trying to keep it in place, but the degraded crumbling adhesive keeps the new adhesive on top from being able to hold, it becomes a lumpy disgusting mess of multiple hack glue jobs. the only real fix is to remove everything and take it down to a squeaky clean base and apply new turning what was supose to be a 30 second open look then close back up into a multi hour ordeal. caught like a mouse in a trap.
That makes a lot of sense, the guy selling this also had 2 846 JBOD's, this one must have been the center of that aray as it had the external HBA next the the LSI 2008 SAS2 internal HBA, but even for all that connectivity needs no one had had ever conneted to this Intel SCU. and now I know why. it will be disabled.
You are right use case for SSD is boot drive, some light home server aplications and some chron jobs. and maybe add a second SSD as a write cache later if preformace needs are more than i have. the hot swap backplane and cage here are cool but not particularly useful to justify the costs.
So this board will do more memory than I have, and EEC DDR3 is cheap, it technically can do up to 1.5TB but 256GB is the maximum where you get to keep 1333Mhz, it took a while going round and round with a chart in the motherboard manual and a video from "art of server" to figure out someone had already done the leg work and maxed it out at a good node, going further will require a speed reduction and/or load reduced ram.
I do in fact have this cutout, runs vertically along the from nealy top to bottom of the back of the case in a sliver of space between the motherboard and power supply, just concerned as the motherbord comes right up to the edge of it. see pictures above, I removed the blanking plate that covering the slot.
One more question I have a tiny dongle thing plugged into JTMP1, manual calls it "TPM (Trusted Platform Module)/Port 80"
Do i want that? is it windows only? useful in the *nix side? on a lan? I have pinned its IP to mac address via my opnsense router, its a known and trusted device, I guess MAC could be spoofed but I am in a small town doubt anyone in this town could do that much less somone on my LAN.
Do i want that? is it windows only? useful in the *nix side? on a lan? I have pinned its IP to mac address via my opnsense router, its a known and trusted device, I guess MAC could be spoofed but I am in a small town doubt anyone in this town could do that much less somone on my LAN.
HAHAHAHAHA - hmm TPM - if the cpu is still supported you might be able to run Win 11 on it!!!! you can keep it or probably better yet just pull it as it may come back to haunt during some upgrade cycle if you forget about it...
Let me give you some food for thought. You *will* it sounds like have a SAS IT Mode HBA connected to your expander backplane. You also should have at least ONE unused SFF-8087 when you hook up that SAS IT Mode HBA (asumming a -8i and you connect both its SFF-8087 to the backplane). Now, you could if you wanted, convey that to the rear bracket panel via SFF-8087 to SFF-8087 bracket and it present it should you want to hook up an external disk shelf.
OR...
you could also get an SFF-8087 to forward SATA breakout cable and run that cable back to your rear 2x rear hot swap... oh look your rear hot swap supports SAS or SATA drives now via your IT Mode HBA. You could if you wanted to be frugal, get a nice $20 used SLC 2.5" SAS SSD (say 100GB would be more than adequate) and have yourself a nice handy dedicated SLOG device (whether you can take advantage of that is subject to your use case and configuration parameters for your pool)... Won't be the fastest SLOG but would probably be more than adequate and an improvement against spinning rust. Save your PCIE slots for NVME (in a mirror maybe) and use that for your local vms, jails, containers etc. etc. etc. However if you have slots and pcie lanes to spare then you could look at OPTANE (P1600X) is a nice (either) size device for SLOG and per GB may seem expensive however dropping $60USD + carrier card $20USD on a nice SLOG really isn't that bad...
yep. use of velcro is use-case appropriate and to each their own. Personally I sprang for the skinny hot swap bays for my CSE-836 and initially used them as boot drives - cause - BLINKIES!!! now theyre just mirroed SSD storage for bhyve vm's that don't require performance. Talk to the supermicro e-store chat and see if they can hook you up with the correct internal SLEDS.
If you use TrueNAS Core for instance and strictly GUI configure it then you burn whatever unused storage is on the boot drives. I use 32 and 64GB SATA DOMS (X10 orange plugs) - minimal form factor - work fine for boot and boot time is very acceptable, I don't waste hot swap bays, they don't run especially hot. Tastes great less filling.
It all depends on your use cases and (data) working set size. If you are working on 100GB's of pics and video simultaneously then 512GB of memory may improve your workflow because of the decreased data access latency with in-memory ARC. If you are trying to get every bit of performance on vms or jails and you're running more than a handful - well then maximize your memory speed.
I think you're at the engineering stool stage of this project.
Its a good place to be however don't let decision paralysis get in the way of moving forward.
edit - And thank you for sharing and letting me kibitz... its been a nice diversion from my daily.
The only problem with going from the expander backplane to the rear cage is that (at least on my BPN-SAS3-826EL1-N4) it doesn't output any sideband signals. I've noticed that some tools for listing out your SAS topology don't like it when you have individual drives hanging directly off an expander.
However, you can still buy 82885T expanders on eBay for $30. That will give you SGPIO (it avoids the aforementioned issue by having a virtual enclosure device), plenty of internal ports, and two external ports all without needing a second HBA.
However, you can still buy 82885T expanders on eBay for $30. That will give you SGPIO (it avoids the aforementioned issue by having a virtual enclosure device), plenty of internal ports, and two external ports all without needing a second HBA.
Making partial progress, parts are trickling in, 9x 14TB drives are here, I still need some more drive tray dummies, I had to return the first set I ordered, they were advertised as the new hollow version but the old version is what I received. when the next set get here I can load up all the drives.
I got enough bits and pieces to move the NVME to my main desktop, and then steal its 1TB Samsung SSD and cobble it into the server. Just required re-installing everything from scratch on my desktop ,
I was a bit concerned about this next part, the HBA, backplane, sff-8087 cables etc were all untested, to just dip a toe in the waters I inserted one spinning rust drive to the running system to test "hot swap" and it showed up in disks a few moments later. everything " just worked " ™
Edit: ignore the the rest of this post; it appears adding -b 4096 will cover up to 17TB, unless somone updates Badblocks its days are numbed we are moving past that limit.
Thought I would test run bad blocks on just one drive first from terminal, but ran into an issue:
badblocks: Value too large for defined data type invalid end block (13672382464): must be 32-bit value
32 bit addressing?!? that appears to top out at 4TB:
In the video above he ran Badblocks on a 10TB drives?
Will this also be a problem with the bulk script as opposed to just running bad blocks directly? Am I running into Debian's notorious old packages?
I got enough bits and pieces to move the NVME to my main desktop, and then steal its 1TB Samsung SSD and cobble it into the server. Just required re-installing everything from scratch on my desktop
,I was a bit concerned about this next part, the HBA, backplane, sff-8087 cables etc were all untested, to just dip a toe in the waters I inserted one spinning rust drive to the running system to test "hot swap" and it showed up in disks a few moments later. everything " just worked " ™
Edit: ignore the the rest of this post; it appears adding -b 4096 will cover up to 17TB, unless somone updates Badblocks its days are numbed we are moving past that limit.
Thought I would test run bad blocks on just one drive first from terminal, but ran into an issue:
badblocks: Value too large for defined data type invalid end block (13672382464): must be 32-bit value
32 bit addressing?!? that appears to top out at 4TB:
In the video above he ran Badblocks on a 10TB drives?
Will this also be a problem with the bulk script as opposed to just running bad blocks directly? Am I running into Debian's notorious old packages?
Last edited:
Could I talk you into posting the command you used? From reading the web it appears I need -b 4096 as opposed to the default 1024 byte blocks.
I'm not sure of your budget or appetite for used drives, but you can find 10TB SAS or SATA drives on Ebay for under $50 if you are patient.
Here's a recent example of 4 for just over $200, and you can do better than that at times
Here's a recent example of 4 for just over $200, and you can do better than that at times
LOT OF 4x HUH721010AL5204 HGST 10TB 12G 7.2K SAS 3.5" LFF HDD HARD DRIVE 0F27371 | eBay
Find many great new & used options and get the best deals for LOT OF 4x HUH721010AL5204 HGST 10TB 12G 7.2K SAS 3.5" LFF HDD HARD DRIVE 0F27371 at the best online prices at eBay! Free shipping for many products!
www.ebay.com