SAS, SATA, U.2, NVME, MiniSAS, HBA or RAID, OcuLink?? - Lost in Diversity

[DecentSystems]

So, I think I need to ask you all for some help because being an old-timer from the 2010's and now looking to enhance my storage and disk systems... I am bedazzled by the enormous amount of various options currently available... in such a variety it makes SCSI standards look like a walk in the park!

What I'll do is make a list here of what I think is going on, and I welcome all your corrections and explanations where mistakes are being made

ASSUMED STATE OF AFFAIRS:

1. SFF-8643 is for 12G SAS cards.
2.MiniSAS is a common name for an SFF-8643 connector
3. SFF-8643 is also used to connect a single NVME or U.2 drive
4. However, some adapters offering SFF-8643 connectors may only support U.2/NVME drives (and NOT SAS/SATA)
5. A single SFF-8643 connector can be connected to only ONE U.2 Drive
6. However, a single SFF-8643 connector can also support 4x "conventional" SAS or SATA drive
7. Speed aside, a SFF-8643 and SFF-8087 are identical when it comes to their SAS/SATA features/properties
8. An OcuLink (SFF-8611) is the same as SFF-8643, just smaller
9. An OcuLink (therefore) can also support 4x SAS/SATA drive (though the host adapter may not?)
10. SFF-8654 is an 8-lane connector, which can be "converted" into 2x SFF-8087 or 2x SFF-8643
11. SFF-8654 is a sneaky little bastard because it actually exists in two versions (4i and 8i) which are both called SFF-8654 (or SlimSAS!) but are NOT the same size
12.For ultimate confusion there also exits a SFF-8654 LP Low Profile connector
13. Finally, there is also an MCIO connector (SFF-TA-1016 8i) which is different from SFF-8654 in appearance but is electrically identical to SFF-8654 8i


Questions:
A. Can a SFF-8087 connector (at least in theory, assuming a card would support it) also be used to connect a U.2 disk?
B. What happens when you attach a U.2 disk to a SAS backplane? Does this even fit?
C. Will an NVME host adapter like the LSI9300-16i support/see/recognize a SAS or SATA disk?
D. Does U.2 always imply the disk is ALSO NVME? Are there U.2 disks without NVME?
E.Is the LSI9300-16i card (and similar NVME-focused adapters) also a RAID card?
F. Why does it seem like the entire concept of RAID is mostly alien/not applicable to U.2 and NVME drives?
G. Why are there so few M.2 adapters which support "old-school" RAID (e.g. RAID 5)
H. Why are there so few M.2 SSD bearing adapters which do not require bifurcation?
I. Can one mix U.2/NVME drives (connected using SFF-8643 to SFF-8639) with SAS/SATA disks (using SFF-8643 to 4x SATA, say) on the same adapter, and would this decrease the performance of some drives?
J. Why does it appear to be the case that M.2 drives/slots can be rather fickle and picky with the drives they support?
K. What is the max speed of a U.2 drive (interconnect)?
L. Why are U.2 disks not keeping pace with M2. disk speeds?
M. How important are cable lengths when it comes to high-speed (2GB/s or more) disk drives (of any of the types mentioned here)
N. How stable/reliable are those funny M.2 cards with 4x (or more!) SATA ports on them? Does this generally work well?
O. Assuming the Motherboard supports SATA RAID mode on the onboard ports, can disks attached to a SATA-via-M.2 adapter be added to a RAID array (with any number of drives connected to the onboard SATA ports)
P. To what end do we have THAT MANY various connector types, standards, speeds and other varieties of what are far fewer ACTUAL differences?
Q. What might be the reasons for there not being an easy to use click and release system for M.2 disks like we've had for "regular" drives since the nineties (for SCSI,SATA and SAS)?

R. And last but not least... The LSI 3Ware 9650SE 24-port SATA Raid card has a 3x 8-lane connector which does not seem to have an SFF "code"/model. Even Broadcom calls it a "multi-lane connector" without bothering to provide more details. I am gonna go ahead and assume that this is NOT identical to an SFF-8654 8i connector, as this would make things simpler, correct?

 

[ano]

SFF-8643 used for nvme has a different ohm rating, than regular sas and should not be mixed, often identified by white plugs on SFF-8643 nvme versions.

mcio is so superior to everything, but hard to get

to answer some stuff
b) yes, it fits, some backplanes has both sas, sata AND nvme, and it just figures out what goes where from type of drive connected, like N4 supermicro backplanes or N10 or whatever
c) 9300-16i is not a nvme adapter
e) almost all trimode adapters seems to have hwraid as well
f) because even a 2024 high end raid card, is shitty slow vs 4 x CM7 drives, and imagine 24 or 32 or more connected
g) because SDS, but you get BOSS, LSI, supermicro etc
h) there are tons, you need somewthing with a switch chip, which makes them way more complex and costly
i) yes
j) they are not finnicky in my experience
k) u.2 is more size form factor, but gen5 pcie is fastest now
l) u.2 are much faster than m.2 because of chip controller/power budget
m) very!!! but mcio and gen5 is a lot better than 4 which is quite picky, gen3 is . mehh everything goes ish
q) m.2 drives / connector are not made for that many inserts/removals, but there are some "quick" systems out there, now nvme hot swapping in general is a "there be dragons" thing, even on gen5

from all our testing, tri mode is still mehhh btw

 

[Tech Junky]

The easy way to sift through this when it comes to U drives is....

U.2 cards/adapters work with both drive types
U.3 only works with U.3

The key is to calculate your number of lanes which is simple... each drive needs 4 lanes so, either a single M2 slot or dumb adapter

If you want multiple drives the easiest / cheapest is the Oculink cards that have dual x8 sockets on them and then bifurcate your slot to x4/x4/x4/x4 aka AMD or server mobo

For the raid... Just use the OS if you really want to raid things but, when you hit U drives in the 16TB+ size they tend to be more tolerant / reliable and you can get away with just doing an rsync cron job instead of wasting money on 5 drives and just sync 2 periodically or mirror them

Cable length.... 50cm or less

 

[CyklonDX]

You don't want hardware raid on SSD's (of any type)
Hardware raid doesn't support TRIM.
vroc "hardware" raid is the only way to get TRIM (its software mostly raid)

(Those hardware raid cards that say they support trim - but only in hba mode or pass-through mode)

 

[mattventura]

10. SFF-8654 is an 8-lane connector, which can be "converted" into 2x SFF-8087 or 2x SFF-8643
11. SFF-8654 is a sneaky little bastard because it actually exists in two versions (4i and 8i) which are both called SFF-8654 (or SlimSAS!) but are NOT the same size
12.For ultimate confusion there also exits a SFF-8654 LP Low Profile connector
13. Finally, there is also an MCIO connector (SFF-TA-1016 8i) which is different from SFF-8654 in appearance but is electrically identical to SFF-8654 8i

Questions:
A. Can a SFF-8087 connector (at least in theory, assuming a card would support it) also be used to connect a U.2 disk?
B. What happens when you attach a U.2 disk to a SAS backplane? Does this even fit?
C. Will an NVME host adapter like the LSI9300-16i support/see/recognize a SAS or SATA disk?
D. Does U.2 always imply the disk is ALSO NVME? Are there U.2 disks without NVME?
E.Is the LSI9300-16i card (and similar NVME-focused adapters) also a RAID card?
F. Why does it seem like the entire concept of RAID is mostly alien/not applicable to U.2 and NVME drives?
G. Why are there so few M.2 adapters which support "old-school" RAID (e.g. RAID 5)
H. Why are there so few M.2 SSD bearing adapters which do not require bifurcation?
I. Can one mix U.2/NVME drives (connected using SFF-8643 to SFF-8639) with SAS/SATA disks (using SFF-8643 to 4x SATA, say) on the same adapter, and would this decrease the performance of some drives?
J. Why does it appear to be the case that M.2 drives/slots can be rather fickle and picky with the drives they support?
K. What is the max speed of a U.2 drive (interconnect)?
L. Why are U.2 disks not keeping pace with M2. disk speeds?
M. How important are cable lengths when it comes to high-speed (2GB/s or more) disk drives (of any of the types mentioned here)
N. How stable/reliable are those funny M.2 cards with 4x (or more!) SATA ports on them? Does this generally work well?
O. Assuming the Motherboard supports SATA RAID mode on the onboard ports, can disks attached to a SATA-via-M.2 adapter be added to a RAID array (with any number of drives connected to the onboard SATA ports)
P. To what end do we have THAT MANY various connector types, standards, speeds and other varieties of what are far fewer ACTUAL differences?
Q. What might be the reasons for there not being an easy to use click and release system for M.2 disks like we've had for "regular" drives since the nineties (for SCSI,SATA and SAS)?

R. And last but not least... The LSI 3Ware 9650SE 24-port SATA Raid card has a 3x 8-lane connector which does not seem to have an SFF "code"/model. Even Broadcom calls it a "multi-lane connector" without bothering to provide more details. I am gonna go ahead and assume that this is NOT identical to an SFF-8654 8i connector, as this would make things simpler, correct?

SlimSAS, SlimSAS LP, and MCIO all have both 4 and 8-lane versions. MCIO also has a less-common 16-lane version.

A. Theoretically, yes. Not common in practice. It's not rated for those speeds.
B. If it's truly a SAS/SATA-only backplane, it won't work at all. U.2 backplanes use a separate set of connectors for the NVMe pins. U.3 backplanes overlap it with the SAS/SATA pins, but that's a different story.
C. 9300-16i is a SAS HBA, it won't recognize NVMe disks. You might be thinking of the 9400-16i. The 9400 and newer "tri-mode" HBAs are weird. You typically have to cable them different, and also they're just kind of a crappy way to connect NVMe drives anyway.
D. Generally, yes. Technically, U.2 is just the pinout/form factor/whatever, but nobody would in good faith advertise a SAS or SATA drive as "U.2".
E. Generally, the xx0x cards (e.g. 9300, 9305, 9400, etc) are HBA only, while the ones ending in 40, 60, or 80 are RAID.
F. Software RAID is fine. Hardware raid less so. The problem, especially with the tri-mode RAID controllers and HBAs, is that they expose the drives as SAS drives, even if they're natively NVMe. This means that your host OS can't take advantage of various NVMe-specific improvements. Passing drives directly to the OS and letting the OS do the RAID is generally much better for a variety of reasons.
G. Same as above.
H. It requires a switch chip.
I. I'm guessing you're talking about the tri-mode adapters. You can mix, but again, these are just kind of bad. The only reason to buy, say, a 9400-16i, over an 8i HBA and a 2-port retimer or switch is if you really don't have spare slots.
J. You have to worry about different M.2 keying (only one specific keying has the PCIe x4 out of the common types).
K. Same as a PCIe x4 link. Theoretical max of 32gb/s for gen3, 64 for gen4, 128 for gen5. Though most gen4+ drives are going to be U.3.
L. They are, I'm not sure where you're getting that. But also, see K - most newer drives use U.3. But you can plug U.3 drives into a U.2 backplane.
M. Relatively important, especially if you aren't using a redriver, retimer, or switch in the middle, or a backplane with any of those.
O. Theoretically, yes.
P. Beats me. Hopefully MCIO will be the last connector we need for a while, since it supposedly is good all the way up to gen7 speeds, but we've also got those Gen-Z connectors, so....
Q. M.2 just isn't meant for hotswapping. We don't have such a thing because there's U.2, U.3, EDSFF, etc for that already.
R. It looks like a bastardized 8087, stretched from 4 lanes to 8. AFAIK there's no official standard for doing this with 8087. Only the newer connectors like SlimSAS and MCIO have 8-lane+ versions.

 

[DecentSystems]

SFF-8643 used for nvme has a different ohm rating, than regular sas and should not be mixed, often identified by white plugs on SFF-8643 nvme versions.

Right. Duly noted, and thank you. I had no idea this actually means something.

c) 9300-16i is not a nvme adapter

Indeed it is not. I should have used "LSI 9400-16i SATA/SAS HBA" as the example.

e) almost all trimode adapters seems to have hwraid as well

By Trimode I am assuming you mean cards that support U.2 (or U.3) , SATA and SAS...? And you're saying that, ideally, even if they do support it, one should avoid mixing them, correct?

f) because even a 2024 high end raid card, is shitty slow vs 4 x CM7 drives, and imagine 24 or 32 or more connected

Okay but is it not true that if I were to attach 4x such a drive to an NVME adapter, giving each card a full 4-lane channel, and configure them in RAID5, I'd get some seriously sick bandwidth/speeds that far exceed those of a single disk...?

h) there are tons, you need somewthing with a switch chip, which makes them way more complex and costly

Yeah no I do know they exist but I'm just surprised that it is not that common "a thing". There's like a whole range of LSI SAS/NVME cards... but I see no equivalent series/ranges of M.2 slot cards... that's what surprised me Also, such a PCIE switch chip, is unlikely to cost in excess of, say, 20 bucks, I imagine..? Add in the cost of a more complex PCB and an adapter not requiring Bifurcation can cost $75 more - yet I am not seeing that many...an ASUS card requires Bifurcation, I think they Gigabyte one does too.. Highpoint has some, I believe... and indeed Supermicro and the Dell BOSS. Not sure about the HPE Z Turbo drive. They exist..but when compared to the many U160 SCSI RAID cards back in the day for example.. it's a meagre showing.

j) they are not finnicky in my experience
m) very!!! but mcio and gen5 is a lot better than 4 which is quite picky, gen3 is . mehh everything goes ish

I've played around mostly with Gen 3 PCIE systems and (attempted) Gen 4 disks.. and this has given me SO much crap I've stopped buying M.2 drives for now. Dell Precision 7810/7820 with an attitude, HP Turbo Drive not recognized, Supermicro X11-series not booting... very annoying.

q) m.2 drives / connector are not made for that many inserts/removals, but there are some "quick" systems out there, now nvme hot swapping in general is a "there be dragons" thing, even on gen5

Okay - it need not be to hotswap/change often - but rather to have the equivalent of a designated 4 to 8 disk bay where you can easily slot in M.2 cards (either with B or M key) It would take up very little space in a chassis and so I am just surprised it is not a thing at all. I guess what I mean is that I find it odd that M.2 drives are treated very differently from 2.5/3.5 inch disks.

Another thing that surprises me is the apparent absence of any M.2 disk-related Bios/configuration options in systems. Most Bios-es treat those disk like they don't exist, or are nothing different from "regular" disks. You need to know stuff like "Bifurcation" and half-guess at what the correct setting might be... instead of it detecting/telling you the slot it is in, and what can be configured/set up. It's a bit strange, that's all

Thank you for your answers, I'll come back to answer the other posts too

Ah! What's wrong with SSD and hardware raid? Why is TRIM missing, and why should I avoid running SSD Hardware RAID? What might I use as an alternative that:

- Ensures data survival when one disk kicks ze bucket;
- Offers better performance than a single disk (whether M.2, SATA SSD or U.2)

 

[nexox]

L. Why are U.2 disks not keeping pace with M2. disk speeds?

I assume you're comparing to desktop M.2 drives (because enterprise M.2 isn't particularly fast,) and basically they target different workloads, M.2 seems to just try to go for high scores on short benchmarks, while U.2/3 must sustain constant performance at high load for years. When people try to run server type workload on consumer SSDs they're often pretty disappointed with the performance or longevity, hopefully before a power loss corrupts their data.

There's like a whole range of LSI SAS/NVME cards... but I see no equivalent series/ranges of M.2 slot cards

Because LSI makes enterprise hardware and M.2 has been relegated to boot drive use in servers.

Also, such a PCIE switch chip, is unlikely to cost in excess of, say, 20 bucks, I imagine..? Add in the cost of a more complex PCB and an adapter not requiring Bifurcation can cost $75 more - yet I am not seeing that many.

Pretty sure they're more than $20 just for PCIe 3.0, more for 4.0 and beyond. I'm sure if there was really much demand 3.0 cards could be made for $75 but they're pretty niche products that don't get made in huge batches.

Supermicro X11-series not booting

There's a BIOS default that seems to break legacy booting NVMe unless it's switched to the other setting, I forget the name. It very conveniently gets reset on some BIOS updates and then boot fails again.

Okay - it need not be to hotswap/change often - but rather to have the equivalent of a designated 4 to 8 disk bay where you can easily slot in M.2 cards (either with B or M key) It would take up very little space in a chassis and so I am just surprised it is not a thing at all. I guess what I mean is that I find it odd that M.2 drives are treated very differently from 2.5/3.5 inch disks.

These exist: MB873MP-B_8 Bay M.2 NVMe SSD PCIe 4.0 Mobile Rack Enclosure for External 5.25" Drive Bay (8 x OCuLink SFF-8612, no Tri-mode support) but again, super low demand so they're expensive if you can even find a retailer that has them in stock at any given time. M.2 just doesn't have the power and volume to be very useful for enterprise applications, so there aren't many products that provide enterprise-like features (such as hot swap.)

Another thing that surprises me is the apparent absence of any M.2 disk-related Bios/configuration options in systems.

That's not specific to M.2, that's NVMe in general, and it's kind of because UEFI handles all that, so the BIOS doesn't have to do much, except perhaps implement legacy boot.

You need to know stuff like "Bifurcation" and half-guess at what the correct setting might be... instead of it detecting/telling you the slot it is in, and what can be configured/set up.

Bifurcation configuration (which isn't specific to M.2 or NVMe, it's just PCIe,) can be confusing, especially on Supermicro boards, but a lot of that stuff isn't really intended for home lab use, you're supposed to buy 1000 or 10000 systems and have the integrator set the BIOS as you require, after you work with an engineer from the manufacturer to come up with those requirements. To be fair, if you use first-party adapters the BIOS can often detect the bifurcation automatically, because they have additional signaling to indicate what's required.

 

[nabsltd]

Ah! What's wrong with SSD and hardware raid? Why is TRIM missing, and why should I avoid running SSD Hardware RAID?

Modern LSI cards (93xx or newer) pass through trim to SSDs if the SSD supports the correct features (RZAT and DRAT).

Luckily, the dirt cheap (used) Intel S3600 series of SATA SSDs support both those features.

 

[CyklonDX]

Modern LSI cards (93xx or newer) pass through trim to SSDs if the SSD supports the correct features (RZAT and DRAT).

There's no support in raid mode (IR); only in IT mode, and software raid's.

TRIM and SG_UNMAP support for LSI HBAs and RAID controllers

TRIM and SG_UNMAP support for LSI HBAs and RAID controllers

www.broadcom.com

 

[nabsltd]

There's no support in raid mode (IR); only in IT mode, and software raid's.

Sorry about that. I should have said "94xx or newer support TRIM in hardware RAID" (from the same document you linked).

 

[Koop]

D. Does U.2 always imply the disk is ALSO NVME? Are there U.2 disks without NVME?

It does these days but you always want to double check. I have a HGST SAS SSD drive sitting on my desk right now that uses U.2/SFF-8639. I was pretty confused when I first got it because of the prolific use of U.2/SFF-8639 being primarily used for NVMe. Drive is from 2018 from an EMC system. Probably plenty of other SAS SSDs like that out there which were used in enterprise during that time frame. I would assume they don't make them anymore like this but I'm not sure to be honest.

I believe someone explained to me that U.2 = NVMe protocol that USES the physical SFF-8639 but U.2 implies the physical, electrical, etc for NVMe. Not sure if that's accurate.

 

[CyklonDX]

Sorry about that. I should have said "94xx or newer support TRIM in hardware RAID" (from the same document you linked).

I agree that Tri-Mode does support TRIM, but worth noting that roc does suffer much more from increased latency when doing heavy lifting vs IR mode.

The Tri-mode is a kind of a software raid running on its own 'system' typically ran on arm chip or chipset for computation and running its software suite; it doesn't pass os trim calls, but does them on its own periodicaly (not sure on the nvme front, its likely all disks are physically visible in system). It has its own limitations as you cannot use TRIM on Raid1, Raid0 with SAS/SATA disks (only R5/6 is supported), and NVMe's have only support it in R0, and R1. (from docs it was made in such to add nvme r0/r1 as cache to raid5/6 sas arrays.)

 

[nabsltd]

D. Does U.2 always imply the disk is ALSO NVME?

Yes. U.2 == NVMe, unlike M.2.

I have a HGST SAS SSD drive sitting on my desk right now that uses U.2/SFF-8639.

If you look carefully, there are different pins on the connector than on a U.2 drive. What your SAS drive has is an SFF-8482.

 

[Tech Junky]

Yes. U.2 == NVMe, unlike M.2.

U/M are the physical drive connectors
NVME is the protocol

Back in the day before NVME the connector was used for SATA/SAS

As for the cabling the different pin outs is what says which protocol to use. U.3 is to combine all 3 types of protocols and connectors eventually.

 

[nexox]

Back in the day before NVME the connector was used for SATA/SAS

SATA and SAS don't use the same connectors and U.2 has far more contacts than the SAS-only version. That said, there's nothing stopping a manufacturer using the U.2 connector on a SAS drive if they also make NVMe drives and can save money using a common part.

 

[Koop]

If you look carefully, there are different pins on the connector than on a U.2 drive. What your SAS drive has is an SFF-8482.

I'm lookin'



I don't have a U.2 NVMe drive to compare though. Can you blame for getting confused lol

Pretty sure it's as @nexox said and it's a physical connector that could be used for either. I assume from a manufacturing perspective, for a period of time this was common for SAS SSDs.

 

[piranha32]

I don't have any U.2 drives handy, but looking at the pinouts, U.2 connector looks like a dual-channel SAS connector with a whole row of extra pins added on the other side

 

[Chriggel]

I'm lookin'

View attachment 36245

I don't have a U.2 NVMe drive to compare though. Can you blame for getting confused lol

Pretty sure it's as @nexox said and it's a physical connector that could be used for either. I assume from a manufacturing perspective, for a period of time this was common for SAS SSDs.

This is the U.2 / SFF-8639 connector. SFF-8482 wouldn't have the lower pins on the key.

SATA and SAS don't use the same connectors and U.2 has far more contacts than the SAS-only version. That said, there's nothing stopping a manufacturer using the U.2 connector on a SAS drive if they also make NVMe drives and can save money using a common part.

Indeed, and that's exactly what's happening in the wild:



On the left: Kioxia CM6 (NVMe)
On the right: Kioxia RM6 (SAS)

I don't really see why that would cause so much confusion. This is just the U.2 / SFF-8639 connector on both drives.

The lower left pins are power and the lower right are SAS port A. Between those is the key that prevents inserting the drive into SATA backplanes. Otherwise, SATA uses the same pins for power and data. But obviously, SATA host to SAS device doesn't work, but SAS host to SATA device does.
The upper pins on the key are used for SAS port B on dual port devices. They don't do anything on this device, as the RM6 doesn't have the second SAS port.

And all the remaining pins (upper left, upper right and the lower pins on the key) don't do anything on SAS drives. Earlier SAS drives don't have them. But they do carry the PCIe lanes and control signals for NVMe drives. And likewise, the SAS pins don't do anything on NVMe drives.

Unsurprisingly, making the connector a common part does have a history. Prior to SFF-8639 existing, when the connector was SFF-8482, there were separate parts for single port and dual port SAS drives. The single port connectors didn't have the pins on the upper part of the key which, in theory, made sense because they didn't need them. But then manufacturers noticed that this was stupid and started fitting the same connector to single and dual port drives. From then on, you couldn't tell apart single and dual port drives just by looking at their connector.

It's just the same today. You need to know, or just look at what it says on the drive, if a drive is SAS or NVMe.

U.3 is the same SFF-8639 connector, but changes the pinout according to the SFF-TA-1001 specification to make Trimode easier. With that, SAS and PCIe use the same data pins and the device can tell the host what type it is so that the host can then talk SATA, SAS or NVMe to the device.

 

[nabsltd]

I'm lookin'

Only at one side:

It gets even worse when you realize that the whole SATA spec is now incorporated into SATA Express, which allows AHCI or NVMe over a standard SATA connector (which is like the above but without the 13 center pins). This is just like M.2 (or NGFF) can be either NVMe or SATA.

But unlike "M.2", "U.2" implies both connector shape and pinout (NVMe). "SAS" is the same...it tells you the connector shape and pinout.

 

[Chriggel]

No one needs to worry about SATA Express anymore, it's obsolete. But this is indeed where the SFF-8639 connector originates from, it was initially specified as the SATA Express device connector (back then with two PCIe lanes). SATA Express was unpopular and subsequently failed very quickly, mostly because the host connector was crap.

U.2 and M.2 came right after it and U.2 re-used the SFF-8639 connector because that was the perfectly reasonable part of SATA Express. M.2 took over the SATA/NVMe compatibility from SATA Express while U.2 focused more on professional use and supported SAS as well as SATA and NVMe.

Trimode with U.2 is a nightmare though when you're sourcing individual parts, this is the only really confusing thing about all this, so U.3 was specified. If you want to get a taste of that, look at the IcyDock website and their compatibility information for their U.2 drive cages and trimode controllers. It's solved now, but this wasn't released all at once, it was a real hot mess and some customers got really angry about it. Took them three iterations of their SKUs to work this out.