Question about bandwith in this type of cards

[cfunk]

this card :

https://es.aliexpress.com/item/4000756443950.html?spm=2114.12010612.8148356.8.584697a9CJPfua&gatewayAdapt=glo2esp

and this card :

https://es.aliexpress.com/item/4000029811733.html?spm=a2g0o.detail.pcDetailTopMoreOtherSeller.5.3887qcwxqcwxfe&gps-id=pcDetailTopMoreOtherSeller&scm=1007.40050.354490.0&scm_id=1007.40050.354490.0&scm-url=1007.40050.354490.0&pvid=39cfd4be-15d5-4afe-b451-3cc93141b34f&_t=gps-id:pcDetailTopMoreOtherSeller,scm-url:1007.40050.354490.0,pvid:39cfd4be-15d5-4afe-b451-3cc93141b34f,tpp_buckets:668%232846%238114%231999&utparam-url=scene%3ApcDetailTopMoreOtherSeller%7Cquery_from%3A

both have the PLX8748 controller, and both cards claim that can carry up to 8 u.2 disks on a PCIe3 16x. My question is, will the disks perform at full bandwith / latency capacity or at least close to full ? Think is, I am trying to cram as much NVME's as I can on a single server and I'm looking for a solution. For m2 cards, I have seen a max of 4 NVME slots on a 16x card

Thanks

 

[MountainBofh]

Your bottleneck will be the pci-express slot in this case. A standard desktop NVME ssd uses 4 pci-express lanes. So a 16x slot will support at full speed 4 NVME drives. Using some sort of PLX card will let you hang more drives off a 16x slot, but now all the drives are going to share the bandwidth of the 16x slot.

I don't think latency will take a major hit, but sustained transfer will be limited by the 16x slot. Keep in mind that a 16x ver 3.0 slot has a pretty damn high bandwidth of 16 gigabytes/sec. That's enough to feed a QSFP28 network card and still have breathing room left over.

 

[Tech Junky]

@MountainBofh has the math right but, typical Gen 3 drives hit 3.5GB/s so, closer to max of 14.

m2 cards, I have seen a max of 4 NVME slots on a 16x card

Makes sense since 16/4=4. The plx chip muxes the additional drives and shares the bandwidth.

The real question should be how much capacity you need and do you really need higher speeds? U drives right now are up to about 30TB but, they cost slot. You can hit 8/16TB though for less. 8's go for about $400 and 16's about $1000. If you need more speed then upgrade the mobo. My Gen 4 hits 6.5GB/s.

 

[cfunk]

thanks for you inputs guys, this really helps. The reason that I thought to go for u.2 it's the following :

1. u.2 are hot plug, and easier to maintain from the point of view of hot hands ( somebody who doesn't know much and is just given orders )
2. I haven't found any card m.2 that supports 8 disks. The m.2 cards support max 4 disks
3. my aim is to make a CEPH cluster, so I need to be able to squeeze to the max the two PCIe3 slots I have on these servers
4. on this case, latency is more important than bandwidth since it's all about databases R/W
5. 8TB u.2 are going for 480-500$ which is way cheaper than m.2

This is why I can't decide if to just get a 4 slot card or an 8. 8 x 8TB puts you at 64TB on just the 16x slot alone, so it seems a good option. But this would only be a good option if latency is close to the original latency. As I said, transfer bandwidth is not the most important since this is basically aimed to be a huge database cluster

 

[Tech Junky]

8TB u.2 are going for 480-500$ which is way cheaper than m.2

This is what made me switch away from an M2 project where I was converting to all flash from spinners.

You could just go with 15.36TB drives and 4 slots for about the same capacity. As to latency.... it's a combo of speed but also bus capacity.

There's a lot of moving parts though from what you've provided so far that have more questions than answers.... if this will be a dedicated machine for DB use then building from the ground up would be a better idea than using an legacy Gen3 system. Not only do you unleash the storage potential but reduce the long term expenses with more efficient hardware.

Another option along the lines of U drives is EDSFF which have the M2 connector but, also larger capacities but, they're also longer.

 

[nexox]

u.2 are hot plug, and easier to maintain from the point of view of hot hands ( somebody who doesn't know much and is just given orders )

U.2 can be hot plug, but every piece of hardware has to support it, it tends to be kind of tricky to actually make it work correctly if you're piecing together a system, don't be surprised if you need a reboot to detect added or swapped drives.

 

[mattventura]

U.2 can be hot plug, but every piece of hardware has to support it, it tends to be kind of tricky to actually make it work correctly if you're piecing together a system, don't be surprised if you need a reboot to detect added or swapped drives.

This, but also a switch card can sometimes get hotplug working in cases where it wouldn't otherwise. In short, the BIOS needs to be able to recognize hotplug events (which has become much more common, probably thanks to Thunderbolt), and the actual slot/port needs a hotplug controller. The latter can be fixed using a switch card that supports hotplug (it needs to be a good quality one), but the former requires BIOS hacking.

 

[cfunk]

Thank you guys for your input and letting me know about the hot plug mechanics, noted

You could just go with 15.36TB drives and 4 slots for about the same capacity. As to latency.... it's a combo of speed but also bus capacity.

You are right here, even tho the price per TB goes a little higher, but it's another good solution. Check this if you feel like reading : Reddit Challenge Accepted - Is 10k IOPS achievable with NVMes? - Ceph