Chelsio NIC breakdown

[NoncarbonatedClack]

Does anyone have a good breakdown on the suffixes for Chelsio NIC's?

I'm looking at the T580 in particular. I'm seeing:

SO-CR
LP-CR

And a few others.

This is going into a temporary virtual storage host (truenas core), until I get the hardware for dedicated storage. This will be passed through to the VM and going into a Brocade ICX-6610 to provide iSCSI connectivity.

I'm just trying to find out if there is any real appreciable difference in the various versions of the card.

 

[Aluminat]

Perhaps something like this?

 

[RTM]

You can probably find the answers you are looking for in this thread on the truenas forum:

Chelsio T520-CR vs Chelsio T520-SO-CR

Hey guys, I know that "SO" means "Server Offload", but what exactly are the differences between - Chelsio T520-CR - Chelsio T520-SO-CR? There is a huge price difference between those two network adapters at our local dealer. In my opinion "Server Offload" sounds like an extra feature which...

www.truenas.com

 

[brokenwindupdoll]

After having stared at their product lineup for a long while before jumping in, here's what I learned:

CR: Means it's a pcie card (as oppossed to an OCP one).
LP: Low-Profile. They often will make a model, then do a low-profile version later. Performance is the same, just the form-factor is different.
LL: Low-Latency. I've only seen a couple models labeled this, and I really haven't got a clue how much faster it is in the real world if you aren't leveraging all of their offload stack.
SO: "Server Offload." Basically a cut-down model without all the features. You have to look at the product sheets to see if it's features you actually care about.

For example, on the T5 series, you miss out of iSCSI and RDMA offload if you go SO:

https://www.chelsio.com/wp-content/uploads/2013/10/T580-LP-CR-PB.pdf

https://www.chelsio.com/wp-content/uploads/2013/10/T580-SO-CR.pdf

 

[i386]

I'm curios if they can be crossflashed like mellanox cards

 

[blinkenlights]

I'm curios if they can be crossflashed like mellanox cards

You can reprogram the vital product data (VPD) with the "chelsio_adapter_config" utility included with the Linux source package (e.g. ChelsioUwire-3.16.0.1/tools/chelsio_adapter_config_v4). I believe Chelsio used "silicon binning" to create the low-latency (LL) card SKUs. From what I have seen, they are 100% identical to the regular card versions in terms of hardware.

Disclaimer: you can and probably will destroy your Chelsio cards by playing with ASIC core & memory clock speeds (I know, I killed two T520s, one T540, and one T6225).

That said, I have two T6225-CRs (500MHz + 950MHz) reprogrammed as T6225-LL-CRs (800MHz + 1050MHz) in my firewall and they're doing just fine. Does it make a difference in real life usage? Probably not.

 

[NoncarbonatedClack]

Perhaps something like this?

Not quite

You can probably find the answers you are looking for in this thread on the truenas forum:

Chelsio T520-CR vs Chelsio T520-SO-CR

Hey guys, I know that "SO" means "Server Offload", but what exactly are the differences between - Chelsio T520-CR - Chelsio T520-SO-CR? There is a huge price difference between those two network adapters at our local dealer. In my opinion "Server Offload" sounds like an extra feature which...

www.truenas.com

That is some of what I was looking for

After having stared at their product lineup for a long while before jumping in, here's what I learned:

CR: Means it's a pcie card (as oppossed to an OCP one).
LP: Low-Profile. They often will make a model, then do a low-profile version later. Performance is the same, just the form-factor is different.
LL: Low-Latency. I've only seen a couple models labeled this, and I really haven't got a clue how much faster it is in the real world if you aren't leveraging all of their offload stack.
SO: "Server Offload." Basically a cut-down model without all the features. You have to look at the product sheets to see if it's features you actually care about.

For example, on the T5 series, you miss out of iSCSI and RDMA offload if you go SO:

https://www.chelsio.com/wp-content/uploads/2013/10/T580-LP-CR-PB.pdf

https://www.chelsio.com/wp-content/uploads/2013/10/T580-SO-CR.pdf

This is what I was looking for, thanks!
Need to see if the Txxx-CR models are the ones with the largest feature set, those seem to be most expensive.
I'd be interested in the LL ones for my use case.

You can reprogram the vital product data (VPD) with the "chelsio_adapter_config" utility included with the Linux source package (e.g. ChelsioUwire-3.16.0.1/tools/chelsio_adapter_config_v4). I believe Chelsio used "silicon binning" to create the low-latency (LL) card SKUs. From what I have seen, they are 100% identical to the regular card versions in terms of hardware.

Disclaimer: you can and probably will destroy your Chelsio cards by playing with ASIC core & memory clock speeds (I know, I killed two T520s, one T540, and one T6225).

That said, I have two T6225-CRs (500MHz + 950MHz) reprogrammed as T6225-LL-CRs (800MHz + 1050MHz) in my firewall and they're doing just fine. Does it make a difference in real life usage? Probably not.

Well this is interesting. So you can essentially "overclock" the ASIC?
I'd be curious to see if lowering the clock speed reduces heat or power use by an appreciable amount.

 

[blinkenlights]

Well this is interesting. So you can essentially "overclock" the ASIC?
I'd be curious to see if lowering the clock speed reduces heat or power use by an appreciable amount.

It's not in the format or at the granularity you might have in mind. There are 36 VPD files in the 3.16.0.1 driver release (./ChelsioUwire-3.16.0.1/tools/chelsio_adapter_config_v4/vpds/), each one used to tweak things like the ASIC core clock speed, memory speed, product identifier, supported media, number of ports, etc. The true intent of the VPD files and the adapter config utility is to configure ports for breakout cables and to recover broken cards. I will include the full list below so you can see what I am talking about.

So with your scenario of lowering the clock speed, you might be able to do it like this:

You bought a T520-LL-CR (low-latency) card that ships, by default, with the following VPD flashed: t520_ll_init_gen3_650_1075_variable_2133_vpd.bin

In concept, you should be able to flash this VPD to lower the ASIC core speed from 650MHz to 250MHz:
t520_cr_init_gen3_250_825_fixed_2133_vpd.bin

However, I can't think of a reason why anyone would ever want to do that The low-latency parts were marketed to high-frequency traders (HFTs), who are notorious for paying a big premium for exotic silicon (see Intel "Black Ops" and Everest processors). If an eBay seller knows what they are doing, the "LL" part will be priced higher - you would not want to convert your pricey low-latency card to the stock version.

It might be a fun science fair project to reverse engineer the format of the VPD files so you actually can set arbitrary clock speeds

Datawise_t540_cr_init_gen3_500_825_t540_xfi_vpd.bin
Datawise_t540_xfi_gen3_500_825_124vf.bin
diamanti_t6240_so_init_800_1050_gen3_x8_248vf.bin
diamanti_t6240_so_init_800_1050_gen3_x8_variable_2x40gxlaui_vpd.bin
t520_bt_init_gen3_250_820_fixed_2133_vpd.bin
t520_cr_init_gen3_250_825_fixed_2133_vpd.bin
t520_ll_init_gen3_650_1075_variable_2133_vpd.bin
t520_ocp_init_gen3_250_825_vpd.bin
t520_so_init_gen3_250_825_fixed_vpd.bin
t540_bt_init_gen3_500_820_variable_2133_vpd.bin
t540_cr_init_gen3_500_825_variable_2133_vpd_2mc.bin
t580_cr_init_gen3_500Mhz_variable_2133_vpd.bin
t580_cr_qsa_variable_2133_vpd.bin
t580_cr_spider_variable_2133_vpd.bin
t580_lp_cr_init_gen3_500Mhz_variable_2133_vpd.bin
t580_lp_cr_qsa_variable_2133_vpd.bin
t580_lp_cr_spider_variable_2133_vpd.bin
t580_lp_so_init_gen3_500Mhz_variable_vpd.bin
t580_lp_so_qsa_variable_vpd.bin
t580_ocp_so_4x10g_vpd.bin
t580_ocp_so_init_gen3_500Mhz_2x40g_vpd.bin
t580_so_spider_variable_2133_vpd.bin
t61100_ocp_so_cr_init_800_950_gen3_x16_variable_15625_1x100g_vpd_mfg.bin
t62100_cr_init_800_1050_gen3_x16_variable_2133_15625_2x100g_vpd_mfg.bin
t62100_lp_cr_init_800_1050_gen3_x16_variable_2133_15625_2x100g_vpd_mfg.bin
t62100_so_cr_init_800_1050_gen3_x16_variable_15625_2x100g_vpd_mfg.bin
t62100_spider_cr_variable_2133_15625_2x100g_vpd.bin
t62100_spider_lp_cr_variable_2133_15625_2x100g_vpd.bin
t62100_spider_so_cr_variable_15625_2x100g_vpd.bin
t6225_cr_init_500_950_gen3_x8_variable_2133_15625_2x25g_vpd_mfg.bin
t6225_ll_cr_init_800_1050_gen3_x8_variable_2133_15625_2x25g_vpd_mfg.bin
t6225_ocp_so_cr_init_500_950_gen3_x8_248vf_mfg.bin
t6225_ocp_so_cr_init_500_950_gen3_x8_variable_15625_2x25g_vpd_mfg.bin
t6225_so_cr_init_500_950_gen3_x8_248vf_mfg.bin
t6225_so_cr_init_500_950_gen3_x8_variable_15625_2x25g_vpd_mfg.bin
t6425_cr_init_250_950_gen3_x4_variable_2133_15625_2x25g_vpd_mfg.bin