Hi all,
In my previous post, I mention that our research lab's mini server room and needs a small storage server running TrueNAS.
I got initial quote from Lenovo and will also try DELL and ixSystem later.
I think I might get a more accurate answer here and people are generally very knowledgeable about varies different brand of servers.
In my scenario, buying from three big brand (DELL, HPE, Lenovo) yields much better discount and support over Supermicro for a research lab in the University (I personally like Supermicro myself for its simplicity and flexibility).
Lenovo SR655 has only two HBA options: 430-8i, 430-16i, based on LSI 3408, 3416 respectively.
In the datasheet, it says:
Lenovo 530 datasheet
Question : Based on both information about 430(HBA) and 530(RAID), Does the datasheet already indicate that 430 HBA is running under IT mode and thus can be used in ZFS without cross-flash LSI vanilla IT firmware?
The reply at TrueNAS forum seems to suggest that cross-flash is necessary
It kind of means that the only two options are Supermicro and ixSystem's server.
Basically, it almost like using new servers from other vendors automatically disqualify the use of TrueNAS.
Just my 2 cents thought:
Summary of post at TrueNAS forum
Our research lab is considering a TrueNAS storage server to store and feed bulk data to our computation node via dual 10G SFP+ or single 40G QSFP.
It will be 8 x 3.5inch 18TB HDD in RAID-Z2, with lz4 compression; future expansion is another 8 HDD RAID-Z2 vdev
Work load
SMB:
Maybe a few other jail plugins or Dockers
Lenovo SR655 2U server with 12 x 3.5in SAS front bay + 4 x 3.5in SAS mid bay + 4 x 2.5in rear SAS bay (mirrored boot drive)
1. EPYC Milan 7313 16 Core 3.0GHz
2. RAM 128GB 3200MHz
3. Mellanox ConnectX-4 Lx 10/25GbE SFP28
4. HBA 430-8i (LSI SAS3408), 430-16i (LSI SAS3416)
5. No slog
6. L2ARC (optional)
7. Mirrored Intel S4610 240GB as boot drive
In my previous post, I mention that our research lab's mini server room and needs a small storage server running TrueNAS.
I got initial quote from Lenovo and will also try DELL and ixSystem later.
I think I might get a more accurate answer here and people are generally very knowledgeable about varies different brand of servers.
In my scenario, buying from three big brand (DELL, HPE, Lenovo) yields much better discount and support over Supermicro for a research lab in the University (I personally like Supermicro myself for its simplicity and flexibility).
Lenovo SR655 has only two HBA options: 430-8i, 430-16i, based on LSI 3408, 3416 respectively.
In the datasheet, it says:
ThinkSystem 430-8i and 430-16i Internal SAS HBAs Product Guide
The Lenovo ThinkSystem 430-8i and 430-16i SAS/SATA 12Gb HBAs are high performance host bus adapters for internal storage connectivity in ThinkSystem servers. They integrate the latest enhancements SAS technology and offer eight or 16 lanes of 12 Gbps SAS. This product guide provides essential...
lenovopress.com
I read the forum discussion about Lenovo 540-8i RAID card, also based on LSI 3408Non-RAID (JBOD mode) support for SAS and SATA HDDs and SSDs (RAID not supported);
End-to-End CRC with Advanced Error Reporting; "
Configuration through
- XClarity Provisioning Manager UEFI interface
- XClarity Controller web interface
- XClarity Administrator Configuration Patterns
- StorCLI command-line interface
- LSI Storage Authority (LSA) GUI interface
- UEFI Human Interface Infrastructure (HII)
Lenovo 530 datasheet
ThinkSystem RAID 530 Series Internal RAID Adapters Product Guide
The ThinkSystem RAID 530 family are entry-level 12 Gb SAS/SATA internal RAID adapters that offer a cost-effective RAID solution for small to medium business customers. These cacheless adapters support RAID levels 0/1/10/5/50, and include an extensive list of RAS and management features. This...
lenovopress.com
Question : Based on both information about 430(HBA) and 530(RAID), Does the datasheet already indicate that 430 HBA is running under IT mode and thus can be used in ZFS without cross-flash LSI vanilla IT firmware?
The reply at TrueNAS forum seems to suggest that cross-flash is necessary
I get his point. But I cannot and Lenovo will not cross-flash the firmware to LSI vanilla IT mode version. (it will void the warranty)Q: 430-8i HBA based on LSI 3408
A [by senior member#1] : Both are supported under the mpr driver but you will likely have to flash them from the OEM firmware to HBA or IT firmware. Ensure that this is possible and available from your OEM, or there are instructions on how to flash your specific OEM card into its LSI equivalent (LSI 9400-8i or 9400-16i I believe)
Q: For 430-8i, if TrueNAS already sees individual drives, could I assume that I don't need firmware flash?
A [by senior member#2]: Unfortunately not. The problem with unsuitable storage controllers is not so much that they "simply don't work". It is instead edge-case situations when things may go south. For more details please check "Don't use RAID" from the recommended readings in my signature.
It kind of means that the only two options are Supermicro and ixSystem's server.
Basically, it almost like using new servers from other vendors automatically disqualify the use of TrueNAS.
Just my 2 cents thought:
- Large business or institution are probably using large scale proprietary solutions, like IBM Spectrum Scale, or other fancy stuff. So TrueNAS is not appealing to them.
- Customers who would consider TrueNAS are medium/small business or individual research labs (I got this impression from a few use case published on ixSystem website, like mentioning of a research lab at UCSD).
- The server option (R, X, M series) that ixSystem offers are quite limited and specs and pricing are not transparent (also old, still on 2nd gen Xeon). If these options are not suitable, or the customer sometimes have to order from a particular vender due to an agreement, or If ixSystem's server option is 30-50% more expensive than the equivalent specs from the big three, the customer most likely simply give up using TrueNAS or ZFS.
- I understand that testing and certification take a lot of effort. But testing an already HBA card like Lenovo 430 seems to be a low hanging fruit (maybe it is very close to a vanilla LSI card). I would love to see that ixSystem to sell subscription for TrueNAS support on limited selections of other OEM brand server. In fact, they did certify a server from Asigra (based on Supermicro).
Summary of post at TrueNAS forum
Our research lab is considering a TrueNAS storage server to store and feed bulk data to our computation node via dual 10G SFP+ or single 40G QSFP.
It will be 8 x 3.5inch 18TB HDD in RAID-Z2, with lz4 compression; future expansion is another 8 HDD RAID-Z2 vdev
Work load
SMB:
- 12TB of bulk "cold" data storage
- Proxmox VM reads some data from storage server, compute, and writes data back. Most file reads are ~10MB image files or other large binary data files and a couple small 1-100kB configuration ASCII text files.
Maybe a few other jail plugins or Dockers
Lenovo SR655 2U server with 12 x 3.5in SAS front bay + 4 x 3.5in SAS mid bay + 4 x 2.5in rear SAS bay (mirrored boot drive)
1. EPYC Milan 7313 16 Core 3.0GHz
2. RAM 128GB 3200MHz
3. Mellanox ConnectX-4 Lx 10/25GbE SFP28
4. HBA 430-8i (LSI SAS3408), 430-16i (LSI SAS3416)
5. No slog
6. L2ARC (optional)
7. Mirrored Intel S4610 240GB as boot drive